Grid DBA

Just a short post linking to my presentations at Oracle OpenWorld 2014 and St. Louis Oracle User Group (SLOUG) Big Bash. Oracle Enteprise Manager Cloud Control 12c - Setting Up Metrics and Monitoring (Top 10 Features for DBAs v1.1) from Leighton Nelson A Second Look at Oracle RAC 12c from Leighton Nelson

For the past couple of months I've been working on a project to automatically clone a three node RAC database on AIX to a single instance database from storage snapshots. The clone was to be done daily after our ETL process had completed and was used to offload reporting as well as other purposes. Initially I thought this would be quite straightforward having done clones using RMAN before. How hard could it be? The procedure to clone the database is summarized as follows: 1. Generate controlfile  creation script 2. Identify cloned LUNS 3. Rename cloned disk groups e.g. DG_DATA -> DG_DATA_CLONE 4. Mount cloned disk groups 5. Modify/Create pfile or use existing pfile for clone 6. Setup environment variables for clone 7. Create audit directories for clone 8. Startup clone instance in nomount mode and re-create controlfile from script 9. Recover clone database (Apply archivelogs from source DB) 10. Open database with resetlogs 11. Rename database using NID ...

I've been lucky enough to have been accepted as a presenter at IOUG Collaborate 12 in Las Vegas, NV to be held April 22-26. I will be presenting the following sessions: High Availability Boot Camp: Looking at RAC, Clusterware/Grid Infrastructure Diagnostic Tools Learning Oracle with Oracle VM VirtualBox As RAC SIG US Events Chair I will also be organizing the RAC SIG Experts Panel and Birds of a Feather.  I'm excited and anxious for what will be my first public presentations. Hope the attendees will enjoy it. See you in Las Vegas!

This is a place holder for a blog post on using Enterprise Manager 12c Cloud Control for managing Oracle 11gR2 RAC and Grid Infrastructure. Hopefully, I'll get some time to actually do a write-up. For now, enjoy the screenshots .

Cluster Health Monitor or CHM (formerly IPD/OS) is a utility used to gather OS metrics such as physical memory, swap, CPU usage etc. It's useful for troubleshooting node reboots or evictions, instance evictions as well as poor performance. CHM gathers data in real time with little resource overhead and stores the metrics which can then be retrieved for root-cause analysis. This is very important as other utilities often cannot provide such detailed metrics if there's a resource issue on a node. CHM was bundled with  Grid Infrastructure for Linux and Solaris (x86-64) platforms in 11.2.0.2. However, the feature has now been made available in Grid Infrastructure 11.2.0.3.0 for AIX. After upgrading from 11.2.0.2 to 11.2.0.3 a new resource ora.crf can be seen in the CRS stack. grid@oradba10t[+ASM1]-/home/grid >crsctl stat res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER ...

The Oracle RAC Assurance team has been working on a handy little goody for quite sometime. RACcheck as the name aptly implies turns out to be quite useful at checking your RAC configuration for any issues as well as provides recommendations for correcting them. It checks database, ASM, Grid Infrastructure and Clusterware components as well as infrastructure level components such as OS, storage and network. Setup is straightforward and you can run it as the database installation owner (oracle). Root or sudo privileges are required for some checks. MOS Note 1268927.1 describes it in detail. So far it works on RedHat Linux, AIX, and Solaris SPARC and SuSE Linux. Please note that bash is required for running this utility so if you're using an OS such as AIX which doesn't have bash installed by default you may want to discuss this with your systems administrator. Here's the HTML output of running RAC Check against a 2-node 11.2.0.2.3 cluster running on AIX 6.1. Results are al...

Here are some steps I used to troubleshoot the failure of a public network used for SCAN in a 2-node RAC cluster. Note: I used an aliased crsstat   for the command: crsctl stat res –t Check the status of the Clusterware resources. You can see that there are several resources that are offline below. The ones highlighted in red are the ones we are interested in. The resource for the local listener is now offline while the VIP has been failed over. You will also notice that all of the SCAN Listeners have been failed over to the surviving node. grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >crsstat -------------------------------------------------------------------------------- NAME           TARGET  STATE        SERVER                   STATE_DETAILS ----------------------------------------------...

Having worked on Oracle RAC since 9i, there hasn't been a time when I haven't faced something new or realized something was not what I thought it was. I think this is one of the reasons why I enjoy working on these systems - always a challenge. I stopped my RAC database to put it into noarchivelog mode using the srvctl command [oracle@oradba10t oracle]$ srvctl stop database -d labrac Next I proceeded to connect to the first instance to startup in mount mode [oracle@oradba10t oracle]$ sqlplus /nolog SQL*Plus: Release 10.2.0.4.0 - Production on Thu May 5 09:47:15 2011 Copyright (c) 1982, 2007, Oracle.  All Rights Reserved. SQL> conn / as sysdba Connected to an idle instance. SQL> startup mount ORA-29760: instance_number parameter not specified Wait, what's this? I've never encountered this error message before. Googled a bit and realized that most of the responses were a bit far fetched since the database had been in perfect working order before I stopped...

While installing Oracle RAC it is important to setup SSH user equivalence for installation of of the binaries.  Oracle 11gR2 RAC now includes automatic ssh setup. While reading one of my favorite blogs by Martin Bach  he describes using a utility called sshUserSetup.sh for setting up user equivalence for RAC . This takes out some of the repetitive steps needed to setup ssh user equivalence when installing a RAC cluster.  As Martin pointed out you can also use OUI to setup user equivalence but it helps if you can run the cluster verification utility (CVU) to check on many pre-requisites before starting the installation.  During installation using OUI on the AIX platform however, OUI is unable to perform automatic SSH setup. This means that SSH user equivalence will need to be setup manually prior to installation. To ease the setup all you need to do is to run the sshUserSetup.sh script and follow the prompts directions. It takes as parameters the user (oracle or gr...

Remote Diagnostic Agent (RDA) is a Oracle utility used to collect information on RAC configurations. It collects a wealth of information that would otherwise be tedious to collect manually. This includes operating system, network, hardware, database, cluster  configuration and logs . This is useful for troubleshooting RAC issues such as node evictions. Once collected this information can then be uploaded to My Oracle Support (MOS) as part of a SR. It also complements Oracle Configuration Manager (OCM) which is used to proactively do health checks and help with the speedy resolution of SRs. RDA can be downloaded from MOS. Unzip the RDA archive oracle@gridora31[gridprd2]-/u02 >cd /mnt/orasource/AIX/utils oracle@gridora31[gridprd2]-/mnt/orasource/AIX/utils >ls RDA422.zip ocm-Production-AIX-ppc.zip swingbench230422.zip cpumonitor02.91.zip orion_aix_ppc64.gz swingbench240688.zip datagenerator040198.zip osw3b.tar traceanalyzer010100.zip oracle@gridora31[gridprd2]-/mnt/orasou...

So User Group Forum day has come and gone. The IOUG sessions were pretty good.  Naturally I was focusing on RAC, Grid Infrastructure/ASM etc.  My favourite session of the day was the Grid 2.0 session by Pythian's Alex Gorbachev - high entertainment value (oh yeah I learned stuff too). Running a close second to that was the Oracle Experts panel - pretty informative discussions but there wasn't enough time to answer all the questions. I also got to meet up with some pretty cool guys from twitterland - Chet (oraclenerd), Sunil Ranka and even ran into Alex  in between sessions. Chet and myself met up at the Welcome Keynote where we were expecting to have front-row seats - with our press/blogger pass and all.  Too late.  Safra Catz gave the opening welcome and introduced the HP keynote speakers.  This proved to be the most agonizing hour for most attendees including myself.  After what seemed like an eternity Larry did his own long pre-introductory ritu...

In my most recent attempt to experience the new and wonderful features that Oracle 11g has to offer I needed to get myself another sandbox. Unfortunately my current employer is unable to provide such tools so it's back to the virtual environment for me. My first journey in the virtualization arena took me on a roller coaster ride in my attempt to install Oracle 10g RAC on VMWare ESX 3. That was my first blog. My first thought was to try out Oracle's VM Server since I am in all honesty attached to Oracle's products by an umbilical cord. However, having moved into a job, state and country I didn't have the necessary resources to do this - namely a machine with enough specs to do what I wanted. So armed with my HP Pavillion dv2715nr laptop with 2GB RAM and dual-core AMD x64 Turion processor running Windows Vista Home Premium and Ubuntu 8.04 I attempted to install Oracle Database 11g. My choice of virtual machine was VirtualBox 1.6.2. Previously I had installed the fr...

Our 2 node 10g RAC database would not startup after being shut down. The database was generating an error after trying to startup with command srvctl start database -d db1 It turns out that one of the other DBAs modified some database parameters sga_target to 1710 and forgot to add the M or convert it to bytes...ouch! Somehow one of the nodes got unregistered from the cluster. Soln: The following command was used to add back the instance to the cluster srvctl add instance -d dbname -i instance_name -n nodename

I'm about to embark on setting up Oracle 10g RAC on a VMware ESX 3 server. I've seen a number of articles on the web on setting up RAC on VMware but not ESX server in particular. There was a previously published article on OTN about setting up RAC on VMware which was subsequently removed. However I've been trying to take out bits and pieces from other article like Tarry Singh's extensive article on DBAsupport.com and one from Oracle-Base.com