Grid DBA

The Oracle RAC Assurance team has been working on a handy little goody for quite sometime. RACcheck as the name aptly implies turns out to be quite useful at checking your RAC configuration for any issues as well as provides recommendations for correcting them. It checks database, ASM, Grid Infrastructure and Clusterware components as well as infrastructure level components such as OS, storage and network. Setup is straightforward and you can run it as the database installation owner (oracle). Root or sudo privileges are required for some checks. MOS Note 1268927.1 describes it in detail. So far it works on RedHat Linux, AIX, and Solaris SPARC and SuSE Linux. Please note that bash is required for running this utility so if you're using an OS such as AIX which doesn't have bash installed by default you may want to discuss this with your systems administrator. Here's the HTML output of running RAC Check against a 2-node 11.2.0.2.3 cluster running on AIX 6.1. Results are al...

Just a quick note to self when adding datafiles located on ASM when using Data Guard. If the initialization parameter STANDBY_FILE_MANAGEMENT is set to AUTO then any datafile added on the primary database is automatically propagated to the standby site. Before adding the datafile on the primary site you need to verify that adequate disk space is available in the disk groups on both primary and standby. If not done, then your managed recovery will cease to operate thus causing risks to your disaster recovery database. Consider the example below. Datafile gets added on the primary database. ALTER TABLESPACE "FTSBCAUDIT" ADD DATAFILE '+DG_DATA' SIZE 8G REUSE AUTOEXTEND ON NEXT 100M MAXSIZE UNLIMITED Fri Oct 21 18:49:22 2011 Starting control autobackup Control autobackup written to DISK device         handle '/archivelogs/ftsora01/ftpprd/c-117026708-20111021-13' Completed: ALTER TABLESPACE "FTSBCAUDIT" ADD DATAFILE '+DG_DATA' SIZE 8G REUSE ...

Preamble: I'm writing this blog from my iPhone in the hopes of getting this out quickly before I forget. Please pardon the lack of screenshots or a step by step due to the limitation of the mobile blogger platform. Fusion Middleware 11g provides the ability to manage its components such as Oracle Identity Manager via Fusion Middleware Control. However if you have multiple Fusion Middleware deployments and Enterprise Manager 12c you can use the latter to centralize and simplify management of your environments. Enterprise Manager 12c uses a feature called auto-discovery to discover and add targets. Once auto-discovery is set up you can view the discovered targets and 'promote' the ones you want to manage. This enables you to quickly add your database targets which includes Grid Infrastructure, listeners ASM and the database instances out of the box. However an extra step is required for your Fusion Middleware targets. Auto-discovery has to be explicitly configured for FMW...

With the recent acquisition of Ksplice by Oracle - I got to discover a really cool tool for Linux. Ksplice allows you to apply updates to a live running kernel without the need for rebooting your machine. Yeah right! That was my initial reaction too. But thanks to  some really smart guys from MIT Ksplice does just that. No need to reboot your linux kernel after an upgrade. Sounds too good to be true? Well the proof is in the pudding they say (I like the potato kind :) Well check out the video below to see how easy it is to setup and use Ksplice. For a quick overview of Ksplice check out the FAQ . If you want more details as to the inner workings read the whitepaper . Ksplice is available on Oracle Linux via subscription through ULN. However it is free for Ubuntu Desktop and Fedora. This means that if you're running Red Hat Enterprise Linux or any other enterprise Linux flavor you're out of luck. A few notes about installing Ksplice on Ubuntu The installation was done on...

Oracle Database 11g XE has been in beta for several months now. On September 2, the production release was made available for Windows 32-bit and Linux 64-bit platforms. This lightweight version of the juggernaut database of world is geared towards students, developer and ISVs. You can find the full feature list here . Having played around with the beta version I decided it was time to take a stab at deploying it in the cloud - more specifically Amazon AWS EC2.  I started off by selecting the  Oracle Linux 5.6 x86_64 - OVM (ami-42778a2b) to launch a micro-instance of Oracle Linux. The micro-instance has 613MB of physical memory, 2GB swap and 15GB EBS storage.  [root@ip-10-242-49-142 ~]# cat /proc/meminfo | grep SwapTotal SwapTotal: 2104504 kB [root@ip-10-242-49-142 ~]# cat /proc/meminfo | grep MemTotal MemTotal: 629760 kB Once you download the installation archive from OTN and save it to your computer, you'll have to transfer it to your EC2 instance. ...

Recently, while trying to patch my test RAC database I ran into an issue where I couldn't mount the shared folders in the guest VMs. Neither manual mount or automount were working. I would get the following message when trying to mount the shared folder manually: [root@racnode2 ~]# mount -t vboxsf orasource /mnt/orastage mount: wrong fs type, bad option, bad superblock on orasource, missing codepage or other error In some cases useful info is found in syslog - try dmesg | tail or so The tail of dmesg showed the following additional errors: sf_read_super_aux err=-22 sf_read_super_aux err=-22 sf_read_super_aux err=-22 After doing a search on Google I finally found this thread in the Virtualbox tickets. Apparently the problem is with Oracle Linux's unbreakable enterprise kernel  2.6.32-200.13.1.el5uek. The thread lists several workarounds. I did the following to get things working correctly [root@racnode2 VBOXADDITIONS_4.1.0_73009]# which gmake /usr/bin/gma...

VirtaThon 2011 is the first Oracle online/virtual conference and is hosted by BrainSurface.com. The conference will feature many well known and respected Oracle experts and speakers and a few unknown ones (including yours truly). This will mark my debut as a public speaker and I am grateful for the opportunity after having only aspirations for some time now. You can sign up and register for VirtaThon here  and the session schedule can be found here .

Here are some steps I used to troubleshoot the failure of a public network used for SCAN in a 2-node RAC cluster. Note: I used an aliased crsstat   for the command: crsctl stat res –t Check the status of the Clusterware resources. You can see that there are several resources that are offline below. The ones highlighted in red are the ones we are interested in. The resource for the local listener is now offline while the VIP has been failed over. You will also notice that all of the SCAN Listeners have been failed over to the surviving node. grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >crsstat -------------------------------------------------------------------------------- NAME           TARGET  STATE        SERVER                   STATE_DETAILS ----------------------------------------------...

There has been a lot of talk in the ubiquitous cloud space recently and more specifically the PaaS service delivery model. Naturally this piqued my curiosity. How does the database fit in the cloud? Just about every major enterprise vendor is jumping on the cloud bandwagon. And rightfully so. Private clouds are now being pushed by Oracle with Exalogic Elastic Cloud and Exadata, IBM with SmartCloud and VMware with Cloud Foundry just to mention a few. Amazon AWS and Rackspace are the major players in the public cloud stratosphere. Add to this community Clouds like OpenStack The easiest to do an evaluation is by using Amazon AWS public cloud services. Amazon AWS provides Relational Database Service (RDS) for MySQL and Oracle Database 11g. Strictly speaking EC2 is a Infrastructure as a Service (IaaS) delivery model. However, the use of AMIs enable PaaS characteristics where complete environments can be easily packaged and deployed in EC2. There are several publicly available Oracle Da...

Ever received this error message in your alert log? I have on a couple of occasions. Perhaps an application server started spawning more database connections than it normally does in turn creating more processes on the database. The Oracle description of this error is: Error: ORA 20 Text: maximum number of processes exceeded   Cause: An operation requested a resource that was unavailable. The maximum number of processes is specified by the initialization parameter PROCESSES. When this maximum is reached, no more requests are processed. Action: Try the operation again in a few minutes. If this message occurs often, shut down Oracle, increase the PROCESSES parameter in the initialization parameter file, and restart Oracle. But what if you cannot connect to your database to shut it down and increase the parameter? oradba01t[labdb01]-/home/oracle/>sqlplus  /as sysdba SQL*Plus: Release 11.1.0.7.0 - Production on Mon Jun 6 11:25:05 2011 Copyright (c) 1982, 2008, Ora...

I don't normally do product reviews, but at the request of my fellow Oracle tweeps (@hillbillytoad) I have attempted to do my first review in this post. Here goes. While working on a project to implement a new enterprise backup software for Oracle (EMC Avamar), I was asked certain questions about the Oracle database backups - how large are the database backups? What is the growth rate of backups? What is the rate of change of databases? How many databases are backed up per hour? How many level 0 and level 1 incremental backups are done over a period of time? It's relatively straightforward to get this information from one or two databases but a little bit more challenging across the enterprise with hundreds of databases. We can query the v$ views in the target databases - v$database, v$backup_set, v$backup_set_details, v$rman_status etc.. or their recovery catalog counterparts rc_database, rc_backup_set, rc_backup_set_details (see summary of RMAN catalog views   here ). Separ...