Troubleshooting RAC Public Network Failure

Here are some steps I used to troubleshoot the failure of a public network used for SCAN in a 2-node RAC cluster.
Note: I used an aliased crsstat  for the command: crsctl stat res –t
Check the status of the Clusterware resources. You can see that there are several resources that are offline below. The ones highlighted in red are the ones we are interested in. The resource for the local listener is now offline while the VIP has been failed over. You will also notice that all of the SCAN Listeners have been failed over to the surviving node.
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >crsstat
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DG_DATA.dg
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
ora.DG_FLASH.dg
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
ora.LISTENER.lsnr
               ONLINE  OFFLINE      tibora30
               ONLINE  ONLINE       tibora31
ora.asm
               ONLINE  ONLINE       tibora30                 Started
               ONLINE  ONLINE       tibora31                 Started
ora.gsd
               OFFLINE OFFLINE      tibora30
               OFFLINE OFFLINE      tibora31
ora.net1.network
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
ora.ons
               ONLINE  ONLINE       tibora30
               ONLINE  OFFLINE      tibora31
ora.registry.acfs
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       tibora31
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       tibora31
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       tibora31
ora.cvu
      1        ONLINE  OFFLINE
ora.oc4j
      1        ONLINE  ONLINE       tibora31
ora.scan1.vip
      1        ONLINE  ONLINE       tibora30
ora.scan2.vip
      1        ONLINE  ONLINE       tibora31
ora.scan3.vip
      1        ONLINE  ONLINE       tibora31
ora.tibora30.vip
      1        ONLINE  INTERMEDIATE tibora31                 FAILED OVER
ora.tibora31.vip
      1        ONLINE  ONLINE       tibora31
ora.tibprd.db
      1        ONLINE  ONLINE       tibora30                 Open
      2        ONLINE  ONLINE       tibora31                 Open
ora.tibprd.tibprd_applog.svc
      1        ONLINE  ONLINE       tibora31
ora.tibprd.tibprd_basic.svc
      1        ONLINE  ONLINE       tibora31
ora.tibprd.tibprd_smap.svc
      1        ONLINE  ONLINE       tibora31
Then look at the CRS logs under $GI_HOME/log/hostname/alerthostname.log  for entries similar to the ones below:
2011-06-21 09:43:57.844
[/u01/11.2.0/grid/bin/orarootagent.bin(21168162)]CRS-5818:Aborted command 'check for resource: ora.net1.network tibora30 1' for resource 'ora.net1.network'. Details at (:CRSAGF00113:) {0:9:2} in /u01/11.2.0/grid/log/tibora30/agent/crsd/orarootagent_root/orarootagent_root.log.
2011-06-21 09:44:00.459
[/u01/11.2.0/grid/bin/oraagent.bin(22413372)]CRS-5016:Process "/u01/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/11.2.0/grid/log/tibora30/agent/crsd/oraagent_grid/oraagent_grid.log"
2011-06-21 09:44:01.112
[/u01/11.2.0/grid/bin/oraagent.bin(22413372)]CRS-5016:Process "/u01/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/11.2.0/grid/bin/oraagent.bin" for action "
check" failed: details at "(:CLSN00010:)" in "/u01/11.2.0/grid/log/tibora30/agent/crsd/oraagent_grid/oraagent_grid.log"
2011-06-21 09:44:01.180
[/u01/11.2.0/grid/bin/oraagent.bin(22413372)]CRS-5016:Process "/u01/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/11.2.0/grid/bin/oraagent.bin" for action "
check" failed: details at "(:CLSN00010:)" in "/u01/11.2.0/grid/log/tibora30/agent/crsd/oraagent_grid/oraagent_grid.log"
Check the status of the VIP on the node
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >srvctl status vip -n tibora30
VIP tibora30-vip is enabled
VIP tibora30-vip is not running
Also check the status of the SCAN resources.
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node tibora30
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node tibora31
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node tibora31
In this particular case the SCAN VIPs were running on both nodes. It turns out that on another cluster which also experienced a network failure the SCAN VIPs were all running on node.
First we need to start the local listener:
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >srvctl start listener
Now check the status of the resources.
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >crsstat
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DG_DATA.dg
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
ora.DG_FLASH.dg
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
ora.LISTENER.lsnr
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
ora.asm
               ONLINE  ONLINE       tibora30                 Started
               ONLINE  ONLINE       tibora31                 Started
ora.gsd
               OFFLINE OFFLINE      tibora30
               OFFLINE OFFLINE      tibora31
ora.net1.network
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
ora.ons
               ONLINE  ONLINE       tibora30
               ONLINE  OFFLINE      tibora31
ora.registry.acfs
               ONLINE  ONLINE       tibora30
               ONLINE  ONLINE       tibora31
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       tibora31
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       tibora31
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       tibora31
ora.cvu
      1        ONLINE  OFFLINE
ora.oc4j
      1        ONLINE  ONLINE       tibora31
ora.scan1.vip
      1        ONLINE  ONLINE       tibora30
ora.scan2.vip
      1        ONLINE  ONLINE       tibora31
ora.scan3.vip
      1        ONLINE  ONLINE       tibora31
ora.tibora30.vip
      1        ONLINE  ONLINE       tibora30
ora.tibora31.vip
      1        ONLINE  ONLINE       tibora31
ora.tibprd.db
      1        ONLINE  ONLINE       tibora30                 Open
      2        ONLINE  ONLINE       tibora31                 Open
ora.tibprd.tibprd_applog.svc
      1        ONLINE  ONLINE       tibora31
ora.tibprd.tibprd_basic.svc
      1        ONLINE  ONLINE       tibora31
ora.tibprd.tibprd_smap.svc
      1        ONLINE  ONLINE       tibora31

Starting the local listener also caused the VIP to relocate to the previous node.
In another situation I had to manually relocate the VIP to the original node.
Next we need to check our nodeapps including ONS.


grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >srvctl status nodeapps

VIP tibora30-vip is enabled

VIP tibora30-vip is running on node: tibora30

VIP tibora31-vip is enabled

VIP tibora31-vip is running on node: tibora31

Network is enabled

Network is running on node: tibora30

Network is running on node: tibora31

GSD is disabled

GSD is not running on node: tibora30

GSD is not running on node: tibora31

ONS is enabled

ONS daemon is running on node: tibora30

ONS daemon is not running on node: tibora31


Here you can see that the ONS daemon is not running on the tibora31.
To start the ONS daemon issue the following command:
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >srvctl start nodeapps -n tibora31

PRKO-2421 : Network resource is already started on node(s): tibora31

PRKO-2420 : VIP is already started on node(s): tibora31
 Check the status of the nodeapps again
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >srvctl status nodeapps
VIP tibora30-vip is enabled
VIP tibora30-vip is running on node: tibora30
VIP tibora31-vip is enabled
VIP tibora31-vip is running on node: tibora31
Network is enabled
Network is running on node: tibora30
Network is running on node: tibora31
GSD is disabled
GSD is not running on node: tibora30
GSD is not running on node: tibora31
ONS is enabled
ONS daemon is running on node: tibora30
ONS daemon is running on node: tibora31
The CVU resource can be started as follows:
grid@tibora30[+ASM1]-/u01/11.2.0/grid/log/tibora30 >srvctl start cvu -n tibora30
You can now verify connectivity to the database/services. I prefer to use SQL Developer to test connectivity to my databases with one connection for each service name.

All the SCAN listeners were running on a single node. At least one needed to be relocated to service the requests coming from the SCAN on that node.
grid@tibora31[+ASM2]-/home/grid >srvctl relocate scan_listener -i 1 -n tibora30
grid@tibora31[+ASM2]-/home/grid >srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node tibora30
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node tibora31
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node tibora31
Once the SCAN was relocated the application connected successfully.

Comments

  1. AnonymousAugust 28, 2012 at 1:19 AM

    I appreciate your invaluable sharing for network failure.it saves me for looking too much garbage on internet.regards.

    ReplyDelete
  2. shamsApril 6, 2016 at 3:08 AM

    Awesome.
    It solved my issue. Great explanation. Thanks you

    ReplyDelete
  3. AnonymousNovember 28, 2016 at 3:02 PM

    it solved my issue too.. Thanks.

    ReplyDelete

Post a Comment

Popular posts from this blog

Setup a Wordpress site in 10 Minutes using Oracle Container Cloud Service

Image
Oracle Container Cloud Service Oracle Container Cloud Service lets you easily deploy applications based on docker images. If you don't have an existing application you can create one on docker. However, the easiest method to get a full application up and running is by deploying a stack. Oracle Container Cloud Service provides several example stacks that you can deploy, including web servers based on NGINX or Apache, MongoDB database, and Wordpress. You can find more examples on Oracle GitHub Site . Let's see just how easy it is to spin up a Wordpress site using the Oracle Container Cloud Service. Prerequisites: Access to Oracle Cloud Account Subscription to Oracle IaaS Service First, go to the Oracle Container Cloud Service console and launch a new service. In this example, I created a service with three containers. One manager and two worker nodes. Choose an appropriate "shape" for your compute instances. Note: Oracle uses the term "shape" to r...
Read more

Oracle Home Cloning: Tools and Techniques

One of the many necessary and frequent tasks that a database administrator performs before creating a database is installing the Oracle database binaries. These binaries form what is known as an Oracle Home (referenced by the ORACLE_HOME environment variable on Linux and UNIX environments). Unfortunately, this is also one of the most mundane and uninteresting tasks a DBA can perform. So like all good DBAs know, the best way to deal with such repetitive task is to automate it. Automating installations of Oracle database binaries enables us to reliably repeat the process to enable faster installation as well as provide identical copies of the software in multiple environments. Fortunately, Oracle allows us to clone the Oracle database binaries from one environment to another. Cloning your Oracle binaries from a source copy with all the necessary patches applied enables us to have gold images which can then be deployed to multiple environments. The procedure for cloning an Oracle home...
Read more

Using XFS on Oracle Linux

Install xfs packages on Oracle Linux 6.6 via Oracle Public Yum [root@matthew ~]# yum install xfsprogs Loaded plugins: refresh-packagekit Setting up Install Process public_ol6_UEKR3_latest | 1.2 kB 00:00 public_ol6_UEK_latest | 1.2 kB 00:00 public_ol6_UEK_latest/primary | 19 MB 00:02 public_ol6_UEK_latest 407/407 public_ol6_addons | 1.2 kB 00:00 public_ol6_addons/primary | 77 kB 00:00 public_ol6_addons 269/269 public_ol6_latest | 1.4 kB 00:00 public_ol6_latest/primary | 46 MB 00:05 public_ol6_latest 29302/29302 http://public-yum.oracle.com/beta/repo/OracleLinux/OL6/uek3/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found" Trying other mirror. Resolving Dependencies --> Running transaction check ---> Package xfsprogs.x86_64 0:3.1.11-1.0.5.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ===================================================================================================...
Read more