Configure and De-configure OMS with SLB

My OEM environment consists of two management servers (OMS) for high availability (level 3 HA). We also have a SLB configured for the OMSs but they haven't been configured with the SLB virtual hostname. I know, this sounds very confusing but there's a difference. Essentially they act as individual OMSs since agents can only upload to one OMS and console is pointed to a single OMS.

The load balancer is a F5 BIG-IP and all the configuration setting were done for using OEM. This whitepaper clearly describes how to do this.  

After upgrading my OEM environment from 12.1.0.2.0 to 12.1.0.3.0 I decided to reconfigure the OMSs to use the SLB virtual hostname. Reconfiguring the SLB involves securing the OMSs with the virtual hostname of the SLB.

[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl secure oms -host oemcc.example.com -secure_port 4899 -slb_port 4889 -slb_console_port 443
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Enter Agent Registration Password :
Securing OMS... Successful
Restart OMS

Once the OMS has been secured you can confirm the configuration by checking with emctl status oms -details.

[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oemcc.example.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7803
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4899
EM Instance Home           : /u01/Oracle/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/Oracle/gc_inst/em/EMGC_OMS1/sysman/log
SLB or virtual hostname: oemcc.example.com
HTTPS SLB Upload Port : 4899
HTTPS SLB Console Port : 443
Agent Upload is unlocked.
OMS Console is unlocked.
Active CA ID: 2
Console URL: https://oemcc.example.com:443/em
Upload URL: https://oemcc.example.com:4899/empbs/upload
WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : oms1.example.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oms1.example.com
WebTier is Up
Oracle Management Server is Up

From the output you can see the the SLB or virtual hostname is using the virtual hostname instead of the hostname of the OMS server - oms1 in this example.

After securing each OMS they need to be restarted.

$ ./emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

Once both OMSs have been secured and restarted, the agents also need to be re-secured with the SLB virtual hostname. To secure the agents use:

emctl secure agent -emdWalletSrcUrl https://oemcc.example.com:4899/em

It was this part that failed during my configuration. I'm not sure why it failed but I think it may have to do with the certificates generated for the SLB. I decided to revert the OMS changes until I could figure out the reason for the failure since all my agents were unable to reach the OMS. This is where I seemingly met a brick wall. I couldn't find any documentation that showed how to de-configure the SLB. Luckily emctl help provides a wealth of information about the commands verbs. This is where I found the solution - emctl secure -no_slb

[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl secure oms -no_slb
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Enter Agent Registration Password :
Securing OMS... SuccessfulRestart OMS

Verify that the SLB configuration is no longer present

[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : oms1.example.com
HTTP Console Port : 7788
HTTPS Console Port : 7803
HTTP Upload Port : 4889
HTTPS Upload Port : 4899
EM Instance Home : /u01/Oracle/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/Oracle/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostnameAgent Upload is unlocked.
OMS Console is unlocked.
Active CA ID: 2
Console URL: https://oms1.example.com:7803/em
Upload URL: https://oms1.example.com:4899/empbs/upload

WLS Domain Information
Domain Name : GCDomain
Admin Server Host : oms1.example.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oms1.example.com
WebTier is Up
Oracle Management Server is Up

After re-securing both OMSs and restarting them my agents should now be able to upload to the OMSs. It turns out that I needed to secure the agents again without the slb. This was easily done through the OEM Console, Setup -> Manage Cloud Control -> Agents and then selecting the agents. Click the secure icon which will create a job for securing the agents. Enter the agent registration password and submit the job.

If I missed something in the documentation, please feel free to leave a comment below.