Look who's tweeting now - Sending OEM Notifications via Twitter
Disclaimer: The steps described in this post have only been tested in a lab environment. Verify that your organization's security policy isn't being violated when doing any of these steps. Twitter is a public social networking platform and has inherent security risks. I will not accept responsibility if you perform this in your production environment.
Many applications built in the past couple of years have integrated social networking widgets (useful or not) to connect a social world via Twitter, Facebook, Google+ etc. However, most enterprise applications still rely on email as the primary notification method.
Oracle Enterprise Manager Cloud Control 12c allows us to send notifications for incidents, events, jobs out of the box via email. In addition, EM12c supports advanced notification methods including OS scripts, PL/SQL scripts and SNMP traps. For the complete description of Notification Methods you can read the chapter in the Oracle Enterprise Manager Cloud Control 12c documentation.
However, wouldn't it be cool if we were able to also get tweets from OEM as a notification? Yes - I'm a twitter junkie!
Oracle Enterprise Manager's OS command should theoretically allow us to write scripts do this.
The next question was - "How do I write a script that calls twitter APIs?". After some googling it became evident that calling twitter APIs would require using OAuth and/or some kind of wrapper
in Python or Java. I came across a few solutions that required creating and registering a new application for accessing Twitter using Python but this seemed like overkill for my use case.
After a few hours of not finding a suitable solution I remembered that I had started using a command-line Twitter client called TTYtter for a few months now on my Ubuntu netbook. TTYtter is completely Perl-based client which has a lot of functionality. It runs in interactive or script mode which makes it a perfect candidate for scripting.
Here are the overall steps for sending Tweets from Oracle Enterprise Manager.
- Install and authorize TTYtter to post tweets
- Define TTYtter script
- Deploy the script on OMS
- Register your OS Script as a new Notification Method
- Assign the notification method to an incident rule
Install and authorize TTytter
The first step is to download and install TTYtter. It's plain text file so you can review the contents and see what it does. TTYtter requires Perl 5.8.6 or higher and curl 7.20 or higher. These are usually available on Linux and Unix operating systems or can be easily installed. I think it can also run on Windows using Cygwin, however I haven't tested this.Installation involves saving the perl script in a path on your OMS server. It is convenient to rename the script to ttytter instead of the default filename. Next, we need to go through a one time setup. This involves authorizing TTYtter to post tweets on your behalf.
It is recommended to create a separate twitter account for OEM notifications. This way you can make the account private so that tweets aren't visible on the public timeline.
A step in the setup requires going to the Twitter website to authorize the token generated by TTYtter. This will generate a PIN which you will provide when prompted.
To start the setup simply execute the perl script.
Next we start ttytter to send a test message from the account.
Now would be a good time to review the TTYtter command-line options.
The command below sends a direct message to my my personal twitter account.
./ttytter -ssl=1 -runcommand="/dm leight0nn Test DM from OEM" –silentNote the HTTP POST now requires that the SSL option is supplied. We can also put the options in a configuration file called .ttytterrc in the user's home directory.
$cat .ttytterrcThe hold option is useful if you want to ttytter to retry if there's a problem with twitter. For example the "fail whale".
ssl=1
hold=1
There are a couple of ways to post messages via ttytter. The simplest method is to call it with the -runcommand option followed by the type of command you want to execute. You can also pass the -script option specifying one or more commands.
After verifying that your tweet has been successfully sent and received create a test script by saving the contents to a file. If you're behind a proxy server you also need to set the http_proxy and https_proxy environment variables.
Define TTYtter script
$cat oemtweet_test.sh#!/bin/bash
export http_proxy=http://guest:guest@192.168.1.2:80
export https_proxy=http://guest:guest@192.168.1.2:80
/opt/oracle/scripts/ttytter -ssl=1 -runcommand="/dm leight0nn Test DM from OEM" -silent
Register your OS Script as a new Notification Method
The next step is to configure the OS script as a notification method in OEM. As described in the Oracle Enterprise Manager Cloud Control 12c documentation:
Notification system can invoke a custom script when an incident rule matches the OS Command advanced notification action. A custom script receives notifications for matching events, incidents and problem through environment variables.
Log into OEM as a user with Super Administrator privileges. Go to Setup, Notifications Notification Methods.
Click OK.
In the Add Actions screen select the "Always execute the actions" radio button for the Condition for Actions. You may choose to create an incident also but not necessary for our testing.
The Notifications sections is where we'll specify how to get notified. Basic notifications sends an email to the addresses/users provided in the Email To, Email Cc and Page fields. Advanced Notifications is where we'll choose to send twitter notifications based on our OS script.
The next screen should be similar to the one below. Click Next.
Review the new rule and click Continue.
Click the Save button to save the rule set.
With our rule set created we need to test it by modifying metrics so that they cross the defined thresholds. In this example we can select a suitable database target though any target should work. From the database home page go to Monitoring, All Metrics.
We'll modify the Tablespace Free Space (MB) metric so that it triggers a warning or critical event.
Select values for the new warning and critical thresholds. You can click the test button to see if the event will be triggered based on the metric value in the chart. Click the button to Save the metrics.
It may take a few minutes for the metric to be evaluated and trigger an event. Refresh the page until event triggers. In our example you'll see that the event was triggered for the UNDOTBS1 tablespace. The Last Comment column tells us that it called our Twitter OS script. The exit code of 0 and null error and output means that our script was executed successfully.
Notification system can invoke a custom script when an incident rule matches the OS Command advanced notification action. A custom script receives notifications for matching events, incidents and problem through environment variables.
Log into OEM as a user with Super Administrator privileges. Go to Setup, Notifications Notification Methods.
Under the Scripts and SNMP Traps section select Add OS Command from the drop down list.
Enter a name for the Notification Method and an optional description as well the path for the script on the OMS host. If there are multiple OMS servers the script should be located in the same path, preferably in a shared location. You can also enable repeat notifications by select the checkbox next to "Support repeat notifications".
Click the "Test OS Command" button to verify that the script is working.
Assign the notification method to an incident rule
Now we need to assign our new Notification method to an Incident Rule. Click Setup, Incidents, Incident Rules to access the Rule Sets. Create a new Rule Set for testing. Apply the rule set to all targets.
Select the Rule Type to apply to Incoming events and updates to events.
For our test we'll create a Rule based on Metric Alert. No filter is needed at this time since we want to test the rule. Click Next.
The Notifications sections is where we'll specify how to get notified. Basic notifications sends an email to the addresses/users provided in the Email To, Email Cc and Page fields. Advanced Notifications is where we'll choose to send twitter notifications based on our OS script.
Specify a name for the Rule and click Next.
The next screen should be similar to the one below. Click Next.
Review the new rule and click Continue.
Click the Save button to save the rule set.
With our rule set created we need to test it by modifying metrics so that they cross the defined thresholds. In this example we can select a suitable database target though any target should work. From the database home page go to Monitoring, All Metrics.
We'll modify the Tablespace Free Space (MB) metric so that it triggers a warning or critical event.
Select values for the new warning and critical thresholds. You can click the test button to see if the event will be triggered based on the metric value in the chart. Click the button to Save the metrics.
It may take a few minutes for the metric to be evaluated and trigger an event. Refresh the page until event triggers. In our example you'll see that the event was triggered for the UNDOTBS1 tablespace. The Last Comment column tells us that it called our Twitter OS script. The exit code of 0 and null error and output means that our script was executed successfully.
If you entered a valid twitter handle in your OS script you should also have received a direct message via twitter.
If there is an error message or the direct message wasn't received you can enable debugging for notification methods by setting the log4j.category.oracle.sysman.em.notification property.
$OMS_HOME/bin/emctl set property -name log4j.category.oracle.sysman.em.notification -value -DEBUG -module loggingWith this set you see whenever incident rules triggers notifications by looking at the $OMS_HOME/gc_inst/sysman/log/emoms_pbs.trc file.
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
SYSMAN password:
Property log4j.category.oracle.sysman.em.notification has been set to value -DEBUG for all Management Servers
OMS restart is not required to reflect the new property value
The example output below shows that the notification methods for our Twitter rule set being called.
2014-01-15 23:43:32,736 [DeliveryThread-OSCMD7] DEBUG notification.pbs logp.251 - Exiting updateDeviceContactTS.Device=Twitter Broadcast type=2
2014-01-15 23:43:32,736 [DeliveryThread-OSCMD14] DEBUG notification.pbs logp.251 - Notification handled for SYSMAN, Twitter Broadcast, 28
2014-01-15 23:43:32,737 [DeliveryThread-OSCMD5] INFO notification.pbs logp.251 - Notifications delivered in this batch: 1
2014-01-15 23:43:32,738 [DeliveryThread-OSCMD14] INFO notification.pbs logp.251 - Notifications delivered in this batch: 1
2014-01-15 23:43:32,739 [DeliveryThread-OSCMD7] DEBUG notification.pbs logp.251 - Notification handled for SYSMAN, Twitter Broadcast, 28
2014-01-15 23:43:32,740 [DeliveryThread-OSCMD7] INFO notification.pbs logp.251 - Notifications delivered in this batch: 2
2014-01-15 23:43:33,628 [NotificationMgrThread] DEBUG notification.pbs logp.251 - queue_ready returned with to=1
2014-01-15 23:43:34,495 [NotificationMgrThread] DEBUG notification.pbs logp.251 - queue_ready returned with to=1
2014-01-15 23:43:34,664 [DeliveryThread-OSCMD12] DEBUG notification.pbs logp.251 - Ready to log getIssueType=1 retValue=1
2014-01-15 23:43:34,666 [DeliveryThread-OSCMD12] DEBUG notification.pbs logp.251 - updateDeviceContactTS.Device=Twitter Broadcast type=2 message=
2014-01-15 23:43:34,667 [DeliveryThread-OSCMD12] DEBUG notification.pbs logp.251 - Exiting updateDeviceContactTS.Device=Twitter Broadcast type=2
2014-01-15 23:43:34,670 [DeliveryThread-OSCMD12] DEBUG notification.pbs logp.251 - Notification handled for SYSMAN, Twitter Broadcast, 282014-01-15 23:43:34,671 [DeliveryThread-OSCMD12] INFO notification.pbs logp.251 - Notifications delivered in this batch: 1
After verifying that the test tweet was successfully sent and delivered it can be enhanced to send some useful information. We can also create a separate script to not just send a direct message but also a broadcast. This way a group of users can follow the OEM Twitter account and receive the tweets. This may be a group of DBAs for example.
Note if you make the account private then the owner of the twitter account will have to approve access to each user who wants to follow the OEM user's tweets.
Passing incidents and events to our TTYtter scripts requires setting and passing environment variables.
There are a number of environment variables that can be passed from OEM. The table below lists the generic environment variables and their descriptions.
| Environment Variable | Description |
|---|---|
NOTIF_TYPE
|
Type of notification and possible values
NOTIF_NORMAL,
NOTIF_RETRY,
NOTIF_DURATION,
NOTIF_REPEAT,
NOTIF_CA,
NOTIF_RCA
|
REPEAT_COUNT
|
How many times the notification has been sent out
before this notification.
|
RULESET_NAME
|
The name of the ruleset that triggered this notification.
|
RULE_NAME
|
The name of the rule that triggered this notification.
|
RULE_OWNER
|
The owner of the ruleset that triggered this notification.
|
MESSAGE
|
The message of the event, incident, or problem.
|
MESSAGE_URL
|
EM console URL for this message.
|
There are environment variables for different event types, incidents and problems. To get the list of environment variables for each event type you can use the query below.
The ENV_VAR_NAME column represents the environment variable that can be passed to the OS script.
Create a new script with the contents below to send a status update (via broadcast).
echo "EM12c: ${RULE_NAME} ${EVENT_NAME} ${TARGET_NAME} ${MESSAGE} ${EVENT_REPORTED_TIME}." | /opt/oracle/scripts/ttytter -script -keyf=/opt/oracle/.ttytterkeyThis will send a status update to Twitter and pass the Rule Name, Event Name, Target Name, Message and Event Reported Time variables. Due to twitter's 140 character limit you may have to be very selective on which variables to include. Save the script and repeat the steps to create a new notification method and Incident Rule Sets using the new notification method. Once complete you should now have a complete framework for sending tweets from OEM.
Comments
Post a Comment