Workaround for Nagios check_linux_raid failure in RHEL / CentOS 6.2

I recently stumbled upon another Nagios plugin that no longer works with SELinux under RHEL / CentOS 6.2: check_linux_raid.

Just like the check_disk plugin, it has the nagios_checkdisk_plugin_exec_t SELinux type. As of May 2012, this problem has not yet been fixed.

The workaround is simple, as with the check_disk plugin:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib64/nagios/plugins/check_linux_raid

Or, for 32-bit systems:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib/nagios/plugins/check_linux_raid

PNP4Nagios with SElinux on CentOS / RHEL 6

PNP4Nagios is commonly used to add performance graphs to a Nagios installation.

For additional security, SElinux is enabled on the monitoring host. There is no standard SElinux policy for applications like PNP4Nagios, so we need to develop a custom policy. This sounds harder than it actually is:

  • Run the software as you normally would (SElinux will interfere, so prepare for errors)
  • Extract audit messages and use them to create or update a local SElinux policy for the software
  • Repeat until everything works

In this example, I am running Nagios 3.2.3 with PNP4Nagios 0.6.16 on EL6, 64-bit.

After configuring Nagios and PNP4Nagios integration in Synchronous Mode (see documentation), I noticed that PNP4Nagios is not logging any performance data to /var/lib/pnp4nagios/.

Normally, PNP4Nagios should automatically create directories and files under /var/lib/pnp4nagios as performance data is received by Nagios. This smells of an SElinux issue, so check /var/log/audit/audit.log for suspicious messages. Sure enough, several audit messages have been logged. They look like this:

type=AVC msg=audit(1329129875.344:198212): avc:  denied  { getattr } for  pid=26692 comm="process_perfdat" \
    path="/var/lib/pnp4nagios/orac/Root_Partition.xml.26692" dev=dm-0 ino=1444378 \
    scontext=unconfined_u:system_r:nagios_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1329129875.344:198212): arch=c000003e syscall=5 success=no exit=-13 a0=3 a1=25440a0 \
    a2=25440a0 a3=0 items=0 ppid=26691 pid=26692 auid=0 uid=498 gid=498 euid=498 suid=498 fsuid=498 egid=498 \
    sgid=498 fsgid=498 tty=(none) ses=14942 comm="process_perfdat" exe="/usr/bin/perl" subj=unconfined_u:system_r:nagios_t:s0 key=(null)

Create a policy

You can run the “audit2allow” command (part of the policycoreutils-python RPM) to display suggested policy improvements based on the audit log:

audit2allow -a

The output can be saved in a file, for example local_nagios.te:

grep nagios_t /var/log/audit/audit.log | audit2allow -l -v -m local_nagios > local_nagios.te

This generates an output file suitable for compiling into a custom SElinux module.

Note: ALWAYS prefix the policy name with something like local_ to prevent overwriting system policies!

Test and refine the policy

Compile and load the SElinux policy module:

checkmodule -M -m -o local_nagios.mod local_nagios.te
semodule_package -o local_nagios.pp -m local_nagios.mod
semodule -v -i local_nagios.pp

Note: The above tools can be found in the checkpolicy and policycoreutils RPMs.

Re-run the software and check for SElinux audit messages. New issues can be captured and translated into a new policy:

grep nagios_t /var/log/audit/audit.log | audit2allow -l -v -m local_nagios > local_nagios.te_NEW

Merge the new results (in local_nagios.te_NEW) with your existing policy (in local_nagios.te). Compile and reload the module.

Lather, rinse, repeat ;-)

Results

After some iterations, your local_nagios.te file will look something like this:

module local_nagios 1.0;

require {
    type nagios_t;
    type var_log_t;
    type var_lib_t;
    class dir { write create add_name remove_name };
    class file { create getattr ioctl lock open read rename unlink write };
}

#============= nagios_t ==============
allow nagios_t var_lib_t:dir { add_name create remove_name write };
allow nagios_t var_lib_t:file { create getattr ioctl lock open read rename unlink write };
allow nagios_t var_log_t:file { read rename unlink };

If all is well, the audit.log should not show any new messages for nagios_t:

clear;tail -f /var/log/audit/audit.log |grep nagios_t

Note: The new SElinux policy will survive reboots; it is automatically copied to /etc/selinux/targeted/modules/active/modules/local_nagios.pp.

Enjoy!

Workaround for Nagios check_disk failure in RHEL / CentOS 6.2

After updating from EL 6.1 to 6.2, the Nagios “check_disk” plugin suddenly stopped working with “Permission denied” errors. This problem is related to the SElinux policy (you *are* running with SElinux enabled, aren’t you?).

By default, these AVC denials are not logged in /var/log/audit/audit.log which makes this problem harder to spot (if you want, you can enable all audit-messages by running semodule -DB).

There are at least two relevant entries in Bugzilla:

  • Bug 771245 – nagios-plugins-disk fails when checking /boot on RHEL6.2 boxes
  • Bug 768055 – SELinux silent denials of Nagios NRPE check of /boot

Fortunately, there is a simple workaround while we wait for an updated selinux-policy package. As root, do the following:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib64/nagios/plugins/check_disk

Or, for 32-bit systems:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib/nagios/plugins/check_disk

No need to restart anything; just wait until Nagios re-checks the service and the problem should be gone. Enjoy!

Twagios 2.0 – Nagios notifications revisited

In a previous blog post, I described how to use Twitter for Nagios notifications – dubbing it “Twagios”.

A couple of months later, Twitter stopped supporting basic authentication (username/password). This meant that the old (simple) way of sending notifications stopped working. In this post, I’ll explain how I replaced the old Twagios with a new script – “Twagios 2.0” ;-)

First of all, a big thank you to Jeff Miller for writing this excellent post on using the Python tweepy library to create a simple command line client. I simply followed Jeff’s instructions…

  1. Install the tweepy library
  2. Create a new OAuth registration at http://twitter.com/oauth_clients
  3. Create the new “Twagios 2.0” client
  4. Configure Nagios notifications

Install the tweepy library

I’m on CentOS 6.0. Assuming that you already have Python installed, the easiest way to get tweepy is through pip, the Python package manager:

  yum install python-pip
  pip-python install tweepy

Note: why name the package python-pip, and the binary pip-python? How intuitive ;-)

Create a new OAuth registration

I am assuming that you already created a separate Twitter account for sending out Nagios notifications – if not, read my old blog post. The next steps were taken from Jeff Miller’s blog:

Go to http://twitter.com/oauth_clients and log in with your “Twitter Bot” account. Register a nice name for your new script as a “client” with “read & write” permissions. Twitter should issue you with a “Consumer Key” and “Consumer Secret” for your new client.

Authorize access to your “Twitter Bot” account by replacing the CUSTOMER_KEY and CUSTOMER_SECRET values in the following temporary script:

  #!/usr/bin/env python

  import tweepy

  CONSUMER_KEY = 'paste your Consumer Key here'
  CONSUMER_SECRET = 'paste your Consumer Secret here'

  auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
  auth_url = auth.get_authorization_url()
  print 'Please authorize: ' + auth_url
  verifier = raw_input('PIN: ').strip()
  auth.get_access_token(verifier)
  print "ACCESS_KEY = '%s'" % auth.access_token.key
  print "ACCESS_SECRET = '%s'" % auth.access_token.secret

Run the script, open the “Please authorize” URL in your browser and confirm authorization for your “Twitter Bot” account. Twitter should now give you a PIN number. Enter this PIN number and the script should display an ACCESS_KEY and ACCESS_SECRET. We now have all the information successfully authenticate the new client with Twitter.

Create your new “Twagios 2.0” client

Create the actual “Twagios 2.0” script; replace the CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY and ACCESS_SECRET with the proper values:

  #!/usr/bin/env python
  #
  # See: http://talkfast.org/2010/05/31/twitter-from-the-command-line-in-python-using-oauth

  import sys
  import tweepy

  CONSUMER_KEY = 'paste your Consumer Key here'
  CONSUMER_SECRET = 'paste your Consumer Secret here'
  ACCESS_KEY = 'paste your Access Key here'
  ACCESS_SECRET = 'paste your Access Secret here'

  auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
  auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
  api = tweepy.API(auth)
  api.update_status(sys.argv[1])

Done! Send your first tweet:

  chmod 755 twagios
  ./twagios "Hello World"

Configure Nagios

The notification commands describedin my old post should be changed as follows:

  ### Twitter ###

  define command{
        command_name    host-notify-by-twitter
        command_line    /usr/local/bin/twagios "@MyTwitterAccount $HOSTALIAS$ is $HOSTSTATE$. $HOSTOUTPUT$. Time: $SHORTDATETIME$"
  }

  define command{
        command_name    notify-by-twitter
        command_line    /usr/local/bin/twagios "@MyTwitterAccount $SERVICEDESC$ @ $HOSTNAME$ is $SERVICESTATE$. $SERVICEOUTPUT$. Time: $SHORTDATETIME$"
  }

Note: Replace “@MyTwitterAccount” with your actual Twitter account so your Twitter Bot account can send you @mentions.

You could even have it send DM’s by prefixing the message with “d “, for example:

  /usr/local/bin/twagios "d @MyTwitterAccount $HOSTALIAS$ is $HOSTSTATE$. $HOSTOUTPUT$. Time: $SHORTDATETIME$"

Update 2011.12.05 – message length check

As suggested by @erik_mol on Twitter, the above script lacks a check for message length (140 characters max.):

Twagios script needs minor tweak; s = sys.argv[1] api.update_status(s[0:140]). Saves other readers headache, thanks

As message length is indeed important, I have also added some sample substitutions (“WARNING” becomes “WARN” and so on), reducing message length where possible.

  #!/usr/bin/env python
  #
  # See: http://talkfast.org/2010/05/31/twitter-from-the-command-line-in-python-using-oauth

  import sys
  import tweepy
  import string

  CONSUMER_KEY = 'paste your Consumer Key here'
  CONSUMER_SECRET = 'paste your Consumer Secret here'
  ACCESS_KEY = 'paste your Access Key here'
  ACCESS_SECRET = 'paste your Access Secret here'

  auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
  auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
  api = tweepy.API(auth)

  # Process the commandline argument, replace words etc. to shorten the message
  s = sys.argv[1]
  s = string.replace(s,"WARNING","WARN")
  s = string.replace(s,"PROBLEM","PROB")
  s = string.replace(s,"CRITICAL","CRIT")

  # Send the message, 140 characters max.
  api.update_status(s[0:140])

Enjoy the script – happy Sinterklaas everyone ;-)

Twagios – Use Twitter for Nagios notifications

Several customer sites use Nagios for monitoring and alerting. Nagios can send out notifications about problems in a variety of ways, for example using email, pager, SMS and even Twitter. I based my configuration on this post.

Configuring Twitter notifications takes a couple of steps:

  1. Set up a new “Twitter Bot” account to be used for Nagios notifications.
  2. Create a new Nagios contact to send out notifications for your hosts and services.
  3. Define the appropriate notify-by-twitter and host-notify-by-twitter commands.

Setting up the Twitter Bot account

  • Go to twitter.com and create a new account for your Twitter Bot.
  • Go to Settings, and mark the updates as Private. Otherwise, everyone can see your Nagios status updates.
  • Log on to Twitter with your own account, and “Follow” your new Twitter Bot account.
  • Log on to Twitter with your Twitter Bot account, and accept your own account as follower.

Nagios contact

We need to tell Nagios to send updates for host- and service-problems to your Twitter Bot. Add a Nagios contact for this purpose (in my installation, in /etc/nagios/conf.d/general/contacts.cfg):

    define contact{
        contact_name                    twitter
        alias                           Twitter Bot
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r
        host_notification_options       d,r
        service_notification_commands   notify-by-twitter
        host_notification_commands      host-notify-by-twitter
        email                           dummy@example.com
 }

The dummy email address was added to make Nagios happy; it complains if you do not specify an email or pager address.

This new Nagios contact must be added to your standard contact group that is notified in case of trouble:

    define contactgroup{
        contactgroup_name       admins
        alias                   Nagios Administrators
        members                 ed,twitter
        }

Notification commands

The final step in configuring Nagios is defining the actual notification commands we used for the “twitter” contact. Nagios will happily send out notifications using any method that we define. In /etc/nagios/commands.cfg, add the following commands (command_line should be one long line):

    ### Twitter ###
    define command{
        command_name    host-notify-by-twitter
        command_line    /usr/bin/curl --connect-timeout 30 --max-time 60 -u TwitterBotName:TwitterBotPassword -d status="@YourTwitterName $HOSTALIAS$ is $HOSTSTATE$. $HOSTOUTPUT$. Time: $SHORTDATETIME$" http://twitter.com/statuses/update.xml
    }

    define command{
        command_name    notify-by-twitter
        command_line    /usr/bin/curl --connect-timeout 30 --max-time 60 -u TwitterBotName:TwitterBotPassword -d status="@YourTwitterName $SERVICEDESC$ @ $HOSTNAME$ is $SERVICESTATE$. $SERVICEOUTPUT$. Time: $SHORTDATETIME$" http://twitter.com/statuses/update.xml
    }

Replace “TwitterBotName” and “TwitterBotPassword” with the username and password for your new account. Replace “YourTwitterName” with your own Twitter username. This way, notifications are sent out as so-called “Mentions”: these are updates that contain your Twitter @username.

Bonus: Push notifications on iPhone

If you have an iPhone, install the Boxcar app and configure it with the Twitter service for your @username. You will now receive push-notifications on your phone for all Nagios status updates. Neat, huh?

Update 2011.08

  • Boxcar is no longer necessary as your Twitter client can send push notifications too.
  • The above script no longer works because Twitter no longer supports basic authentication.
  • Wrote an update to this blog post, Twagios 2.0, replacing curl with a new client that supports OAuth.