Workaround for Nagios check_disk failure in RHEL / CentOS 6.2

After updating from EL 6.1 to 6.2, the Nagios “check_disk” plugin suddenly stopped working with “Permission denied” errors. This problem is related to the SElinux policy (you *are* running with SElinux enabled, aren’t you?).

By default, these AVC denials are not logged in /var/log/audit/audit.log which makes this problem harder to spot (if you want, you can enable all audit-messages by running semodule -DB).

There are at least two relevant entries in Bugzilla:

  • BugΒ 771245 – nagios-plugins-disk fails when checking /boot on RHEL6.2 boxes
  • BugΒ 768055 – SELinux silent denials of Nagios NRPE check of /boot

Fortunately, there is a simple workaround while we wait for an updated selinux-policy package. As root, do the following:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib64/nagios/plugins/check_disk

Or, for 32-bit systems:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib/nagios/plugins/check_disk

No need to restart anything; just wait until Nagios re-checks the service and the problem should be gone. Enjoy!

Making Cobbler work with SElinux on CentOS / RHEL 6

By default, Cobbler will not work properly on a CentOS / RHEL 6 machine with SElinux enabled. The easy way out is to disable SElinux entirely, but I prefer to write a custom policy instead – it is not that difficult.

The basic approach is this:

  1. Use Cobbler as you normally would (you will trigger several SElinux denials, so expect errors)
  2. Extract the relevant SElinux audit messages; convert them into a local policy
  3. Load your local policy
  4. Repeat steps 1..3 until everything works as expected

First attempt: the “cobbler import” command fails; rsync cannot access files on the mounted DVD ISO. Time to start writing a local policy!

The following command generates a basic SElinux policy from the SElinux audit messages:

  cat /var/log/audit/audit.log | audit2allow -l -v -m local > local.te

The resulting local.te file (ASCII, open it in your favorite editor) will list various items, some if which are not related to the Cobbler / Rsync operations. Edit the file to taste. Now, compile and load that policy:

  checkmodule -M -m -o local.mod local.te
  semodule_package -o local.pp -m local.mod
  semodule -v -i local.pp

Note: every invocation of “audit2allow -l” will overwrite your local.te policy file with new events since the last time a policy module was loaded. This is why you should keep backup copies of the previous versions so you can merge new events in with the existing ones.

In the end, you will end up with a policy in local.te like this:

module local 1.0;

require {
    type cobblerd_t;
    type cobbler_var_lib_t;
    type iso9660_t;
    type public_content_t;
    type rpm_var_lib_t;
    type rsync_etc_t;
    type security_t;
    type tmp_t;
    class capability { sys_module fsetid };
    class dir { add_name create getattr open read remove_name rmdir search write };
    class file { create getattr open read unlink write };
    class lnk_file create;
    class unix_dgram_socket create;
}

#============= cobblerd_t ==============
allow cobblerd_t cobbler_var_lib_t:lnk_file create;
allow cobblerd_t iso9660_t:dir { open read search getattr };
allow cobblerd_t iso9660_t:file { open read getattr };
allow cobblerd_t public_content_t:dir { write rmdir remove_name };
allow cobblerd_t rpm_var_lib_t:dir { open read search getattr write };
allow cobblerd_t rsync_etc_t:file create;
allow cobblerd_t security_t:dir read;
allow cobblerd_t self:capability fsetid;
allow cobblerd_t self:unix_dgram_socket create;
allow cobblerd_t tmp_t:dir { add_name create remove_name rmdir write };
allow cobblerd_t tmp_t:file { create getattr open read unlink write };

Panic after installing new kernel on CentOS 6

I just finished updating all of my CentOS 6 machines, resulting in a couple of machines (VMs) no longer booting:

Kernel panic - not syncing: No init found. Try passing init= option to kernel.

It appears that “yum update” installs the new kernel:

[root@c3p0 boot]# rpm -qa |grep ^kernel-
kernel-2.6.32-131.17.1.el6.x86_64
kernel-firmware-2.6.32-131.17.1.el6.noarch
kernel-firmware-2.6.32-71.29.1.el6.noarch
kernel-2.6.32-71.29.1.el6.x86_64

… but no initramfs is created under /boot:

[root@c3p0 boot]# ls -l /boot
total 25561
-rw-r--r--. 1 root root 100203 Oct 6 20:44 config-2.6.32-131.17.1.el6.x86_64
-rw-r--r--. 1 root root 97911 Jun 27 21:08 config-2.6.32-71.29.1.el6.x86_64
drwxr-xr-x. 3 root root 1024 Aug 16 12:58 efi
drwxr-xr-x. 2 root root 1024 Dec 1 17:15 grub
-rw-r--r--. 1 root root 13435614 Aug 16 12:59 initramfs-2.6.32-71.29.1.el6.x86_64.img
drwx------. 2 root root 12288 Aug 16 12:54 lost+found
-rw-r--r--. 1 root root 165827 Oct 6 20:47 symvers-2.6.32-131.17.1.el6.x86_64.gz
-rw-r--r--. 1 root root 160602 Jun 27 21:11 symvers-2.6.32-71.29.1.el6.x86_64.gz
-rw-r--r--. 1 root root 2279162 Oct 6 20:44 System.map-2.6.32-131.17.1.el6.x86_64
-rw-r--r--. 1 root root 2228188 Jun 27 21:08 System.map-2.6.32-71.29.1.el6.x86_64
-rwxr-xr-x. 1 root root 3882992 Oct 6 20:44 vmlinuz-2.6.32-131.17.1.el6.x86_64
-rwxr-xr-x. 1 root root 3795744 Jun 27 21:08 vmlinuz-2.6.32-71.29.1.el6.x86_64

Also, in /etc/boot/grub.conf, the initrd entry is missing for the new kernel:

[root@c3p0 boot]# cat /etc/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/VolGroup-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS Linux (2.6.32-131.17.1.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-131.17.1.el6.x86_64 ro root=/dev/mapper/VolGroup-lv_root rd_LVM_LV=VolGroup/lv_root rd_LVM_LV=VolGroup/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet
title CentOS (2.6.32-71.29.1.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro root=/dev/mapper/VolGroup-lv_root rd_LVM_LV=VolGroup/lv_root rd_LVM_LV=VolGroup/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet
	initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img

I remember seeing this problem before on a fairly recent Fedora system, probably Fedora 14 or 15, but never on older CentOS / RHEL releases.

My guess is that this is a problem with Dracut. You can re-generate the initramfs by running dracut (as root, of course), but I’d rather not have to perform emergency surgery on lots of (virtual) machines…

Anyway, the command line would look like this:

dracut initramfs-${version}.img ${version}
dracut initramfs-2.6.32-131.17.1.el6.x86_64.img 2.6.32-131.17.1.el6.x86_64

Steve Jobs 1955 – 2011

Steve Jobs 1955 - 2011
Earlier today, Steve Jobs passed away at only 56 years old. He’s had an incredible impact on the tech industry and touched millions of people. To me, his keynotes (“SteveNotes”) and his incredible attention to detail in all of Apple’s designs have been a huge inspiration.

XKCD paid him a fitting tribute:

Eternal Flame

“There’s always the hope that if you sit and watch for long enough,
the beachball will vanish and the thing it interrupted will return.”

Twagios 2.0 – Nagios notifications revisited

In a previous blog post, I described how to use Twitter for Nagios notifications – dubbing it “Twagios”.

A couple of months later, Twitter stopped supporting basic authentication (username/password). This meant that the old (simple) way of sending notifications stopped working. In this post, I’ll explain how I replaced the old Twagios with a new script – “Twagios 2.0” ;-)

First of all, a big thank you to Jeff Miller for writing this excellent post on using the Python tweepy library to create a simple command line client. I simply followed Jeff’s instructions…

  1. Install the tweepy library
  2. Create a new OAuth registration at http://twitter.com/oauth_clients
  3. Create the new “Twagios 2.0” client
  4. Configure Nagios notifications

Install the tweepy library

I’m on CentOS 6.0. Assuming that you already have Python installed, the easiest way to get tweepy is through pip, the Python package manager:

  yum install python-pip
  pip-python install tweepy

Note: why name the package python-pip, and the binary pip-python? How intuitive ;-)

Create a new OAuth registration

I am assuming that you already created a separate Twitter account for sending out Nagios notifications – if not, read my old blog post. The next steps were taken from Jeff Miller’s blog:

Go to http://twitter.com/oauth_clients and log in with your “Twitter Bot” account. Register a nice name for your new script as a “client” with “read & write” permissions. Twitter should issue you with a “Consumer Key” and “Consumer Secret” for your new client.

Authorize access to your “Twitter Bot” account by replacing the CUSTOMER_KEY and CUSTOMER_SECRET values in the following temporary script:

  #!/usr/bin/env python

  import tweepy

  CONSUMER_KEY = 'paste your Consumer Key here'
  CONSUMER_SECRET = 'paste your Consumer Secret here'

  auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
  auth_url = auth.get_authorization_url()
  print 'Please authorize: ' + auth_url
  verifier = raw_input('PIN: ').strip()
  auth.get_access_token(verifier)
  print "ACCESS_KEY = '%s'" % auth.access_token.key
  print "ACCESS_SECRET = '%s'" % auth.access_token.secret

Run the script, open the “Please authorize” URL in your browser and confirm authorization for your “Twitter Bot” account. Twitter should now give you a PIN number. Enter this PIN number and the script should display an ACCESS_KEY and ACCESS_SECRET. We now have all the information successfully authenticate the new client with Twitter.

Create your new “Twagios 2.0” client

Create the actual “Twagios 2.0” script; replace the CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY and ACCESS_SECRET with the proper values:

  #!/usr/bin/env python
  #
  # See: http://talkfast.org/2010/05/31/twitter-from-the-command-line-in-python-using-oauth

  import sys
  import tweepy

  CONSUMER_KEY = 'paste your Consumer Key here'
  CONSUMER_SECRET = 'paste your Consumer Secret here'
  ACCESS_KEY = 'paste your Access Key here'
  ACCESS_SECRET = 'paste your Access Secret here'

  auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
  auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
  api = tweepy.API(auth)
  api.update_status(sys.argv[1])

Done! Send your first tweet:

  chmod 755 twagios
  ./twagios "Hello World"

Configure Nagios

The notification commands describedin my old post should be changed as follows:

  ### Twitter ###

  define command{
        command_name    host-notify-by-twitter
        command_line    /usr/local/bin/twagios "@MyTwitterAccount $HOSTALIAS$ is $HOSTSTATE$. $HOSTOUTPUT$. Time: $SHORTDATETIME$"
  }

  define command{
        command_name    notify-by-twitter
        command_line    /usr/local/bin/twagios "@MyTwitterAccount $SERVICEDESC$ @ $HOSTNAME$ is $SERVICESTATE$. $SERVICEOUTPUT$. Time: $SHORTDATETIME$"
  }

Note: Replace “@MyTwitterAccount” with your actual Twitter account so your Twitter Bot account can send you @mentions.

You could even have it send DM’s by prefixing the message with “d “, for example:

  /usr/local/bin/twagios "d @MyTwitterAccount $HOSTALIAS$ is $HOSTSTATE$. $HOSTOUTPUT$. Time: $SHORTDATETIME$"

Update 2011.12.05 – message length check

As suggested by @erik_mol on Twitter, the above script lacks a check for message length (140 characters max.):

Twagios script needs minor tweak; s = sys.argv[1] api.update_status(s[0:140]). Saves other readers headache, thanks

As message length is indeed important, I have also added some sample substitutions (“WARNING” becomes “WARN” and so on), reducing message length where possible.

  #!/usr/bin/env python
  #
  # See: http://talkfast.org/2010/05/31/twitter-from-the-command-line-in-python-using-oauth

  import sys
  import tweepy
  import string

  CONSUMER_KEY = 'paste your Consumer Key here'
  CONSUMER_SECRET = 'paste your Consumer Secret here'
  ACCESS_KEY = 'paste your Access Key here'
  ACCESS_SECRET = 'paste your Access Secret here'

  auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
  auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
  api = tweepy.API(auth)

  # Process the commandline argument, replace words etc. to shorten the message
  s = sys.argv[1]
  s = string.replace(s,"WARNING","WARN")
  s = string.replace(s,"PROBLEM","PROB")
  s = string.replace(s,"CRITICAL","CRIT")

  # Send the message, 140 characters max.
  api.update_status(s[0:140])

Enjoy the script – happy Sinterklaas everyone ;-)