Happy Birthday, Koetje ;-)

Vandaag viert Koetje alweer haar tweede verjaardag – de tijd vliegt.

Ze hangt wel eens in de gordijnen, soms probeert ze Bommel te vangen maar eigenlijk is ze heel lief. Blacky en Koetje kunnen het heel erg goed vinden, dus het is een gezellige beestenboel met onze 3 katten…

Hier is ze in de weer met een duivenveer die ze in de tuin gevonden heeft – blijkbaar héél interessant!

♫ Happy Birthday to Koe,
Happy Birthday to Koe… ♫

Workaround for Nagios check_linux_raid failure in RHEL / CentOS 6.2

I recently stumbled upon another Nagios plugin that no longer works with SELinux under RHEL / CentOS 6.2: check_linux_raid.

Just like the check_disk plugin, it has the nagios_checkdisk_plugin_exec_t SELinux type. As of May 2012, this problem has not yet been fixed.

The workaround is simple, as with the check_disk plugin:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib64/nagios/plugins/check_linux_raid

Or, for 32-bit systems:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib/nagios/plugins/check_linux_raid

FreeNAS on HP ProLiant MicroServer N40L

After the NLUUG presentation on FreeNAS, I bit the bullet and bought the HP MicroServer N40L for experimentation. A small and quiet server, with 4 HDD slots that make for a nice NAS setup. Of course, the system has limited CPU power but it should be enough for basic file serving.

Hardware

I decided to install a low-profile Intel NIC and upgrade the RAM memory to run ZFS comfortably.

HP have published some videos on (dis-)assembling the server, making the job a lot easier. There was only one issue: the mini-SAS connector is a pain to remove. Some Googling later, I found this post:

    To unplug a Mini-SAS x4 cable, squeeze the clip on the plug, then push the plug in before pulling it out. (Source: Oracle)

ZFS is very RAM-hungry, so I installed 2x 4GB Unregistered ECC DIMMs, giving me the maximum supported 8GB configuration:

    Crucial 4GB 240-pin DIMM 512Mx72 DDR3 PC3-10600 ECC (CT51272BA1339)

The Crucial DIMMs were detected without problems. It is possible to use standard non-ECC memory, but I wanted the extra reliability offered by having RAM with Error Correction.

Software

There are two ways to run FreeNAS: bare metal, or in a virtual machine. If you want to go the virtual route, HP has made a customized version of VMware ESXi 5 available for ProLiant servers. To minimize complexity, I run FreeNAS on bare metal. This avoids having to deal with raw device passthrough in VMware and ensures maximum performance.

FreeNAS installation

Installation was a breeze. I created a bootable USB stick (4GB, but 2GB should be fine) using VMware Fusion on my Mac; create a new VMware guest (FreeBSD 8, 64-bit) and set it to boot from the ISO image. Connect the USB stick to the VM; the FreeNAS installer will detect it and ask if you want to install to the USB drive. After installation, shut down the VM and plug the USB stick into the internal USB-port in the MicroServer. Done.

Next, you’ll want to add your harddisks and create a ZFS Volume. I enabled the “4k sectors” option for my 2TB Western Digital drives.

You can simply share this entire volume, or create ZFS Datasets within the ZFS Volume. This gives you more fine-grained control over permissions and sharing.

I mainly use NFS and SMB (CIFS) shares at the moment. They can be used from Mac OSX without problems. Having a central LDAP directory (or perhaps even NIS) helps when setting the correct ownership and permission.

Practical experience

So far, I’m quite happy with FreeNAS performance and ease of use.

FreeNAS 8.2.0-BETA3 appears quite stable; I haven’t found any major bugs yet.

Links

 

PNP4Nagios with SElinux on CentOS / RHEL 6

PNP4Nagios is commonly used to add performance graphs to a Nagios installation.

For additional security, SElinux is enabled on the monitoring host. There is no standard SElinux policy for applications like PNP4Nagios, so we need to develop a custom policy. This sounds harder than it actually is:

  • Run the software as you normally would (SElinux will interfere, so prepare for errors)
  • Extract audit messages and use them to create or update a local SElinux policy for the software
  • Repeat until everything works

In this example, I am running Nagios 3.2.3 with PNP4Nagios 0.6.16 on EL6, 64-bit.

After configuring Nagios and PNP4Nagios integration in Synchronous Mode (see documentation), I noticed that PNP4Nagios is not logging any performance data to /var/lib/pnp4nagios/.

Normally, PNP4Nagios should automatically create directories and files under /var/lib/pnp4nagios as performance data is received by Nagios. This smells of an SElinux issue, so check /var/log/audit/audit.log for suspicious messages. Sure enough, several audit messages have been logged. They look like this:

type=AVC msg=audit(1329129875.344:198212): avc:  denied  { getattr } for  pid=26692 comm="process_perfdat" \
    path="/var/lib/pnp4nagios/orac/Root_Partition.xml.26692" dev=dm-0 ino=1444378 \
    scontext=unconfined_u:system_r:nagios_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1329129875.344:198212): arch=c000003e syscall=5 success=no exit=-13 a0=3 a1=25440a0 \
    a2=25440a0 a3=0 items=0 ppid=26691 pid=26692 auid=0 uid=498 gid=498 euid=498 suid=498 fsuid=498 egid=498 \
    sgid=498 fsgid=498 tty=(none) ses=14942 comm="process_perfdat" exe="/usr/bin/perl" subj=unconfined_u:system_r:nagios_t:s0 key=(null)

Create a policy

You can run the “audit2allow” command (part of the policycoreutils-python RPM) to display suggested policy improvements based on the audit log:

audit2allow -a

The output can be saved in a file, for example local_nagios.te:

grep nagios_t /var/log/audit/audit.log | audit2allow -l -v -m local_nagios > local_nagios.te

This generates an output file suitable for compiling into a custom SElinux module.

Note: ALWAYS prefix the policy name with something like local_ to prevent overwriting system policies!

Test and refine the policy

Compile and load the SElinux policy module:

checkmodule -M -m -o local_nagios.mod local_nagios.te
semodule_package -o local_nagios.pp -m local_nagios.mod
semodule -v -i local_nagios.pp

Note: The above tools can be found in the checkpolicy and policycoreutils RPMs.

Re-run the software and check for SElinux audit messages. New issues can be captured and translated into a new policy:

grep nagios_t /var/log/audit/audit.log | audit2allow -l -v -m local_nagios > local_nagios.te_NEW

Merge the new results (in local_nagios.te_NEW) with your existing policy (in local_nagios.te). Compile and reload the module.

Lather, rinse, repeat ;-)

Results

After some iterations, your local_nagios.te file will look something like this:

module local_nagios 1.0;

require {
    type nagios_t;
    type var_log_t;
    type var_lib_t;
    class dir { write create add_name remove_name };
    class file { create getattr ioctl lock open read rename unlink write };
}

#============= nagios_t ==============
allow nagios_t var_lib_t:dir { add_name create remove_name write };
allow nagios_t var_lib_t:file { create getattr ioctl lock open read rename unlink write };
allow nagios_t var_log_t:file { read rename unlink };

If all is well, the audit.log should not show any new messages for nagios_t:

clear;tail -f /var/log/audit/audit.log |grep nagios_t

Note: The new SElinux policy will survive reboots; it is automatically copied to /etc/selinux/targeted/modules/active/modules/local_nagios.pp.

Enjoy!

Workaround for Nagios check_disk failure in RHEL / CentOS 6.2

After updating from EL 6.1 to 6.2, the Nagios “check_disk” plugin suddenly stopped working with “Permission denied” errors. This problem is related to the SElinux policy (you *are* running with SElinux enabled, aren’t you?).

By default, these AVC denials are not logged in /var/log/audit/audit.log which makes this problem harder to spot (if you want, you can enable all audit-messages by running semodule -DB).

There are at least two relevant entries in Bugzilla:

  • Bug 771245 – nagios-plugins-disk fails when checking /boot on RHEL6.2 boxes
  • Bug 768055 – SELinux silent denials of Nagios NRPE check of /boot

Fortunately, there is a simple workaround while we wait for an updated selinux-policy package. As root, do the following:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib64/nagios/plugins/check_disk

Or, for 32-bit systems:

chcon -t nagios_unconfined_plugin_exec_t /usr/lib/nagios/plugins/check_disk

No need to restart anything; just wait until Nagios re-checks the service and the problem should be gone. Enjoy!