Automatically deploy Dell OMSA with Puppet

I’m pretty sure that nowadays every sysadmin out there still managing bare metal hardware and in particulary Dell servers know and use Open Manage Server Administrator (OMSA), which is a very nice and convenient software. The problem with OMSA is that it can be a little cumbersome to install, with all it’s repos and packages (especially on not-officially-supported distros like Ubuntu, Debian or CentOS). As a Puppet user I am, I was looking for a decent module to install the most recent OMSA version on all my machines, but I didn’t find any.

So, I wrote this brand new Open Manage Server Administrator puppet module!

It sports Ubuntu & CentOS support, granular package install and SNMP integration.
If you want the SNMP integration, be sure that your /etc/snmp/snmpd.conf contains this lines:

view systemview included .1.3.6.1.4.1.674.10892
view systemview included .1.3.6.1.4.1.674.10893
smuxpeer .1.3.6.1.4.1.674.10892.1
smuxpeer .1.3.6.1.4.1.674.10893.1


As I said in the module’s README, you can enable all of these lines through Puppet snmp module only if you use this PR, otherwise you won’t have the second smuxpeer line (whose OID is the StorageServices one).

So, after installing the OMSA module with SNMP support, you can for example easily check all your hardware with this Zabbix template or, if you are a Nagios user, with this plugin

Very slow Hadoop on PowerEdge R815

We have a little internal Hadoop cluster for development and testing, two very powerful Dell PowerEdge R815 with Debian and a bunch of Xen VMs to reproduce a production environment. Problem is that the cluster, even with a relatively small amount of data, was sloooow. And when I say slow I mean almost unusable for Hadoop development (a mapreduce on a small dataset took 5x more than on the big one in production). Even an insignificant

$ hadoop fs -ls

took more than 4s to list the content of HDFS. strace was showing tons of wait() syscalls for no apparent reason, while in the production system the same operation takes 1s and no wait() at all.
After trying almost everything (even without Xen and running Hadoop on the bare metal), I changed by chance a Power Management option in the R815 BIOS. By default it was set to Active Power Controller. Changing it to Maximum Performance did the trick! The ls now takes about a second, just like the production environment. My guess is that probably the default value (which is some kind of automagical load detection) wasn’t able to see that the machine really needed power when running Hadoop, leaving the CPU underclocked to save energy. Maximum power probably is not so green but it solved the problem

HOWTO: Install DELL OpenManage System Administrator on exotic Linux distributions

If you happen to have to manage some Dell server running some exotic-not-supported-at-all distribution like, for example, Slackware or Gentoo, there’s a trick you can use that can saves you a lot of PITA. It envolves using debootstrap and some well known unofficial Debian Dell repository. Here we go:

1. First of all, check your kernel has IPMI support. If it doesn’t, rebuild it to have IPMI enabled (Device Drivers -> Character Devices -> IPMI). Build every module you find here.
2. Load the just built modules:
   modprobe ipmi_msghandler
modprobe ipmi_si
modprobe ipmi_devintf
3. Install debootstrap. With Gentoo, it’s a simple emerge debootstrap
4. mkdir -p /var/debian
5. debootstrap --arch amd64 etch /var/debian http://http.us.debian.org/debian
   You can obviously change the –arch parameter with the one that better fits your needs, say i386
6. Now, let’s wait while debootstrap does all the dirty work :)
7. mount -o bind /dev /var/debian/dev
8. mount -t proc none /var/debian/proc
9. mount -o bind /sys /var/debian/sys
10. cp /etc/resolv.conf /var/debian/etc
11. chroot /var/debian/ /bin/bash
12. Now we are in our new Debian-lite environment.
13. edit /etc/apt/sources.list to look like this, for example:
    

    deb http://ftp.belnet.be/debian/ etch main non-free contrib
    deb-src http://ftp.belnet.be/debian/ etch main non-free contribdeb http://security.debian.org/ etch/updates main contrib non-free
    deb-src http://security.debian.org/ etch/updates main contrib non-free

    deb ftp://ftp.sara.nl/pub/sara-omsa dell sara
    deb http://linux.dell.com/repo etch dell-software

14. apt-get update
15. apt-get install dellomsa
16. And we are done! If all went well, you should now be able to run tools like “omreport” to inspect your hardware status and “omconfig” to change BIOS settings and much more!

On a side note, after a reboot you have to repeat steps 7,8 and 10 and once in the chroot, issue a /etc/init.d/dataeng start