Installing Debian 9 (Stable) on Dell Gen 14 servers with Perc H740P RAID Controllers

Dell’s 14th Generation servers have the option of shipping with Perc H740P RAID controllers. These are stonkingly good, as compared to the old H730 controllers, sporting 8GB of NVRAM as compared to the old 2GB. However, Linux kernel 4.14 is the earliest kernel with driver support and Debian 9.3 (at time of writing) runs 4.9.0-5. Debian can and does backport drivers into stable kernels but, at time of writing, they haven’t done so. I suspect that’s why you’re here reading this!

Foreword – Update your firmware!

These controllers are very new and, clearly, very unproven! Before you start, ensure they are running the latest firmware. We have had multiple perfectly working SSDs rejected from an array under heavy read load to the point that the array failed completely. Dell claim the latest firmware resolves it. We’ll see. If I’m honest, I regret buying 25 of the H740P and should probably have gotten the H730 until the issues with the H740P are ironed out. But we live and learn…

Anyway. Back to drivers. You have a few options…

Option 1: Use Debian 10 (Testing)

Debian strongly discourage the use of their “Testing” versions in production, though if you’re running a staging server (or you’ve got massive balls)… maybe it’s an option. Debian 10 works fine, at time of writing, and I suspect it will continue to do so.

Option 2: Install Debian on a disk not behind the RAID controller

As far as I’m aware, this isn’t how Dell ships servers with RAID controllers. The entire HBA is connected to the controller. If you can bodge in a disk or two that is not connected to the H740P, you can install Debian onto that. Once Debian is booted, you can either install a later kernel from Backports or you can compile and load the driver from Dell (see instructions below).

Once you’ve gotten Debian installed and able to detect arrays behind the RAID controller, it’s feasible that you can use something like an Ubuntu live CD (something with a Kernel new enough to use the Perc 740P) to image your disk onto the RAID array and then boot from that. I’ve not tried this, but it’s probably possible.

Option 3: Install Debian 9 with the driver supplied by Dell

Dell supply drivers for Linux. They are listed as RPMs inside .tar.gz files for RHEL and SUSE. The SUSE driver works on Debian 9 – I’ve not tried the RHEL driver.

Download the .tar.gz (e.g. UnifiedDriver_7.700.52.00_SLES12.tar.gz) from Dell. Open it with 7zip, as that will allow you to browse through the many layers of archive. Do this:

  • Open the .tar.gz with 7zip
  • Open the tar
  • Open the folder
  • Open the -src RPM
  • Open the .cpio
  • Open the .tar.bz2
  • Open the .tar
  • Open the folder

Herein lie the source files!

Compilation

You’ll need to compile these on a Debian machine with an identical version of the kernel to that on which you wish to run the drivers. If you wish to install Debian from USB, Disk, etc. then you’ll need to load the drivers into the installer. As such, you’ll need to make sure that the kernel version you’re running is the same as the installer. To be safe, download a fresh install CD from the Debian website and use this to create your server and your compile machine. The compile machine can be a VirtualBox VM, or similar. It doesn’t need to have a Perc controller.

Your compile machine will need the Linux Kernel headers.

  1. apt-get install linux-headers-$(uname -r)

Get a copy of the source files onto your compile machine and `cd` into your source directory.

Run the compile script:

  1. ./compile.sh

This will create a file called megaraid_sas.ko. That is your driver. You can copy this off your compile machine.

Loading driver such that the Debian installer can “see” your RAID array

If you have physical access to the server, put this on a USB stick. If you don’t, the easiest way to use this in the Debian installer on a Dell is to make it into a floppy disk image and mount that with iDrac. You can use MagicISO on Windows for this. Open MagicISO and:

File→New→Disk Image→2.88MB. Drag and drop your megaraid_sas.ko into the file pane on the right. Click File→Save As and save the file as something like driver.img.

The driver.img can be remotely loaded into iDrac (assuming iDrac Enterprise).

Start the Debian installer, ensuring that you do a Graphical Installation.

After network setup, hit Ctrl-Alt-F2 to enter a shell.

Find the device name of the USB or floppy disk image using fdisk (e.g. /dev/sdb). You can identify it based on size (floppy disk will be about 3MB).

  1. fdisk -l

Mount the disk (changing the device name and filesystem type appropriately. ext2 is right for the floppy image):

  1. mount /dev/sdb /mnt -t ext2

Copy the driver across:

  1. cp /mnt/megaraid_sas.ko /lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/

Load the module:

  1. modprobe megaraid_sas

Unmount the USB/floppy disk:

  1. umount /mnt

Hit Ctrl-Alt-F5 to get back to the GUI.

Loading the driver into the installed OS, such that it boots correctly

Once you’ve installed Debian and are still in the installer, on last screen before reboot, hit Ctrl-Alt-F2 to enter shell again.

Load the module into initramfs and update:

  1. cp /lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/megaraid_sas.ko /target/lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/
  2. chroot /target
  3. echo megaraid_sas >> /etc/initramfs-tools/modules
  4. update-initramfs -u
  5. exit

Reboot, and Debian should boot.

When you update your kernel

On every kernel update, you’ll need to recompile and install the driver. DKMS can do this for you, but here’s the manual instructions. It’s really important that you do this before rebooting with the new kernel else it will not boot. You also won’t be able to boot older kernels… so beware!

Compile the driver against the kernel that you have upgraded to, as per the instructions above. You can do this on the server itself or on a separate compilation machine, as long as the right kernel headers are installed.

Copy the driver to the server and `cd` into its directory. Now copy it to the right place and update initramfs:

  1. cp ./megaraid_sas.ko /lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/
  2. update-initramfs -u

You can now reboot into the new kernel.

I updated my kernel and rebooted already, now it’s broken

Oh dear. You can fix this by booting something like an Ubuntu Live CD (i.e. something that has the drivers for the controller already). In this, you can mount the partitions from your Debian install. If you have multiple partitions, ensure that at least /, /boot and /var are mounted.

You can now chroot into your root mount (e.g. chroot /mnt/debian) and then following instructions above (copy the driver and update initramfs).

Some more revelations

[10/03/2018] – The latest firmware to unlock the full 8GB of cache has been released. Be sure to upgrade to it.

[05/03/2018] – We had 3 disks in a 4 disk RAID 10 array fail. That was messy. Turns out it’s a bug in the controller firmware and the latest firmware fixes it. Be sure to upgrade!

[05/02/2018] – Today my colleague noticed that megacli only reports 4GB of cache on this controller. It turns out that Dell couldn’t get it working with 8GB and have promised to release a firmware update some time in March to unlock the full 8GB. For now, you’ve got 4… which is still better than the 2GB of the older models.

[23/02/2018] – An SSD failed and needed replacing. It was possible to mark the disk as offline, but attempts to mark it as missing or prepare it for removal failed. The assumption is that megacli isn’t compatible with these controllers at the moment. Storcli didn’t work either. In the end, my colleague had remote hands remove and replace the drive. This caused the controller to mark it as Foreign. megacli was used to remove the foreign flag from the drive. Following this, he ensured that auto rebuild was enabled for the controller and then set the disk as a hot spare. This caused the controller to add it back to the array and rebuild. If anyone else has found a nice way to replace disks in an array, please add a comment!

Proxmox Containers: Manage network interfaces file yourself

Proxmox will manage the network interfaces file (/etc/network/interfaces on Debian) for you, on containers. This is sometimes annoying – for example when you want to add static routes with a post-up.

If you simply touch /etc/network/.pve-ignore.interfaces inside the container then it will let you manage it yourself and will thus obey any post-up lines that you add.

Quick Fix: Filesystem notification initialization error — contact your mail administrator (check for configuration errors with the FAM/Gamin library)

This error can be seen in your mail client and is often caused by Courier. If you are the system administrator of the mail server, simply install libgamin as follows:

If you are root:

apt-get install gamin

If not:

sudo apt-get install gamin

Restart all of your courier services as follows (requires you to be using bash shell):

If you are root:

find /etc/init.d/ | grep courier | while read line; do $line restart; done

If not:

find /etc/init.d/ | grep courier | while read line; do sudo $line restart; done

Solved! Debian Wheezy – SASL LOGIN authentication failed: no mechanism available

Solved! Debian Wheezy – SASL LOGIN authentication failed: no mechanism available

I’m not sure why I ever run an `apt-get upgrade` on Debian as I usually spend the next week fixing the issues it’s created. Today’s problem was preventing postfix authenticating smtp users with saslauthd and thus stopping e-mail sending. The log file (/var/log/mail.err) produced the oh so helpful message “SASL LOGIN authentication failed: no mechanism available”.

Google wasn’t much use in diagnosing this as a few people had reported having the issue on Debian but no solution was available. I fixed it by downgrading 5 sasl packages to their Squeeze releases. Commands below:

  1. mkdir /tmp/saslfix
  2. cd /tmp/saslfix
  3.  
  4. wget http://ftp.us.debian.org/debian/pool/main/c/cyrus-sasl2/sasl2-bin_2.1.23.dfsg1-7_i386.deb
  5. wget http://ftp.us.debian.org/debian/pool/main/c/cyrus-sasl2/libsasl2-2_2.1.23.dfsg1-7_i386.deb
  6. wget http://ftp.us.debian.org/debian/pool/main/c/cyrus-sasl2/libsasl2-modules_2.1.23.dfsg1-7_i386.deb
  7. wget http://ftp.us.debian.org/debian/pool/main/c/cyrus-sasl2/libsasl2-modules-sql_2.1.23.dfsg1-7_i386.deb
  8. wget http://ftp.us.debian.org/debian/pool/main/c/cyrus-sasl2/libsasl2-dev_2.1.23.dfsg1-7_i386.deb
  9.  
  10. dpkg -i *.deb
  11.  
  12. /etc/init.d/saslauthd restart
  13. /etc/init.d/postfix restart