JunOS error: In routing-instance default-switch interface ae3.0 has vlan and vxlan mixed which is not supported

You might see this error when using VXLAN on Juniper kit after you set an interface to be a member of “all” VLANs. This is because “all” includes VLAN 1. Per this documentation, you can’t actually use VLAN 1 with VXLAN but you must map it to a VNI. For example:

  1. /* You need to map VLAN 1 to a VNI else you can't use `vlan members all` on interfaces - DO NOT USE THIS VLAN */
  2. VLAN001 {
  3. vlan-id 1;
  4. vxlan {
  5. vni 1;
  6. }
  7. }

Set syntax is:

  1. set vlans VLAN001 vlan-id 1 vxlan vni 1
Tags: , , ,

Installing Debian 9 (Stable) on Dell Gen 14 servers with Perc H740P RAID Controllers

Dell’s 14th Generation servers have the option of shipping with Perc H740P RAID controllers. These are stonkingly good, as compared to the old H730 controllers, sporting 8GB of NVRAM as compared to the old 2GB. However, Linux kernel 4.14 is the earliest kernel with driver support and Debian 9.3 (at time of writing) runs 4.9.0-5. Debian can and does backport drivers into stable kernels but, at time of writing, they haven’t done so. I suspect that’s why you’re here reading this!

Foreword – Update your firmware!

These controllers are very new and, clearly, very unproven! Before you start, ensure they are running the latest firmware. We have had multiple perfectly working SSDs rejected from an array under heavy read load to the point that the array failed completely. Dell claim the latest firmware resolves it. We’ll see. If I’m honest, I regret buying 25 of the H740P and should probably have gotten the H730 until the issues with the H740P are ironed out. But we live and learn…

Anyway. Back to drivers. You have a few options…

Option 1: Use Debian 10 (Testing)

Debian strongly discourage the use of their “Testing” versions in production, though if you’re running a staging server (or you’ve got massive balls)… maybe it’s an option. Debian 10 works fine, at time of writing, and I suspect it will continue to do so.

Option 2: Install Debian on a disk not behind the RAID controller

As far as I’m aware, this isn’t how Dell ships servers with RAID controllers. The entire HBA is connected to the controller. If you can bodge in a disk or two that is not connected to the H740P, you can install Debian onto that. Once Debian is booted, you can either install a later kernel from Backports or you can compile and load the driver from Dell (see instructions below).

Once you’ve gotten Debian installed and able to detect arrays behind the RAID controller, it’s feasible that you can use something like an Ubuntu live CD (something with a Kernel new enough to use the Perc 740P) to image your disk onto the RAID array and then boot from that. I’ve not tried this, but it’s probably possible.

Option 3: Install Debian 9 with the driver supplied by Dell

Dell supply drivers for Linux. They are listed as RPMs inside .tar.gz files for RHEL and SUSE. The SUSE driver works on Debian 9 – I’ve not tried the RHEL driver.

Download the .tar.gz (e.g. UnifiedDriver_7.700.52.00_SLES12.tar.gz) from Dell. Open it with 7zip, as that will allow you to browse through the many layers of archive. Do this:

  • Open the .tar.gz with 7zip
  • Open the tar
  • Open the folder
  • Open the -src RPM
  • Open the .cpio
  • Open the .tar.bz2
  • Open the .tar
  • Open the folder

Herein lie the source files!


You’ll need to compile these on a Debian machine with an identical version of the kernel to that on which you wish to run the drivers. If you wish to install Debian from USB, Disk, etc. then you’ll need to load the drivers into the installer. As such, you’ll need to make sure that the kernel version you’re running is the same as the installer. To be safe, download a fresh install CD from the Debian website and use this to create your server and your compile machine. The compile machine can be a VirtualBox VM, or similar. It doesn’t need to have a Perc controller.

Your compile machine will need the Linux Kernel headers.

  1. apt-get install linux-headers-$(uname -r)

Get a copy of the source files onto your compile machine and `cd` into your source directory.

Run the compile script:

  1. ./compile.sh

This will create a file called megaraid_sas.ko. That is your driver. You can copy this off your compile machine.

Loading driver such that the Debian installer can “see” your RAID array

If you have physical access to the server, put this on a USB stick. If you don’t, the easiest way to use this in the Debian installer on a Dell is to make it into a floppy disk image and mount that with iDrac. You can use MagicISO on Windows for this. Open MagicISO and:

File→New→Disk Image→2.88MB. Drag and drop your megaraid_sas.ko into the file pane on the right. Click File→Save As and save the file as something like driver.img.

The driver.img can be remotely loaded into iDrac (assuming iDrac Enterprise).

Start the Debian installer, ensuring that you do a Graphical Installation.

After network setup, hit Ctrl-Alt-F2 to enter a shell.

Find the device name of the USB or floppy disk image using fdisk (e.g. /dev/sdb). You can identify it based on size (floppy disk will be about 3MB).

  1. fdisk -l

Mount the disk (changing the device name and filesystem type appropriately. ext2 is right for the floppy image):

  1. mount /dev/sdb /mnt -t ext2

Copy the driver across:

  1. cp /mnt/megaraid_sas.ko /lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/

Load the module:

  1. modprobe megaraid_sas

Unmount the USB/floppy disk:

  1. umount /mnt

Hit Ctrl-Alt-F5 to get back to the GUI.

Loading the driver into the installed OS, such that it boots correctly

Once you’ve installed Debian and are still in the installer, on last screen before reboot, hit Ctrl-Alt-F2 to enter shell again.

Load the module into initramfs and update:

  1. cp /lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/megaraid_sas.ko /target/lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/
  2. chroot /target
  3. echo megaraid_sas >> /etc/initramfs-tools/modules
  4. update-initramfs -u
  5. exit

Reboot, and Debian should boot.

When you update your kernel

On every kernel update, you’ll need to recompile and install the driver. DKMS can do this for you, but here’s the manual instructions. It’s really important that you do this before rebooting with the new kernel else it will not boot. You also won’t be able to boot older kernels… so beware!

Compile the driver against the kernel that you have upgraded to, as per the instructions above. You can do this on the server itself or on a separate compilation machine, as long as the right kernel headers are installed.

Copy the driver to the server and `cd` into its directory. Now copy it to the right place and update initramfs:

  1. cp ./megaraid_sas.ko /lib/modules/$(uname -r)/kernel/drivers/scsi/megaraid/
  2. update-initramfs -u

You can now reboot into the new kernel.

I updated my kernel and rebooted already, now it’s broken

Oh dear. You can fix this by booting something like an Ubuntu Live CD (i.e. something that has the drivers for the controller already). In this, you can mount the partitions from your Debian install. If you have multiple partitions, ensure that at least /, /boot and /var are mounted.

You can now chroot into your root mount (e.g. chroot /mnt/debian) and then following instructions above (copy the driver and update initramfs).

Some more revelations

[10/03/2018] – The latest firmware to unlock the full 8GB of cache has been released. Be sure to upgrade to it.

[05/03/2018] – We had 3 disks in a 4 disk RAID 10 array fail. That was messy. Turns out it’s a bug in the controller firmware and the latest firmware fixes it. Be sure to upgrade!

[05/02/2018] – Today my colleague noticed that megacli only reports 4GB of cache on this controller. It turns out that Dell couldn’t get it working with 8GB and have promised to release a firmware update some time in March to unlock the full 8GB. For now, you’ve got 4… which is still better than the 2GB of the older models.

[23/02/2018] – An SSD failed and needed replacing. It was possible to mark the disk as offline, but attempts to mark it as missing or prepare it for removal failed. The assumption is that megacli isn’t compatible with these controllers at the moment. Storcli didn’t work either. In the end, my colleague had remote hands remove and replace the drive. This caused the controller to mark it as Foreign. megacli was used to remove the foreign flag from the drive. Following this, he ensured that auto rebuild was enabled for the controller and then set the disk as a hot spare. This caused the controller to add it back to the array and rebuild. If anyone else has found a nice way to replace disks in an array, please add a comment!

Proxmox: Shrinking disk of an LVM backed container

First, shut down the container and ensure it’s not running.

List out the LVM logical volumes:

  1. lvdisplay | grep "LV Path\|LV Size"

Choose the disk you want to resize – you’ll see they’re named by container ID and disk number. For example, /dev/vg0/vm-205-disk-1.

Check and fix the filesystem, just incase:

  1. e2fsck -fy /dev/vg0/vm-205-disk-1

Resize the file system. It is advisable, at this point, to set this to 1GB smaller than you actually want it (e.g. 7G when you want 8G):

  1. resize2fs /dev/vg0/vm-205-disk-1 7G

Resize the LVM LV to the actual size you want it to be:

  1. lvreduce -L 8G /dev/vg0/vm-205-disk-1

Resize the file system to fill the LVM LV:

  1. resize2fs /dev/vg0/vm-205-disk-1

Finally, edit the configuration for the container such that Proxmox reports the correct size for the disk. You will find this at /etc/pve/lxc/205.conf where 205 is your container ID.

You can now start your container and check the disks’ sizes:

  1. df -h

Proxmox Containers: Manage network interfaces file yourself

Proxmox will manage the network interfaces file (/etc/network/interfaces on Debian) for you, on containers. This is sometimes annoying – for example when you want to add static routes with a post-up.

If you simply touch /etc/network/.pve-ignore.interfaces inside the container then it will let you manage it yourself and will thus obey any post-up lines that you add.

Hetzner: Routing IPv6 through a firewall such as pfSense

Hetzner offer very well specified physical servers at extremely low prices. I’ve used them for many years and they’ve proved to be extremely reliable. With each server, Hetzner will give you a single IPv4 IP and a /64 IPv6 subnet. You can also run virtualization software such as Proxmox and it’s often desirable to run a firewall such as pfSense on a virtual machine to protect the other virtual machines.

All good in principal, but the /64 IPv6 subnet has caused some confusion. Surely you need some more address space to be able to route the /64 subnet? It turns out, no. Hetzner don’t use NDP or proper IPv6 routing… they seem to just deliver the address space to the server (probably via static NDP entries mapping your /64 to your server’s MAC address). This actually works to our advantage because you do not need to assign the physical server any IPv6 addresses in the issued /64.

Broadly, the setup looks like this:

  • Physical server does not have an IPv6 address assigned to its physical interface
  • Physical server has IPv6 forwarding turned on
  • Proxmox (thus the physical server) has a private IPv6 address assigned to the bridge (vmbr) interface that it shares with pfSense
  • pfSense WAN interface has another private IPv6 address in the same subnet as the vmbr assigned to it
  • pfSense “LAN” interface has an address from your public /64 assigned to it
  • pfSense uses SLACC to assign IPs in your /64 to the VMs behind it
  • Physical server has a route to your assigned /64, via the private IP you assigned to your pfSense WAN interface
  • Physical server has a default IPv6 route to fe80::1

Here’s a picture where the assigned /64 is 2a01:4f8:66b:12d9::/64 and the private IPv6 /64 used between Proxmox and pfSense has been chosen fairly randomly using this:

Hetzner IPv6 Routing

Here’s the relevant parts of the network config on the physical Proxmox server:

  1. auto vmbr0
  2. iface vmbr0 inet static
  3. address
  4. netmask
  5. ovs_type OVSBridge
  6. post-up route -A inet6 add default gw fe80::1 dev enp0s31f6
  7. post-up route -A inet6 add 2a01:4f8:66b:12d9::/64 gw fda2:5d88:d5a3:1d4d::2
  8. post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  9. post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  11. iface vmbr0 inet6 static
  12. address fda2:5d88:d5a3:1d4d::1
  13. netmask 64

If you have any questions, leave a comment.