Chassis clustering a Juniper SRX firewall via a switch

Intro

It is recommended that clustered SRX devices are directly connected. To do this, you need to run 2 cables, one for the control plane and the other for the fabric. This is sometimes not easy (or cheap) in a data centre environment where the firewalls are in different racks – especially given that the control link must be copper, on most SRX devices, and is thus limited to 100m.

You can also cluster SRX devices by connecting the links into a switch. A common use for this would be to cluster 2 firewalls, each in different racks, via your core switching chassis cluster.

tldr; (sorry, it’s still quite long)

You’ll need to read the chassis cluster guide. Here’s the one for the SRX 300, 320, 340, 345, 550 and 1150. On pages 44 and 45 you will see diagrams of how the devices must be connected. Most SRX devices enforce the use of a particular port for the control plane. When clustered, the control port will be renamed to something like fxp1. The fabric can usually be any port you like.

Connect the control and fabric ports of each SRX device into your switch.

The switch ports need to be configured like so:

  • MTU 8980
  • Access port (no VLAN tagging)
  • A unique VLAN – control and fabric need their own VLAN (e.g. control = 701, fabric = 702). The VLAN should have only 2 ports in it (e.g. firewall 1 control port and firewall 2 control port)
  • IGMP snooping turned off
  • CDP/LLDP/other junk turned off

You must first delete the configuration for the control interface, on each firewall, if it exists. If you don’t do this, you’ll be stuck in a strange state when the firewalls come back up as they will error when loading the configuration. If you can, you may as well delete all interfaces:

  1. edit
  2. delete interfaces
  3. commit

Log into each firewall via its console port. On firewall 1:

  1. set chassis cluster cluster-id 1 node 0 reboot

On firewall 2:

  1. set chassis cluster cluster-id 1 node 1 reboot

Wait for the firewalls to finish rebooting. Check the status of the cluster like so:

  1. show chassis cluster status

One node should be primary and the other secondary. Make sure you wait for all the “Monitor-failures” to clear before continuing.

Now you can work solely on the primary node… so you can log out of the secondary. You’ll need to assign the physical ports that you connected up for the fabric to the interfaces fab0 and fab1. Note that the ports on the secondary device will have been re-numbered. That is to say the on-board ports will no longer be ge-0/0/something, but will rather be something like ge-5/0/something. The number prefix depends on the model of SRX and, specifically, how many PIM slots it has. You’ll need to read the chassis clustering guide to work out what to do for your model.

  1. set interfaces fab0 fabric-options member-interfaces ge-0/0/2
  2. set interfaces fab1 fabric-options member-interfaces ge-5/0/2
  3. commit

Check the full cluster status:

  1. run show chassis cluster interfaces

You should see both control and fabric as Up.

Config for Juniper EX Series Switches

The below is the config for an EX series virtual chassis (VC). It’s simpler than if you had unclustered switches as you don’t need to worry about carrying VLANs between switches. If you don’t have a VC, you’ll need to do a little more on top of this.

  1. vlans {
  2. VLAN701 {
  3. description fw_control_link;
  4. vlan-id 701;
  5. }
  6. VLAN702 {
  7. description fw_fabric_link;
  8. vlan-id 702;
  9. }
  10. }
  11. protocols {
  12. igmp-snooping {
  13. vlan VLAN701 {
  14. disable;
  15. }
  16. vlan VLAN702 {
  17. disable;
  18. }
  19. }
  20. lldp {
  21. interface ge-0/0/17.0 {
  22. disable;
  23. }
  24. interface ge-4/0/17.0 {
  25. disable;
  26. }
  27. interface ge-0/0/18.0 {
  28. disable;
  29. }
  30. interface ge-4/0/18.0 {
  31. disable;
  32. }
  33. }
  34. }
  35. interfaces {
  36. ge-0/0/17 {
  37. description FW-01_Control_Link;
  38. mtu 8980;
  39. unit 0 {
  40. family ethernet-switching {
  41. port-mode access;
  42. vlan {
  43. members VLAN701;
  44. }
  45. }
  46. }
  47. }
  48. ge-0/0/18 {
  49. description FW-01_Fabric_Link;
  50. mtu 8980;
  51. unit 0 {
  52. family ethernet-switching {
  53. port-mode access;
  54. vlan {
  55. members VLAN702;
  56. }
  57. }
  58. }
  59. }
  60. ge-4/0/17 {
  61. description FW-02_Control_Link;
  62. mtu 8980;
  63. unit 0 {
  64. family ethernet-switching {
  65. port-mode access;
  66. vlan {
  67. members VLAN701;
  68. }
  69. }
  70. }
  71. }
  72. ge-4/0/18 {
  73. description FW-02_Fabric_Link;
  74. mtu 8980;
  75. unit 0 {
  76. family ethernet-switching {
  77. port-mode access;
  78. vlan {
  79. members VLAN702;
  80. }
  81. }
  82. }
  83. }
  84. }

Debugging

Check the status of nodes in the cluster:

  1. show chassis cluster status

Find out which interfaces are in the cluster:

  1. show chassis cluster interfaces

This will show you if data is being sent/received over the control and fabric links:

  1. show chassis cluster statistics

Check if the arp table has entries for the other firewall (i.e. they have layer 2 connectivity):

  1. show arp | match fxp

Configuring Node Specific Things

When you change the configuration on one node, it will be automatically applied on the other nodes. However, you will want some settings that are specific to a single node – for example hostname and management IP. You can set these settings into groups <nodename>, e.g. groups node0.

You’ll also need to set apply-groups “${node}” in order to have the node specific configuration apply to the right nodes.

Example config below for configuring hostname and management IP:

  1. groups {
  2. node0 {
  3. system {
  4. host-name fw-01;
  5. }
  6. interfaces {
  7. fxp0 {
  8. unit 0 {
  9. family inet {
  10. address 192.168.1.1/24;
  11. }
  12. }
  13. }
  14. }
  15. }
  16. node1 {
  17. system {
  18. host-name fw-02;
  19. }
  20. interfaces {
  21. fxp0 {
  22. unit 0 {
  23. family inet {
  24. address 192.168.1.2/24;
  25. }
  26. }
  27. }
  28. }
  29. }
  30. }
  31. apply-groups "${node}";

[SOLVED] pfSense – pfsync_undefer_state: unable to find deferred state

This pfSense bug has been present since 2.2 and is still present at the time of writing this. It occurs when using limiters on a pair of pfSense servers set up in a HA cluster. When connections are received which match the firewall rule you have put in place to force the traffic into the relevant limiter, those states fail to correctly sync and pfSense will experience extremely high load values. You will see the message pfsync_undefer_state: unable to find deferred state on the console and in logs. pfSense may also become inaccessible over HTTP and SSH.

The “fix” is to disable all synchronization of state. To do this, go to System -> High Avail. Sync and untick the Synchronize states box. You will need to do this on all nodes in the HA cluster.

It should be noted that this isn’t a particularly good fix. If states do not synchronize between nodes in the cluster, TCP connections will be (not very cleanly) terminated and must be re-established. Some applications may not respond too well to this.

 

Configuring the permissions of a Samba share

Configuring the permissions of a Samba share

Samba share permissions can be a bit fiddly. The user and group IDs which own the file on the Samba server will propagate over to the client machines, which will enforce local permissions themselves.

Ideally, you want to have the same users/groups on all machines. This isn’t always practical but could be achieved with a config management tool such as Puppet or SaltStack, or indeed by backing your local users from an LDAP server.

If this is not possible, the following is suggested:

On your Samba server

  • Create a group which will own all the files, for example samba-users
  • Add all of your Samba users to the group you created – e.g. adduser downloader samba-users
  • Chown all of your shared files and folders to root:samba-users
  • Chmod all of your shared files to 660
  • Chmod all of your shared folders to 770
  • Add the below to the config for your share to enforce the above for all new files and folders:
  1. create mask = 0664
  2. force create mode = 0664
  3. directory mask = 0775
  4. force directory mode = 0775
  5. force group = samba-users

On your client server(s)

  • Create a group which will be able to access all the files on the share, for example samba-users
  • Obtain the group ID (GID) from /etc/group for this group
  • In the mount options of the share (in /etc/fstab) add the uid 0 (root) as in the below example
  • In the mount options of the share (in /etc/fstab) add the gid as in the below example where the GID is 1002
  1. //192.168.1.123/downloads /mnt/downloads cifs username=downloader,password=foobarbaz,iocharset=utf8,uid=0,gid=1002 0 0

If you u(n)mount and remount the share you will see that all the files are now owned by the group you specified in fstab.

Disclaimer

There might be a better way… feel free to comment if you know what it is.

SOLVED – “mount error(13): Permission denied” when doing cifs mount on LXC container (Proxmox)

When trying to do a command like this on a system running inside an LXC container on Proxmox:

  1. mount -t cifs '\\172.55.0.60\downloads' -o username=myuser,password=mypass /mnt/downloads

 

Linux threw the error mount error(13): Permission denied. `tcpdump` showed that no traffic was leaving the container and `strace` didn’t throw up a lot of useful info.

dmesg said this:

  1. [171150.670602] audit: type=1400 audit(1471291773.083:167): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/shm/" pid=59433 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"

This reddit post finally yielded the answer. You need to edit /etc/apparmor.d/lxc/lxc-default and below the last deny mount line, add this:

  1. allow mount fstype=cifs,

The final config file will look something like this:

  1. # Do not load this file. Rather, load /etc/apparmor.d/lxc-containers, which
  2. # will source all profiles under /etc/apparmor.d/lxc
  3.  
  4. profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  5. #include <abstractions/lxc/container-base>
  6.  
  7. # the container may never be allowed to mount devpts. If it does, it
  8. # will remount the host's devpts. We could allow it to do it with
  9. # the newinstance option (but, right now, we don't).
  10. deny mount fstype=devpts,
  11. allow mount fstype=cifs,
  12. }

Now restart apparmour:

  1. systemctl restart apparmor.service

Shut down your VM and start it again.

Your mount command might well work now. If not, check logs again to be sure it’s not a secondary problem (e.g. incorrect hashing algorithm).

HP ProLiant MicroServer Gen8 as a home server

I picked up a HPE ProLiant Gen8 G1610T with the intention of turning it into a Virtual Machine host to run a number of things, most prevalent a Plex media server. I then got a bit carried away and did a few upgrades to it. Details are below:

The Base Server

HP regularly have cashback deals on these servers. That makes them ludicrously cheap. I got mine on eBuyer for £114.98 after cash back. They come, stock, configured as follows:

  • CPU: Intel Celeron G1610T (Dual Core) @ 2.3 GHz
  • RAM: 4GB PC3-12800 Unbuffered ECC
  • Network: 2 x 1Gbit interfaces
  • Remote Management: iLO 4 Essentials presented as a separate (third) 1Gbit network interface
  • RAID controller: HP Dynamic Smart Array B120i with 4 removable (not hotswap) front bays. It’s fake RAID, but it supports 0, 1 and 10
  • Expansion card slots: 1 x PCI-E 16 slot
  • PSU: 1 x 200W. Bit of a shame there’s no option to have a redundant pair… but there’s just not enough space
HPE ProLiant Gen8 G1610T

HPE ProLiant Gen8 G1610T

Doing the Work

All of the work was ludicrously easy to do. The motherboard is on a tray that slides out the back of the server. You just need to unplug the 4 cables first.

There’s a tool on the front of the server which can be used to remove the heatsink from the CPU… and a standard flat-head screwdriver will do also.

HPE ProLiant Gen8 G1610T Without Case

HPE ProLiant Gen8 G1610T Without Case

HPE ProLiant Gen8 G1610T Motherboard Removal

HPE ProLiant Gen8 G1610T Motherboard Removal

CPU Upgrade

The CPU that ships with the server is absolutely fine for most workloads. It benches at 2349 on CPU Benchmark. It supports ECC RAM, which makes it great for a file server, and packs enough punch to run Plex with a single transcoding channel. It supports VT-x making it good for a low-usage virtualization server, however it doesn’t support VT-d so you cannot pass the disk controller straight through to a VM.

The stock CPU is 45w and can, apparently, be swapped for any low power Sandybridge i3, i5 or i7. I tried an Ivybridge I5 but this didn’t work particularly well.

I swapped mine for an Intel Xeon E3-1240 V2 (quad core with HT) @ 3.40GHz. This is actually 69w but the board handles it fine. I turned off auto-management of the chassis cooling fan and instead opted to have it run on full power to help circumvent any heat related issues. This CPU benchmarks at a whopping 9264 – almost 4x more powerful than the stock. It also supports VT-d.

RAM Upgrade

4GB is a bit low, by today’s standards. I opted to upgrade to 16GB (2x8GB) of Kingston PC3-12800 Unbuffered ECC RAM.

If you swap out the CPU for a Sandybridge i3/i5/i7, I understand you still need to use ECC RAM despite the CPU not explicitly supporting this.

HPE ProLiant Gen8 G1610T RAM Upgrade

HPE ProLiant Gen8 G1610T RAM Upgrade

Real RAID Controller

I used the spare PCI-E slot to run a 3WARE 9650SE-8LPML RAID controller, with write cache and BBU. This can be passed directly to a VM, using the VT-d functionality of the CPU. The HBA cable inside the Microserver plugs directly into this, so installation was simple. This gives phenomenal disk write performance as well as the ability to hot swap hard drives, which the native “fake” RAID doesn’t support.

3WARE 9650SE-8LPML RAID controller with BBU in HPE ProLiant Gen8 G1610T

3WARE 9650SE-8LPML RAID controller with BBU in HPE ProLiant Gen8 G1610T

SSD

An SSD can be mounted in the optical drive bay that comes in the server. Many people that I saw online were using gaffa tape to attach this. I used a cheap 9.5mm SATA hard disk caddy to mount a 1TB Samsung SSD. There is a spare SATA port on the board that this can plug into and there is a 4 pin floppy disk power connector that it can be powered off.

These caddys turn SATA into slimline SATA, thus to do this, you’ll need a slimline SATA adaptor which takes its power from a 4 pin floppy disk connector. These are, in this day and age, like rocking horse shit. You may need to butcher one together using a few other readily available connectors and some heat shrink.

In order to boot from this SSD, you need to change the RAID mode of the on-board controller to AHCI SATA. This presents a bit of a problem if you’re using it with other drives, as it’ll only see this SSD. Apparently you can change it to Legacy SATA which will allow you to access the other drives, but not in RAID. If you don’t want to boot from the SSD, you don’t need to change anything.

SSD in 9.5mm caddy in HPE ProLiant Gen8 G1610T

SSD in 9.5mm caddy in HPE ProLiant Gen8 G1610T

Spinning Disks

I simply fitted 4x 3TB disks into the bays at the front. Because of the real RAID controller, these have become hot-swappable… unlike if they were on the stock controller. I opted for faster 7200 RPM disks, in the hope of getting a bit more speed out of them. These are configured in RAID5, giving just over 8TB total storage.

Front Disk Slots of HPE ProLiant Gen8 G1610T

Front Disk Slots of HPE ProLiant Gen8 G1610T

iLO 4 Advanced

The iLO that ships with the server isn’t great. It doesn’t do remote management, remote media, etc. If you look hard enough (or not very hard at all), you might be able to find a way to upgrade it to iLO 4 advanced. This supports full remote administration.

HP ILO 4

HP iLO 4

Network

The server comes with 2 x 10/100/1000 network interfaces, plus a separate 10/100/1000 interface for the ILO. I have used an LACP bond to provide additional throughput and redundancy across the primary pair of NICs.

Software

I have used Proxmox as the virtualization platform. It’s free and fairly feature-some. Having used it for a few years in production environments, at work, it’s proved to be reliable and useful.

What Now?

I use the box to run various VMs including a NAS running Samba, Plex, Asterisk, a Radius server and some home automation stuff. It’s been reliable and the server is hardly sweating yet.

Resolving poor network throughput performance on pfSense running on Proxmox

There exists a bug in the FreeBSD VirtIO network drivers that massively degrades network throughput on a pfSense server. VirtIO is the interface of choice for Proxmox users and this problem can become troublesome.

The solution is to disable Hardware Checksum Offloading in pfSense. This is in System -> Advanced -> Networking tab. Tick the Disable hardware checksum offload box. You now need to reboot pfSense for this to take effect.