Hetzner: Routing IPv6 through a firewall such as pfSense

Hetzner offer very well specified physical servers at extremely low prices. I’ve used them for many years and they’ve proved to be extremely reliable. With each server, Hetzner will give you a single IPv4 IP and a /64 IPv6 subnet. You can also run virtualization software such as Proxmox and it’s often desirable to run a firewall such as pfSense on a virtual machine to protect the other virtual machines.

All good in principal, but the /64 IPv6 subnet has caused some confusion. Surely you need some more address space to be able to route the /64 subnet? It turns out, no. Hetzner don’t use NDP or proper IPv6 routing… they seem to just deliver the address space to the server (probably via static NDP entries mapping your /64 to your server’s MAC address). This actually works to our advantage because you do not need to assign the physical server any IPv6 addresses in the issued /64.

Broadly, the setup looks like this:

  • Physical server does not have an IPv6 address assigned to its physical interface
  • Physical server has IPv6 forwarding turned on
  • Proxmox (thus the physical server) has a private IPv6 address assigned to the bridge (vmbr) interface that it shares with pfSense
  • pfSense WAN interface has another private IPv6 address in the same subnet as the vmbr assigned to it
  • pfSense “LAN” interface has an address from your public /64 assigned to it
  • pfSense uses SLACC to assign IPs in your /64 to the VMs behind it
  • Physical server has a route to your assigned /64, via the private IP you assigned to your pfSense WAN interface
  • Physical server has a default IPv6 route to fe80::1

Here’s a picture where the assigned /64 is 2a01:4f8:66b:12d9::/64 and the private IPv6 /64 used between Proxmox and pfSense has been chosen fairly randomly using this:

Hetzner IPv6 Routing

Here’s the relevant parts of the network config on the physical Proxmox server:

  1. auto vmbr0
  2. iface vmbr0 inet static
  3. address
  4. netmask
  5. ovs_type OVSBridge
  6. post-up route -A inet6 add default gw fe80::1 dev enp0s31f6
  7. post-up route -A inet6 add 2a01:4f8:66b:12d9::/64 gw fda2:5d88:d5a3:1d4d::2
  8. post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  9. post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  11. iface vmbr0 inet6 static
  12. address fda2:5d88:d5a3:1d4d::1
  13. netmask 64

If you have any questions, leave a comment.



[SOLVED] pfSense – pfsync_undefer_state: unable to find deferred state

This pfSense bug has been present since 2.2 and is still present at the time of writing this. It occurs when using limiters on a pair of pfSense servers set up in a HA cluster. When connections are received which match the firewall rule you have put in place to force the traffic into the relevant limiter, those states fail to correctly sync and pfSense will experience extremely high load values. You will see the message pfsync_undefer_state: unable to find deferred state on the console and in logs. pfSense may also become inaccessible over HTTP and SSH.

The “fix” is to disable all synchronization of state. To do this, go to System -> High Avail. Sync and untick the Synchronize states box. You will need to do this on all nodes in the HA cluster.

It should be noted that this isn’t a particularly good fix. If states do not synchronize between nodes in the cluster, TCP connections will be (not very cleanly) terminated and must be re-established. Some applications may not respond too well to this.


Resolving poor network throughput performance on pfSense running on Proxmox

There exists a bug in the FreeBSD VirtIO network drivers that massively degrades network throughput on a pfSense server. VirtIO is the interface of choice for Proxmox users and this problem can become troublesome.

The solution is to disable Hardware Checksum Offloading in pfSense. This is in System -> Advanced -> Networking tab. Tick the Disable hardware checksum offload box. You now need to reboot pfSense for this to take effect.

How to disable ICMP redirects in pfSense

When a router’s next hop gateway is in the same subnet as the previous hop, it’ll send an ICMP redirect to the previous router in order to cut itself out of the routing. In some setups, this may not be desirable.

To disable this on pfSense, go to System->Advanced and change to the System Tunables tab. Edit net.inet.ip.redirect and/or net.inet6.ip6.redirect to change their values to 0 (zero).

Creating a “Guest Wifi” with 802.1Q VLAN Tagging in pfSense and DD-WRT

Creating a “Guest Wifi” with 802.1Q VLAN Tagging in pfSense and DD-WRT

Preface: For explanation of VLAN tagging, see this post

This seems quite a common thing to do yet it’s proved to be a huge ball-ache with DD-WRT. Here’s the result of many days of faffing about attempting to get this working.

The usecase is thus… We have an established wifi network comprising of D-Link DIR-615 access points flashed with DD-WRT, a router running pfSense and a managed Gigabit switch supporting 802.1Q VLAN tagging. It was required to have a second wifi network for guests of the office to get online. The guest wifi had to be separate from the main wifi in such that the main network was secure from intrusion, guests should be sent to the company website when they log in and we didn’t want to have to double up the access points or run more structured cabling to connect them.

To cut a long story short, the D-Link DIR-615 didn’t work out. This is because only Broadcom based hardware supports hardware VLAN tagging on DD-WRT and the DD-WRT software VLAN tagging just plain doesn’t work – even on Broadcom based hardware. I ended up, after days of faffing, buying some Linksys E1000 and flashing them with DD-WRT myself. Any of the devices listed here that support 802.1Q VLAN tagging should be fine.

The start point was easy. On pfSense, under Interfaces -> (assign) -> VLANs, I created 2 VLANs on the interface that is the LAN. In my case, sk1. These were tagged 1 (primary LAN VLAN ID) and 4 (guest LAN VLAN ID).

pfSense VLAN Config

pfSense VLAN Config

This created 2 new “Network ports” under Interfaces -> (assign) -> Interface assignments called “VLAN1 on sk1” and “VLAN4 on sk1”. VLAN1 was assigned to the existing LAN interface, in place of sk1, and I added another interface called “GuestNet” and assigned VLAN4 to this.

pfSense Interface Assignment

pfSense Interface Assignment

On setting up the switch to tag the port that the pfSense interface was connected to, into VLANs 1 and 4, the router worked as expected.

I set up a static IP address on the GuestNet interface and also set up DHCP to issue IP addresses. I initially added an “allow all” firewall rule to the interface and enabled Captive Portal. Finally, I set up outbound NAT so this interface could access the Internet. Our remote server firewall rules permit traffic from our primary NAT IP so I NAT’ed to an alternative public IP for added security. I won’t go into the details of these – they’re all fairly self explanatory. Drop me a comment to this post if you’re stuck with any of this.

I tested the Captive Portal setup by hooking a laptop up to a port on the switch which was set to untagged VLAN 4. This got an IP address from DHCP and was forced to authorize through Captive Portal before getting onto the network.

Now the bit that took me a long time to figure out – DD-WRT. The first step, which works great, is to add a second wifi network. This is done as a Virtual Interface under Wireless -> Basic Settings. 

DD-WRT Virtual Wifi Network

DD-WRT Virtual Wifi Network

Assign security to this, if you require. If you’re using Captive Portal, this shouldn’t be needed.

Next, you’ll need to break this away from the primary bridge. To do this, under Setup -> Networking, create a new bridge called br1 and click “Apply Settings”. Assign the guest wifi interface that DD-WRT created to this bridge. This will typically be called wl0.1 on Broadcom devices. All being well, you shouldn’t be able to connect to guest Wifi but you should be able to connect to the private wifi.

Now we have to jump into Telnet to VLAN tag ports. Apparently, on some devices, this is possible via the Setup -> VLANs web interface but support is seemingly very flaky. There’s some great docs on doing this via Telnet on the wiki. It seems the best support is on the LAN ports and much of the time you cannot VLAN tag the WAN port.

First look at the existing LAN VLAN (often VLAN1) and see how it’s setup.

root@Propeller Communications 1:~# nvram show | grep vlan1ports
vlan1ports=1 2 3 4 5*

You can see which VLAN is the LAN one from br0 on the Setup -> Networking page. It’ll always be the lowest numbered VLAN of the two in the bridge. So if you have vlan1 and vlan2 bridged, vlan1 is the LAN. If you have vlan0 and vlan1 bridged, vlan0 is the LAN.

You’ll see here that this VLAN has ports 1-4 (all the LAN ports) as well as 5*. The 5 is the router’s internal CPU and must be in all VLANs; apparently this is 8 on Gigabit routers. The * signifies that this is the default VLAN for untagged packets and should only be on one VLAN (this doesn’t work on some hardware, like the E1000). Port 0 is typically the WAN interface – as I said, this often can’t be tagged. To tag ports, you simply suffix the port number with a t. Thus, to assign port 1 tagged to VLAN1 and VLAN4, we’d do this:

nvram set vlan1ports=”1t 2 3 4 5*”
nvram set vlan4ports=”1t 5″
nvram set port1vlans=”1 4″
nvram set vlan4hwname=et0
nvram commit

This should hopefully be self explanatory. The vlan4hwname activates VLAN4.

Be sure to tag the port on your switch into the appropriate VLANs.

NOTE: On some routers (E1000 included), the ports are labeled backwards. If it doesn’t work as intended when you plug the network into Port1, try plugging into Port4. What I did was to untag the port on the switch and ping the DD-WRT device. I then swapped the network cable between the LAN ports until the ping didn’t work, even after waiting a few moments. I enabled tagging on the switch port and the pinging started up.

Once the device has rebooted, you should be able to assign vlan4 to br1 under Setup -> Networking.

DD-WRT Network Setup

DD-WRT Network Setup

All being well, your guest wifi network should start working from the DHCP server on VLAN4.

Finally, I removed the temporary “Allow All” rules on the pfSense firewall for this and replaced them with a rule to allow ICMP echo requests to the GuestNet interface, a block of all IPv4 traffic to private networks (via an alias called PrivateNets), a block of all IPv6 traffic and finally an allow of everything else. This means guest devices cannot access any private networks by IP address, including their own, but can access the Internet.

pfSense Guest Firewall Rules

pfSense Guest Firewall Rules

And that’s it. A bit of a balls but great when you get it right. Comment if you have any Qs.