Network: Rejiggered (also: EdgeRouter Pro, Pt II)
Over the last couple of days, I spent more time figuring things out on my modified EdgeRouter Pro. Part way through, I got bored and decided to install OpenBSD, just to be sure. Installation was very easy, and it was starting to look like we might be on to a winner - I did’t have the read-only filesystem I wanted to protect the flash, but others have already solved that problem. With pf(4) disabled, we were pushing packets around at pretty-much gigabit speeds (not quite, but close enough) between iperf3 on two different instances:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 981 MBytes 822 Mbits/sec receiver
Unfortunately, the story is not the same with pf(4) enabled (with just the default ruleset, nothing crazy and no scrubbing):
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 502 MBytes 421 Mbits/sec receiver
That’s… suboptimal. I don’t have gigabit internet currently (maybe one day), but what I do have is a Samba server on one VLAN and clients that need to access it on another VLAN, and at this stage the routing between them needs to happen at the router.
So it was back to OpenWRT (which due to installing a modified bootloader to boot BSD meant flashing back the EdgeRouter firmware, but it didn’t take long to do via serial console), I configured everything and started looking at getting it set up. I swapped the wireless clients over to it on Thursday night, which honestly caused more grief than it solved, but on Friday night I swapped everything over to it and unplugged the USG-3.
I spent a while figuring out how to get BGP working, in fact that’s what I spent most of Friday night doing. I started out with BIRD2 (the combination ipv4 and ipv6 version), but it crashed immediately on the octeon architecture… a better individual would have made a debug build and tried creating a bug report, but I simply moved on.
Next up, installed FRR, and configured it. I gotta say, if you rewrite something like Quagga, and don’t take the opportunity to re-do the configuration file format, I really have to question your motivations. It might be that they didn’t want to alienate Quagga nerds, but from what I can tell those folks are all still using Quagga anyway so I’m not sure what’s gained. Anyway, after quite some time I was pretty sure I had FRR configured correctly… except it wasn’t routing. It was accepting the routes (Showing as *> (active route) and *= (perfectly fine route but not the active path)), but simply not injecting them into the kernel routing table.
I spent hours on this, before finally giving up and installing bird1-ipv4, since I don’t need BIRD to handle the IPv6 portions of my network. I started out with the “luci” module for it, but since I didn’t really know what I was doing, this caused more harm than good and I learned that if the configuration is invalid, OpenWRT doesn’t seem to have any back-off on the start-up of this daemon, resulting in me OOMing the device several times before I gave up, blue away all the UCI configuration, and configured it by hand with a file like Dog intended.
But once I got that working, it didn’t take very long at all to make it work and MetalLB was announcing routes correctly and I was able to get to bed.
I still have a few more things I’d like to do with it:
- IPv6 basically doesn’t work at all.
- I’ll probably, since I have 8 ethernet ports to work with, split the VLANs out onto their own physical interfaces, to increase bandwidth at the router.
- None of the LXD services are back up, due to network changes. I’ll probably just move those services to K8s, because the BGP stuff for LXD doesn’t look super convincing.
- HTTPS on the router, probably via LetsEncrypt.
fwagglePublished:
Modified:
Filed under:
Location:
Horsham, VIC, AustraliaNavigation: Older Entry Newer Entry