EdgeRouter Pro, Pt III: BGP with LXD
Up late again last night, dicking around. One of the primary issues remaining on our home network was the fact that I hadn’t stood up any of the LXD services, chief amongst them is Samba, which also provides backups for all our machines to the ZFS pool (where we then snapshot things, which should protect us against ransomware unless the ZFS server is compromised as well, which really isn’t in my threat model).
So it’s pretty important, so I spent a while looking at it last night, fully intending to - finally - move it over to K8s. I can’t recall what problems I had with moving it to Docker which led me to leave it on LXD, and of course I didn’t write it down either.
But the reason these services are down is that formerly my VM host straddled the two VLANs (work + home), so that the services were available on both. The reason for this was the inter-VLAN routing performance of the USG wasn’t great, but it does mean that everything on that machine is available on both networks. I could firewall it at the machine, but if it’s not immediately obvious from the last few blog posts I don’t tend to do things the easy way for some reason.
Anyway, I’m waffling - it struck me yesterday that since I use BGP to connect MetalLB for K8s onto my LAN, maybe I can do the same thing with LXD? The first thing to remember is that I have exactly one LXD host, so I could have done it with a static route, but again… easy way, pah!
The first limitation: BIRD (the BGP service on the router, my kingdom for an OpenBGPd port!) will flatly refuse to initiate a connection to a neighbor with the same IP as one that’s already active. This is probably sensible, since this is generally a misconfiguration, but in my case there really were two BGP hosts on the same IP (MetalLB initiates connections and LXD waits for them)! No matter, just configure an IP alias in my netplan config to get an extra IP, and boom, done.
Next up, I need to move the LXD bridge’s IP range to something that doesn’t conflict with anything else on my network, and while I’m at it let’s turn NAT off since the boundary router will handle that:
lxc network set lxdbr0 ipv4.nat=false
lxc network set lxdbr0 ipv4.address=10.1.0.1/16
LXD itself needs to be configured to use BGP:
lxc config set core.bgp_address=10.0.0.3:179
lxc config set core.bgp_asn=65536
This tells LXD to listen to BGP connections, but after I configured BIRD to connect to it, it was dropping the connection immediately after the first BGP packet - you need to tell LXD to expect a connection first, and you do that at the interface level, not globally:
lxc network set lxdbr0 bgp.peers.router.address=10.0.0.1
lxc network set lxdbr0 bgp.peers.router.asn=65535
You can specify a password here too, but I’ve ommitted that for blogging purposes.
Oh, and one other thing - I’m not sure what I’ve got misconfigured or what’s going on, but I had to add the statement interpret communities off; to the protocol bgp section for LXD. I’ll have to research why LXD is sending something that BIRD doesn’t want, but without this I get rejected by protocol in the logs. I could also configure BIRD to not send routes to LXD, but it’ll drop them unceremoniously anyway and it’s not like I do BGP for a living.
Now you can restart the containers (mine were all stopped), and they should have access to the internet and the router should have a single route to the LXD bridge’s entire subnet via the LXD host’s IP. The documentation for LXD’s BGP support suggests that each instance will have its own route, but I’m suspecting that’s only the case when you’re using a cluster, if there’s only one host it doesn’t make sense to fill up the routing tables with a million routes that all say the same thing.
Next up, I don’t want LXD’s dnsmasq to shift the IP around on me if I say, recreate the container, so I need to set a static IP for my LXD container. This is unintuitive, but it does work: you remove the interface from the default profile, then manually add it back, and then it lets you override the ipv4.address value with whatever you like:
lxc network attach lxdbr0 samba eth0 eth0
lxc config device set samba eth0 ipv4.address 10.1.0.3
lxc start samba
Finally, something that was specific to my network, but I’ll be kicking myself if I don’t document it here: when I first started the containers, they didn’t have a default route. I thought this was a BGP thing at first, but it was a leftover from the “straddles two VLANs” days. I had DHCP on both addresses, and wanted the default route to go out over a specific network, so I had the lxdbr0 interface configured to send an empty router list. I did this by adding dhcp-option=3 to dnsmasq.raw on the interface, but I had to remove that and restart networking on the containers to get the default route back.
Once that’s all done, all that was left was to add a firewall rule to allow machines on the “Home” VLAN to access these services and I’m all set.
Long-term, the plan is still to move the last of these services over to K8s, but this will still be handy to have if I ever want to use LXD as a KVM manager (I tried unsuccessfully to run Windows NT 4 on it).
Status Update:
With this in place, that means that the only things left on my list of shit to fix are:
- IPv6.
- SSL cert on the router.
- A VPN on the off chance I travel for work (will likely use wireguard for this).
fwagglePublished:
Modified:
Filed under:
Location:
Horsham, VIC, AustraliaNavigation: Older Entry Newer Entry