Kubernetes: BGP

So one of the primary problems I have with my shiny new Kubernetes cluster (aside from not knowing my arse from my elbow when it comes to this stuff) is that, like my Docker and LXD containers before it, the containers are inconveniently tethered to the node. I am using a NodePort for Traefik, which is fine, and then everything else that uses disk access uses local mounts.

The latter is a problem for another day, but the first thing to sort out is some sort of a LoadBalancer in order to make my network stuff work wherever I need it. When I was first starting, I was having a crack at things like kubectl expose and the LoadBalancer always stayed in PENDING state, which I later learned is normal out of the box… K8s doesn’t come with a load balancer, you have to make one (or configure one provided by your cloud provider).

The going one for bare-metal clusters seems to be MetalLB, even though it’s advertised as “alpha quality software” that “may break”, it sounds good enough to me. So I setup metallb per the instructions, and then I configure the LB (IPs changed to protect the guilty):

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 192.168.0.1
      peer-asn: 64517
      my-asn: 64518
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.168.15.0/24

Configure config.gateway.json on Unifi controller:

{
    "protocols": {
        "bgp": {
            "64517": {
                "neighbor": {
                    "192.168.0.2": {
                        "remote-as": "64518"
                    },
                    "192.168.0.38": {
                        "remote-as": "64518"
                    }
                },
                "parameters": {
                    "router-id": "192.168.0.1"
                }
            }
        }
    }
}

Next, I force-provision the gateway.

Finally, configure a service to use the LoadBalancer type. At first it did not work, because I omitted the control plane node (192.168.0.38) from the config.gateway.json, and for some reason metallb decided to attempt to announce the route from that node instead of the node the service was running on (not sure why that is). That results in errors something like this:

{"caller":"level.go:63","error":"read OPEN from \"192.168.0.1:179\": read tcp 192.168.0.38:36771-\u003e192.168.0.1:179: read: connection reset by peer","level":"error","localASN":64513,"msg":"failed to connect to peer","op":"connect","peer":"192.168.0.1:179","peerASN":64512,"ts":"2022-02-18T01:53:18.504319341Z"}

Once I corrected this, it announced fine:

root@router:/var/log# netstat -rn | grep '\(^Destination\|192.168\)'
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.15.10    192.168.0.38    255.255.255.255 UGH       0 0          0 eth1

… and it’s available everywhere.

What does this do now that nodeport doesn’t? Not much, but it will become more useful once I ditch the old Xeon 55XX machine and replace it with something cheaper to run, because I’ll run a couple of them in a proper cluster and services will be able to freely move around the nodes, without worrying about what IP goes where.

Now to work out either NFS or iSCSI to decouple the containers from the local filesystem…

Horsham, VIC, Australia

Author:

fwaggle

Published:


Modified:


Filed under:


Location:

Horsham, VIC, Australia

Navigation: Older Entry Newer Entry