Kubernetes NFS issues fixed?
Way back at the start of last year, I started “scaling out” our K8s cluster, and used NFS where possible to provide a filesystem for containers. The plan is to switch most of these out for Longhorn, but I’ll still need some of it… like Plex for instance, I am not buying enough SSDs to have longhorn be the storage backend for the media (though I probably will put the SQLite database on it).
This has presented a single nagging issue: every so often, I’ll try to schedule a pod (usually due to an upgrade or whatever) and it’ll just sit there “pending” with the error message in the log something to the effect of connection refused, which is really odd because the NFS server is up and functioning fine.
It finally did it the other morning before work, and I managed to think to look at syslog at the time, and found something slightly more helpful:
svc: failed to register lockdv1 RPC service (errno 111).
That’s still a pretty shit error message, but it’s a slightly more helpful Google-snack than the previous one. I spent a stupid amount of time trying to work out what was wrong with the RPC daemon on the NFS server to no avail, and I thought it was weird because in the past, if I drained+rebooted the client node, it would typically work fine after that.
It was only after a shameful amount of screwing around that I realized what I actually seem to need is the rpcbind service on each of the client nodes. This raises questions like “why?” (I am not an expert in NFS by a long shot, I do not know how the different pieces interact with each other) and more importantly “how did it ever work without it?”
My best guess is that it works fine when there’s only one mounted directory for each NFS export, and what’s probably happening is something is not cleaning up after itself and a directory is still mounted in another namespace, which is why rebooting the client node fixed it? Complete guess though.
