Kubernetes: Certificate Expired
I woke up this morning to find a critical security update for Mastodon, which I better apply lickety-split! Grab the hash for the latest container version, have a quick squizz at the changelog, and then throw it in one of the sidekiq deployment manifests to start the container download and…
error validating "apps/mastodon/config.yml": error validating data: failed to download openapi: the server has asked for the client to provide credentials; if you choose to ignore these errors, turn validation off with --validate=false
Huh, that’s new. What about…
$ kubectl get pods
E0215 07:20:28.488070 1962 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0215 07:20:28.494899 1962 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0215 07:20:28.500210 1962 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0215 07:20:28.506239 1962 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0215 07:20:28.512440 1962 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)
… interesting. It seems that in a feat of homeprod success, I’d finally, some two and a bit years after first experimenting with Kubernetes, managed to keep a cluster up long enough for my certificate to expire. I didn’t even know that could happen, and judging by comments when mentioning it on Mastodon, it seems I wasn’t alone. Hooray for blowing shit up on a regular basis!
Anyway, fixing it up is easy enough: kubeadm certs renew all
Then copy /etc/kubernetes/admin.conf to my workstation’s ~/.kube/config and it works again. I need to restart kube-apiserver and a few other processes, but that’s a job for another day as I have work in a few minutes.
Update: 2024-02-17: Not so! I mentioned on the thread that among the reasons for burning it all down and rebuilding my cluster was that at one point I let it get two versions behind. Kubernetes supports multiple versions concurrently, so they’re not EOL, but it doesn’t support upgrading across multiple minor versions. Since I use Alpine Linux for most of my hosts, it only lets you install the latest version, so if you wait too long, there’s literally no way to get the minor version you require.
I noted in early January that I was in such a situation there, but Alpine still had 1.28 so I reasoned that I still had time. I figured this weekend was the time, and guess what? I did not have time, Alpine carries 1.29. So despite having kept the cluster alive for >1 year, I burned it down a few days later.
At least I’m good at it now, nothing major went wrong and I had the entire thing back up in under an hour of actual downtime… the major delay was that I fat-fingered the upgrade command for the one Ubuntu host in the cluster, and required a second reboot, and since it’s the disk server nothing else could come up in the mean time.
fwagglePublished:
Modified:
Filed under:
Location:
Horsham, VIC, AustraliaNavigation: Older Entry Newer Entry