Kubernetes: Busted again…
It’s actually been a hot minute since my homeprod Kubernetes cluster did anything wayward… and I’ve mostly just left it alone because if it ain’t broken…
Alas, Ubuntu finally pushed out patches for CVE-2026-98432 (a rather nasty bug in TCP/IP that theoretically can lead to RCE with any open socket if the stars align) so I needed to reboot everything. Should be easy enough, drain all the nodes, apply the updates, reboot, uncordon the nodes and wait three to five business days for Mastodon to start again.
Unfortunately I managed to not pin the kubelet package version, and it got upgraded. A smarter individual would have gone “whoops, that was not part of the upgrade plan” and rolled it back, but I am no such individual. I have once again let Kubernetes slip more than one release beyond me, soI cannot upgrade the cluster, but I am pretty good at destroying and rebuilding my cluster these days, so fuck it.
Doubling down on my misfortune was the fact that several things have gone from deprecated to gone in the time I’ve been neglecting it, so it took me nearly two and a half hours to get everything back up and running (and Longhorn still isn’t running, but I haven’t managed to put it into production yet anyway). MetalLB was outdated and refused to start, among other things.
But I’m back up to date now and everything else is running.
In the process though, I learned that one of the fans in my disk server has ceased spinning. It’s a Nidec UltraFlo V80E12BHA5-57 and holy fuck are they expensive. I have them choked the fuck down anyway, so I’m considering my options for switching them out for Noctuas before summer gets here. At the moment I’m just down one fan though.
Solar panels: washed
It’s been not quite 5 years since we installed the new PV array, well over five years since we bought the house, and who the heck knows how long since the original PV array was installed. In that time, I have “washed” the panels exactly once - I sprayed truck wash on, and then hosed off, the old solar array a few years back.
While having extensive monitoring at 15 second intervals of our entire house’s power consumption and generation, I don’t actually keep anything but coarse aggregates for any length of time, which is really freakin’ annoying in retrospect. But I did lament last year that even during the optimal summer days, I never really saw the old 2kW system ping off it’s 2000W nameplate capacity any more. I reasoned, since the installation is old enough that the section of the inverter manual that covers software shows Windows Fucking XP that I was probably starting to get into the period where the panels begin to noticeably degrade. I figured I was down around 10% at this point, and they’d had a fair innings.
Anyway, a few weeks back we noticed that hey these actually look kinda filthy. And here’s the thing about solar panels, the vast majority of literature out there suggests that as long as the panels aren’t flat, the rain washes them. The loudest voices telling you that they need additional cleaning are the people selling solar panel washing services. So I was extremely skeptical to say the least that it was worth my effort.
But this weekend, we had a nice cool day (I did not want to spray cold garden hose water on the panels when they’d been out in 40C weather, they are glass after all), so I went and bought another bottle of truck wash (the 5L we bought nearly 10 years ago being exhausted the last time I washed the house windows), and I sprung for a nice squeegee with a 6’ handle, climbed up on the roof like a dickhead and began the task.
The filth that came off the old panels? The water looked like coffee. Definite, visible improvement on them. The newer ones were substantially cleaner, but there was still a few patches of bird-shit and fuck it, in for a penny and all that, so I washed them off too. All told it took nearly two hours to do. While I was up there I washed out and switched on the evaporative cooler for good measure.
Update: 2026-10-27: Monday and Tuesday were finally half-decent solar days, and the results of my work became clear. Not even full sun (old array is west-facing, and I was looking at about 11:30 or so) and it’s doing 2008W. So I guess I have a new chore now, periodically.
My working theory about this is that most of the literature surrounding it assumes panels that are at a much steeper angle than mine are - they’re generally around 30 degrees, my roof pitch is only 10 degrees (or is it 15? Either way it’s a very shallow roof) and the panels are mounted perfectly flat against it, so I’m thinking that the rain doesn’t wash off fast enough to peel the dust away before it gets stuck on? That’s my guess anyway.
I guess it’s also possible that my data was incorrect, but I spot checked quite a few days last year and I would think if it had actually gone to 2kW output I’d have noticed.
More Dryer Trouble!
It’s been barely four months since I “fixed” our dryer, even though I was fairly sure it was not a permanent fix… I was still disappointed yesterday morning when we noticed it making a horrific squeal, the telltale noises of a bearing gone wrong. At first I rang the local mobile repair guy who’s pretty highly rated, but he didn’t get back to me (not sure if he’s on holidays or what). So today after work I popped the back off it, cracked into it, and sure enough one of the bearings (the front one, which kept marching out of it’s hidey-hole previously) is very chunky inside when spinning it by hand.
These look like standard bearings (in fact they look exactly like the ones used in vacuum cleaners?), so into Wimmera Bearings to get some newies to get me out of the woods today. A$8 each or thereabouts, but also while I was looking at it I noticed a suspicious ring on the dust cover where it looks like something has rubbed on it. That’s not supposed to be there, and likely explains why the thing has died prematurely, so a hypothesis began forming.
The outer bearing shell on both bearings (front and back) sit in a plastic housing, not held in particularly well in any direction except the permiter (so they’re not really retained front-to-back at all). The bearings are located front to back by the pulley pressing against the inner bearing shell, a shim between the inner bearing shells of both bearings, and then the squirrel-cage fan is supposed to be pressing against the front inner shell, which holds everything together against the center shim (I would post exploded diagrams, but there aren’t any - the Fisher and Paykel docco is missing components on it).
That’s the plan anyway, but when I look at the rear of the fan on mine, it doesn’t look like any of the photos of new spare parts… it actually looks like a piece is missing. Indeed there’s a telltale wear mark that looks suspiciously like a mirror image of the marks on the front bearing?
So I bought a small M8 washer to act as a shim to hopefully get the fan away from the bearing dust cover and let us continue to use the dryer, but I think the “real” solution is to buy a new bearing housing (so I have the proper F+P bearings, in the event that they’re not just standard) and a new fan, because I’m fairly sure that the one that I have is in some way deformed and that’s what’s been bothering the bearings.
So now we’ll see how it goes ordering parts from F+P, and hopefully we’re nearly at the end of this saga.
Laptop: Busted, won’t boot
My new(-ish now) laptop has mostly performed great over the just-over-a-year I’ve had it, and surprisingly I hadn’t managed to ratfuck my Linux installation in that time. Well, no time like the present, because on Monday I ran pacman -Syu tcpdump, it installed a pile of packages including a kernel update, I thought nothing of it and shut the machine down. Yesterday when I went to use it, and I was faced with a dreaded message: Please enter the recovery key for disk BLAH BLAH BLAH
Whoops.
I figured that something had failed to sign, but it took me a bit to figure out. First of all, I couldn’t get into the recovery immediately, because I was typing the key wrong. Note to future self: the hyphens in the recovery key are part of it, they’re not optional for aesthetic purposes.
But I managed to boot from a USB key after turning secureboot off, but then I was unable to decrypt and mount the disk, so I figured it was maybe because my USB key was too old (from ~2025) so I made a new one. It was at that point I figured out the hyphen thing, got into it, and no, it looks like everything worked correctly?
I started thinking about what the hell could have happened, noting that it was exactly 451 days since I installed it and maybe something expired. I checked sbctl verify and everything is signed, everything should be happy, but in the end re-enrolling a new key into luks and the TPM worked:
systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 /dev/sdX
I rebooted, flipped secureboot back on, and it’s straight into it without needing the key, happy days. But what caused it?
I was racking my brain trying to work it out when Avi came through today with the release notes I didn’t read:
Starting with package version
42-1, the mkinitcpio systemd hook now includessystemd-pcrosseparator.service(as intended by systemd v261).This affects the measurements of PCR values
0-7,9and12-14. If you configured the systemd hook and unlock LUKS partitions, that depend on these values, you need to re-enroll the TPM2 in use.
I checked my pacman.log and sure enough, I upgraded mkinitcpio to that exact version right before the shit hit the fan, so problem solved I guess? Operator error.
Whoops.
Frenzal Rhomb again - Barwon Heads
It’s still going fairly slow as far as shows go, so we saw Frenzal Rhomb for the third time, and bought tickets for the fourth! You see, we originally bought tickets for the show in Melbourne many months ago. Sabriena liked one of their openers ages ago, Sick Visor, and we were optimistic that they’d be opening for them again as they’ve been collaborating a bit. Alas, they were not the opener for the show we bought. Aww well.
However, in the months between when we bought the tickets and showtime, two confounding factors became apparent. First of all, Sabriena had a local running event in Horsham on Sunday, and the show was Saturday night. That’s doable, but uncomfortable. Either we leave Melbourne immediately after the show, and drive all the way through and get home at about 3am. After a long day, that’s not particularly smart. The alternative is we stay the night in Melbourne, and get up really early (probably out of bed and on the road by about 6:30!) to make it back in time for the event. Either way is not particularly palatable.
But a few weeks back, Sabriena found out an interesting piece of information: Frenzal Rhomb were playing a show the night before, in Barwon Heads (down by Geelong), and Sick Visor were opening for them. The tickets weren’t expensive, so we made the decision: we’d buy tickets for that one, go to it, and I’d try see if someone else wanted to use the Melbourne ones. We tried pretty hard, but were unable to find anyone free to use them, so we just considered it a token donation to support live music.
Sabriena organized for us to stay in a cabin down at Barwon Heads, not walking distance, but not an obscene price either. We left fairly late, and got to check-in after the office was closed, but they left the keys in a lockbox for us. The cabin was pretty nice, in a golf course! Much too fancy for us!
In to the town proper, and we parked up in the parking lot of the pub so we’d not have any trouble finding a spot. I was apprehensive leaving a late-model V8 Commodore in public after dark as they have a tendency to come up missing, but it all worked out.
Wandered around the town, found a place to eat: “Tommy Gunns”. Pretty great dinner, nice neat restaraunt, loads of people eating outside but we were not dressed for it and ended up inside. Afterwards, went for a wander around to walk off the food-baby, including along the boardwalk across the river, and then it was just about time to go into the show.
Show was great, accoustics were good, band room is spacious (they’re apparently talking about getting rid of it?!), staff were very nice. Sabriena was feeling a little under the weather so we didn’t stay for the encore but it was a really good show. Despite being fairly close to the front (by no means in the pit) we didn’t get covered in beer.
In to bed, woke up bright and early, golfers fucking everywhere. Managed to get out, traffic through Geelong absolutely sucks, significantly worse than when Dad and I picked up my arcade machine in Cliffton Springs. Oh, and the roads suck arse, I thought people were just having a sook on Facebook but they really were absolutely terrible, dodging holes the entire way.
We spent Saturday chilling, went to bed in our own bed, and Sabriena had a very nice time at the running event without having to wake up first thing in the morning.

