K8s: Plex Troubles!
After yesterday’s changes to the cluster, I played some video games with my cousin and then went to bed, finding Sabriena watching TV. After “one more episode”, the Plex UI on the TV froze up, so we went to sleep, but I made sure to make a mental note that it stopped working at about 11:20. She also noted that it did this earlier, so she switched to Netflix, before checking it about an hour later and it was working.
So at this point, since it’s never done this, I’m pretty sure it’s related to the changes that I made - specifically, moving the /data and /config directories over to NFS mounts and shifting it to a different node… I reasoned that it was probably the second issue.
After a bit of digging through the logs, I found:
Feb 11, 2023 21:37:29.410 [0x7f6d77813b38] DEBUG - [Req#fae] 7 threads are waiting on db connections held by threads: 0x7f6d7564bb38,0x7f6d72e0fb38,0x7f6d7625db38,0x7f6d75448b38,0x7f6d73012b38,0x7f6d75245b38,0x7f6d74836b38,0x7f6d7422db38,0x7f6d7584eb38,0x7f6d74633b38,0x7f6d7402ab38,0x7f6d72c0cb38,0x7f6d73e27b38,0x7f6d72806b38,0x7f6d73418b38,0x7f6d7381eb38,0x7f6d73a21b38,0x7f6d73215b38,0x7f6d7361bb38,0x7f6d73c24b38
That’s around the time Sabriena said it acted up the first time, and sure enough further on, around 11:20, I found a lot more threads stacked up.
After some research, I found a Reddit thread where someone mentioned that the database for Plex is a modified SQLite, which famously does not like NFS. Shit.
I spent a little while trying to think how I could solve this, when I reasoned that I could make the configuration data an iSCSI mount instead - this is actually mentioned in a different Reddit thread. But since I didn’t know how big it could get with the covers/banners/etc, I decided to instead create a mount for just the database, and everything else could stay NFS.
So on my Ubuntu server, I installed the tgt package, then created the zvol and target:
zfs create -V 10G zeefus/k8s/plex-db
tgtadm --lld iscsi --op new --mode target --tid 1 --targetname iqn.2023-02.some.goofy.server:zeefus.plex-db
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 --backing-store /dev/zeefus/k8s/plex-db
Alright, try to mount it in the container… empty at this stage, I would let Plex create an empty database, then stop the container, clobber it with my one, and start it again.
Uh oh, command not found and other errors. I have to install the iSCSI initiator package - open-iscsi. After a bit more screwing around, I started to get no records found, but that turned out to be just a typo in my configuration. Finally, I got an error about /dev/disk/by-path not existing, which I remembered these are typically created by udev, which is a systemd thing now. A bit more internet sleuthing and I found the eudev package which is a fork of udev.
So the finishing touchest to get it working on Alpine:
apk add open-iscsi eudev
rc-update add udev
rc-update add iscsid
service udev start
service iscsid start
After all this, it works on both nodes (I drained the first one to make sure it worked on the second after my Ansible playbook ran). Hopefully that’s the last of the grief!
I should experiment with authentication on iSCSI, but I’m not too bothered at the moment, NFS will accept any connections on the physical LAN (as opposed to the VLANs we use for client machines), so despite some mitigations I’d probably still be in a fair mount of trouble if someone had access to that LAN anyway, so authentication on the target that hosts my Plex database doesn’t seem super critical.
fwagglePublished:
Modified:
Filed under:
Location:
Horsham, VIC, AustraliaNavigation: Older Entry Newer Entry