Mastodon: Scheduler woes

Ages ago (2017 I think?) I signed up for Mastodon, but no one I knew was using it so I didn’t really do much with it. Last year, I went all-in as it turned out, I’ve essentially not touched Twitter except by accident since October, and in November I went to Mastodon-proper that I self-host.

I’m not sure for how much of that time, but it turns out my scheduler has been broken. At some point they broke that out into separate queues, and I remember making one set of changes but I never arranged a runner to handle the scheduler queue. It was as simple as adding -q scheduler to sidekiq’s command line and restarting it.

This actually resolved a pile of things that were minor annoyances, but not enough to do anything about. For instance, trending hashtags works now. Full-text search works now. I’m not 100% sure, but I think automatic cleanup of foreign media files works now (I was previously doing that manually via tootctl when it got outta hand).

But getting there was a whole different thing. The full-text search indexer for instance really broke shit, filling up all five threads of my sidekiq process and halting everything else. Similar to bug #23985 where IndexingScheduler starts before the prior one is finished.

The solution, of which I still don’t quite understand why it worked so YMMV:

  • redid the tootctl deploy step for fulltext search, not sure if this helped.
  • stop the container(s) running sidekiq.
  • remove any locks that were left hanging because sidekiq was killed.
  • delete the scheduler queue.
  • disable the IndexingScheduler recurring task.
  • start sidekiq containers, with one that’s dedicated to scheduler so it doesn’t hold other queues up.
  • enable and enqueue IndexingScheduler exactly once, then disable the recurring task again.
  • wait for it to finish.
  • enable IndexingScheduler recurring task again for the last time.

After doing this, the subsequent runs finished very fast, so I think the issue is solved. I did have similar grief with first runs of VacuumScheduler as well, but it runs infrequently enough that it definitely finishes before the next start time.

I’m pretty sure fixing this screw-up (entirely on my part, I didn’t pay close enough attention to the documentation) resolved at least three different minor annoyances that I was just living with, so I’m a pretty happy camper now.

Update 2023-07-04: Someone else on the #mastoadmin hashtag had what I suspected to be the same drama, so I told them the above steps and it looks like it solved their issue too.

Update 2023-07-07: The bug’s now closed, what they’ve done from the looks of things (I don’t really speak Ruby, as mentioned) is make it only do small bursts, so it’s basically guaranteed to finish inside of five minutes. That does mean it’ll take longer to chug through everything, but at least it’ll get through it all eventually.

I’m still gonna leave scheduler as a separate queue though, because it’s only about 300MB of RAM total, and that’s if the entire RSS reported by Sidekiq is spent RAM.

Horsham, VIC, Australia

Author:

fwaggle

Published:


Modified:


Filed under:


Location:

Horsham, VIC, Australia

Navigation: Older Entry Newer Entry