Patching Mastodon's 'toolctl' in Kubernetes

Dec 28, 2022

Mastodon has a storage problem. The current version (which is 4.0.2 at time of writing) currently stores remote users’ avatars and header images in the cache, but no function exists to clean down these cached images. For the last month, the instance I manage has ballooned to 73GB of these files. The ‘fix’ was to manually remove the files and then run a ‘remove orphans’ job to clean up the mess, Not exactly ideal.

Thankfully, Evan Philip submitted a PR to clean them up using the existing tootctl command. The change consisted of a single file and it is easy to patch into an existing installation while awaiting 4.0.3 or 4.1. For me, our Mastodon instance is run in a home Kubernetes cluster, so it’s not a simple case of replacing a file on a filesystem. This type of patching can be done in Kubernetes, it’s just not as obvious how to do it.

ConfigMap and Secret objects can be mounted into a Pod much like a PVC. So by creating a ConfigMap and defining it as a volume in a Pod you can override any files within the container itself:

First of all, you want to create a ConfigMap with the new file. The filename here doesn’t matter, but for ease, you can keep it consistent:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: media-cli-patch
  namespace: web
data:
  media_cli.rb: |
    # frozen_string_literal: true

    require_relative '../../config/boot'
    require_relative '../../config/environment'
    require_relative 'cli_helper'

    module Mastodon
      class MediaCLI < Thor
        include ActionView::Helpers::NumberHelper
        include CLIHelper

    ...

Then, you need to update your Pod definition to load in the ConfigMap object as a volume. First, add it to the volumes definition and give it a name:

  volumes:
    - name: system
      persistentVolumeClaim:
        claimName: mastodon-system
    - name: media-cli-patch
      configMap:
        name: media-cli-patch

Then, add it to your volumeMounts section:

  volumeMounts:
    - name: system
      mountPath: /opt/mastodon/public/system
    - name: media-cli-patch
      mountPath: /opt/mastodon/lib/mastodon/media_cli.rb
      subPath: media_cli.rb

The mountPath is the full path within the Pod where it is to be mounted, and the subPath value refers to the filename you gave it in the ConfigMap. Push the resource to the cluster and you’ll have a pod running with the modified file. I use a CronJob to run these routine clean-up jobs, the Web and Sidekiq instances don’t need the patched file as it only applies to commands run via tootctl.

Here is my full CronJob for context:

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: mastodon-cron
  namespace: web
spec:
  schedule: "0 * * * *"
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 1
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: mastodon-cleanup
              image: tootsuite/mastodon:v4.0.2
              command: ["/bin/sh"]
              args:
                ["-c", "tootctl media remove --days=1 --prune-profiles && tootctl preview_cards remove --days=14"]
              envFrom:
                - configMapRef:
                    name: mastodon-config
                - secretRef:
                    name: mastodon-secrets
                - secretRef:
                    name: mastodon-postgresql-auth
              volumeMounts:
                - name: system
                  mountPath: /opt/mastodon/public/system
                - name: media-cli-patch
                  mountPath: /opt/mastodon/lib/mastodon/media_cli.rb
                  subPath: media_cli.rb
          restartPolicy: Never
          volumes:
            - name: system
              persistentVolumeClaim:
                claimName: mastodon-system
            - name: media-cli-patch
              configMap:
                name: media-cli-patch