Upgrades & rollback

Upgrade Eddytor safely - snapshot Postgres, read the release notes, and roll back with eddytoradm migrate undo if you must.

Upgrading Eddytor is pulling a newer image. The server applies new database migrations at boot, under a Postgres advisory lock, so several server replicas are safe. The engine skips migrations (EDDYTOR_SKIP_MIGRATIONS=true).

Heads up

Snapshot Postgres before every upgrade. A restore from that snapshot is the only rollback that loses no data.

Before you start

Read the release notes for the version you are about to run. Every image carries them (from 2.10.0):

# Notes for one release (no database needed)
docker run --rm ghcr.io/nordalf/eddytor-ce-server:<version> eddytoradm release-notes
# All notes, as a file
docker run --rm ghcr.io/nordalf/eddytor-ce-server:<version> cat /usr/share/doc/eddytor/CHANGELOG.md

Then take the snapshot:

# Bundled Postgres. External or managed: use your provider's snapshot, or pg_dump against it.
docker compose exec -T postgres pg_dump -U eddytor -Fc eddytor > eddytor-$(date +%F).dump

Upgrade

A normal rolling upgrade is fine. The release notes name anything that can fail during the rollout.

  1. Bump EDDYTOR_VERSION in .env.
  2. Pull and recreate:
docker compose pull && docker compose up -d
  1. Wait for health and confirm migrations ran:
docker compose ps                               # eddytor-server (healthy)
docker compose logs eddytor-server | grep -iE "preflight|migrat|listening"
  1. Bump the image tag and upgrade in place:
helm upgrade eddytor oci://ghcr.io/nordalf/charts/eddytor -n eddytor \
  --reuse-values --set image.tag=<new-version>
  1. Watch the rollout and confirm migrations:
kubectl -n eddytor rollout status deploy/eddytor-server --timeout=300s
kubectl -n eddytor logs deploy/eddytor-server | grep -iE "preflight|migrat|listening"
helm test eddytor -n eddytor    # hits /healthz on server + engine

The Web UI tracks its own version - bump ui.image.tag independently.

Roll back

Prefer to roll forward to a fixed release. Roll back only when you cannot wait. An older release refuses to start on a database a newer one migrated:

Failed to run migrations: migration 64 was previously applied but is missing in the resolved migrations

So revert the migrations first, then deploy the older release. Run eddytoradm migrate undo from the newer image (only it has the down migrations) with server and engine stopped (a running server re-applies them). Without --yes, it only prints its plan. The target (--to) is the newest migration of the release you go back to:

Roll back fromTo--to
2.10.02.9.161

For other releases, eddytoradm migrate status in the older image ends with This release embeds migrations up to NNN.

cd ~/eddytor
# Target for another release:
#   EDDYTOR_VERSION=<older-tag> docker compose run --rm --no-deps eddytor-server eddytoradm migrate status

# 1. Stop server and engine. Postgres keeps running.
docker compose stop eddytor-server eddytor-engine

# 2. Preview, then revert. .env still names the newer EDDYTOR_VERSION.
docker compose run --rm --no-deps eddytor-server eddytoradm migrate undo --to 61
docker compose run --rm --no-deps eddytor-server eddytoradm migrate undo --to 61 --yes

# 3. Pin the older release and start it.
sed -i.bak 's/^EDDYTOR_VERSION=.*/EDDYTOR_VERSION=v2.9.1/' .env
docker compose pull && docker compose up -d

Don't re-run the installer to roll back - it pins the latest release again.

NS=eddytor
NEW=v2.10.0-k8s            # image you roll back FROM
SECRET=eddytor-secrets     # chart Secret with EDDYTOR_DATABASE_URL, or your secrets.existingSecret

# 1. Stop server and engine. Repeat the get until it shows no pods.
kubectl -n $NS scale deploy/eddytor-server deploy/eddytor-engine --replicas=0
kubectl -n $NS get pods -l 'app in (eddytor-server,eddytor-engine)'

# 2. One-off pod with the newer server image.
kubectl -n $NS apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: eddytor-migrate
spec:
  restartPolicy: Never
  containers:
    - name: eddytoradm
      image: ghcr.io/nordalf/eddytor-ce-server:${NEW}
      command: ["sleep", "3600"]
      envFrom:
        - secretRef:
            name: ${SECRET}
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        seccompProfile:
          type: RuntimeDefault
EOF
kubectl -n $NS wait --for=condition=Ready pod/eddytor-migrate

# 3. Preview, then revert.
kubectl -n $NS exec eddytor-migrate -- eddytoradm migrate undo --to 61
kubectl -n $NS exec eddytor-migrate -- eddytoradm migrate undo --to 61 --yes
kubectl -n $NS delete pod eddytor-migrate

# 4. Deploy the older chart with a pinned image (the default tag k8s floats to the
#    newest release), then scale back up yourself: the HPA ignores a Deployment at 0.
helm upgrade eddytor oci://ghcr.io/nordalf/charts/eddytor -n $NS --version 2.9.1 \
  --reuse-values --set image.tag=v2.9.1-k8s
kubectl -n $NS scale deploy/eddytor-server deploy/eddytor-engine --replicas=1

What a rollback loses

A down migration deletes what only the newer release stores. From 2.10.0 to 2.9.1:

MigrationLost on rollback
064 - storage links per organisation2.10.0 keeps one Azure/Google link per user per organisation. The undo keeps one link per user and provider and deletes the others; those users link their account again. Links that 2.10.0 dropped at upgrade (users in no organisation) don't come back.
063 - sign-in organisation bindingSign-ins in progress lose their organisation binding. The codes expire within minutes.
062 - SSO email-claim fallbackThe emailClaimFallback setting of SSO connections is deleted.

If the server won't come up

  • Stuck in Init (Kubernetes): re-check EDDYTOR_DATABASE_URL host/creds, and on an external DB make sure waitForDb is off - see Install the chart.
  • Boot fails with a secret-decrypt error: the EDDYTOR_ENCRYPTION_KEY changed or is wrong - see Backups & key rotation.

On this page