Skip to content

Upgrade & rollback — operator runbook

Audience: operator upgrading a running TLSStress.Art bench from one release to the next, and — if the new release misbehaves — rolling it back to the previous known-good version.

Outcome: you move the bench to a new tag through the dashboard's Self-Upgrade channel, verify it, and keep a tested escape hatch. The rollback section is fully manual so it works even when the Self-Upgrade orchestrator is the thing that broke, and it covers the air-gapped path where no channel fetch is possible.

Scope

Two directions:

  • Upgrade — move to a newer tag via the Self-Upgrade channel (connected) or the air-gapped bundle (offline).
  • Rollback — return to the previous tag. Because a bad upgrade can leave the orchestrator itself broken, the rollback steps here are manual and do not depend on the dashboard being healthy.

This runbook is the connective tissue between four references:

Prerequisites

  • Admin role in the dashboard.
  • A maintenance window sized to the scope (single MÓDULO vs full bench).
  • The current tag recorded (the version you are upgrading from). Read it in Settings → Self-Upgrade → Current version, or: 
    kubectl get deploy -n web-agents web-agent-dashboard \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
    Write it down. This is your rollback target.
  • The target tag chosen and already verified per verify-release.md. Never upgrade to a tag you have not verified.

Pre-checks (before you touch anything)

Run these and only proceed if all pass:

  1. Cluster healthy — no pods crash-looping, no pending alerts: 
    kubectl get pods -A | grep -vE 'Running|Completed' || echo "✓ all pods healthy"
  2. Backup is recent + Completed — a fresh restore point must exist before you upgrade (this is the state you roll back to): 
    velero backup get | head -5
    If the newest Completed backup is stale, take a snapshot first (next section).
  3. Target tag verified — you completed verify-release.md for the target tag.
  4. Channel matches intent — production benches run recommended; do not upgrade a production bench off an rc or beta pointer. See RELEASE_CHANNELS.md.

Step 1 — Backup / snapshot (your rollback point)

Take a fresh backup so you can roll state back if the upgrade corrupts data (not just images). Full procedure: restore-from-backup.md.

velero backup create pre-upgrade-$(date +%Y%m%d-%H%M) \
  --include-namespaces platform,web-agents,validator \
  --wait
velero backup get | head -3

Record the backup name — this is what a state rollback restores from. Image-only rollbacks (the common case) do not need it, but a schema-changing upgrade does.

Step 2 — Apply via channel

The operator drives every upgrade from the dashboard — never kubectl set image by hand (that bypasses the release-feed digest pinning). See RELEASE_CHANNELS.md.

  1. Settings → Self-Upgrade → Channel — confirm the channel (recommended for production).
  2. The dashboard shows the newest tag on that channel and its release-feed digest. Confirm the tag equals the one you verified.
  3. Click Stage upgrade. The orchestrator pulls the pinned digests and updates the manifests in a controlled order (data plane last).
  4. Watch the rollout: 
    kubectl rollout status deploy -n web-agents web-agent-dashboard --timeout=300s

Air-gapped bench: there is no channel to fetch. Use the offline bundle instead — AIRGAP_INSTALL.md §"Upgrades to a newer version". airgap-deploy.sh is idempotent and loads the new images alongside the old, so the previous tag's images stay on the node — which is exactly what makes the air-gap rollback below a fast, offline operation.

Step 3 — Verify the running release

After the rollout settles, verify the bench is actually on the new tag and is healthy:

# Images now match the target tag?
kubectl get deploy -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}' \
  | grep -i tlsstress

# Signature + SBOM + digest of the now-running image
# (full procedure: verify-release.md)

Then run the post-upgrade smoke checks:

  • Dashboard loads, Settings → Self-Upgrade → Current version shows the new tag.
  • A test agent run completes end-to-end (agent → NGFW → persona).
  • Prometheus targets are all up, no alert flap for 10 minutes.

If all pass, the upgrade is complete. If any fail, go to rollback.

Rollback (manual)

Use this when the new release misbehaves. It is deliberately manual so it works even if the Self-Upgrade orchestrator itself is the broken component. There are two layers — do the image rollback first; only do the state rollback if the upgrade also changed persisted data.

Rollback A — image rollback (connected bench, common case)

Every deployment keeps its previous ReplicaSet, so Kubernetes-native rollback returns to the prior image without any registry fetch:

# Roll each affected deployment back one revision. Example: dashboard.
kubectl rollout undo deploy -n web-agents web-agent-dashboard

# Confirm the previous revision exists first:
kubectl rollout history deploy -n web-agents web-agent-dashboard

Repeat rollout undo for every deployment the upgrade touched, data plane first on the way back down (reverse of the upgrade order). If you need an explicit tag rather than "one revision back", pin the recorded previous tag by editing the manifest / kustomize overlay and re-applying — this is the same path the air-gap rollback uses.

Verify with the Step 3 checks against the previous tag.

Rollback B — state rollback (only if data changed)

If the upgrade ran a schema migration or otherwise mutated persisted state, an image rollback alone leaves new-schema data under old-schema code. Restore state from the pre-upgrade backup taken in Step 1:

  1. Follow restore-from-backup.md using the pre-upgrade-* backup name from Step 1.
  2. Scale the affected namespace to zero, restore, scale back up — the restore runbook has the exact commands.
  3. For PostgreSQL, use the PITR procedure in that runbook with a recovery target before the migration ran.

Order matters: image rollback first, then state rollback. Never kubectl apply over a partially-restored namespace — wipe and restore cleanly (see restore-from-backup.md).

Rollback C — air-gapped bench

No channel, no registry, no rollout undo fetch — but the previous tag's images are already on the node because airgap-deploy.sh loads new images alongside the old (never replacing them). So the rollback is purely a manifest edit:

  1. Edit the manifests / kustomize overlays to point image tags back at the previous tag (the one you recorded in Prerequisites).
  2. Re-apply: 
    kubectl apply -k k8s/
kubectl apply -k personas/
  3. If the previous tag's images were pruned from the node (e.g. after a crictl rmi cleanup), re-stage the previous bundle from your media per AIRGAP_INSTALL.md and re-run airgap-deploy.sh --bundle <previous-bundle> — it is idempotent.
  4. For state, use Rollback B; Velero/PITR work identically offline.

Success criteria

  • After upgrade: all deployments on the target tag, smoke tests pass, no alert flap for 10 min.
  • After rollback: all affected deployments back on the previous tag, smoke tests pass, audit log captures the rollback event.
  • Backup/restore point from Step 1 recorded and (if state rollback ran) the restore completed cleanly.

Last verified against shipping code: v4.0.0 (2026-07-05).