Backup / Restore / DR — Operator Guide

Implements ADR 0017. This guide is the operator-facing how-to for the foundation manifests shipped in k8s/backup/ plus the Layer-3 UI suite shipped 2026-06-11 (#1332/#1333): one-click config backup, UI restore with anti-fraud guard-rails, and scheduled offsite routines — all at /admin/backup. Full-namespace Velero restore execution remains CLI (documented below).

Scope status (post-Scope-Freeze 2026-05-10) — Operational runbooks for this guide live at: - runbooks/dr-drill.md — quarterly DR drill (RTO < 30 min + RPO < 5 min) - runbooks/restore-from-backup.md — partial NS or full bench restore - runbooks/soc2-evidence-collection.md — compliance evidence collection

Cross-refs: ADR 0018 (compliance), VALIDATOR.Art rebuild runbook.

4-layer architecture (recap)

Layer 1 — Continuous backup
   WAL-G (Postgres)         every 5 min
   Restic (catalog YAML)    every 5 min
   →  in-cluster MinIO
   Retention: 24h continuous + 30 daily snapshots

Layer 2 — Scheduled snapshots
   Velero (full namespace)  nightly 02:00 UTC + weekly Sunday 03:00 UTC
   →  in-cluster MinIO
   Retention: 30 days nightly, 90 days weekly

Layer 3 — UI suite (/admin/backup)   ✅ SHIPPED 2026-06-11 (#1332/#1333)
   One-click signed CONFIG backup + UI restore (one transaction,
   anti-fraud guard-rails) + scheduled offsite routines (HTTP PUT /
   S3 SigV4 / SCP) + Velero restore-point browser (read-only).
   Full-NS Velero restore execution + impact preview: still CLI
   (impact preview lands in Backup PR-3).

Layer 4 — Off-site DR replication   (lands in Backup PR-4)
   Continuous WAL replication to a second MinIO instance (separate
   failure domain)
   Quarterly DR drill — automated

Bootstrap (one-time, ~10 minutes)

1. Create the backup namespace + MinIO target

kubectl apply -f k8s/backup/00-namespace.yaml

2. Set MinIO root credentials

The shipped manifest has placeholder credentials (REPLACE_BEFORE_APPLY_*). Replace them with real values:

ACCESS_KEY=$(openssl rand -hex 16)
SECRET_KEY=$(openssl rand -hex 32)
kubectl -n web-agents-backup create secret generic minio-root-credentials \
  --from-literal=accesskey="$ACCESS_KEY" \
  --from-literal=secretkey="$SECRET_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

Save the credentials in a sealed-secret store (1Password, Bitwarden, HashiCorp Vault) — they're needed for the Velero credential below.

3. Apply the MinIO deployment + bucket init

kubectl apply -f k8s/backup/10-minio-target.yaml
kubectl -n web-agents-backup wait --for=condition=Ready pod -l app=minio --timeout=300s
kubectl apply -f k8s/backup/20-bucket-init.yaml

The bucket-init Job creates the three buckets (velero, restic, postgres-wal) with versioning enabled on the two that benefit from it. Idempotent — re-running is safe.

4. Install Velero

Velero itself is a separate install (its CRDs aren't shipped in this repo to avoid coupling our release cadence to theirs). Use the upstream CLI:

# Replace ACCESS_KEY + SECRET_KEY with the values from step 2.
cat > /tmp/credentials-velero <<EOF
[default]
aws_access_key_id = $ACCESS_KEY
aws_secret_access_key = $SECRET_KEY
EOF

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket velero \
  --secret-file /tmp/credentials-velero \
  --use-volume-snapshots=false \
  --use-node-agent \
  --backup-location-config region=us-east-1,s3ForcePathStyle=true,s3Url=http://minio.web-agents-backup.svc.cluster.local:9000

# Wait for Velero pods to be ready.
kubectl -n velero wait --for=condition=Ready pod -l deploy=velero --timeout=300s

shred -u /tmp/credentials-velero

5. Apply the project-specific Velero CRDs

# Edit 30-velero-config.yaml first to replace REPLACE_BEFORE_APPLY_*
# placeholders with the same ACCESS_KEY + SECRET_KEY from step 2.
kubectl apply -f k8s/backup/30-velero-config.yaml

This installs: - BackupStorageLocation/default — points Velero at the MinIO bucket - VolumeSnapshotLocation/default — uses the same target for snapshots - Schedule/web-agents-nightly — nightly backup at 02:00 UTC, 30-day retention - Schedule/web-agents-weekly-full — weekly full backup at Sunday 03:00 UTC, 90-day retention

6. Verify

# Trigger a manual backup
velero backup create test-bootstrap-$(date +%s) \
  --include-namespaces web-agents \
  --wait

# List backups
velero backup get

# Check the bucket directly
kubectl -n web-agents-backup exec -it deploy/minio -- \
  mc ls minio/velero/

You should see at least one tarball.gz.gz and a .json metadata file under velero/backups/.

Layer 3 — the /admin/backup UI suite (shipped)

Everything below is operator-clickable; every action is recorded in the sealed audit chain.

One-click config backup

"Backup agora" produces a SIGNED .json.gz (HMAC of the deployment key) covering a curated registry of configuration tables — cluster config, targets, personas, test plans, DUT devices, lab deployments, templates, saved views. Telemetry (runs/metrics/logs), the WORM sealed audit chain and monetary token state are deliberately excluded: they belong to the Layer-1 full-DB backup, and restoring them from a file could forge history or double-spend TSU.

UI restore (with anti-fraud guard-rails)

Upload any backup archive → signature + per-table SHA-256 validation → impact ConfirmModal → applied in one transaction (all-or-nothing). Archives from another installation require the explicit allowForeign checkbox.

Token-economy guard-rails (CI-enforced invariant):

• FORBIDDEN_TABLES + unit tests guarantee monetary/WORM tables (tokens_notes, license_envelopes, pending_redemptions, sealed_audit_log, runs, …) can never enter the backup registry.
• auth_epoch is never downgraded by a restore — sessions revoked by the global killswitch stay revoked.
• cloud_lease_failmode always keeps the local value — the charging posture only changes through the individually-sealed toggle, never by riding in on an archive. Preserved fields are reported in the restore response and the sealed audit event.

Scheduled offsite routines

Daily or weekly (UTC hour, weekday) to destinations outside the product, executed by the reaper singleton:

Destination	Use for	Notes
http_put	WebDAV (Nextcloud/ownCloud), MinIO presigned, any HTTP drive gateway	optional Basic auth
s3	AWS S3, MinIO, Wasabi, R2	native SigV4, no SDK dependency
scp	Any SSH host	host scp binary + key mounted as a Secret (default /etc/backup-ssh/id_ed25519); honest error if absent

Each routine has "Send now" (tests the destination immediately), pause/resume/delete, last/next-run visibility; failures land in the backup_jobs history and the backup_scheduled_runs_total Prometheus counter — a dead destination never takes the product down. Secrets are write-only (GET redacts them).

Velero restore points (read-only)

The same page lists backups.velero.io CRs with phase/expiry/errors. Restore execution for full-namespace recovery remains CLI — see the next section.

Restore (CLI — full-namespace Velero recovery)

# List restore points
velero backup get

# Restore the most recent backup into a fresh namespace
velero restore create restore-$(date +%s) \
  --from-backup web-agents-nightly-20260509-020000 \
  --wait

For a point-in-time restore (continuous WAL streaming), use WAL-G's restore command — see WAL-G docs for the full syntax.

Disaster Recovery drill (quarterly)

The full drill (Layer 4 with off-site replication) lands in Backup PR-4. For now, the manual drill checklist:

1. Spin up a parallel cluster on different hardware
2. Apply the backup namespace + MinIO + bucket-init manifests
3. Sync the MinIO bucket from production (one-time mc mirror)
4. Install Velero pointing at the restored bucket
5. Run velero restore create dr-drill-$(date +%s) --from-backup latest-weekly --wait
6. Validate: dashboard /api/health returns 200, sample report renders
7. Document RTO + RPO observed; file in compliance artifact log
8. Tear down the parallel cluster

Cadence: every 90 days. Failure pages the on-call operator.

What's NOT covered today

• Pre-restore impact preview — Backup PR-3
• Off-site DR replication — Backup PR-4
• Self-Upgrade rollback integration — Backup PR-5 (cross-ref ADR 0013)
• Encrypted-at-rest with customer-managed KMS — out of scope for v4.7; in-cluster MinIO uses bucket-level encryption with a key Velero generates

Related ADRs

• ADR 0017 — full design rationale
• ADR 0013 — Self-Upgrade uses Layer 2 snapshots for the rollback path
• ADR 0018 — enterprise compliance bar requires documented + tested DR