Skip to content

ADR-0101: SaaS↔on-prem loop — license JWT + manifest pull + heartbeat + usage drain

  • Status: Accepted
  • Date: 2026-05-27
  • Drivers: User wants paid SaaS signups to translate into working on-prem deployments without the operator chasing image registries, license files, or quota state.

Context

After go-live ([[../memory/project_marketing_customer_app_golive_2026_05_27]]) the cloud was live but the on-prem story stopped at "we will ship a TBI image some day". Customers paid us in Stripe, got tokens_quota in Postgres, and then had nothing to install. The two surfaces — paying SaaS and on-prem deployment — were architecturally disconnected.

The discussion at the end of the post-go-live session ([[../memory/project_post_golive_stripe_ses_admin_2026_05_27]]) nailed down a two-stage bootstrap model and locked four decisions:

  1. ECR Private + presigned URLs (not a public registry, not OCI sideload)
  2. TSU rate metering on-prem (not per-feature gating)
  3. 24h grace window for cloud connectivity (not air-gapped)
  4. Cosign root for image trust (key custody pending)

This ADR records the implementation we shipped to close the loop.

Decision

Six new surfaces close the loop:

Cloud side (Next.js, pkg/octopus/customer-app/)

  1. pkg/octopus/customer-app/src/lib/db/migrations/0008_licenses_and_usage.sql — three tables:
  2. licenses (one per deployment site, JWT jti is PK shadow)
  3. usage_events (append-only, idempotent on (license_id, module, client_seq))

  4. license_heartbeats (append-only, audit trail for liveness)

  5. pkg/octopus/customer-app/src/lib/license/jwt.ts — HS256 sign/verify with LICENSE_JWT_SECRET env (stored in Secrets Manager as tlsstress/license-jwt-secret). The on-prem controller does not verify; cloud verifies on every API call. So one symmetric secret in one place, easy to rotate.

  6. pkg/octopus/customer-app/src/lib/license/catalog.ts — source-of-truth module list per tier. Returns OCI image refs. Production rewrites refs to ${ECR_REGISTRY}/<image> when ECR_REGISTRY env is set.

  7. POST /api/license/issue — customer mints a JWT bound to their account, label and validity (≤ 730 days). Persists a licenses row. Rate-limit 10/hr/account.

  8. POST /api/install/manifest — first call binds the JWT to the reporting hardware fingerprint; later calls from a different fingerprint are rejected. Returns module list + cloud config (heartbeat URL, intervals, grace period).

  9. POST /api/license/heartbeat — 1-hour cadence. Updates last_heartbeat_at, appends to license_heartbeats, returns the new graceExpiresAt (now + 24h). Controller refuses new test runs after that timestamp passes.

  10. POST /api/usage/report — batch TSU consumption. Idempotent on (license, module, client_seq). Atomically increments customer_accounts.tokens_consumed, returns remaining quota.

  11. /account/license + /account/download UIs — customer self-service for issuing/revoking license tokens and downloading the bootstrap image.

On-prem side (Go, pkg/bootstrap-controller/)

A new module sibling to pkg/tbi-agent. Self-contained binary bootstrap-controller:

  1. internal/fingerprint — SHA-256 over /etc/machine-id + product_uuid + hostname + first stable MAC (skipping docker*/flannel*). Returns simulated=true on macOS/CI so the cloud knows.

  2. internal/cloudclient — typed HTTP client for the three endpoints. IsLicenseRejected(err) lets the loop bail cleanly when the cloud revokes mid-run.

  3. internal/state — atomic-rename persistence of license JWT, last manifest, heartbeat record, per-module client_seq counter. Survives restart without re-issuing duplicate usage rows.

  4. internal/manifest — renders cloud's module list into per-namespace k8s YAML + a combined all-modules.yaml. Opt-in modules (kali-art, hyperbridge-art) are emitted commented out — operator opts in via Dashboard.

  5. internal/usage — FileSource watches /var/lib/tlsstress/spend/*.jsonl from module sidecars, rotates them on drain so retries don't double-bill. NoopSource for the period before sidecars are wired.

  6. pkg/bootstrap-controller/cmd/bootstrap-controller/main.go — first-boot sequence (fetch manifest → initial heartbeat → initial usage drain) then a 1h/1h/24h ticker loop (heartbeat / usage / refresh manifest).

Wiring

  • Terraform aws_secretsmanager_secret.license_jwt_secret (64-char random).
  • IAM apprunner_secrets_read policy extended to include the new secret ARN.
  • App Runner runtime_environment_secrets.LICENSE_JWT_SECRET mapped from the secret.

Consequences

Positive

  • Customer can mint a token at /account/license, download an image at /account/download, paste the token on first boot, and the loop closes itself. No operator intervention per customer.
  • Idempotent usage reporting means the on-prem controller can be flaky (drops, restarts) without double-billing.
  • Fingerprint pinning makes copy-paste piracy of JWTs measurably annoying: every additional lab needs its own token; revoking one doesn't kill the rest.
  • HS256 with one cloud-side secret is operationally simpler than RS256 + per-box public key bundles. Swap path stays open if a longer offline grace becomes a requirement.

Negative / deferred

  • ECR Private + cosign root — catalog.ts resolves to ECR-rewritten refs but no presigning + cosign verify is shipped yet. The manifest payload carries plain image refs; controller pulls them directly. Operator builds on a private registry (or local) for now. Next session: STS:GetSessionToken flow + cosign verify before kubectl apply.
  • Bootstrap image build — /account/download lists the SKUs but the build pipeline is the existing build/tbi/mkosi.conf plus a wrapper that bakes in the controller binary + signs with cosign. CI workflow is the next operator task.
  • Spend-reporter sidecar — FileSource is wired but the per-module sidecar that writes /var/lib/tlsstress/spend/<module>.jsonl is not yet emitted by the modules. Each pkg/<module> has internal TSU counters; they need a MeteringWriter shim that appends to the JSONL.
  • Tier change propagation latency — manifest is refreshed daily on the controller. Customer upgrades on the SaaS surface won't reflect on-prem for up to 24h. Acceptable for v1; add a "force refresh" admin button in v2.

Risks

  • JWT theft — token in plaintext on disk under /var/lib/tlsstress/bootstrap/license.jwt (0600). Anyone with root on the box can copy it. Mitigated by fingerprint pinning + 1-hour heartbeat — a stolen token bound to box A can't be used on box B; revocation propagates within ~1h.
  • Secret rotation — rotating LICENSE_JWT_SECRET instantly invalidates every outstanding license. Documented as expected behavior (operators notify customers + re-mint).

How to verify

# Cloud side (run from customer-app):
cd pkg/octopus/customer-app
npx tsc --noEmit src/lib/license/*.ts src/app/api/license/**/*.ts \
  src/app/api/install/**/*.ts src/app/api/usage/**/*.ts
npx vitest run

# On-prem side:
cd pkg/bootstrap-controller
go test ./...
go build -o /tmp/bootstrap-controller ./cmd/bootstrap-controller
/tmp/bootstrap-controller --once  # requires license.jwt + reachable cloud

End-to-end smoke (operator-only):

  1. Sign into app.tlsstress.art, go to /account/license, issue a token.
  2. On a Linux box: mkdir -p /var/lib/tlsstress/bootstrap && \ echo "$TOKEN" > /var/lib/tlsstress/bootstrap/license.jwt && chmod 600 …
  3. Run controller with --once. Expect: manifest fetched, heartbeat ok, usage report sent.
  4. Hit /api/license/list (cloud) — your license should show boundFingerprint set and lastHeartbeatAt < 1 min ago.

Pointers

  • Memory: [[../memory/project_saas_onprem_loop_closed_2026_05_27]]
  • Prior: [[0100-token-economy-security-and-admin]] (TSU primitive)
  • Prior: [[../memory/project_tbi_image_phase_i_2026_05_24]] (mkosi scaffold)
  • Prior: [[../memory/project_lab_staging_wizard_complete_2026_05_24]] (operator wizard)
  • Patent claims unchanged. The loop is plumbing; novelty is in the modules it carries.