Skip to content

ADR-0103: Token Economy production hardening

  • Status: Accepted
  • Date: 2026-06-09
  • Driver: Project owner request — full audit + remediation of the token economy chain (acquisition → credit → on-prem consumption → reporting → re-purchase), "no pending items, production-ready".
  • Relates to: ADR-0099 (Token Economy v5 / UTXO ledger), ADR-0101 (SaaS↔on-prem loop)

Context

An end-to-end audit of the token economy found the ledger core (UTXO mint/spend, DB CHECK constraints, transfer settlement) and the TLS transport (SPKI-pinned, fail-closed) to be sound, but surfaced a cluster of correctness and revenue-integrity defects concentrated in the on-prem consumption accounting, the coexistence of two balance models, and the Stripe webhook failure paths. This ADR records the decisions taken to close them.

Decision

D1 — The UTXO ledger is the single source of truth for balance (C1)

getBalance() (ledger) is authoritative everywhere a spendable balance is shown or returned: /api/usage/report returns ledger balance on every path (debit and all-duplicates alike), and the finance / cost-explorer screens read the ledger instead of tokens_quota − tokens_consumed. The legacy tokens_quota/tokens_consumed fields remain only as a cycle-utilisation display and a fast cache that spendTSU keeps coherent; they no longer drive any balance number, eliminating the divergence introduced by monthly resets and boost packs.

D2 — Atomic debit↔credit via a composable spendTSU (A5, M1)

spendTSU now accepts an optional Drizzle transaction (like mintUTXO), so the three multi-step value movements run in ONE transaction: - /api/usage/report: insert usage rows + debit in one tx (no crash window leaving usage recorded-but-not-debited; InsufficientBalance rolls back, no delete-compensation); - token transfers: limit re-check + debit + record insert in one tx (a failed re-check can no longer strand a debit with no record to refund); - MSSP allocation: debit + credit in one tx, plus optional Idempotency-Key request de-dup (a retried POST no longer double-allocates).

D3 — On-prem usage delivery is at-least-once with stable idempotency (C2, C3)

The bootstrap-controller no longer discards a batch on read. Drain rotates *.jsonl into *.inflight with a stable ClientSeq stamped per event; the caller Commits only after the cloud accepts. A failed send (network/5xx/402) keeps the inflight files, and the next cycle re-sends the SAME seqs — nothing is lost and the retry is idempotent on (license_id, module, client_seq). A 402 (pool exhausted) is now correctly detected (IsQuotaExceeded), pauses new runs, holds the batch for retry after top-up, and reports are chunked ≤200 events.

D4 — Stripe webhooks fail safe and reverse on refund/dispute (C4, A1, A4, M8)

  • Transient handler errors return 5xx (Stripe re-delivers) and the event row stays non-terminal so claimStripeWebhookEvent reprocesses it; all handlers are idempotent. A paid invoice can no longer be lost on a transient blip.
  • charge.refunded and charge.dispute.created reverse the tokens the charge minted via clawbackChargeTokens (burn the unspent remainder of those notes; the ledger has no negative notes).
  • A paid invoice that maps to 0 tokens (unknown price) is logged CRITICAL and NOT recorded as a silent applied.
  • Boost mint requires payment_status === "paid"; async_payment_succeeded is handled for delayed methods.

D5 — Low-balance alerts, auto-refill, and reconciliation (A2, A3, M6)

  • The low-tokens email fires from the REAL consumption path (/api/usage/report) via a shared crossing-edge helper (single threshold, 15%).
  • Auto-refill is now executed: /api/cron/auto-refill charges the saved card off-session for the configured boost pack when balance crosses the threshold, opt-in only, anti-flap (≤1/hour), Stripe-idempotent, auto-disable after 3 failures.
  • /api/cron/reconcile-ledger is a watchdog for non-terminal webhooks (possible paid-but-uncredited) and ledger integrity (over-spent notes).

D6 — License key rotation and resilient pinning (M3, M5)

  • verifyLicense selects the HMAC secret by the JWT kid from LICENSE_JWT_SECRETS (a {kid: secret} map), so a key can be rotated without invalidating the outstanding 1-year licenses; alg is pinned to HS256.
  • The controller can override the baked-in SPKI pins via TLSSTRESS_SPKI_PINS (CSV) without rebuilding the image — the escape hatch for a legitimate Google Trust Services CA rotation (runbook: docs/token-economy/spki-pin-rotation.md).

D7 — Billing suspension preserves paid inventory (M4)

A suspension whose reason starts with subscription_ (lapsed/cancelled/past_due) still lets the account spend its PAID boost notes; a fraud/refund/dispute/manual suspension remains a hard block on all spending; deleted is always blocked.

Consequences

  • Money paths are now self-correcting: idempotent webhook retries, at-least-once usage delivery, atomic value movement, and a reconciliation watchdog.
  • New env: LICENSE_JWT_SECRETS (optional, rotation), TLSSTRESS_SPKI_PINS (optional, controller). New crons: auto-refill (hourly), reconcile-ledger (daily) — K8s CronJobs + EventBridge variants under pkg/octopus/customer-app/infra/cron/.
  • The Source interface in the controller gained Drain(seq)/Commit; the metering writer is unchanged (it never produced client_seq).
  • Residual by design: on-prem tsuConsumed remains self-reported (inherent to a customer-run reporter); the reconciliation watchdog + anomaly logging are the compensating controls.

Addendum — 2026-06-10 deep-audit remediation (PR #1275)

A full-chain audit of this ADR's implementation found and fixed defects in the partial-failure paths of several decisions above; the decisions stand, their implementations were corrected:

  • D4: the boost handler swallowed REAL mint errors and finalized terminal (ignored) — a paid boost could be silently lost on a transient blip, contradicting this ADR. It now propagates → 5xx → idempotent re-delivery (same contract as invoice.paid). Clawback is now cumulative-delta per charge (Stripe's amount_refunded is cumulative; per-event burning over-clawed successive partial refunds).
  • D5: auto-refill recovers ORPHANED PaymentIntents (crash between charge and credit) before deciding to charge again — closing a charged-but-never-credited
  • re-charge window. The reconcile watchdog also reports staleActiveTickets.
  • D3: the drain rotation is now an atomic claim→stamp→persist protocol (.rotating → .inflight); the old write-then-remove sequence had a crash window that re-stamped the same events with new seqs (double-billing).
  • Minor: mint integer guard + SAVEPOINT for caller-transaction composition; fraud suspension surfaces as terminal 403 account_suspended (the controller stops instead of retry-looping); clawback takes the account lock first (same order as spendTSU).

Full finding list and self-healing matrix: docs/token-economy/README.md §12.1.