Skip to content

ADR-0104: Usage attestation & anti-fraud for the on-prem reporter

  • Status: Accepted — L1 substrate + L2 implemented, adopted & Phase-2 redeem (Dashboard gate via controller lease broker) + L3 attestation POSTURE (TPM presence policy); the L3 cryptographic quote (sealed key + PCR) is Wave 11-B / hardware
  • Date: 2026-06-09
  • Driver: Close the residual fraud risk left explicit in ADR-0103 — on-prem tsuConsumed is self-reported, and the customer controls the host.
  • Relates to: ADR-0103 (token-economy hardening), ADR-0101 (SaaS↔on-prem loop)

Context

Consumption (tsuConsumed) is produced by pkg/metering on the customer's own Kubernetes hosts, written to /var/lib/tlsstress/spend/*.jsonl, drained by the bootstrap-controller, and POSTed to /api/usage/report. The transport is sound (SPKI-pinned TLS + license JWT bound to the hardware fingerprint), but the value being reported is generated on infrastructure the customer controls. ADR-0103's reconciliation watchdog detects gross drift; it does not prevent a customer from under-reporting.

Threat model

The customer is a potentially adversarial party for billing purposes. Assume they can: read and edit any file on disk (incl. the JSONL), inspect and modify any process and its memory, extract any secret delivered to or stored on the host, and forge arbitrary HTTP payloads. They cannot: forge an Ed25519/HMAC signature without the private key, break TLS, or read cloud-side secrets.

Decision — the governing principle

Signing usage events with a key the host holds is NOT anti-fraud. A customer who controls the host extracts the key and signs whatever numbers they like. Any design that hands the signing key to the customer's host buys only non-repudiation and tamper-evidence against third parties — not against the host operator.

Real solidity requires one of (and ideally a layering of):

  • (A) make the key inextractable (hardware) AND attest the integrity of the binary that uses it — so a signature proves an unmodified reporter produced the value; or
  • (B) move the billing measurement out of the customer's control — the cloud authorizes/debits before execution, so self-reporting becomes reconciliation, not the source of the charge.

(B) is the one that delivers solidity without hardware, and is the recommended baseline.

Layered architecture

Layer Mechanism Guarantee Stops the host operator?
L0 (have) Reconciliation + anomaly detection (cloud-side) Detects gross/statistical drift No — detect only
L1 Signed usage envelope, per-deployment key Reporter identity, non-repudiation, anti-tamper vs 3rd parties + naive JSONL edits, anti-replay No (software key) — substrate for L3
L2 Pre-authorization tickets (cloud debits first) Customer can never spend more than the tickets they bought Yes — measurement is cloud-side
L3 Hardware-rooted key (TPM/Nitro/SEV) + remote attestation Key inextractable + binary integrity attested Yes — for the detailed report
L4 Independent corroboration (DUT/NGFW telemetry, cloud sidecar) Cross-check reported vs observed volume Partial — defense-in-depth

L1 — Signed usage envelope (substrate, landed as PoC)

Each drained batch is wrapped in a canonical envelope and Ed25519-signed:

canonical = "tlsstress-usage-v1\n"
          + licenseJti + "\n"
          + deploymentFingerprint + "\n"
          + seqStart + "\n" + seqEnd + "\n"
          + nonce + "\n"                      // 128-bit random, anti-replay
          + sha256_hex(canonicalEventsJSONL)  // binds the exact events
signature = Ed25519(privKey, canonical)
  • Key lifecycle: the private key is generated on the host (never sent by the cloud — that would be theatre). The cloud learns only the public key, enrolled on first /api/install/manifest and bound to (accountId, licenseJti, fingerprint); rotation/revocation via the same channel.
  • Cloud verification (in /api/usage/report): recompute canonical, verify the signature against the enrolled public key, reject on mismatch, and enforce monotonic seq + unseen nonce (anti-replay) on top of the existing (license_id, module, client_seq) idempotency.
  • Honest limit: with a software key this raises the bar from "edit a text file" to "extract the key + re-sign", gives non-repudiation, and is the prerequisite for L3 — but a determined host operator defeats it. Do not market L1-software as fraud-proof.

Flip the model from post-paid self-report to pre-authorized:

  1. Before a run, the module requests authorization: POST /api/usage/authorize { module, estimatedTsu }.
  2. The cloud debits estimatedTsu from the UTXO ledger (atomically, the ADR-0103 spendTSU) and returns a signed ticket { ticketId, accountId, module, tsuQuota, expiresAt, sig }.
  3. The module runs only within the ticket; it may not exceed tsuQuota.
  4. On completion it reports actual usage ≤ quota; the cloud reconciles (refunds the unused remainder via a transfer_refund-style mint).

Because the debit happens cloud-side at authorization time, the customer can never consume more than they paid for, regardless of what the host reports — this closes the risk at its root with no hardware. Trade-offs: an authorization round-trip per run (mitigated by batching/leasing larger tickets), and an offline-grace policy (reuse the heartbeat grace window: allow a bounded pre-authorized lease to run during a cloud outage, reconciled on reconnect).

Implemented (this layer's core): usage_tickets table (migration 0020); POST /api/usage/authorize (debits via the atomic spendTSU and issues a ticket; 402 when out of balance) and POST /api/usage/redeem (idempotent reconciliation, refunds the unused remainder, optional allowRefund=false for strict use-it-or-lose-it); src/lib/usage/tickets.ts core; Go cloudclient Authorize/RedeemTicket + a usage.TicketLease helper that gates on-prem consumption (Consume refuses once the pre-paid quota is exhausted). Adopted: the Dashboard's single license gate (gate.authorize, the chokepoint for the k6 / Playwright / preflight run-starts) now pre-authorizes against the cloud via a lease broker on the controller (internal/leasebroker) — opt-in, transparent to the agents, with the JWT + SPKI pinning staying on the controller (the Dashboard never holds them). Phase-2 IMPLEMENTED: the run-complete handlers (k6/runs/complete, runs/complete) redeem the ticket — reporting actual usage (from the real duration) so the cloud refunds the unused remainder of the pre-paid lease (cloud-lease.cloudRedeem → broker → /api/usage/redeem); the TSU estimate is calibrated to the cloud's intensity multipliers (tsu-estimate.ts). The TicketLease Go helper remains for controller-side modules that gate their own loop.

L3 — Hardware attestation (for enterprise/defense tiers)

The full guarantee: generate the L1 key inside a TPM 2.0 (or AWS Nitro Enclave / AMD SEV-SNP / Intel SGX); the key is sealed and never exported. At boot, measure the reporter binary into PCRs and send a quote to the cloud at enrollment/heartbeat; the cloud accepts usage signatures only from a reporter whose attestation matches a known-good measurement. This makes the L1 signature meaningful against the host operator: the value came from an unmodified reporter using an inextractable key.

Implemented — L3 POSTURE (not the cryptographic quote): the controller detects its TPM device mode (/dev/tpmrm0tpm2-rm, /dev/tpm0tpm2-raw, else absent) and reports it in the heartbeat; the cloud records it per license (licenses.tpm_attestation_mode) and the /api/usage/authorize policy (attestation-policy.ts) can require TPM 2.0 for defense/defense_unlimited tiers — advisory by default (logs usage.authorize.attestation_gap), hard-deny under REQUIRE_TPM_ATTESTATION=true.

⚠️ Honest limit: posture is device PRESENCE, which a determined host operator can spoof. The cryptographic quote (sealed key + PCR measured-boot, the part that actually binds the running binary) is ztp-prem Wave 11-B and requires TPM/enclave-capable UCS hardware + an attestation verifier. Posture is a visibility + soft-gate layer on the path to that guarantee, not the guarantee.

L4 — Independent corroboration

Where a cloud-controlled vantage point exists (e.g. DUT/NGFW flow telemetry, or a cloud-operated sidecar measuring at a point the customer doesn't author), cross-check observed volume against the report and flag divergence. Detection, not prevention — but it raises the cost of collusion.

Recommendation

  1. Solidity baseline, no hardware: ship L2 (pre-authorization tickets). This is the decision that makes the system robust against an adversarial host.
  2. L1 (signed envelope): land now as the cryptographic substrate (PoC in this change) — non-repudiation + anti-tamper + the foundation L3 builds on.
  3. L3 (TPM attestation): for defense/enterprise tiers and any SLA that requires the detailed per-module report to be trustworthy.
  4. L0 + L4: keep reconciliation; add independent corroboration where telemetry exists.

Consequences

  • This change lands the L1 crypto core (pkg/metering/attest.go sign/verify
  • customer-app/src/lib/usage/attestation.ts verify, interop-tested) so the envelope format is proven implementable. It is NOT wired into the live path yet (key enrollment + /api/usage/report verification + TPM sealing are the next steps) and is explicitly insufficient on its own.
  • L2 is the next implementation decision and the recommended path to actual anti-fraud solidity; it touches the controller, the modules, and the cloud (/api/usage/authorize + ticket reconciliation). It does not depend on L1/L3.
  • Until L2 (or L3) ships, tsuConsumed remains self-reported and the compensating controls are L0 (reconciliation/anomaly) — this must be stated to customers and in any billing SLA.

Addendum — 2026-06-10: L2 ticket lifecycle completed (PR #1275)

The deep audit closed the two lifecycle gaps the original L2 implementation left open:

  • Expiry sweep (H2): the expired ticket state previously had no actor — a crash between authorize and run-complete stranded the pre-paid debit in an active ticket forever. The hourly expire-tickets cron now sweeps active tickets past expires_at + a 24h grace (late legitimate redeems inside the grace still refund). Expiry forfeits the lease — no automatic refund: an auto-refund would reopen the exact fraud this layer closes (run the test, suppress run-complete, wait for the money back). Stranded TSU is logged (usage.tickets.expired) for manual ops refund of legitimate crashes, and reconcile-ledger reports staleActiveTickets (0 when the sweep is healthy).
  • Lease hand-back (M4): a local gate deny AFTER the cloud lease was debited (HSM gate, local spend failure) now hands the ticket back immediately via redeem with actualTsu=0 (full refund — the run never started), instead of leaking it to the expiry sweep.

With these, the L2 state machine is fully governed: active → redeemed (run-complete or deny hand-back) or active → expired (sweep, forfeit). The maximum-fraud bound stated in this ADR is preserved on every path — including abandonment, where suppressing the report now costs the customer the lease rather than refunding it.