Skip to content

ADR 0060 — Project OCTOPUS Wave-4 Hyperscale (50k+ clientes)

Status Date Author Supersedes Superseded by
Accepted 2026-05-16 André Luiz Gallon — —

Context

ADR 0053 (cell-based hyperscale) shipped Wave-1 com cells de até 10k clientes cada e Coordinator HA Raft-based via JetStream KV (W3.5-4). Wave-3 PRs (#777-#786) entregaram NATS JetStream, Cloudflare Workers edge 7-layer, ClickHouse petabyte stub (5 tables + 2 MVs), e cross-cloud coordinator HA.

Wave-4 escala TLSStress.Art de 10k para 100k clientes ativos com mudanças fundamentais:

  • LWW (Last-Write-Wins) atual cria conflitos em writes concorrentes cross-region (perda de updates). Solução: CRDTs (OR-Set + LWW-Map).
  • DynamoDB Global Tables atual tem latência write inconsistente cross-region. Solução: FoundationDB ou Spanner com strong multi-master consistency.
  • ClickHouse atual é single-shard. Solução: 12 shards × 3 replicas = 36 nodes para petabyte sustained ingestion.
  • Sidecar mTLS overhead. Solução: eBPF service mesh (Cilium ou Istio Ambient — sidecarless).
  • Coordinator dependency central. Solução: gossip protocol entre cells para autonomous self-heal.

Trigger gates (do not start until BOTH met)

  1. 50k+ clientes ativos sustained 4 weeks (não apenas pico)
  2. >3 cells per region saturadas (>80% sustained 1h) em 2+ regions

Sem ambos, custo de operação Wave-4 (~$30-50k/mês infra incremental) não justifica vs Wave-3 atual.

Esta ADR formaliza a arquitetura antes dos triggers para que PR reviewers tenham contrato claro do que aceitar quando os triggers forem alcançados.

Architectural decision

10 LOCKED decisions.

D1: Multi-region state = FoundationDB OR Spanner (escolha em PR-W4-1.1)

DynamoDB Global Tables atual usa LWW para conflict resolution — perde updates em escrita concorrente cross-region. Para 50k+ clientes com N regions ativas-active, strong consistency é mandatory.

Opções:

  • FoundationDB: open-source, multi-master, ACID transactions cross-shard, mas auto-hospedado (operational burden ~2 SRE FTE).
  • Google Spanner: managed, global strong consistency, mas multi-cloud lock-in tradeoff.

Decisão LOCKED em PR-W4-1.1 (separate ADR) após RFP comparativo com benchmarks reais. Default starting point: FoundationDB (open-source preserva soberania de dados).

D2: CRDTs para coordinator state — OR-Set + LWW-Map

Coordinator state (cell health, primary election, region routing) atual usa LWW em JetStream KV. Em multi-region write concurrence, substituir por:

  • OR-Set (Observed-Remove Set) para colecções (peers em cell, certs válidos, IPs allowlisted)
  • LWW-Map com vector clocks para single-key updates (cell health score, last heartbeat)

Implementação Go em pkg/octopus/crdt/. Stdlib only.

D3: Anycast unification active-active

Atualmente cada cloud tem seu próprio anycast (Cloudflare Magic Transit US/EU, AWS Global Accelerator pendente, Azure Front Door isolated). Wave-4 ativa active-active:

  • 3 clouds simultaneamente respondendo ao mesmo IP virtual
  • Health-check based withdrawal (cloud unhealthy → BGP withdraw)
  • Client-side latency awareness (Happy Eyeballs v2)

D4: ClickHouse 12 shards × 3 replicas = 36 nodes

Atual: 1 ClickHouse cluster por cell, ~10TB/cell.

Wave-4: dedicated 36-node ClickHouse cluster cross-cell para analytics agregadas, com shard key = cityHash64(deployment_id) para uniform distribution. ~1.2 PB capacity após retention 90 dias hot tier.

Cold tier S3 Glacier Deep Archive para 5y compliance retention (legal billing + audit chain).

D5: OpenTelemetry Collector cluster (substitui direct exporters)

Cada MÓDULO exporta direto para Prometheus + Tempo + Loki atualmente. Wave-4: OTel Collector cluster (3 replicas per cell + 3 cross-cell) agrega + filtra + samples antes dos backends.

  • Tail-based sampling de traces (keep 100% errors, 10% successes)
  • Metric filtering (drop high-cardinality labels antes do storage)
  • Loki Promtail substitute

Reduz custo de observability storage ~40%.

D6: eBPF service mesh (Cilium ou Istio Ambient — escolha em PR-W4-7)

Sidecar Envoy atual: 50MB RAM + 50ms p99 latency overhead por pod. Wave-4 elimina sidecars:

  • Cilium: eBPF dataplane, native K8s integration, mTLS via WireGuard
  • Istio Ambient: ztunnel per-node, waypoint proxies on-demand, mais maduro para enterprise

Decisão em PR-W4-7 após PoC paralela.

D7: Cell autonomous self-heal (gossip protocol)

Coordinator central atual é SPOF (apesar do Raft HA). Wave-4: cells gossipam state entre si (SWIM ou HashiCorp memberlist), eliminando dependência do coordinator para failover.

  • Cell A detecta cell B unhealthy via gossip → re-routes traffic local
  • Coordinator central torna-se "tiebreaker" + ARRR audit, não path crítico
  • Resilience: full coordinator outage NÃO degrada steady-state ops

D8: Multi-region admin console (master-master Postgres pgactive)

Admin console (admin.tlsstress.art) Wave-3 é single-region (us-east-1). Wave-4 ativa multi-region master-master via pgactive (Postgres extension AWS-mantida, ex-BDR).

  • Operator em São Paulo escreve para sa-east-1 master
  • Operator em Tóquio escreve para ap-northeast-1 master
  • Conflicts resolved deterministically (last-writer per row with vector clock metadata)

D9: Petabyte cold storage tiering (S3 Glacier Deep Archive)

Atual: tudo hot tier ClickHouse $0.023/GB. Wave-4: 90d hot → 1y warm S3 → 5y cold Glacier Deep Archive.

  • Hot ClickHouse: 1.2 PB capacity
  • Warm S3 Standard: 12 PB (90d → 1y)
  • Cold S3 Glacier Deep Archive: 50 PB (1y → 5y)

Reduce storage cost de $300k/mês (all-hot 50PB) para ~$50k/mês (tiered).

D10: Wave-4 Sigstore admission policy strict

ADR 0048 ZTP-prem PR-ZTP-10 ainda permite unsigned images com warning. Wave-4 endurece: ALL images must be Sigstore-signed by an approved identity. Unsigned = K8s admission webhook rejects → image pull fails.

Approved identities: - GitHub Actions OIDC (CI + manual builds) - 3rd-party allowlist (cert-manager, Prometheus operators with verified signatures)

Architecture overview

                       ┌─────────────────────────────────────────┐
                       │  Wave-4 Global State Layer              │
                       │  FoundationDB/Spanner (3 regions)       │
                       │  Strong consistency multi-master        │
                       └──────────────────┬──────────────────────┘
                                          │
              ┌───────────────────────────┼───────────────────────────┐
              │                           │                           │
        ┌─────▼────────┐         ┌────────▼─────┐         ┌──────────▼────┐
        │ AWS us-east-1│         │GCP us-central│         │ Azure eastus  │
        │ 10 cells     │         │ 10 cells     │         │ 10 cells      │
        │ Cilium eBPF  │         │ Cilium eBPF  │         │ Cilium eBPF   │
        │ gossip mesh  │←────────│ gossip mesh  │────────→│ gossip mesh   │
        └──────────────┘         └──────────────┘         └───────────────┘
              │                           │                           │
              └───────────────────────────┼───────────────────────────┘
                                          │
                       ┌──────────────────▼──────────────────────┐
                       │  Cross-cell ClickHouse                  │
                       │  36 nodes (12 shards × 3 replicas)      │
                       │  1.2 PB hot + 12 PB warm + 50 PB cold   │
                       └─────────────────────────────────────────┘
                                          ▲
                                          │ filtered/sampled
                       ┌──────────────────┴──────────────────────┐
                       │  OpenTelemetry Collector cluster        │
                       │  per-cell (3 replicas) + cross-cell (3) │
                       └─────────────────────────────────────────┘

Capacity targets

Metric Wave-3 (now) Wave-4 target
Concurrent customers 10k 100k
Cells per region 3-5 20-30
Connect.Art requests/sec/cell 5k 15k
ClickHouse ingestion rate 100k events/s 1M events/s
Cross-region failover RTO 5 min 60 sec
Multi-region write consistency LWW (eventual) Strong (CRDT + global state)
Coordinator outage tolerance 0s (cells fail) 24h+ (gossip steady-state)
Storage cost / PB / month $25k $4k tiered
Sigstore admission policy warn enforce

Wave-4 PR roadmap

PR Scope Estimated LoC Dependencies
PR-W4-1 ADR 0060 hyperscale umbrella (this) docs none
PR-W4-2 ADR 0061 CRDT OR-Set + LWW-Map formalization docs none
PR-W4-3 ADR 0062 multi-region global state (FDB/Spanner RFP) docs none
PR-W4-4 pkg/octopus/crdt/ scaffold (OR-Set + LWW-Map types + interfaces) ~600 PR-W4-2
PR-W4-5 OR-Set implementation + tests ~800 PR-W4-4
PR-W4-6 LWW-Map impl + migration helpers from current LWW state ~700 PR-W4-5
PR-W4-7 ClickHouse 12×3 sharded schema + Helm chart ~500 PR-W4-3
PR-W4-8 OTel Collector cluster Helm + scrape configs ~400 none
PR-W4-9 Coordinator gossip protocol (memberlist) ~1200 PR-W4-4
PR-W4-10 Cilium eBPF service mesh PoC + benchmarks ~600 none
PR-W4-11 pgactive master-master admin DB ~800 none
PR-W4-12 S3 Glacier tiering Terraform + lifecycle policies ~300 PR-W4-7
PR-W4-13 Sigstore admission webhook strict mode ~500 none
PR-W4-14 Observability dashboard + alerts ~400 all above
PR-W4-15 E2E chaos + multi-region failover tests ~800 all above
PR-W4-16 Reviewer checklist + runbook + ops docs docs all above
Total ~7600 LoC

Note: PR-W4-1.1 (FDB/Spanner choice) is a follow-up that consumes PR-W4-3 benchmarks + RFP data.

Backwards compatibility

Wave-4 ships incrementally over 24-32 weeks. Each PR maintains production compatibility via feature flags:

  • OCTOPUS_CRDT_ENABLED=false default — falls back to existing LWW path
  • OCTOPUS_GOSSIP_ENABLED=false default — coordinator central
  • OCTOPUS_GLOBAL_STATE_BACKEND=dynamodb default — keeps Wave-3 stack

Cutover gate: per-cell flag flip after E2E chaos suite green for 7 days.

Patent claim mapping

  • Claim 21 (Family E): Multi-region active-active operator state with CRDT conflict resolution + audit chain integration
  • Claim 22 (Family E): Cell autonomous self-heal via gossip without central coordinator dependency
  • Claim 23 (Family E): Tiered petabyte storage with audit chain Merkle root replication across hot/warm/cold tiers

Provisional target Q1 2027 (after Wave-4 production proof points).

Failure modes + mitigations

Failure Mitigation
FDB/Spanner outage Wave-4 cells degrade gracefully to last-known-good state for 4h before forced ops alert
Gossip protocol partition each side keeps local view; reconverges on rejoin (eventual consistency tolerated for non-routing state)
eBPF service mesh kernel bug per-cell fallback to sidecar mode (feature flag)
Sigstore policy false-positive blocks deploy break-glass emergency unsigned-image allowlist with 4-eyes approval + audit
pgactive master-master conflicts per-row vector clock metadata + admin UI surface conflicts to ops
Cold-tier Glacier restore SLA (12-48h) hot+warm tiers cover 99.9% of queries; cold restores only for compliance/legal

Closes audit gap

Gap #20 — Wave-4 hyperscale not formalized; PR reviewers had no contract for 50k+ scale changes Gap #21 — CRDT migration path from current LWW not documented Gap #22 — Multi-region global state backend choice not formalized

Cross-references

  • ADR 0053 — Wave-1 cell architecture
  • ADR 0054 — PQC mandate (preserved in W4)
  • ADR 0056 — Wave-1 provisioning
  • ADR 0058 — IPv4+IPv6 mandate
  • Roadmap memo
  • W4-2 sister ADR 0061 (next PR)
  • W4-3 sister ADR 0062 (after benchmarks)