ADR 0062 — OCTOPUS Multi-Region Global State RFP (Wave-4 D1)¶
| Status | Date | Author | Supersedes | Superseded by |
|---|---|---|---|---|
| Proposed (RFP) | 2026-05-16 | André Luiz Gallon | partial: DynamoDB Global Tables | TBD by PR-W4-1.1 |
Context¶
ADR 0060 D1 (Wave-4) mandates substituir DynamoDB Global Tables como multi-region global state backend. Atual posture LWW perde writes cross-region em escalas Wave-4 (50k+ clientes).
Este ADR é um RFP (Request for Proposal) que enumera os candidates, critérios de seleção, e o experimental method para escolher o backend. Decisão final em PR-W4-1.1 após PoC + benchmarks de 4 semanas.
Selection criteria¶
Critérios prioritários (weighted), em ordem decrescente:
| Criterion | Weight | Rationale |
|---|---|---|
| Strong consistency cross-region | 30% | sem isso, perdemos updates concorrentes (Wave-4 mandate) |
| Open-source / self-host option | 20% | sovereignty data, no vendor lock-in |
| Operational maturity in 50k+ scale prod | 15% | risk avoidance |
| Cross-cloud compatibility | 15% | AWS+GCP+Azure all in scope |
| Cost at petabyte scale | 10% | $/PB/month vs DynamoDB baseline |
| Driver quality + ecosystem | 10% | Go SDK, k8s operators, observability |
Candidates¶
A. FoundationDB (Apple OSS)¶
Pros:
- Open-source Apache 2.0
- Strong serializable consistency cross-shard
- ACID transactions across keys (rare in distributed)
- Production proven (Apple iCloud, Snowflake metadata layer)
- Self-host across any cloud, no vendor lock-in
- Bindings for Go via
apple/foundationdb-go
Cons:
- Operational burden: cluster management, backup/restore, capacity planning
- ~2 SRE FTE incremental for operation
- Smaller community than Spanner / DynamoDB
- Multi-region replication via sync layer (FDB itself is single-region) — adds complexity
Cost estimate (3 regions, 10TB total):
- AWS: 9× r6i.4xlarge ($1500/mo each) + 30TB EBS = ~$15k/mo
- GCP/Azure equivalent
- Total: ~$45k/mo across 3 clouds
- Plus 2 SRE FTE: ~$30k/mo loaded
- Grand total: ~$75k/mo
B. Google Cloud Spanner¶
Pros:
- Managed (zero ops burden — biggest single advantage)
- True global strong consistency (TrueTime atomic clocks)
- Production proven (Google AdWords, Stripe, Salesforce)
- Auto-scaling, automated backups, point-in-time recovery
- Good Go SDK with native protobuf
Cons:
- GCP-only (multi-cloud lock-in tradeoff)
- Closed-source (no inspection of internals)
- Cost per request can dominate at high QPS
- Latency cross-region (~50-100ms even with TrueTime) — slower than FDB single-region
Cost estimate (3-region, 100k req/sec):
- Compute: 3× regional×3 nodes minimum = 9 nodes × $650/mo = ~$6k/mo
- Storage: 10TB × $0.30/GB/mo = ~$3k/mo
- Network egress: ~$2k/mo cross-region
- Backup: ~$500/mo
- Grand total: ~$11.5k/mo
(Note: significantly cheaper than FDB but trades cost for vendor lock-in)
C. CockroachDB Enterprise¶
Pros:
- Open-source core + commercial Enterprise license
- Self-host any cloud (similar to FDB)
- Postgres wire protocol compatibility (lower learning curve)
- Multi-region native (no sync layer needed — built-in)
- Strong consistency via Raft per range
Cons:
- Commercial license cost can dominate (~$10k/year per node × N nodes)
- Less production proven than FDB or Spanner in true global scale
- Performance hit on multi-region writes (Raft consensus latency)
Cost estimate (3 regions, 10TB):
- Compute: 9× n2-highmem-8 (or equivalent) = ~$8k/mo
- License: ~$30k/mo (Enterprise tier for HA + multi-region)
- Storage: 30TB SSD = ~$3k/mo
- Grand total: ~$41k/mo
D. YugabyteDB¶
Pros:
- Open-source AGPL with commercial license option
- Postgres + Cassandra wire compatibility (dual protocol)
- Multi-region native
- Growing ecosystem
Cons:
- Less production-proven than alternatives (newer)
- Performance characteristics still being validated in industry
- Smaller talent pool for hiring SRE familiar
Cost estimate (3 regions, 10TB):
- Compute: 9× m5.2xlarge = ~$5k/mo
- License: free OSS or ~$15k/mo Enterprise
- Storage: ~$3k/mo
- Grand total: ~$8-23k/mo depending on edition
PoC plan (PR-W4-1.1)¶
Duration: 4 weeks parallel evaluation.
Workloads:
- Cell health write storm: 1000 writes/sec to OR-Set state from 3 regions simultaneously. Measure: convergence time, lost-write rate.
- Sustained read: 10k reads/sec across the 3 regions. Measure: p50/p95/p99 latency, throughput ceiling.
- Failover drill: kill one region's primary, verify reads continue from healthy regions. Measure: failover detection + recovery time.
- CRDT envelope migration: simulate Wave-3 LWW → Wave-4 CRDT migration on the backend. Measure: migration window duration, error rate.
Decision matrix scored 1-5 per criterion, weighted total determines winner.
Owners:
- PR-W4-1.1 lead: octopus-platform on-call rotation
- Reviewers: Cisco-side architecture review board (3 senior staff)
Default starting point (revisable by RFP)¶
Per ADR 0060 D1: default starting point is FoundationDB if PoC doesn't conclusively favor another candidate.
Rationale:
- Open-source sovereignty matches TLSStress.Art posture (already PolyForm dual license; FDB Apache 2.0 is permissive)
- Apple iCloud production track record at billion-scale gives confidence
- Cross-cloud compatibility (run on AWS, GCP, Azure equally well)
- ACID cross-shard semantics simplify CRDT integration
Risk register¶
| Risk | Severity | Mitigation |
|---|---|---|
| FDB ops burden underestimated | HIGH | hire dedicated SRE per region OR pivot to Spanner if ops cost dominates |
| Spanner GCP lock-in | MEDIUM | abstract via pkg/octopus/globalstate/ adapter pattern (every backend implements interface) |
| Cockroach license costs at scale | MEDIUM | negotiate enterprise discount OR fall back to OSS limits |
| Latency tradeoff Spanner cross-region | LOW | within SLO bounds for non-real-time coordinator paths |
| PoC compromise on workload realism | HIGH | use real Wave-3 traffic shadow-replayed, not synthetic |
Adapter pattern¶
Regardless of winner, all callers use pkg/octopus/globalstate/
interface:
type Store interface {
Get(ctx context.Context, key []byte) ([]byte, error)
Put(ctx context.Context, key, value []byte) error
Delete(ctx context.Context, key []byte) error
Watch(ctx context.Context, prefix []byte) (<-chan Event, error)
Transact(ctx context.Context, fn func(Tx) error) error // ACID
}
Backend swap is a configuration change, not code rewrite. PR-W4-1.1 ships the chosen backend implementation; subsequent PRs may add alternatives.
Closes audit gap¶
Gap #22 — Multi-region global state backend choice not formalized