ADR-0085: TREX.Art — DPDK stateful traffic generator (30 Mpps/core, 40M flows)¶
- Status: Accepted (2026-05-24) — full TREX-B wave (TREX-B1..TREX-B4) materialized on top of the pre-existing scaffolds (TREX-1 pod manifest, TREX-2 preflight, partial TREX-5 Grafana, partial TREX-10 docs skeleton)
- Date: 2026-05-24
- Driver:
discuss_module_trex_art_2026_05_10locked TREX.Art as the missing DATA-plane line-rate stateful traffic generator between K6.Art (HTTP-aware, ~10G ceiling) and iperf3 (raw L4 throughput baseline). Memo identified one decisive use case unreachable from either side: 100G+ NGFW certification with realistic stateful TCP/UDP/IPSec workloads - Related: PRs #1040 (TREX-B1), #1041 (TREX-B2), #1042 (TREX-B3), this PR (TREX-B4); strategic memo
discuss_module_trex_art_2026_05_10; sister-MÓDULO ADRs ADR-0083 (KALI.Art Phase B) + ADR-0084 (HAR.Art L7 replay) + ADR-0036 (HyperBridge.Art); patent claim #17
Context¶
TLSStress.Art needs a credible answer to "can your bench saturate 100G NGFW datasheets?". The existing engines top out short of that target:
- PW.Art (Playwright fleet) — browser-realistic at ~50 sessions/host, fundamentally limited to ~10G aggregate even at hundreds of replicas
- K6.Art (Go HTTP load gen) — scriptable HTTP-aware, ~10G/host practical ceiling (kernel TCP stack + scheduler contention)
- iperf3 (via Cloud Endpoint) — raw L4 throughput; the wrong primitive for stateful inspection validation
The competitive landscape (Spirent BreakingPoint, Keysight CyberFlood, Ixia BreakingPoint Cloud) all ship DPDK-backed line-rate stateful traffic generators because the math is unforgiving: Linux kernel TCP stack maxes out near 10 Gbps per core; DPDK kernel-bypass unlocks 30 Mpps/core. Closing the gap requires either: - Building our own DPDK stack (3-5 engineer-year effort) - Wrapping the open-source Cisco TRex (BSD license, actively maintained, ~30 Mpps/core proven)
TRex won the build-vs-buy by a wide margin. The remaining decision was scope.
Decision¶
Ship MÓDULO TREX.Art as a thin Go orchestrator wrapping the upstream Cisco TRex binary, deployed to slot .220 (primary) and .221 (HA standby, Enterprise tier only). The orchestrator is the operator surface; TRex itself is operator-installed out-of-band on the bench node (the binary is ~200 MB and shouldn't bloat the container image).
Materialized in 4 sub-PRs after the original TREX-1/TREX-2 scaffolds:
| Sub-PR | Scope | Status |
|---|---|---|
| TREX-B1 | Library 5 → 10 (memo-locked set) + types.TierLimits + tier-gate enforcement at Start(). 4 new orchestrator tests |
✅ #1040 |
| TREX-B2 | New pkg/trex-art/internal/sandbox/ Python-profile scanner (20 banned patterns / 6 categories, docstring-aware) + POST /scenarios/upload + Dashboard dropzone with per-line violation render. 17 new tests |
✅ #1041 |
| TREX-B3 | New pkg/trex-art/internal/runsummary/ ring buffer (50 entries) + GET /runs/summaries + Dashboard "Recent runs (ground-truth)" panel for SPAN.Art cross-verification. 10 new tests |
✅ #1042 |
| TREX-B4 | This ADR + docs/modules/trex-art.{md,pt-BR.md,es.md} expansion + recovery memo project_trex_art_complete_2026_05_24.md |
✅ |
Locked defaults — 5 questions from the memo¶
The strategic memo locked 5 high-level decisions before B1 started; this ADR records them in canonical form:
| # | Question | Locked answer |
|---|---|---|
| 1 | Hardware prereqs (DPDK NIC + hugepages 1G + dedicated cores + kernel ≥ 5.4) | Acceptable for 100G+ tier — TREX-2 preflight blocks launch when missing, surfaces remediation steps |
| 2 | Scope Freeze exception | Granted — closes the gap vs Spirent / Keysight / Ixia at the 100G+ stateful tier |
| 3 | Marketing branding | Aggressive "Powered by Cisco TRex" — dashboard footer + marketing site + Investor Deck + sales sheets |
| 4 | Cisco TRex aging risk | Fork-commitment locked in TLSStress.Art governance — tlsstress-art/trex-fork to be created if upstream is ever discontinued (currently active maintenance) |
| 5 | External customer-owned TRex cluster mode | OUT OF SCOPE — TREX.Art is bench-deployed only |
Tier policy (locked)¶
Per the memo's "Team+ minimum" mandate:
| Tier | Access | Max cores | Max duration | Max pps/core | Custom profiles |
|---|---|---|---|---|---|
| free | ❌ Disabled (upsell modal) | — | — | — | — |
| indie | ❌ Disabled (upsell modal) | — | — | — | — |
| team | ✅ | 4 | 30 min | 10 Mpps | ❌ |
| enterprise | ✅ | 16 | 24 h | 30 Mpps | ✅ |
Tier-gate enforcement runs at the top of orchestrator.Start():
- Empty StartRequest.Tier defaults to "team" (the minimum tier that can launch any TREX scenario)
- "free" / "indie" / unknown → rejected with Team+ required error
- tier < scenario.MinTier (e.g. team launching http3-quic-saturation or mixed-enterprise) → rejected with requires tier X error
- Duration / pps-per-core / cores clamped to min(Config caps, TierLimits caps) — tighter wins
10-profile library (locked)¶
Matches the Investor Deck pitch "10 bundled enterprise traffic profiles, no script writing required":
| Profile | Category | MinTier | Default pps/core |
|---|---|---|---|
tcp-syn-flood |
ddos | team | 100k |
udp-flood |
ddos | team | 200k |
http-stateful |
stateful | team | 50k |
ipsec-aggressive |
ipsec | team | 30k |
dns-amplification-defense-test |
defense-test | team | 80k |
https-handshake-storm |
stateful | team | 40k |
sip-call-setup-storm |
stateful | team | 25k |
http3-quic-saturation |
stateful | enterprise | 60k |
voip-rtp |
stateful | team | 50k |
mixed-enterprise |
stateful | enterprise | 70k |
Operator-uploaded custom profiles unlock at enterprise tier and are validated through the pkg/trex-art/internal/sandbox/ scanner before acceptance.
Sandbox scanner (TREX-B2 — Python AST-free heuristic)¶
A Go-side line-based scanner runs on every upload. Why heuristic rather than a full Python AST walker (bandit): - Upload latency budget is ~200 ms inline — spinning a Python interpreter triples it - Container is Go-only / distroless — adding a Python runtime fights the image-minimization posture - The patterns we reject are deliberately conservative — false positives are fine; false negatives are not
20 banned patterns across 6 categories: shell-out (subprocess / os.system / os.popen / os.exec), network (socket / urllib / requests / httpx / http.client), dynamic-code (exec / eval / compile / import), FFI (ctypes / cffi), env-writes (os.environ[).
Comment + triple-quoted-string stripping means """os.system is bad""" in a docstring doesn't trip the rule.
True byte-exact AST validation is deferred to an operator-opt-in CI hook (out of scope).
Ground-truth cross-verification (TREX-B3)¶
Every completed scenario produces a runsummary.Summary containing the intended-packet count (cores × pps × elapsed) alongside scenario name + tier + exit phase. The orchestrator exposes:
- GET /runs/summaries — newest-first array of last 50 runs
- GET /runs/summaries/latest — most-recent (200) or 404
External correlator contract: SPAN.Art polls /api/trex-art/runs/summaries/latest for intended counts + SPAN's /stats?since&until for wire-observed counts; drift > 5% over the run window → "DUT fast-path bypass suspected" (a SPAN.Art Pillar 5 finding).
In-memory ring instead of Postgres because TREX.Art is a single-pod control surface; persistent storage belongs to the Dashboard Drizzle layer (lands in a follow-up if/when "history older than 50 runs" becomes a real ask).
Patent posture¶
Patent claim #17 from the project-wide claims list — "heterogeneous traffic-generation engine selection within unified test orchestration platform, providing operator-explicit selection per test plan among (a) browser-realistic L7 generation engine (PW.Art), (b) scriptable HTTP load generation engine (K6.Art), (c) DPDK-accelerated stateful traffic generation engine (Cisco TRex backend), (d) raw L4 throughput generation engine (iperf3 backend); with hardware-prerequisite preflight validation per engine type; unified observability pipeline (SPAN ground truth + ML cortex pattern recognition) across all engines; and tier-gated capability unlocking aligned with token-metered consumption."
Total patent claims after TREX.Art: 17.
Consequences¶
Positive
- Closes the most-visible competitive gap (100G+ stateful) with a thin orchestrator (~2000 LoC in
pkg/trex-art/) instead of a multi-engineer-year DPDK build - 10-profile library lets a customer-facing demo run start to finish without writing Python
- Tier-gate at the orchestrator layer (not just UI) means raw
curl POST /scenarios/startfrom the SDK is also gated — single source of truth - Sandbox scanner means operators can safely accept community profiles + Cisco-published profiles without manual review
- Ground-truth run summaries close the SPAN.Art cross-verify loop without a database
Negative / deferred
- Custom profile PVC persistence — TREX-B2 validates but doesn't persist; storage lands when a customer asks (matches HAR-B2 pattern)
- TREX → SPAN.Art push (synchronous notification on run end) — currently SPAN polls; push semantics defer to a follow-up if poll latency becomes a UX issue
- Cisco methodology alignment —
ngfw-cert-baseline.pyis named in the memo but ships in the operator-upload path (custom profile), not the bundled 10. Lands as a bundled scenario after the first NGFW vendor certification engagement validates the exact ASTF profile - Cloud Endpoint TRex peer mode — memo flagged it as opt-in; deferred behind the first customer ask for cross-region traffic origination
- Profile marketplace (community contributions, signed by TLSStress.Art QA) — Phase v5.x
Alternatives considered¶
- Build our own DPDK stateful traffic generator — rejected; 3-5 engineer-year effort, no clear competitive advantage over wrapping TRex
- Use Spirent / Keysight / Ixia SDK — rejected; commercial license costs + closed source defeats the "open-platform" positioning
- Make TREX.Art free-tier — rejected; hardware-prereq friction + token-economy alignment with the 3x baseline rate matches the cost model
- External customer-owned TRex cluster mode — rejected per memo question #5; would dilute the bench-as-product story
- Full Python AST sandbox (bandit-style) — rejected for B2; latency + container-size cost don't match the false-positive-tolerant validation we actually need
- Persistent run-summary database (Postgres) — rejected for B3; 50-entry ring covers the realistic operator shift, persistence is the Dashboard's concern
Compliance¶
Per CLAUDE.md: "Dashboard is the ONLY operator interface." TREX-B1 through TREX-B3 all ship the UI surface (start / stop / upload / recent runs) at /admin/trex-art — operator never types kubectl against the oobi-fabric namespace for TREX.Art.
Per the locked ZTP-prem posture: gosec W0 baseline preserved (the new pkg/trex-art/internal/{sandbox,runsummary}/ packages are pure stdlib — bufio, strings, sync, time).
Per ADR-0044 (OOBI Intrusion Detection): the trex-art HTTP server listens on :8085 inside the OOBI fabric only; ingress is fronted by the Dashboard proxy which carries the auth surface.
References¶
discuss_module_trex_art_2026_05_10— strategic memo (locked the 10-PR roadmap + 5 questions + Team+ tier + scope-freeze exception + "Powered by Cisco TRex" branding + fork commitment + no-external-cluster scope)- ADR-0083 (KALI.Art Phase B), ADR-0084 (HAR.Art L7 replay), ADR-0036 (HyperBridge.Art) — sibling MÓDULOs in the offensive + advanced-traffic family
- PRs #1040 (TREX-B1), #1041 (TREX-B2), #1042 (TREX-B3), this PR (TREX-B4)
- Upstream: cisco-system-traffic-generator/trex-core
- Patent claim #17