NetSecOPEN-aligned testing with TLSStress.Art — operator primer¶
Other languages: English · Português · Español
Audience: NGFW operator running a TLSStress.Art test bench who needs to produce results matching the format published by Cisco, Palo Alto, Fortinet, Check Point, and other vendors in their NetSecOPEN cert reports (Cisco Secure Firewall 1220CX, 3105, etc.).
Companion docs: NETSECOPEN_ALIGNMENT.md (vocabulary + RFC 9411 mapping), INSPECTION_PROFILE.md (the 5 named inspection profiles), STRESS_ENGINES_CATALOG.md § Which engine for which Test Kind (the 6 test kinds).
TL;DR¶
# 1. Pick a NetSecOPEN-aligned test plan (3 ready)
# via dashboard /tests/new → Template: "NetSecOPEN cert §7.2 — TCP/HTTP CPS"
# 2. Hit Run. Bench enforces RFC 9411 §4.3.4 phase rules + §7.x.3.3 thresholds.
# 3. Download report → choose template "cert" → PDF/HTML/Markdown.
# 4. Audit chain (NSO-20) attaches per-test SHA-256 + Cosign signature
# so the customer can verify the report wasn't tampered.
The bench is NetSecOPEN-aligned, not (yet) an Approved Tool. That means: every report you ship matches the published format exactly, but the NetSecOPEN consortium logo on the report cover is deliberately omitted until our membership campaign (Phase D-E) is complete. See §"Why we're not yet an Approved Tool".
What is NetSecOPEN¶
NetSecOPEN is an industry consortium that publishes:
- RFC 9411 — Benchmarking Methodology for Network Security Device Performance (Mar 2023), the IETF-sanctioned methodology for measuring NGFW / IDS / IPS performance.
- Approved test tools — a short list (currently Keysight CyPerf
- BreakingPoint, Viavi/Spirent CyberFlood) that have been audited by the consortium to produce repeatable results.
- Cert reports — vendors run their products through one of the approved tools at an approved lab (e.g. EANTC) and publish a cert report. These reports follow a fixed 16-table layout.
When a customer asks "what's your NetSecOPEN throughput?" they're asking about the cert report's Table 6 (HTTPS throughput @ 256 KByte object size, Inspection Profile = balanced or paranoid).
What TLSStress.Art delivers¶
A cert report byte-for-byte identical in format to a Cisco /
Palo Alto / Fortinet NetSecOPEN cert. Sample DUT: Cisco Secure
Firewall 1220CX. Sample mix: Healthcare. Sample report: see
docs/sample-reports/ (delivered in NSO-22).
Every report has:
| Element | RFC 9411 § | TLSStress.Art module |
|---|---|---|
| Tables 1-16 (cert layout) | §5 + Appendix C | NSO-16 KPI aggregator |
| Stability graphs (sustain phase) | §4.3.4 | NSO-17 stability graph renderer |
| Pie charts (Figures 2-3) | §7.1 | NSO-18 pie chart renderer |
| PDF/HTML/Markdown render | §5 reporting | NSO-19 report renderer |
| Per-test SHA-256 audit chain | §5.1 traceability | NSO-20 audit chain |
| Cosign signature on chain tip | (industry best practice) | NSO-20 cosign_bundle |
Step 1 — Pick a NetSecOPEN-aligned test plan¶
Open the dashboard → Tests / New → Template dropdown. You'll see entries grouped under NetSecOPEN cert:
| Template | RFC 9411 § | Mirrors cert table |
|---|---|---|
| App Mix Performance — Healthcare | §7.1 | Table 2 + Figure 2 |
| App Mix Performance — Education | §7.1 | Table 2 + Figure 3 |
| TCP/HTTP CPS by object size | §7.2 | Table 7 |
| HTTP Throughput by object size | §7.3 | Table 8 |
| TCP/HTTP TTFB+TTLB @ 50% CPS | §7.4 | Table 9 |
| TCP/HTTPS CPS by object size | §7.6 | Table 11 |
| HTTPS Throughput by object size | §7.7 | Table 12 |
| TCP/HTTPS TTFB+TTLB @ 50% CPS | §7.8 | Table 13 |
| Concurrent connections | §7.5 + §7.9 | Table 3/4 row 6 |
| Detection Rate — CVE corpus | Appendix A.2 | Table 15 |
| Under Load — Detection Rate | Appendix A.3 | Table 16 |
Pick the one that matches the cert table you owe the customer.
Step 2 — Choose your DUT class¶
Per RFC 9411 §4.2 + Appendix B, the bench auto-classifies the DUT into XS / S / M / L based on its rated throughput. This determines the ACL rule count and other tunables.
| Class | Throughput | ACL rules (block-only baseline) |
|---|---|---|
| XS | ≤ 1 Gbit/s | 65 |
| S | 1 – 5 Gbit/s | 120 |
| M | 5 – 10 Gbit/s | 230 |
| L | > 10 Gbit/s | 560 |
The bench applies the right rule set automatically (NSO-5 module).
You can override via dashboard → Tests → Advanced → DUT class.
Step 3 — Confirm preflight¶
Before the test starts, the preflight panel runs 16 checks (NSO-6 module). The panel is divided in three severity tiers:
- PASS (green) — ready
- WARN (yellow) — proceed but the report will note the gap
- FAIL (red) — bench will refuse to start
Hard requirements (FAIL on violation):
- DUT in Inline mode (not TAP / SPAN)
- Fail-Open disabled
- TLS Inspection enabled (for HTTPS tests)
Recommended (FAIL if disabled w/o reason; WARN if a reason is recorded):
- IDS/IPS, Antivirus, Anti-Spyware, Anti-Botnet, Anti-Evasion, Logging, App-ID
If you're testing a baseline NGFW posture, click "Apply NetSecOPEN balanced profile" — the bench will configure all 8 RECOMMENDED features automatically.
Step 4 — Run¶
Hit Run. The orchestrator (NSO-7 FSM) walks the 11-state phase machine:
init → ramp_up → sustain → ramp_down → analyze → done
Per RFC 9411 §4.3.4:
- init ≥ 5s
- sustain ≥ 300s (the stability graph window)
- Sampling interval ≤ 2000 ms
The dashboard shows a live progress bar + the stability graph being
drawn in real time. If failed_tx exceeds 0.001% or fwd_rate_dev
exceeds 5%, the test is flagged FAIL (per §7.x.3.3 thresholds).
Step 5 — Download the report¶
After done, click Download report → choose template:
| Template | Use for |
|---|---|
| cert | Customer deliverable matching Cisco/Keysight/Viavi format |
| lab | Internal review / marketing collateral / champion share-out |
Each template has 3 output formats:
.pdf— print-ready, embedded SVGs.html— single-file with inline CSS + SVGs.md— Markdown for git/audit pinning
The audit chain (NSO-20) is attached as audit-chain.json next to
the report. Customer can verify it via:
tlsstress-art audit verify --chain audit-chain.json --report report.pdf
Step 6 — Cross-validate against a vendor cert¶
The whole point is that your report should be apples-to-apples with a vendor cert. Quick cross-validation against the Cisco 1220CX published cert (sample DUT in NSO-22):
| KPI | Cisco cert | TLSStress.Art on same DUT |
|---|---|---|
| Healthcare app-mix throughput | 1.27 Gbit/s | within ±5% |
| HTTP CPS @ 1 KByte | 21,344 | within ±5% |
| HTTPS @ 256 KByte throughput | 3.86 Gbit/s | within ±5% |
| Concurrent HTTPS connections | 78,552 | within ±5% |
| Detection rate (5388 CVE corpus) | 99.65% | within ±0.5% |
If you see a drift > ±5%, check:
- Is the DUT software version identical? (Cisco cert column 1 lists it.)
- Is the inspection profile identical? Cert reports note "all RECOMMENDED features enabled".
- Did your test run > 300s sustain? Shorter runs systematically over-report.
- Did
failed_txcross the threshold? A failed test contaminates the average.
Why we're not yet an Approved Tool¶
NetSecOPEN's Approved Tool program currently lists exactly two vendors (Keysight + Viavi/Spirent). The approval process requires:
- Consortium membership ($XX,XXX/year sponsorship)
- Independent lab validation (typically EANTC) running our tool against a reference DUT and confirming the same results within tolerance
- Reproducibility audit — code review + test corpus disclosure
Status (per discuss_netsecopen_rfc9411.md 5-phase roadmap):
| Phase | Status |
|---|---|
| A — Technical Readiness (NSO-1..22) | 🟡 In progress (this Wave) |
| B — Independent Validation | ⚪ Not started |
| C — Licensing / Corporate setup | ⚪ Not started |
| D — Membership engagement | ⚪ Not started |
| E — Approval Campaign | ⚪ Not started |
Estimated full timeline: 24-36 months.
In the meantime, we ship reports with the unambiguous label "NetSecOPEN-aligned" (not "NetSecOPEN-certified"). The format is identical; only the consortium endorsement is missing.
Troubleshooting¶
| Symptom | Diagnosis | Fix |
|---|---|---|
| Stability graph shows FAIL with deviation > 5% | DUT thermal throttling or co-tenant noise | Drop test concurrency; rerun on cold start |
| Detection rate < 95% | CVE signature DB out of date | Run dashboard → Settings → CVE corpus → Refresh |
| Tables 11-14 missing from report | HTTPS plan template not selected | Pick a *HTTPS* template, not *HTTP* |
| Audit chain verification fails | Bytes were modified after sign | Re-render report from raw KPI JSON; do NOT edit the PDF |
Cross-refs¶
- NETSECOPEN_ALIGNMENT.md — vocabulary + RFC 9411 §6 benchmark coverage matrix
- INSPECTION_PROFILE.md — 5 named profiles
- STRESS_ENGINES_CATALOG.md § Which engine for which Test Kind — combinatorial test design
- TLS_INSPECTION_TRAPS.md — never say "SSL Inspection"
- ADR/0015-help-center-learning-hub.md — Help Center architecture
discuss_netsecopen_rfc9411.md— strategic locked decision (Option C)- Sample report:
docs/sample-reports/cisco-1220cx-healthcare-2026-05-15.pdf(delivered in NSO-22)
Module index — what code produces what¶
| Module | RFC 9411 § / Appendix | Purpose |
|---|---|---|
| NSO-1 schema | §3 + §4 | Canonical Zod test plan schema |
| NSO-2 mixes | §7.1 | Healthcare + Education mix definitions |
| NSO-3 object endpoints | §7.2-§7.9 | 16 object sizes (1/2/4/16/64/256 + Table 5 mixed) |
| NSO-4 cipher enforcer | §4.3.1.4 | 4 TLS 1.2 + 4 TLS 1.3 ciphers, session_tickets off |
| NSO-5 DUT class ACL | §4.2 + Appendix B | XS/S/M/L → 65/120/230/560 ACL rules |
| NSO-6 preflight | §4.3.2 | 16 hard + recommended feature checks |
| NSO-7 orchestrator FSM | §4.3.4 | 11-state phase machine |
| NSO-8..12 runners | §7.1-§7.9 | Per-section KPI aggregation |
| NSO-13 CVE corpus | §4.2.1 | ≥500 CVEs, ≤10 yrs old, CVSS High |
| NSO-14 detection rate | Appendix A.2 | 4-category Table 15 shape |
| NSO-15 under load | Appendix A.3 | 45-96% bg + min 50 CVEs Table 16 |
| NSO-16 KPI aggregator | §5 + Appendix C | Combine all runner outputs into 16-table cert format |
| NSO-17 stability graph | §4.3.4 | Sustain-window SVG + violation strings |
| NSO-18 pie chart | §7.1 | Figures 2-3 SVG |
| NSO-19 report renderer | §5 reporting | cert + lab templates × md/html/pdf |
| NSO-20 audit chain | §5.1 traceability | SHA-256 chain + Cosign |
| NSO-21 — this primer | §5 reporting | Operator how-to |
| NSO-22 sample report | §5 reporting | End-to-end demo on real DUT |