Skip to content

API Feature Catalog — what TLSStress.Art unlocks via REST APIs

Read in your language: English · Português · Español

Scope status (post-Scope-Freeze 2026-05-10) — API features now live inside MÓDULO API INFRA.Art (MGMT-light plane). Catalog scope expanded to 7+ vendor profiles per SSH-4 vendor knowledge base. See modules/api-infra-art.md for canonical MÓDULO reference.

This document catalogs every concrete feature unlocked by talking to the lab elements' REST APIs (Cisco FTD, Cisco Nexus 9000 NX-API DME, Cisco UCS CIMC Redfish, future Palo Alto + Fortinet). It is the roadmap for the API integration: what is shipped, what is wired, what is planned.

For implementation details see DUT_API_INTEGRATION.md.

Status legend

Symbol Meaning
✅ Shipped — adapter method exists and the dashboard / report consumes it
🟡 Adapter method exists but UI / dashboard wiring pending
🔵 Designed — adapter method scheduled
⚪ Idea captured, not yet planned

Three pillars of telemetry

Pillar Source What it gives
Metrics SNMP, node-exporter, kubelet Time-series numerical state
Events Syslog, K8s events Reactive event stream
API (this catalog) Vendor REST APIs Live config, applied policies, hardware health, topology, write actions

The three combined produce the level of forensic completeness the Test Run Report needs.

A. Pre-flight checks (refuse to run if state is wrong)

Before allowing a Test Plan to start, the engine queries the relevant API endpoints to confirm the lab is in a runnable state. If any check fails, the run is refused with a specific, actionable error.

# Feature Source Status
A-1 Confirm NGFW deploy status is DEPLOYED (not pending) FTD /operational/deploy 🟡
A-2 Confirm decrypt policy is enabled and applies to test interfaces FTD /policy/sslpolicies 🟡
A-3 Confirm NGFW NTP source matches lab expectation FTD /devicesettings/default/ntp 🟡
A-4 Confirm Nexus has LLDP and CDP enabled Nexus sys/lldp/inst + sys/cdp/inst 🟡
A-5 Confirm UCS CIMC reports no critical hardware faults UCS LogServices/Faults/Entries 🟡
A-6 Confirm UCS thermal sensors below threshold UCS Chassis/<id>/Thermal 🟡
A-7 Confirm HA pair active+standby roles correct on FTD FTD /devices/default/ha 🟡
A-8 Confirm FTD smart license has sufficient capacity FTD /license (planned v6.x) 🔵
A-9 Confirm Nexus QoS profile matches expected baseline Nexus sys/qos 🔵
A-10 Confirm UCS firmware revision against approved list UCS Systems/<id> Bios + Firmware 🔵

The pre-flight runs in parallel against every registered device. Total time budget: ≤30 s. Failure summary is shown to the operator with a single button "Override and run anyway (records non-forensic)" — explicit operator opt-in for unusual cases.

B. Real-time metrics during runs

Once a run is live, the polling worker increases its cadence on the active devices and streams metric snapshots into Prometheus via a custom exporter — surfaced as Grafana panels alongside the agent-side metrics already collected.

# Feature Source Status
B-1 NGFW CPU utilization during the test FTD /operational/systemmonitoring/cpu 🔵
B-2 NGFW memory utilization FTD /operational/systemmonitoring/memory 🔵
B-3 NGFW active connections + new connections/sec FTD /operational/systemmonitoring/connections 🔵
B-4 NGFW current decrypt-rule hit counts FTD /operational/policy/sslpolicy/<id>/hitcount 🔵
B-5 Nexus interface counters (bytes, errors, drops) at 1 Hz Nexus sys/intf/<id>/counters 🔵
B-6 UCS fan RPM during run UCS Chassis/<id>/Thermal 🔵
B-7 UCS CPU temperature during run UCS Chassis/<id>/Thermal 🔵
B-8 UCS power draw (watts) UCS Chassis/<id>/Power 🔵
B-9 NGFW IPS hit counts FTD /operational/policy/intrusionpolicy/<id>/hitcount 🔵
B-10 Nexus QoS queue depth + drops Nexus sys/qos/<class>/queue ⚪

These metrics give the operator a real-time view of "what the device sees" — without ever needing to log into the NGFW management console or the Nexus CLI.

C. Topology and inventory

# Feature Source Status
C-1 Auto-build lab topology diagram from LLDP/CDP Nexus sys/lldp/inst + sys/cdp/inst ✅ adapter / 🟡 dashboard
C-2 Hardware inventory annex (S/N + firmware + PSU + DIMM) Nexus sys/ch?rsp-subtree=full + UCS Systems/<id> ✅ adapter / 🟡 dashboard / 🔵 report
C-3 Service profile binding check (UCSM-managed UCS only) UCSM serviceProfile/<id> ⚪
C-4 NGFW interface state + cable status FTD /operational/interfaces ✅ adapter / 🟡 dashboard
C-5 UCS NIC link state + speed negotiation UCS Systems/<id>/EthernetInterfaces ✅ adapter / 🟡 dashboard

The Test Run Report Annex A — auto-generated topology — is produced from C-1.

D. Forensic and compliance

# Feature Source Status
D-1 Pre-run config snapshot of every device + SHA-256 All adapters ✅ adapter / 🔵 wired into Test Plan engine
D-2 Post-run config diff against pre-run snapshot Internal dut_api_snapshots query 🔵
D-3 Compliance attestation against policy baseline FTD /policy/* + Nexus running-config 🔵
D-4 Per-snapshot SHA-256 cited in Report Annexes B + C + D Internal ✅ field exists / 🔵 report wiring
D-5 Tamper-evident chain — Snapshot SHA-256 anchored in PDF Future Cosign signing 🔵

E. Cross-source correlation

The unique value: combine the three pillars to detect things no single pillar can.

# Feature Source combination Status
E-1 NGFW syslog "decrypt error" + simultaneous API check of decrypt policy state. Diff = config drift mid-run Syslog (decrypt error) + API (decrypt policy state) 🔵
E-2 UCS CIMC fault + correlated agent failures. Same time window = root cause attribution API (UCS faults) + Postgres (runs table failures) 🔵
E-3 Nexus interface error counter spike + correlated p99 latency spike API (Nexus counters) + Prometheus (agent p99) 🔵
E-4 TLS Decrypt Probe says "off" + API confirms decrypt policy enabled. Discrepancy → NGFW broken Probe metric + API (decrypt policy) 🔵
E-5 NGFW deploy state changes mid-run = run invalidated API polling delta 🔵

F. Write operations (operator-confirmed)

Read-only is shipped; writes require operator confirmation, audit trail, and automatic rollback on failure.

# Feature Source Status
F-1 Set NGFW NTP server via API FTD PUT /devicesettings/default/ntp/<id> + POST /operational/deploy 🔵
F-2 Toggle decrypt policy state for A/B testing FTD PUT /policy/sslpolicies/<id> 🔵
F-3 Apply test-bed-specific QoS policy on Nexus pre-run, revert post-run Nexus DME PUT 🔵
F-4 Reset interface counters before run Nexus DME action ⚪
F-5 Stage rollback configuration before any write Internal: snapshot before, restore after 🔵
F-6 UCS BIOS power profile change for "max-perf" runs UCS Redfish Systems/<id>/Bios PATCH ⚪

Each write requires: 1. Admin re-auth at request time 2. Explicit operator confirmation in the UI ("about to change NGFW NTP source — proceed?") 3. Audit log entry with before/after snapshots 4. Automatic rollback if the post-write read confirms divergence from intent

G. Power, sustainability, performance-per-watt

# Feature Source Status
G-1 UCS power consumption time-series during runs UCS Chassis/<id>/Power 🔵
G-2 "Throughput per watt" metric per plan run UCS Power + Prometheus throughput ⚪
G-3 Thermal throttle detection (warn if CPU temp exceeds threshold) UCS Thermal + alerting 🔵
G-4 Carbon-equivalent estimate per run (operator-supplied grid factor) Computed from G-1 ⚪

H. Hardware health for SOAK runs

24-hour-plus plans demand hardware-level vigilance. The API integration provides what node-exporter cannot.

# Feature Source Status
H-1 UCS DIMM error counts (correctable + uncorrectable) UCS Memory?$expand=. ✅ adapter / 🔵 alerting
H-2 PSU redundancy status during 24 h+ runs UCS Chassis/<id>/Power ✅ adapter / 🔵 alerting
H-3 NIC error rate baseline + during-run comparison UCS + Nexus interface counters 🔵
H-4 Disk SMART data on UCS local storage UCS SimpleStorage/<id> ⚪
H-5 SOAK-end report: peak fan RPM, peak temp, total power, peak watts Aggregation over collected snapshots 🔵

I. Auto-rollback

# Feature Source Status
I-1 Pre-run capture full config of all 3 device types Adapters + new pre_run_config table 🔵
I-2 Post-run automatic restore (operator opts in via plan param) Adapters write methods 🔵 (depends on F-*)
I-3 Mid-run drift detection → auto-pause + alert Polling diff against pre-run 🔵
I-4 Rollback verification: post-restore snapshot matches pre-run SHA-256 compare 🔵

J. Test plan integration

Test Plan YAML can express "the lab MUST be in this state for this plan to be valid":

# Feature YAML field Status
J-1 Plan declares ngfw_state_required: decrypt-on and adapter verifies already exists ✅
J-2 Plan declares ntp_source_must_match: lab-relay new field 🔵
J-3 Plan declares expected_ngfw_models: [FPR1010, FPR1140] and refuses on mismatch new field 🔵
J-4 Plan declares pre_run_capture_config: true for compliance runs new field 🔵
J-5 Plan completion triggers final config snapshot automatically engine hook 🔵
J-6 Plan failure triggers automatic post-mortem snapshot engine hook 🔵

K. Multi-vendor abstraction

Same endpoint_label queries return data regardless of vendor — Dashboard and Report code stays vendor-agnostic.

# Feature Source Status
K-1 Same query "show NGFW NTP config" works across FTD / Palo Alto / Fortinet Adapter abstraction ✅ FTD / 🔵 Palo Alto / 🔵 Fortinet
K-2 Operator dashboard shows "all NGFWs" agnostically Postgres view + Grafana 🟡
K-3 Future Palo Alto + Fortinet adapters slot in transparently Architecture supports ✅
K-4 Adapter feature matrix surfaced in admin UI ("FTD has decrypt_policy, Palo Alto does too, Fortinet does not") Adapter introspection 🔵

L. Reporting

The Test Run Report (Phase 3 → 5) consumes API snapshots heavily.

# Feature Source Status
L-1 Annex A — auto-generated topology from LLDP/CDP Nexus adapter ✅ adapter / 🔵 SVG render
L-2 Annex B — Nexus inventory + sanitized running-config Nexus adapter ✅ adapter / 🔵 report wiring
L-3 Annex C — NGFW inventory + decrypt policy + IPS state FTD adapter ✅ adapter / 🔵 report wiring
L-4 Annex D — UCS hardware inventory + sensor readings UCS adapter ✅ adapter / 🔵 report wiring
L-5 Per-annex SHA-256 chain-of-custody (already in shape) All adapters ✅
L-6 "What changed during the run" appendix — pre vs post diff Internal computation 🔵

M. UI-side surfacing — the operator never opens the NGFW console

The operator's "single pane of glass" goal — everything visible in Grafana / Dashboard / Report.

# Feature Surface Status
M-1 Grafana — DUT API Status dashboard dashboards/dut-api-status-cm.yaml ✅
M-2 Grafana — DUT Live State (NGFW + Switch + UCS) dashboard dashboards/dut-live-state-cm.yaml ✅
M-3 Dashboard — admin page to register / test devices New /admin/dut-api/ 🔵
M-4 Dashboard — per-device snapshot browser ("show me what FTD-1 said about its NTP config 30 min ago") New page 🔵
M-5 Dashboard — manual snapshot trigger button per device New API endpoint 🔵
M-6 Dashboard — write operation confirmation modal New component 🔵 (depends on F-*)
M-7 Test Run Report — full DUT context auto-embedded Phase 3 report builder 🔵

N. Long-tail and future ideas

# Feature Status
N-1 Cisco Intersight integration (cloud-managed UCS) ⚪
N-2 UCS Manager (UCSM) XML API for B-series chassis ⚪
N-3 Cisco DNA Center integration for switch fabric ⚪
N-4 NetConf/YANG support as an alternative to NX-API DME ⚪
N-5 gNMI streaming telemetry from Nexus (sub-second metrics) ⚪
N-6 OpenConfig YANG models — vendor-neutral interface ⚪
N-7 Ansible playbook export — "TLSStress.Art configured my NGFW this way; here's the playbook to reproduce" ⚪
N-8 Terraform provider — TLSStress.Art as a resource type ⚪
N-9 Webhook to ServiceNow / Jira / PagerDuty when API write fails ⚪
N-10 API call replay — record + replay a sequence of API calls from one engagement to another ⚪

Based on operator value vs implementation cost:

  1. PR-B (highest ROI) — Polling worker + admin UI for device registration. Without this, the adapters do not run on a schedule. ~1 week of work.
  2. PR-C — Pre-flight checks (A-1 through A-7) integrated into Test Plan engine. Refuses runs in invalid states. ~3 days of work.
  3. PR-D — Test Run Report Annex B / C / D wiring (L-2, L-3, L-4). Closes the "API-derived data into PDF" loop. ~1 week of work.

The remaining categories (B real-time, F writes, I auto-rollback) follow once the foundation is solid.

Capability comparison vs commercial alternatives

Feature group Spirent CyberFlood Ixia BreakingPoint TLSStress.Art (this catalog)
Pre-flight checks via API partial (vendor-locked DUTs) partial full (multi-vendor open architecture)
Real-time NGFW config visibility none none full
Auto topology discovery manual manual LLDP/CDP automatic
Hardware health during SOAK none none UCS Redfish full
Forensic chain-of-custody on config none none SHA-256 per snapshot
Multi-vendor abstraction vendor-locked vendor-locked open adapter pattern
Report integration proprietary closed format proprietary closed format versioned JSON + signed PDF

References