Skip to content

Troubleshooting — operator quick reference

Audience: operator running a TLSStress.Art bench who hit a symptom and wants the fastest path to a cause + action. Symptoms are grouped by area. Deeper procedures live in the runbooks and MÓDULO primers linked at the bottom.

Remember: the Dashboard is the only operator interface. Every fix below is expressed as something you do from the UI or read-only kubectl/logs. Fixing a MÓDULO live is prohibited.

Bench bring-up

Symptom Likely cause Action
Personas stuck ImagePullBackOff Air-gapped node missing the image, or tag mismatch On a connected bench, re-pull; on air-gap, re-stage the bundle (AIRGAP_INSTALL.md). Confirm the manifest tag matches what's loaded.
Persona pod runs but agents get certificate signed by unknown authority on leg 2 persona-ca-issuer not applied, or cert missing IP SANs kubectl apply -k platform/ (PKI phase). Check the persona cert has the public IP as an IP SAN.
kubectl kustomize personas/ fails Generated manifests stale after a personas.yaml edit Re-generate personas/_generated/; never hand-edit generated files.
BGP routes for a persona /24 never appear on the DUT ISP/BGP MÓDULO (VyOS+FRR) not up, or route sync not run Check the BGP MÓDULO pod; run tools/dut-persona-routes.py sync. Pre-flight: tools/validate-ip-pool.sh.

TLS / two-leg inspection

Symptom Likely cause Action
Agents fail leg 1 with cert error to the DUT Agents don't trust the NGFW CA Confirm the ngfw-ca ConfigMap is applied and mounted (NODE_EXTRA_CA_CERTS / SSL_CERT_FILE). This is intentional — agents trust only the NGFW CA on leg 1.
HTTP/3 (QUIC) traffic never reaches personas but HTTP/2 does UDP/443 blocked, or host UDP buffers too small Verify DUT allows QUIC inspection; confirm net.core.rmem_max/wmem_max = 64MB from the node-tuning DaemonSet (85-node-tuning.yaml).
CPS far below expected on the DUT Session-ticket resumption masking new-connection cost, or agent scale too low Personas disable session tickets by design; raise the HPA target / agent count from the Dashboard, watch tcp_sockets_open.
Intermittent TLS handshake failures under load Host UDP/TCP tuning not applied on this node Confirm the 85-node-tuning DaemonSet is Running on the node; it sets BBR+FQ and UDP buffers.

Agents (Playwright / k6)

Symptom Likely cause Action
An agent is stuck, CPU pinned at 100% with no throughput Wedged worker / busy-spin The Dashboard's per-target circuit breaker should trip; if not, scale that agent deployment to 0 and back up from the UI. Check tcp_sockets_open for a connection leak.
Duplicate runs after a retry Missing/duplicated idempotency key Agents use Idempotency-Key on control calls; a duplicate means a retry raced — confirm in audit_log, do not re-submit manually.
Agents won't scale past a ceiling HPA max reached (blast-radius cap) This is a safety cap (Playwright max=80, k6 max=200). Raising it is a config change through the Dashboard, not kubectl edit.

Dashboard / control plane

Symptom Likely cause Action
A config change didn't take effect CPOS 2-phase commit rolled back (all-or-nothing) Check the CPOS status in the UI — a partial change is reverted by design. Re-submit the full change. See CPOS primer.
Dashboard login rejected on the operator LAN HMAC cookie mismatch / clock skew Re-authenticate; check host clock. ZTP-prem assumes a hostile LAN, so auth is strict by design.
A production write silently didn't run PIE-PA gate awaiting operator approval On production DOM mode, writes need an explicit click-through. Approve it in the UI. See PIE family primer.
"Upgrade" button applied but a service is now broken Bad release Follow upgrade-and-rollback.md — do the manual image rollback.

Observability & metrics

Symptom Likely cause Action
Prometheus target down for a MÓDULO eth0 metrics path broke, or pod not Running Metrics ride eth0 (not the isolated net1). Check the pod is Running and :9091//api/metrics is reachable on eth0.
SNMP metrics from the DUT missing SNMP exporter not deployed or wrong community/creds Check the 60-snmp-exporter deployment; verify DUT SNMP config.
Grafana panels empty but Prometheus has data Datasource / time-range issue Confirm the Prometheus datasource; widen the time range.

SaaS / billing edge

Symptom Likely cause Action
Bench refuses to run: "quota exhausted" TSU quota spent The bench spends TSU as it runs; top up on app.tlsstress.art. Usage reconciles via the hourly report.
Hourly usage report rejected by the cloud HMAC secret rotated on one side only Re-sync the report secret; check the bootstrap-controller logs.

When to escalate

If none of the above resolves it, capture the state and file a bug — don't mutate a MÓDULO live to work around it:

kubectl get pods -A | grep -vE 'Running|Completed'
# attach the relevant pod logs (read-only) to the bug report

Open a bug report.