Monitoring test validity¶

Scope status (post-Scope-Freeze 2026-05-10) — See ARCHITECTURE.md for the canonical 37 MÓDULOs + 7 Test Kinds + DOM/CPOS/PIE-PA safety architecture. ADRs 0014, 0019-0025 cover post-Freeze additions.

Goal of this document: in seconds, not hours, an operator should be able to answer the question "is the NGFW measurement I just collected actually meaningful, or was the test bed itself the bottleneck?"

The cluster's purpose is to load a Next-Generation Firewall and measure how much of the request rate / latency degradation is the NGFW's TLS-decryption cost. That's only true if the load-generating infrastructure (UCS hosts, Linux kernel, Caddy webservers, browser-engine + synthetic-load fleets, Nexus 9000 trunk, in-cluster NFS) keeps headroom while the test runs. If the host or fleet saturates first, every NGFW number printed by the dashboard is wrong.

This page describes the metrics, alerts, and Grafana dashboard that exist to make that condition obvious.

The four signals¶

Signal	"Test invalidated" threshold	Where it shows up
TestBedInfrastructureBottleneck	persona p99 > 1 s and any host CPU > 85 %, or any HPA at maxReplicas	Grafana "Test bed status" stat panel turns red. PrometheusAlert fires `severity=critical`.
HostUDPBufferOverflow	UDP RcvbufErrors > 0 for 2 min on any host	Any HTTP/3 timing measurement involving that host is invalid (kernel dropped QUIC packets).
HostConntrackNearFull	Conntrack table > 80 % full	All measurements involving that host are invalid (kernel will drop packets imminently).
HostNetworkPacketDrops	> 100 packets/s dropped on a real NIC	RTT and throughput numbers from that host are wrong.

Any of these alerting invalidates the in-flight test run. Stop the run, fix the underlying issue, re-run.

What you look at first¶

Open the Grafana dashboard Test-Bed Infrastructure Health (UID ai-forse-infra-health). Top row, four big stat panels:

Panel	Green means	Red means
Test bed status	persona latency stable AND host CPU < 85 % AND no HPA pinned	Run is suspect. Investigate the next three panels.
Hosts with tuning missing	All hosts ran `host-tuning.sh apply`	At least one host is at default sysctls (conntrack 262 K, ports 28 K) — that host will saturate first.
HPAs at maxReplicas	None pinned	Fleet is at full capacity; test bed is the limit, not the NGFW.
OOMKills (15 m)	Zero	At least one container hit its memory limit. If that's a persona Caddy, the run is invalid.

If everything is green and the alerts panel is empty, the NGFW measurements are honest — keep running.

If anything is red, the rows below the stat panels show where to look: - Per-host CPU saturation — line chart per node, threshold red at 90 % - Memory + conntrack pressure — same per-node breakdown - Network drops / UDP overflow / TCP retransmits — three side-by-side charts; non-zero on any of them is bad - Pod CPU throttle + restarts — confirms which container is the actual constraint - Sysctl effective values — table, lets you spot the host that didn't get tuned

What gets monitored, and how¶

Three new components ship with this stack:

node_exporter (k8s/dut/90-node-exporter.yaml) — a DaemonSet that runs on every UCS node and exports kernel-level metrics on :9100 over the OOBI control plane. Without this DaemonSet there is no host-level visibility at all.
kube-state-metrics (k8s/dut/91-kube-state-metrics.yaml) — a Deployment that exposes Kubernetes object state (pod restart counts, OOMKill reasons, HPA replicas, etc.) as Prometheus metrics. Required for half of the existing alerts to fire at all.
PrometheusRule web-agent-cluster-infra-health (k8s/dut/92-infra-prometheus-rules.yaml) — 14 alerts split across host, pod, and "test validity" groups, plus 7 recording rules that pre-aggregate the heavy joins so evaluation stays fast at a 30 s interval.

The dashboard config map is at platform/observability/dashboards/infra-health-cm.yaml — Grafana picks it up via the grafana_dashboard: "1" label as long as the standard dashboard sidecar is running.

Confirming tuning is live¶

Two complementary checks:

# Per-host, from the operator workstation (k3s example)
sudo scripts/host-tuning.sh status

# Cluster-wide, from Prometheus
sum(node_nf_conntrack_entries_limit) by (instance) > 1000000

Both should show every host at the tuned value (2 097 152 for conntrack). If a host is at the kernel default (~262 K), the TestBedSysctlMissing alert will fire automatically — no need to remember to look.

Interpreting alerts during a run¶

A typical NGFW capacity test produces the following signals when the NGFW is the bottleneck (the goal):

Persona p99 latency rises smoothly with VU count
Persona CPU rises but stays below 70 %
Host CPU < 50 %, conntrack < 30 % full
No HPA at max
Zero TestBedInfrastructureBottleneck firings

The same signals when the test bed is the bottleneck (failure mode):

Persona p99 latency rises sharply at low VU count
A single host (or fleet) hits its ceiling before persona Caddy CPU does
TestBedInfrastructureBottleneck fires within 3 minutes of starting
HPAMaxedOut fires
Often coincides with HostUDPBufferOverflow or HostConntrackNearFull

The alerts are designed so the second case is impossible to miss.

Coverage extended in PR-V¶

PR-U (#167) shipped the host + pod baseline. PR-V closes the gaps that survived:

New ServiceMonitor / Probe / Rule	What you can now see / alert on
`Probe` snmp-nexus9000 + snmp-ngfw-dut (`k8s/dut/61-snmp-probes.yaml`)	NGFW (SUT) and Nexus 9000 trunk metrics actually scraped — without these the project was blind to the very devices it exists to measure.
`ServiceMonitor` kubelet + cAdvisor (`k8s/dut/93-kubelet-cadvisor.yaml`)	`container_cpu_`, `container_memory_`, `container_network_`, `container_fs_` — real container resource usage. The dashboard's "Top 20 by CPU/RSS" panels need this.
`ServiceMonitor` reloader (`k8s/87-stakater-reloader.yaml`)	`reloader_reload_executed_total` — silent failures of cert rotation.
PrometheusRule `extended-health` (`k8s/dut/94-extended-prometheus-rules.yaml`)	16 new alerts in 7 groups (NIC, disk, system pressure, coverage, NFS, SNMP, reloader).

The 16 new alerts (extended set)¶

Group	Alerts
NIC	NICLinkDown, NICCarrierFlapping, NICRingBufferOverflow
Disk	DiskIOLatencyHigh, DiskNearFull
System	LoadAverageHigh, FileDescriptorExhaustion, NTPClockSkew
Coverage	NodeTuningDaemonSetIncomplete, NodeExporterCoverageIncomplete, CNIDHCPDaemonIncomplete
NFS	NFSClientOperationFailures, NFSClientHighLatency
SNMP	SNMPProbeFailure, SNMPProbeSlow
Reloader	ReloaderRolloutFailure

Total alert + panel count after PR-V¶

	Before PR-U	After PR-U	After PR-V
Host + pod alerts	0	15	31
Composite test-validity alerts	0	3	3
Grafana dashboard rows	5 (persona-only)	6	10
ServiceMonitors	1	3	6
Probes (SNMP)	0	0	2

New dashboard panels (`Test-Bed Infrastructure Health`)¶

NGFW + Nexus SNMP probes up — single stat that turns red when the SUT or trunk is invisible
SNMP probe duration — per-device timeseries; warns at 25 s (timeout = 30 s)
NIC link state — 1/0 per real NIC, drops to 0 instantly on cable / SFP failure
Disk I/O busy % — per device, > 50 % sustained = NFS server bottleneck
File descriptors used % — flags FD exhaustion before it crashes pods
Top 20 containers by CPU / RSS — cAdvisor; spot the heaviest container vs its limit
NFS RPC error rate / latency — slot-side view of cloned-sites delivery

New-vs-reused connections per generator (CPS drift)¶

The bench's headline decrypt-cost axis is CPS — full TLS handshakes per second the NGFW must decrypt. A request that reuses an existing connection costs the DUT almost nothing; a request that dials a fresh connection costs a full handshake. The PrometheusRule web-agent-cluster-cps-drift (k8s/dut/100-cps-drift-rules.yaml) therefore needs an explicit new-vs-reused signal from every generator. Each one measures it differently:

Generator	Signal	How it is measured	Limits
Playwright (web-agent)	`web_agent_connections_total{state="new"\\|"reused"\\|"unknown", protocol="h1"\\|"h2"\\|"h3"\\|"unknown"}`	The agent classifies every response from Playwright's `request.timing()` (Chromium's CDP timing block in HAR style): a fresh dial reports a connect phase (`connectStart`/`connectEnd` ≥ 0); a request served over an already-established connection reports `connectStart == connectEnd == -1` while `requestStart` ≥ 0. Per-cycle counts (≤ 12 keys) ship in the completeRun payload; the dashboard folds them into the counter on `/api/metrics` — the same scrape path (eth0 ServiceMonitor) every other fleet metric takes. Implementation: `agent/src/connection-reuse.ts`.	Attribution is per request, not per socket (1 connection serving 30 requests = 1 `new` + 29 `reused`). Service-worker / cache / `data:` responses classify as `unknown`, never `reused`. QUIC 0-RTT resumed dials count as `new` (matches the DUT's view — still a fresh connection to track). The protocol label depends on `nextHopProtocol` being present in the page's performance buffer; missing entries degrade to `protocol="unknown"` without losing the count.
k6	`http_req_tls_handshaking_count` (+ `http_reqs`)	k6 records an `http_req_tls_handshaking` sample for every request. The k6 fleet pins `noConnectionReuse: true` + `noVUConnectionReuse: true` in `k6-agent/src/k6-runner.ts`, so every request dials a fresh TLS connection — requests ≈ handshakes ≈ new connections. The existing signal therefore suffices; no duplicate new-vs-reused counter was added.	`_count` counts samples (one per request), not handshakes — it only equals new connections because reuse is disabled at the script level. If that posture ever regressed, the count would keep tracking requests while real handshakes vanish; the visible symptom is `http_req_tls_handshaking` p95 collapsing toward 0 ms while `http_reqs` continues.
h3loadgen	`h3loadgen_handshakes_total` vs `h3loadgen_requests_total`	Inference by comparison: in cps mode every request must dial a fresh QUIC connection, so handshakes/s ≈ requests/s. A sustained gap (handshakes < 85 % of requests) means silent connection reuse — the self-contained `H3LoadgenCPSConnectionReuseLeak` alert fires.	Reuse is inferred from two aggregate rates, not attributed per request. In throughput (non-cps) modes the gap is expected and the alert's premise does not apply.

How this feeds the drift rules in k8s/dut/100-cps-drift-rules.yaml:

web_agent_cluster:cps:observed_playwright — rate of web_agent_connections_total{state="new"}: fresh connections/s (full handshakes) the browser fleet put on the wire.
web_agent_cluster:cps:observed_playwright_requests — all classified Playwright requests/s, the denominator for reuse-share panels. There is deliberately no Playwright reuse-leak alert: a browser fleet legitimately reuses connections (h2/h3 multiplexing is the whole point of browser realism).
web_agent_cluster:cps:observed_total_all_generators — h3loadgen + k6 + Playwright new-connections. Kept as a separate series from web_agent_cluster:cps:observed_total so the CPSIntendedObservedDrift alert — whose intended target was defined against the synthetic fleets — keeps its semantics unchanged.

Frozen per-run verdict in the Test Run Report¶

Since report schema v2 (2026-07-05, #1585) the dashboard folds these signals into the frozen Test Run Report at generation time. The report's validity field is computed once — from the same Prometheus, over the exact run window — and then frozen with the rest of the snapshot:

invalid — any of the four run-invalidating signals above fired, or H3LoadgenCPSConnectionReuseLeak fired (cps mode silently reusing connections — the run was not measuring CPS at all).
degraded — CPSIntendedObservedDrift fired (observed CPS more than 15% off the intended target — measured at the wrong operating point).
valid — validity samples were present in the window and none of those alerts fired. The verdict always carries its evidence in reasons.
unknown — Prometheus was unreachable at freeze time, or the run window had no validity samples (rules not deployed / retention expired). Fail-closed: the verdict is never silently "valid".

The report also embeds the persona-plane node hardware profile plus the single-node ceiling disclaimer (HLD §10.1, ADR-0107), so every result is read against the generator's own ceiling, not just the DUT's.