Skip to content

TLS Decrypt Mode Verification

Read in your language: English · Português · Español

Scope status (post-Scope-Freeze 2026-05-10) — See ARCHITECTURE.md for the canonical 37 MÓDULOs + 7 Test Kinds + DOM/CPOS/PIE-PA safety architecture. ADRs 0014, 0019-0025 cover post-Freeze additions.

Status: enabled by default in DUT mode since this PR (no operator action needed beyond updating the regex pattern in the probe ConfigMap if your NGFW vendor's CA Subject doesn't match the default catch-all).

Why this matters

The headline measurement of this test bed is the cost the NGFW pays for TLS inspection — agents (browser-engine + synthetic-load) generate HTTPS load, the NGFW under test decrypts and re-encrypts every connection, and the cluster reports the latency / throughput / error-rate impact.

That measurement is only meaningful if the NGFW is actually doing the inspection. Three failure modes silently invalidate test results:

Failure What the operator believes What's actually happening
Silent bypass "We benchmarked the NGFW with TLS decryption ON" NGFW had decryption disabled → results measure routing speed, not inspection capacity
Partial bypass "All test traffic was decrypted" NGFW had a policy excluding one VLAN / destination → mixed (some decrypted, some not) results
Mid-run flip "Decrypt mode held steady through the test" Config drift / rollback in the middle of a run → results before and after are not comparable

A NGFW vendor demoing the test bed has a strong incentive to silently disable decryption to make their device look faster. A customer running their own evaluation has a strong reason to detect that.

How the verification works

The cluster ships a dedicated, lightweight TLS Decrypt Mode Probe (k8s/dut/15-tls-decrypt-probe.yaml) that runs alongside the agent fleet. Once a minute, the probe:

  1. Opens a TLS handshake to a persona webserver on the same VLAN agents use (default: shop.persona.internal:443 over the browser-engine VLAN 20)
  2. Captures the leaf certificate the peer presented
  3. Extracts the Issuer Distinguished Name (CN, O)
  4. Classifies the issuer:
  5. Matches the NGFW CA pattern (configurable regex, default catches Cisco FTD / Palo Alto / FortiGate / Check Point names) → decrypt ACTIVE
  6. Matches the persona-ca-issuer Subject → decrypt BYPASS
  7. Matches neither → UNCLASSIFIED (probe can't tell — usually means the NGFW pattern needs adjustment for the vendor in use)
  8. Emits Prometheus metrics

The probe shares the same network attachment as the browser-engine agents (same Multus NAD dut-pw, same macvlan VLAN, same routing through the NGFW). What the probe sees IS what the agents see — there's no possible drift between the two.

What this exposes

Real-time mode indicator (Grafana)

A dedicated Grafana dashboard TLS Decrypt Mode — NGFW Inspection Verification (dashboard JSON) shows the live verdict as a colored stat panel:

┌──────────────────────────────────────────────────────────┐
│  Current decrypt mode      │  Peer cert issuer (CN)      │
│  ┌───────────────────┐    │  Cisco FTD CA               │
│  │  DECRYPT ACTIVE   │    │  cn=Cisco FTD CA            │
│  │  (green)          │    │  o=Cisco Systems            │
│  └───────────────────┘    │                             │
└──────────────────────────────────────────────────────────┘

Color encoding: - 🟢 DECRYPT ACTIVE — the NGFW is intercepting TLS as expected - 🟡 DECRYPT BYPASS — the NGFW is just routing, not inspecting (test results measure routing speed) - 🔴 UNCLASSIFIED — probe couldn't classify the issuer (usually means the NGFW CA pattern in tls-decrypt-probe-config needs an update to match your vendor's CA Subject) - 🔴 PROBE OFFLINE — probe pod is down or hasn't reported in > 5 min

Mode-change detection (alerts)

k8s/dut/16-tls-decrypt-rules.yaml defines three Prometheus alerts:

Alert Severity Trigger
TLSDecryptModeChanged warning Probe observed a flip (ON ↔ OFF) within 10 min — fires within 1 min of the change
TLSDecryptModeUnknown warning Probe completed handshake but issuer matched neither CA pattern for 3 min
TLSDecryptProbeStale warning Probe hasn't reported fresh metrics in > 5 min

TLSDecryptModeChanged is the killer feature: any test run that spans a flip in TLS decrypt mode is automatically invalidated. The annotation tells the operator: "Stop the current run, confirm desired NGFW state, restart cleanly."

Why this is a positive differentiator

Commercial appliances at this price point — Spirent CyberFlood, Ixia BreakingPoint — don't ship this kind of verification. They generate load, they measure throughput, but they trust the operator to declare the NGFW state correctly. There's no ground-truth check.

This test bed: - Verifies the NGFW state independently of operator declaration - Detects silent bypass automatically - Surfaces mode changes as alerts, not buried in a log - Costs ~10m of CPU and ~16Mi of RAM (the probe is busybox + alpine + a sleep loop)

That makes test results from this cluster defensible in customer evaluations: the operator can point at the dashboard history and demonstrate that decrypt mode held steady throughout the measurement window. A vendor cannot quietly flip the NGFW into bypass without it being visible in a Grafana panel and an alert.

Configuration

Edit k8s/dut/15-tls-decrypt-probe.yaml ConfigMap tls-decrypt-probe-config to adjust:

Field Default What to change to
ngfw_ca_subject_pattern catch-all regex matching common vendor names A specific pattern that matches your NGFW's CA Subject CN/O. Examples: Cisco FTD CA, PAN Forward Trust, FGT_CA, Check Point Forward Trust. Use a regex if you support multiple NGFW vendors
probe_target shop.persona.internal:443 Any persona DNS:port. Defaults to the first synthetic persona, always available
poll_interval_seconds 60 Smaller values (15–30s) for faster mode-change detection at the cost of slightly more probe load

Apply the ConfigMap change with kubectl apply -k k8s/dut/. The Stakater Reloader integration restarts the probe pod automatically.

How to find your NGFW's CA Subject

The ngfw_ca_subject_pattern regex must match a substring of your NGFW's intercept-CA Subject DN. To find that substring:

# 1. Probe a persona BEFORE configuring the regex (probe will return UNCLASSIFIED)
kubectl exec -n web-agents deploy/tls-decrypt-probe -c probe -- \
  /bin/sh -c "openssl s_client -connect shop.persona.internal:443 -servername shop.persona.internal -CAfile /etc/ssl/combined/ca.crt 2>/dev/null | openssl x509 -noout -issuer"

# Output (example):
# issuer=C = US, ST = California, L = San Jose, O = Cisco Systems, OU = TLS Inspection, CN = Cisco FTD Forward Trust CA

# 2. Extract a unique substring — for the example above, "Cisco FTD" is plenty.
# 3. Update the ConfigMap:
kubectl edit configmap tls-decrypt-probe-config -n web-agents
#   ngfw_ca_subject_pattern: "Cisco FTD"

# 4. Probe will pick up the new pattern within ~30s and reclassify the cert as ACTIVE.