Skip to content

ADR 0034 — SPAN.Art 3-Way Fusion Cross-Correlator

  • Status: Accepted (2026-05-13) — ADR merged #688; backend SPAN-7 shipped #689. Pillar 1 (lifecycle correlator) and Pillar 2 (deny_ineffective verdict) extensions tracked in ADR 0035 §"Pillars 1-2" rather than as follow-ups to this ADR.
  • Date: 2026-05-13
  • Deciders: TLSStress.Art project
  • Targets: v3.8.x (Phase 1 Materialization — SPAN-7/7a/7b)
  • Patent claim family: claim #16 (extension — adaptive multi-source correlation with operator-configurable source set)
  • Supersedes: nothing — extends ADR 0024 (SPAN.Art line-rate capture)

Context

SPAN.Art shipped in v3.7.0 as a 5-layer pipeline (SPAN-1 libpcap → SPAN-2 TLS extractor → SPAN-3 cross-correlator → SPAN-4 K8s polish → SPAN-5 E2E + Annex). The shipping cross-correlator pairs two sources:

  1. Wire metadata — TLS ClientHello observations from span-tls-extractor (SPAN-2)
  2. DUT syslog — normalized records from Promtail (SyslogRecord JSON)

That 2-way correlation emits 4 verdicts (match / miss_log / miss_wire / misclassified), which already gives operators a strong DUT-validation Annex. But the bench has a third first-class telemetry channel that today does not enter the correlation: NetFlow / IPFIX records exported by the DUT.

Customer-facing problem this leaves uncovered:

  • Today's miss_log verdict means the DUT didn't emit a syslog within the window. The operator cannot distinguish:
  • (a) DUT logged but syslog pipeline throttled
  • (b) DUT logged via NetFlow but not syslog (log daemon misconfigured)
  • (c) DUT was deaf on both reporting channels (TLS inspection bypass — much more serious)
  • Today there is no DUT-internal consistency check — e.g. "DUT syslog says deny, NetFlow says 50MB transferred." Catching that disagreement is the smoking gun for a deny rule that did not actually deny.

Both gaps need a third signal and a richer verdict matrix.

Decision

Extend SPAN-3 (span-correlator) into an adaptive multi-source fusion correlator with operator-configurable source set:

  1. Accept a third input — NetFlowRecord JSON — via a new POST /netflow endpoint
  2. Expose three operator toggles controlling which sources are active in the correlation:
  3. ENABLE_WIRE_INPUT
  4. ENABLE_SYSLOG_INPUT
  5. ENABLE_NETFLOW_INPUT
  6. Auto-derive the correlation mode from the active set
  7. Emit an extended 8-verdict vocabulary that adapts to the active mode (only verdicts achievable in that mode can fire)
  8. The HTTP endpoints always listen — input is silently discarded (with tlsstress_span_correlator_input_disabled_total counter incrementing) when the corresponding flag is off, so forwarders (Promtail, FLOW.Art) need no awareness of the correlator's mode

The 4 legitimate correlation modes

A 3-toggle config gives 2³ = 8 states, but only 4 produce meaningful correlation:

# Mode Active sources Verdicts Use case
1 observe exactly 1 source — telemetry collection only; no DUT validation
2 wire_syslog SPAN + Syslog 4 today's SPAN-3 behavior (backward-compat default)
2 wire_netflow SPAN + NetFlow 4 DUT only exports NetFlow (e.g., service-provider deployments)
3 dut_internal Syslog + NetFlow 4 no SPAN tap available — validates DUT against itself
4 fusion_3way SPAN + Syslog + NetFlow 8 highest fidelity; surfaces dut_silent + wire_capture_gap

Mode #3 (dut_internal) is novel — it serves clients who cannot install a SPAN tap (compliance, MSP-managed DUT, etc.) and still want DUT validation through the two DUT-side reports.

The 8-verdict unified vocabulary

All modes share the same enum; each verdict has an applicability matrix:

Verdict Meaning Applicable in
match All enabled sources agree All modes
miss_wire DUT signals agree; wire did not observe Modes with SPAN active
miss_log Wire + other signals present; syslog absent Modes with Syslog active
miss_flow Wire + other signals present; NetFlow absent Modes with NetFlow active
misclassified DUT classified flow wrongly vs wire ground truth Wire-grounded modes (SPAN + ≥1 DUT signal)
internal_disagreement DUT syslog disagrees with own NetFlow dut_internal + fusion_3way
dut_silent Wire observed; DUT silent on both channels fusion_3way only
wire_capture_gap Both DUT signals concur; wire did not see fusion_3way only

The renderer (dut-annex) reads the active mode from the correlation_mode field on ValidationReport and only renders tables for verdicts applicable to that mode.

Per-signal windowing

NetFlow/IPFIX records arrive much later than syslog (Cisco IOS default active-timer = 60s; cache flush + transport adds more — realistic arrival 60-180s after first packet of the flow). Each signal therefore needs an independent TTL:

Signal Default window
Wire (SPAN) — (always primary; never expires before sweep)
Syslog 30 s
NetFlow / IPFIX 300 s (5 min)

CLI flags --syslog-window and --netflow-window let operators tune per deployment.

Probabilistic verdict under NetFlow sampling

NetFlow exporters frequently sample (1:N where N ≫ 1). With sampling, the absence of a NetFlow record for a given flow does not prove the DUT failed to export — only that this flow wasn't in the sample bucket.

NetFlowRecord.SamplingRate is therefore mandatory. The verdict engine adapts:

  • SamplingRate == 1 — deterministic verdicts (today's behavior under SPAN-3 unchanged for non-NetFlow modes)
  • SamplingRate > 1 — verdicts involving NetFlow absence are marked _or_sampled:
  • miss_flow → miss_flow_or_sampled
  • dut_silent → dut_silent_or_netflow_sampled
  • The Annex renderer surfaces sampling rate prominently so the operator weighs verdict confidence accordingly

A future refinement (out of SPAN-7 scope) is statistical aggregation across many flows — "of 10000 flows, 50 should have shown up at sampling 1:200; we saw 20, which is 1σ below expectation; flag the DUT export". That belongs in dut-annex or a downstream analyzer, not in SPAN-3's per-flow verdict logic.

internal_disagreement — the DUT-internal consistency check

Triggered in dut_internal and fusion_3way modes when:

  1. Action–volume mismatch: syslog.action ∈ {deny, drop, block} AND netflow.bytes_total > 0
  2. Volume asymmetry on supposedly bidirectional flow: netflow.packets_in > 0 AND netflow.packets_out == 0 with syslog.classification == "tls" (handshake requires bidirectional exchange — one-sided suggests truncation or DUT inspection dropped one direction)
  3. Classification–payload mismatch: syslog.classification == "dns" AND netflow.bytes_total > 10_000 (DNS flows ≤ a few KB typically; 10KB+ classified as DNS is suspicious)
  4. Negative duration: netflow.flow_end < netflow.flow_start (DUT clock skew / template misalignment — itself a DUT bug)

The decision table is encoded in pkg/span-correlator/internal/correlator/internal_check.go (new file), table-driven so future rules drop in as a slice append.

Observe-only mode (1 source active)

When exactly one source is active, no correlation is possible. The correlator:

  • Does not emit ValidationReport JSON (stream stays clean)
  • Increments Prometheus counters so operator sees activity
  • Passes ingest through to the sink (downstream FLOW.Art / Loki receive the raw input as-is)

This honors the operator's intent — they chose to collect, not validate. The Annex renderer detects observe-only mode (via correlation_mode field absent from input stream) and emits a short paragraph stating no validation was performed, rather than a misleading "all-match" verdict.

UX — the Dashboard Source Selector

Three orthogonal checkboxes; the active correlation mode is derived and shown live:

┌─ DUT Validation correlator ──────────────────────────────┐
│                                                          │
│ Active sources:                                          │
│   ☑ Wire metadata        (SPAN.Art line-rate capture)    │
│   ☑ NGFW syslog          (Promtail receiver)             │
│   ☑ NetFlow / IPFIX      (FLOW.Art forwarder)            │
│                                                          │
│ → Mode: fusion_3way (highest fidelity)                   │
│   Active verdicts: 8                                     │
│   Windows: syslog 30s · netflow 5min                     │
│                                                          │
│   ┌─ Source health ──────────────────────────────┐       │
│   │ SPAN.Art:   1284 events/min  ✅              │       │
│   │ Syslog:      980 records/min ✅              │       │
│   │ NetFlow:      42 records/min ⚠️ underactive  │       │
│   └──────────────────────────────────────────────┘       │
│                                                          │
│  [Why is NetFlow underactive?] [Edit NetFlow template]   │
└──────────────────────────────────────────────────────────┘
  • Mode is read-only ("derived"), not a separate dropdown — operator manipulates fonts, mode follows
  • Source-health row surfaces upstream misconfig (Promtail dead, NetFlow exporter wrong template) without operator hunting
  • Toggling a checkbox writes the corresponding ConfigMap flag and triggers a rolling restart of the SPAN-3 pod (~5s unavailability — acceptable; correlator state is in-memory and re-warms on first packet)

FLOW.Art → SPAN-3 forwarder

flow-art (the NetFlow/IPFIX collector that ships in pkg/flow-art/ — separate scope) gains a new sink mode analogous to the Promtail → SPAN-3 pattern:

# flow-art/config.yaml
sinks:
  - kind: span-correlator-http
    url: http://span-correlator.web-agents.svc.cluster.local:8091/netflow
    timeout: 5s
    normalize: cisco-ftd | pan | fortinet  # vendor template

The forwarder runs inside flow-art — vendor template parsing stays where the binary NetFlow decoder lives. SPAN-3 receives already-normalized JSON only. This mirrors how SPAN-3 today treats syslog (Promtail does vendor normalization, SPAN-3 receives canonical JSON).

Push-based (not pull) — chosen because it matches the existing Promtail-style architecture, is lower-latency, and avoids SPAN-3 having to keep a query loop running against FLOW.Art's TSDB.

Architecture changes

pkg/span-correlator — internal layout

internal/
  types/
    netflow_record.go   (new — NetFlowRecord schema + Validate)
    ...
  window/
    (no change — generic Buffer[T] reused for NetFlow)
  correlator/
    correlator.go       (extended — 3-buffer match logic)
    internal_check.go   (new — internal_disagreement rules table)
  server/
    server.go           (extended — /netflow handler + mode logic)

ValidationReport schema additions

type ValidationReport struct {
  // ... existing fields ...

  // New in v3.8:
  CorrelationMode string         `json:"correlation_mode"`   // "fusion_3way" | "wire_syslog" | ...
  NetFlowRecord   *NetFlowRecord `json:"netflow,omitempty"`  // present when NetFlow is part of the match
  Sampled         bool           `json:"sampled,omitempty"`  // true when verdict involves a sampled NetFlow
}

Backward-compat:

  • All existing fields remain (no breaking changes)
  • New fields are optional; consumers that ignore them keep working
  • Verdict values expand, never change meaning:
  • match, miss_log, miss_wire, misclassified (existing) keep their semantics — they now mean "this verdict was reachable given the active sources"
  • 4 new verdicts (miss_flow, internal_disagreement, dut_silent, wire_capture_gap) only fire under modes that include NetFlow

ConfigMap additions (SPAN-4 follow-up)

# k8s/span-correlator/10-configmap.yaml
data:
  ENABLE_WIRE_INPUT:    "true"
  ENABLE_SYSLOG_INPUT:  "true"
  ENABLE_NETFLOW_INPUT: "false"   # opt-in default — preserves SPAN-3 v3.7 behavior

  SYSLOG_WINDOW:        "30s"
  NETFLOW_WINDOW:       "5m"
  WINDOW:               "30s"     # legacy — still respected for
                                  # SPAN-3 v3.7 deployments

Default ENABLE_NETFLOW_INPUT: "false" means existing v3.7 deployments upgraded in-place keep the wire_syslog mode they have today. No regression.

Consequences

Pros

  • DUT-validation depth grows from 4 verdicts → 8 verdicts; dut_silent is a smoking gun for inspection bypass
  • New dut_internal mode addresses clients without SPAN tap — expands TAM materially
  • Patent claim #16 family sharpens — adaptive multi-source correlation with operator-configurable source set is novel
  • Backward-compatible: defaults preserve v3.7 behavior; opt-in via single ConfigMap flag
  • Source-health UX surfaces upstream config errors that previously required operator hunting

Cons / risks

  • Memory footprint grows ~3× under fusion_3way (three buffers per flow key, 5-min window for NetFlow). At 10k pending flows ≈ 30MB — tractable but worth a load-test
  • NetFlow sampling complicates verdict confidence — partially mitigated by _or_sampled verdict variants but operator must understand the caveat
  • flow-art needs the forwarder sink shipped before SPAN-7 end-to-end works in production — coordination with FLOW.Art roadmap required
  • Dashboard component (SPAN-7a) is the highest-effort piece — introduces new React state + WebSocket subscription for live health data
  • Provisional patent draft (SPAN-7b) needs a freedom-to-operate scan before filing; IDS/IPS triple-source correlation is a populated patent space

Compatibility

  • v3.7 deployments upgrade in-place with ENABLE_NETFLOW_INPUT=false → zero behavioral change
  • v3.8 default install can opt in via Dashboard toggle (no manifest redeploy required after SPAN-7a ships)
  • dut-annex v3.8 can read both v3.7 and v3.8 ValidationReport JSON (new fields are optional)

Rollout plan — SPAN-7 wave

PR Scope Estimated effort
SPAN-7 pkg/span-correlator — NetFlowRecord schema, /netflow endpoint, 3-buffer correlation engine, 8-verdict decision table, mode auto-derivation, internal_disagreement rules, ~40 unit tests covering all 4 modes ~6 h
SPAN-7a Dashboard <SourceSelector> + PATCH /api/span-correlator/sources + source-health tile + WebSocket live counter feed ~4 h
SPAN-7b pkg/dut-annex mode-aware rendering (mode-specific verdict tables, sampling disclaimer); pkg/span-e2e cobertura dos 4 modos × IPv4/IPv6 (8 paths); flow-art HTTP sink to /netflow; provisional patent draft + FTO scan summary ~5 h

Total: ~15 h spread across 2-3 sessions. SPAN-7 ships first (the correlator extension is the patent-claim-bearing artifact); 7a and 7b can ship in either order after.

References

  • ADR 0024 — SPAN.Art line-rate capture (parent ADR)
  • ADR 0021 — PURE Discovery Hub source ranking (Discovery Hub source F = SPAN.Art today; this evolution does not change source ranking)
  • ADR 0035 — DUT Inspection Effectiveness Validation (6-pillar evolution that builds on this ADR; absorbs SPAN-7b follow-ups + opens SPAN-8 and SPAN-9 waves)
  • Patent claim #16 family — SPAN tiered ingest + cross-correlator (this ADR sharpens claim #16 with the adaptive-multi-source element; ADR 0035 proposes two additional claim families — Decryption Posture + Fast-Path Detection)
  • pkg/span-correlator/README.md — current 2-way correlator
  • pkg/dut-annex/README.md — current Annex renderer
  • docs/SPAN_DUT_VALIDATION_ANNEX.md — current Annex sample renderings
  • pkg/flow-art/ — NetFlow/IPFIX collector (separate module — receives the new HTTP sink config in SPAN-7b)