Skip to content

ADR 0038 — Universal DSCP Marking Policy (CS7 Control Plane + CS1 Test Plane)

  • Status: Proposed (2026-05-13) — materialization in flight (PRs #710-#717)
  • Date: 2026-05-13
  • Deciders: TLSStress.Art project
  • Targets: v4.x (rollout incremental, non-breaking)
  • Builds on: ADR 0019 (OOBI fabric), ADR 0037 (pkg/oobi/client/), ADR 0036 (HyperBridge.Art — explicit non-rewrite of passthrough)

Context

The TLSStress.Art bench mixes three radically different traffic classes on overlapping physical links:

  1. Control plane — MÓDULOs talking to each other over OOBI fabric. Heartbeats, audit logs, PacketBatch JSON, slot lookups, RELAY commands. Losing this = losing orchestration = bench-wide failure.

  2. Test plane — Agents (Playwright, K6, Iperf3) generating synthetic load against the DUT, which forwards to Personas. Heavy bandwidth; can saturate the bench. Losing this is expected behavior under stress — it's the load we're testing the DUT under.

  3. Customer / Passthrough plane — Real customer traffic crossing a HyperBridge.Art bridge inline. Indistinguishable from the customer's production. We never touch this — the DUT inspects it and we forward verbatim.

Pre-v3.10, none of these planes were DSCP-marked. Consequences:

  • Underlying network treated everything as Best-Effort (DSCP 0). Customer's Nexus / Catalyst / cloud underlay had no signal to prioritize OOBI control plane during congestion.
  • Test traffic could starve OOBI heartbeats when the agents hit line rate — operator lost visibility precisely when they needed it most.
  • Multi-tenant cloud deployments had no way to politely cede bandwidth to the customer's production traffic — our test load competed equally with their real apps.

Decision

Adopt a two-class deterministic DSCP marking policy anchored at the extremes of the Class Selector hierarchy (RFC 4594):

Plane DSCP ToS byte Origin
Control / Management CS7 0xe0 MÓDULOs (38, including OOBI client) + Fabric VTEP encapsulation
Test / Data CS1 0x20 Agents (PW + K6 + Iperf3) + Personas (61 synthetic + 39 cloned)
Passthrough (inalterado) (do cliente) HyperBridge.Art bridge L2 forwarding

Why CS7 + CS1 (not e.g. EF + AF11)

  • CS7 = "Network Control" (RFC 4594). Universally interpreted by routers/switches as "never drop these unless physically impossible." Aligns with the MÓDULO/OOBI semantic.
  • CS1 = "Low-Priority Data" ("scavenger"). Universally interpreted as "drop these first under any congestion." Aligns with synthetic test load semantic.
  • Both are Class Selectors — backward-compatible with the IPv4 IP Precedence (0–7) era. Any equipment that respects even basic QoS (3-bit IP Precedence) honors the relative ordering of CS7 > CS1.
  • Maximum gap — by picking the extremes, we guarantee unambiguous prioritization across every QoS-aware device.
  • AF + EF require Diffserv-PHB-aware infrastructure; CS classes work everywhere.

What this guarantees

  1. In multi-tenant networks (cloud, customer underlay), our test traffic stays below customer's real traffic — we cannot accidentally DoS their production network.
  2. In our own bench, OOBI never loses to test traffic — the operator always retains control even under maximum DUT stress.
  3. When DUT or upstream congests, test load is the first to be dropped — realistic and safe behavior.
  4. Even when OOBI runs in degraded mode (VXLAN piggybacking on Data Plane when dedicated underlay fails), CS7 marking on the outer header keeps control plane prioritized.

What we explicitly do NOT mark

  • HyperBridge.Art passthrough — the customer's traffic crossing inline through our bridge keeps its original DSCP. Rewriting it would (a) violate transparency, (b) break inspection-of-QoS by the DUT, (c) potentially impact the customer's downstream QoS policies.
  • Heartbeat Unix sockets (orchestrator ↔ watchdog) — Unix sockets have no IP layer; DSCP is not applicable.
  • Localhost loopback — kernel doesn't propagate DSCP on loopback by default; not a concern.

Implementation — 4 enforcement layers (defense in depth)

Layer 1 — Canonical constants (pkg/oobi/canon.go)

Single source of truth referenced by code + manifests + CI gates:

const (
    // Control plane (CS7) — MÓDULO + OOBI Fabric traffic
    DSCPControlPlaneCS7 uint8 = 0x38  // 6-bit DSCP value (= 56)
    ToSControlPlaneCS7  uint8 = 0xe0  // 8-bit ToS byte (CS7<<2 | ECN=0)

    // Test plane (CS1) — Agents + Personas (100 total)
    DSCPTestPlaneCS1 uint8 = 0x08     // 6-bit DSCP value (= 8)
    ToSTestPlaneCS1  uint8 = 0x20     // 8-bit ToS byte (CS1<<2 | ECN=0)
)

Layer 2 — Application-level setsockopt

pkg/oobi/client/ — extended Control hook applies IP_TOS=0xe0 (IPv4) and IPV6_TCLASS=0xe0 (IPv6) on every socket. Every MÓDULO adopting the client gets CS7 for free.

K6 agent — custom dialer wraps net.Dialer with a Control hook that applies IP_TOS=0x20 + IPV6_TCLASS=0x20.

Iperf3 agent wrapper — always passes --tos 32 (CS1 ToS in decimal) on the iperf3 CLI.

Layer 3 — Pod-level iptables MARK (covers Chromium, third-party)

Init container in each agent + persona pod (privileged):

iptables  -t mangle -A POSTROUTING -j DSCP --set-dscp 0x08
ip6tables -t mangle -A POSTROUTING -j DSCP --set-dscp 0x08

Idempotent; namespace-scoped; survives restart. Covers everything the application layer doesn't (Chromium, Caddy passthrough, etc.).

Layer 4 — Network-level (VXLAN outer + 802.1p PCP)

VTEP DaemonSet (k8s/oobi/30-vtep-daemonset.yaml) configures the vxlan device with explicit outer ToS:

ip link add oobi0 type vxlan id 254254 dstport 4789 \
  local <node-IP> nolearning \
  tos 0xe0   # CS7 outer header

DUT overlay NADs (k8s/dut/20-network-attachments.yaml) configure egress-qos-map so DSCP CS1 propagates to 802.1p PCP on VLAN-tagged egress:

egress-qos-map "0x08:1"   # DSCP CS1 → PCP 1

Verification

  1. Unit tests — pkg/oobi/client/ test verifies socket ToS after dial via getsockopt(IP_TOS) matches 0xe0. K6 dialer test verifies 0x20.
  2. Integration test — pcap capture + tshark filter ip.dsfield == 0xe0 on OOBI traffic; ip.dsfield == 0x20 on DUT-side test traffic. CI gate (#717) runs this on a sample topology.
  3. Prometheus metric — oobi_dscp_mismatch_total{plane,direction} on both client + agent pods. Should be zero; alert if non-zero.
  4. Manifest CI gate (#717) — script parses K8s manifests, verifies every pod with role=agent or role=persona has the DSCP init container; every VXLAN interface config has tos 0xe0.

Consequences

Pros

  • Determinism — Two values, two planes. Documented in canon. Audit-friendly.
  • Universal compatibility — CS7 + CS1 honored by every QoS-aware device, from cheap unmanaged switches with IP Precedence support up to enterprise Diffserv PHB.
  • Multi-tenant safe — Our test traffic can't accidentally starve customer's production traffic in shared networks.
  • Operator UX — One Prometheus alert tells the operator if the policy is violated anywhere in the fleet.

Cons / risks

  • iptables init containers require privileged — Each agent + persona pod needs privileged: true for the init step. Acceptable in our threat model (these pods already run in dedicated namespaces with NetworkPolicy isolation), but raises the bar for SecurityContext audits. Mitigation: the privilege scope is CAP_NET_ADMIN only, and only the init container needs it — the main container drops caps.
  • DSCP rewriting by intermediate equipment — Some carrier-grade routers strip/remap DSCP at trust boundaries. If our bench traverses such a boundary, marking can be lost. Mitigation: document in the LDS wizard preflight check; warn if the underlay doesn't preserve DSCP.
  • ECN bits forced to 0 — We use ToS = DSCP << 2 | 0 (ECN=0). TCP connections that want ECN-Capable Transport won't get it on OOBI traffic. Tradeoff: OOBI is control plane; ECN matters for bulk data. The agents (Layer 2 K6) can opt out of the marking if testing ECN on a specific scenario — they explicitly set their own ToS in that case.

Compatibility

  • Pre-adoption MÓDULOs continue sending DSCP 0 (Best-Effort). Once a MÓDULO migrates to pkg/oobi/client/, it inherits CS7 automatically. No coordinated deploy needed.
  • Bench operators see no behavior change in normal operation; the marking is invisible until the network starts congestion- dropping, at which point the right traffic gets dropped first.
  • DUT sees DSCP CS1 on test traffic — DUTs that perform QoS-aware steering may behave differently than with unmarked traffic. This is intentional and a feature: it's how customer production traffic looks (their apps mark DSCP), so our tests are more realistic.

References

  • Implementation: PRs #710-#717
  • Constants: pkg/oobi/canon.go
  • Client integration: pkg/oobi/client/ (ADR 0037)
  • VTEP: k8s/oobi/30-vtep-daemonset.yaml
  • Agent overlays: agent/, k6-agent/, iperf3-agent/
  • Persona generator: personas/ (61 synthetic) + k8s/clone-personas/ (39 cloned)
  • RFC 4594 — Configuration Guidelines for DiffServ Service Classes
  • RFC 2474 — Definition of the Differentiated Services Field (DS Field)