Skip to content

ADR 0043 — OOBI VLAN (No VXLAN) Refactor

  • Status: Accepted (2026-05-14)
  • Date: 2026-05-14
  • Deciders: TLSStress.Art project (operator directive locked in memo project_oobi_no_vxlan_directive_2026_05_14)
  • Supersedes (in part): ADR 0019 (OOBI slot allocation — carrier mechanism changes; slot map unchanged), ADR 0036 (HB references to overlay), ADR 0037 (oobi/client source-bind device), ADR 0038 (DSCP marking — preserved verbatim), ADR 0039–0042 (references to the VXLAN overlay updated; behaviour unchanged)
  • Targets: v4.0.0 (major breaking change)

Context

Before 2026-05-14 the Out-of-Band Infrastructure (OOBI) management plane was carried over a VXLAN overlay with:

  • VNI 254254 (mnemonic = inner /24 octets .254.254)
  • UDP port 4789 (IANA standard)
  • Inner addressing 100.127.252.0/24 + IPv6 ULA fd5a:7c5e:a72::/64
  • Underlay MTU floor 1550 (1500 inner + 50 VXLAN overhead)
  • Carrier VLAN 99 by default (operator-configurable)
  • VTEP DaemonSet (k8s/oobi/30-vtep-daemonset.yaml) creating vxlan0/oobi0 on every node, maintaining a head-end replication (HER) peer FDB read from a ConfigMap

Reasons that posture was chosen originally:

  1. Customer collision avoidance: the customer's L2 might already use any RFC 1918 carve-out; VXLAN gave us encapsulation independence.
  2. Slot stability across topologies: the overlay made the inner addressing identical in single-node lab and multi-node production, regardless of physical L2 plan.
  3. Multicast-free BUM: HER mode let us avoid the underlay needing multicast.

In the 9 months since the overlay shipped (v3.5.0), the actual deployment experience surfaced trade-offs that the operator considered unacceptable:

  • Encap/decap overhead in the kernel hot path; even at MTU 1500 inner the underlay needed jumbo to keep wire frames at line rate.
  • Operator confusion between OOBI (the management network) and OOBI fabric (the VXLAN overlay) — the two terms drifted apart in the codebase and documentation despite naming the same thing.
  • VTEP DaemonSet was an additional moving part: peer list maintenance, FDB drift detection, an additional namespace (oobi-fabric), additional privileges (NET_ADMIN + SYS_ADMIN + hostNetwork).
  • pkg/oobi/canon-verify.sh exit codes depended on inner-VTEP kernel state being just right (vxlan0 exists with correct VNI, dstport, MTU, ToS, no learning) — flaky across kernel versions.
  • VXLAN as a test payload against the NGFW DUT (TREX.Art and HyperBridge.Art can generate it as data plane stress) is a perfectly legitimate use case — but having VXLAN simultaneously as management transport AND test payload created semantic confusion in test reports.

The 2026-05-14 operator directive declared:

Não teremos mais Fabric OOBI com VXLAN. Teremos somente OOBI, tradicional, sem VXLAN como overlay. Precisamos ajustar todos os modulos, kernels, runtimes para operar com VLAN. A única exceção que possuirá VXLAN são os elementos que farão túneis VTEP com o NGFW DUT em determinados testes de performance com VXLAN. A rede 100.127.252.0/24 e a VLAN 99 não devem mais existir no projeto.

The directive is captured verbatim in memo project_oobi_no_vxlan_directive_2026_05_14.

Decision

OOBI becomes a traditional 802.1Q VLAN carved out of the management uplink — no overlay. The canonical surface:

Property Value Rationale
Carrier VLAN ID 2777 Replaces previous "VLAN 99"; chosen for low collision probability with customer DUT-VLANs (most networks reserve <1000 + lab spaces 3000+)
Host subinterface oobi0 Bring-up: ip link add link eth0 name oobi0 type vlan id 2777. Name is uplink-agnostic (works for eth0, ens3, bond0, etc.)
IPv4 subnet 100.127.252.0/24 RFC 6598 CGNAT carve-out; exotic enough to minimise customer collision while remaining administratively scoped
IPv4 supernet 100.127.252.0/22 Parent prefix; .253–.255 reserved for HPA expansion + HA replicas + future
IPv6 subnet fd5a:7c5e:a72::/64 Preserved verbatim — RFC 4193 ULA was uplink-agnostic and never depended on the overlay
MTU 1500 Standard Ethernet floor; jumbo (9000) supported when both NICs + switch allow
DSCP marking CS7 mandatory on every OOBI packet Preserved verbatim from ADR 0038. The removal of VXLAN does NOT relax this — the DSCP bits live in the only IPv4/v6 header now in flight
Slot map unchanged (slot 0–255 → 100.127.252.<slot>) The slot map (SlotPW=79, SlotSPANPrimary=230, etc.) is preserved — only the IPv4 prefix changed

VXLAN does not disappear. It changes role:

  • ❌ NOT management transport (was pre-2026-05-14)
  • ✅ Test payload generated by TREX.Art and HyperBridge.Art against the DUT, to measure DUT VXLAN inspection capacity in scale. Test payload is emitted on the data plane (VLAN 20 / VLAN 30 / data-plane macvlans), never on the OOBI VLAN.

The expurgo is total: 100.127.252.0/24, VLAN 99 (as OOBI), VXLAN VNI 254254, and oobi0 interface name must not appear anywhere in the codebase except as historical comments and regression-guard test cases.

Topology compatibility matrix

Deployment OOBI carrier
Single-node lab oobi0 subinterface on the single host; no switch needed; pods receive 100.127.252.x via Multus macvlan
Dual-node (UCS-1 + UCS-2 with cross-cable) Direct link between hosts; each host runs oobi0 subinterface; 802.1Q trunking on the cable
Dual/tri/multi-node (UCS + Nexus switch) Nexus 9000 trunks VLAN 2777 between the UCS ports; standard configuration

Single-node intra-host transport works without any switch (loopback in the kernel). The same canonical surface (oobi0 + slot map) exists across all three topologies.

Consequences

Positive

  • Performance: no encap overhead. Postgres queries, Dashboard → MÓDULOs RPC, agent heartbeats — all gain ~5% throughput without jumbo, more with jumbo.
  • Simplicity: VTEP DaemonSet deleted. One less moving part.
  • Debuggability: tcpdump -i oobi0 works natively. No need to decapsulate VXLAN headers in capture analysis.
  • Single-node parity: VXLAN VTEP in single-node was a loopback hack; VLAN subinterface is naturally local.
  • Semantic clarity: VXLAN now means "test payload generated by TREX.Art / HyperBridge.Art against the DUT" — no overlap with the management plane.

Negative

  • Major version bump: any installation on v3.x is incompatible with v4.0+. CHANGELOG records this as a breaking change.
  • Documentation drift: 144 files referenced the retired identifiers (10.254.254, VLAN 99, VNI 254254, oobi0). All must be expurgated across 7 PRs in the OOBI-VLAN wave.
  • Customer collision risk shifts: now in 100.127.252.0/22 (CGNAT carve-out). Acceptable — no customer should be using CGNAT internally; if they are, operator picks a different /22 within 100.64.0.0/10 via CPOS override.
  • Stale operator workflows: any operator CPOS document referencing oobi.canon.vni / oobi.canon.udp_port / oobi.canon.inner_mtu will hit a clear validation error (we keep them in FORBIDDEN_OOBI_FIELDS as a tripwire).

Implementation plan (7 PRs)

PR Scope
PR-OOBI-VLAN-1 (this commit) Foundation: pkg/oobi/canon.go + pkg/oobi/client/ defaults + TS mirror + airgap-oobi-canon-verify.sh Check 1 + this ADR + memo
PR-OOBI-VLAN-2 Go libs sweep: pkg/customer-mgmt-receiver/internal/nad/*, pkg/hyperbridge-art/internal/netflow/exporter.go, pkg/flow-art/internal/server/source_bind_*.go, pkg/span-collector/cmd/*, plus 22 Go test files
PR-OOBI-VLAN-3 Infra-as-code: DELETE k8s/oobi/30-vtep-daemonset.yaml + DELETE k8s/vyos-vtep/ + recreate NAD oobi-overlay as macvlan on oobi0 + update 31 k8s/ files + scripts/airgap-oobi-canon-verify.sh Checks 2–4 rewritten
PR-OOBI-VLAN-4 Dashboard TS sweep: 22 TS files (lib/oobi/, lib/lab-staging/, lib/sustainability/, lib/tests/ip-canon)
PR-OOBI-VLAN-5 VXLAN-as-test-payload isolation in TREX.Art + HyperBridge.Art with explicit --vxlan-test-payload-mode flag; documentation that VXLAN here is payload, never transport
PR-OOBI-VLAN-6 Docs sweep: CLAUDE.md + 38 docs + READMEs + ADRs 0036–0042 with "Refactored 2026-05-14" header + CHANGELOG v4.0.0 entry
PR-OOBI-VLAN-7 Smoke validation refeita: P1.1 K8s manifests + P1.2 NetFlow + P1.3 SPAN + P1.4 CMR + PW agent — all on the new 100.127.252.0/22 carve-out

Compliance

  • airgap-oobi-canon-verify.sh Check 1 enforces the new constants bash ↔ go ↔ ts agreement (this commit).
  • airgap-oobi-canon-verify.sh Checks 2–4 enforce runtime state (PR-OOBI-VLAN-3).
  • pkg/oobi/canon_test.go::TestNoVXLANRemnants regression guard against accidental reintroduction (this commit).
  • dashboard/src/lib/oobi/__tests__/canon.test.ts regression guard matching the Go-side guard (this commit).
  • CPOS schema rejects mutations to canon fields (post-refactor + pre-refactor names) — FORBIDDEN_OOBI_FIELDS in dashboard/src/lib/dom/cpos-schema.ts.

References

  • Memo: project_oobi_no_vxlan_directive_2026_05_14.md
  • Memo: discuss_oobi_immutable_gateway_art_2026_05_10.md (now partially superseded)
  • ADR 0019: OOBI slot allocation (slot map preserved)
  • ADR 0038: Universal DSCP Marking Policy (preserved verbatim)
  • CHANGELOG v4.0.0 entry (lands in PR-OOBI-VLAN-6)