Skip to content

ADR-0107: Cross-node persona plane is a prerequisite for benching NGFWs above ~10 Gbps decrypt

  • Status: Accepted — roadmap decision. The single-node persona plane is the accepted v4.x posture; the cross-node work is scoped here so the ceiling is a documented product boundary, not a surprise in a customer engagement.
  • Date: 2026-07-05
  • Driver: HLD §10.1 (known scalability ceiling of the persona plane) and issue #1599. The whole persona plane — 100 persona pods (61 baked-in + 39 cloned, schema v4.3) plus the VyOS/FRR BGP gateway that announces their public per-country /24s — is pinned to a single node (role=ngfw-dut), because macvlan and the L2 BGP fabric are node-local. That node is a hard server-side ceiling on how much decrypt load the bench can present to a DUT.
  • Relates to: ADR-0007 (public-IP persona plane), ADR-0011 (topology axes, l2_fabric), HLD §10.1, and the frozen-report validity work (#1585 — the per-run report now embeds the persona-node hardware profile and a ceiling disclaimer so every result is read against the generator's own limit).

Context

The persona plane's data path is deliberately node-local:

  • Every persona pod attaches to its country VLAN (101–120) via a macvlan net1 interface — macvlan sub-interfaces exist only on the node that hosts the parent NIC. A persona scheduled on another node simply has no L2 adjacency to its VLAN.
  • The VyOS+FRR BGP MÓDULO that is the L3 gateway for all 20 public /24s runs on the same node and peers over those node-local VLANs.
  • Kustomize therefore pins all 100 personas + the gateway to the role=ngfw-dut node in every deployment mode (single/dual/tri/multi — the multi-node modes distribute agents, never the persona plane).

Consequences of that pinning, measured and documented in HLD §10.1: a single UCS/NUC-class node saturates a mid-range NGFW but tops out around ~10 Gbps of inspected/decrypted traffic — its CPU (TLS server-side crypto for 100 Caddy instances), NIC line rate, and PCIe/memory bandwidth are all single-node resources. Benching a high-end NGFW (25/40/100 Gbps decrypt classes) above that point measures the bench, not the DUT.

Two short-term mitigations shipped alongside this ADR (#1585/#1599):

  1. The frozen per-run report embeds personaNodeProfile — the role=ngfw-dut Node's CPU/memory capacity, kubelet/OS info — plus a fixed disclaimer that single-node persona-plane throughput is ceilinged by this node (HLD §10.1).
  2. The report's test-validity verdict (validity) turns generator under-delivery into an explicit degraded/invalid verdict instead of a silently-wrong NGFW number.

Those make the ceiling honest; they do not raise it.

Decision

  1. Single-node macvlan persona plane is the accepted v4.x posture. No v4.x release will attempt to span the persona plane across nodes. The ~10 Gbps decrypt ceiling is a documented product boundary, surfaced in the HLD, in this ADR, and in every frozen run report.
  2. Benching beyond the node ceiling requires cross-node L2 for the persona plane — a persona pod on node B must keep L2 adjacency to its country VLAN and its BGP gateway. Candidate designs, to be evaluated in that order:
  3. Cilium ClusterMesh / cross-node L2 CNI (primary candidate): replace node-local macvlan with a CNI that can stretch the persona L2 segments across nodes, keeping personas.yaml and the public-IP pool (platform/network/public-ip-pool.yaml) as the unchanged source of truth.
  4. Multi-node VLAN trunk fabric via the existing topology.yaml l2_fabric axis (ADR-0011): trunk VLANs 101–120 to every persona-capable node through the external switch (l2_fabric: nexus, future arista/catalyst/generic), keep macvlan per node, and shard personas per country across nodes — the BGP gateway either stays put (VLANs stretched in hardware) or gains per-node peers.
  5. Sequencing gate: cross-node persona plane is a prerequisite for any roadmap item that claims NGFW benching above ~10 Gbps decrypt. Marketing/product material must not promise higher figures until this lands.
  6. Whatever design wins, the operator surface stays dashboard-only (topology selection via topology.yaml + UI; no kubectl steps for the operator), and personas keep their public per-country IPs (ADR-0007 — never RFC1918).

Consequences

  • v4.x results remain defensible: every frozen report states the node the persona plane ran on and its hardware, so a reviewer can check the claimed DUT load against the generator's ceiling.
  • High-end DUT engagements (>10 Gbps decrypt) are explicitly out of scope until the cross-node work lands — sales/labs must scope engagements accordingly.
  • The Cilium ClusterMesh path implies a CNI migration for the persona plane (today Multus+macvlan): a spike must validate BGP gateway placement, per-country VLAN semantics, and that agents→NGFW→persona two-leg TLS routing is preserved before committing.
  • The VLAN-trunk path keeps the current CNI but couples scale-out to physical switch capacity and makes l2_fabric: none (single-node lab, first-class since ADR-0011) permanently single-node — acceptable, labs don't bench 25 Gbps DUTs.
  • Report schema v2 (validity + personaNodeProfile) is already forward-compatible: when the persona plane spans nodes, the profile field grows to a per-node list in a future schema version; frozen v2 reports remain immutable and honest about what they captured.

Alternatives considered

  • Scale up the single node (bigger UCS, 100G NIC, more cores). Postpones the ceiling instead of removing it; TLS server-side crypto for 100 personas still contends with the BGP gateway and the NIC on one PCIe/memory complex. Rejected as the long-term answer (still the pragmatic v4.x answer for mid-range DUTs).
  • L3-only persona spread (route /32s to personas on any node, no stretched L2). Breaks ADR-0007 realism: per-country VLAN semantics and the BGP-announced /24s are part of the product's defensibility; turning the persona plane into NAT-ed ClusterIP-style L3 would change what the DUT sees on the wire. Rejected.
  • Second independent single-node persona plane (two benches side by side). Doubles hardware without letting a single run exceed the per-node ceiling — a DUT decrypt test is one traffic aggregate, not two half-tests. Rejected.
  • SR-IOV VFs per persona. Solves NIC sharing, not cross-node L2 — personas still cannot leave the node that owns the VLAN trunk; and 100 VFs × certs × BGP is operationally hostile. Rejected as the primary lever (may complement the trunk-fabric option later).