Skip to content

ADR-0088: VALIDATOR.Art ML cortex — Phase E sidecar + opt-in switchover

  • Status: Accepted (2026-05-24) — Phase E scaffold (ML-1..ML-4) shipped in PRs #1051 / #1053 / #1055 / this PR. Trained model + operator validation loop land in Phase F
  • Date: 2026-05-24
  • Driver: discuss_module_validator_tbi_2026_05_10 §"ML cortex embedded (8 functions)" + discuss_validator_tech_stack_2026_05_10 §"Layer 4: ML cortex (Python sidecar)" + ADR-0086 §"Pure-heuristic role assignment in Phase A; ML cortex in Phase E"
  • Related: PRs #1051 (ML-1), #1053 (ML-2), #1055 (ML-3), this PR (ML-4); paired wave ADR-0086 (VALIDATOR.Art receiver) + ADR-0087 (TBI image)

Context

VALIDATOR.Art Phase A (ADR-0086) shipped role assignment as a pure Go heuristic — interpretable, fast, deterministic, but unable to learn from operator overrides. The strategic memo named this as Phase E: a Python ML sidecar replacing the heuristic with a trained model that improves over time as operator-validated examples accumulate.

The defensible patent novelty (Patent Family E — "Enrollment Cortex") sits in the combination:

  1. Hardware-fingerprint → role classification via heuristic-replay-bootstrapped model
  2. Operator-validation feedback loop that re-trains weekly without breaking the deterministic baseline
  3. Heuristic fallback that guarantees enrollment continuity when the model degrades or the sidecar is unreachable
  4. Tier-specific role taxonomy (DPDK-aware, GPU-aware, NGFW-DUT-aware) tied to the broader TLSStress.Art bench operator workflow

No commercial product ships this combination today. Talos / Kairos / Cluster API have role taxonomies but no ML; AWS / GCP / Azure ML services don't ship with K8s enrollment integration; MAAS has hardware probes but assigns roles via static tagging only.

Decision

Ship the ML cortex as a Python FastAPI sidecar running alongside validator-art in the same K8s pod (slot .97 Deployment). The validator-art orchestrator consults the sidecar via 127.0.0.1 HTTP/JSON when --ml-cortex-url is set; falls back to the heuristic silently on error/timeout.

Materialized in 4 sub-PRs:

Sub-PR PR Scope
ML-1 #1051 pkg/validator-ml-cortex/ Python scaffold — Pydantic schemas mirroring Go types, heuristic.py (line-for-line port of roleassign.Compute), inference.py (ONNX stub that returns None until ML-2 trains), server.py (FastAPI /healthz /metrics /predict-role). Dockerfile non-root UID 2087. 16 pytest
ML-2 #1053 training/ subpackage — schemas.TrainingExample, featurize (positional 12-element float vector + 7-class label encoder, pinning test), generate_replay (deterministic-per-seed dataset generator), train.py (scikit-learn DecisionTree + skl2onnx export with deferred imports), verify.py (onnxruntime inference + per-class mismatch + threshold gate). datasets/seed-heuristic-replay.jsonl 200 examples (committed for reproducibility)
ML-3 #1055 pkg/validator-art/internal/mlclient/ Go HTTP client — Predictor interface + ErrDisabled sentinel + 5 tests. Server /enroll opt-in switchover (consult ML first, fall back to heuristic on error). types.Config.MLCortexURL + --ml-cortex-url CLI flag. K8s manifest adds ml-cortex sidecar container in slot .97 Deployment
ML-4 This PR ADR-0088 + docs/modules/ml-cortex.{md,pt-BR.md,es.md} + docs/patents/family-e-enrollment-cortex-provisional-draft.md (claim #18 reservation) + recovery memo + MEMORY.md index entry

Locked decisions (Phase E)

# Decision Rationale
1 Same-pod sidecar (not separate Deployment) 127.0.0.1 cross-container call, zero network hops, no service-mesh integration needed; matches the "low-latency role decision" requirement of the enrollment path (operator boots a node, expects it to be ready in seconds, not minutes)
2 HTTP/JSON not gRPC Sidecar is in-pod; gRPC + .proto + buf workflow would triple the surface for zero perf benefit. Operator can curl localhost:8087/predict-role for debugging
3 DecisionTree for Phase E baseline (not deep net / random forest) Interpretable (tree dump is human-auditable) + compact (ONNX < 100 KB) + mirrors the heuristic structure (the heuristic IS nested if-else); the right model class to LEARN that exact shape. Deeper nets would overfit the 200-example seed
4 Heuristic-replay bootstrap The trained model MUST match the Go heuristic on the seed set with ≥ 99% accuracy before ML-3 enables the opt-in switchover. This protects against the regression "ML model picks a worse role than the deterministic baseline would have"
5 Operator opt-in, not auto-rollout The K8s manifest leaves --ml-cortex-url flag COMMENTED OUT by default. Operator un-comments after they've verified the model's accuracy on their own dataset (operator-validated examples, not just our seed)
6 Heuristic fallback always available When the sidecar is unreachable / times out / returns 5xx / model is corrupt, the validator falls through to the heuristic silently. Enrollment NEVER fails because ML is unavailable — degraded service, never failed enrollment
7 Source attribution in every prediction RoleAssignment.Source field carries "heuristic" vs "onnx" so operator can audit which side served each enrollment. Telemetry counter validator_ml_cortex_predictions_total{source=…} for fleet-wide visibility
8 Pure Python, no PyTorch in sidecar Sidecar uses ONNX Runtime only (~50 MB wheel via [inference] extras). PyTorch (1+ GB) lives in the training pipeline ([training] extras), which runs on the operator's laptop / CI, not in the cluster
9 JSONL dataset format Append-only, line-by-line streamable, human-readable. Lets the operator manually curate examples + diff datasets across releases. Parquet / Arrow would be premature optimization for the 200-1000 example scale
10 Patent Family E provisional draft in this PR Claim #18 reservation — see docs/patents/family-e-enrollment-cortex-provisional-draft.md. Total claims after this wave: 18

Training → inference → serving → validation flow

Phase E baseline (this wave):

   1. generate_replay.py  →  datasets/seed-heuristic-replay.jsonl (200 ex)
   2. train.py            →  models/role-predictor.onnx (DecisionTree)
   3. verify.py           →  exit 0 iff accuracy ≥ 0.99 on seed (CI gate)
   4. ml-cortex sidecar loads ONNX at boot (ML_CORTEX_MODEL_PATH env)
   5. /predict-role serves ONNX prediction (source="onnx") OR heuristic
      fallback (source="heuristic") when no model loaded / fails
   6. validator-art /enroll consults sidecar when --ml-cortex-url set
   7. Heuristic fallback if sidecar errors / times out / unreachable

Phase F adds the feedback loop:

   8. Operator validates assignment via Dashboard "Approve / Override"
      → captured in operator_validated dataset
   9. Weekly retraining job (GradientBoosting once examples > 1000)
   10. New model auto-verified vs current; auto-deployed if accuracy
       ≥ 0.99 on combined seed + operator dataset; rolled back if not

Patent Family E — provisional draft (claim #18 reserve)

Detailed in docs/patents/family-e-enrollment-cortex-provisional-draft.md. Headline claim:

"A method for assigning compute roles to newly-enrolled nodes in a heterogeneous test-bed cluster, comprising: (a) collecting a hardware fingerprint at node first-boot; (b) classifying the node into a role using a heuristic-replay-bootstrapped ONNX model deployed as a same-pod sidecar to a central enrollment validator; (c) silently falling back to a deterministic heuristic baseline when the model is unavailable or fails verification; (d) capturing operator-validated role assignments via a dashboard approve/override workflow; (e) re-training the model weekly on the combined seed + operator-validated dataset; (f) auto-deploying only models that match the heuristic baseline accuracy on the seed set; (g) attributing each served prediction with a source field for fleet-wide observability."

Provisional filing target: 2026 Q3 once Phase F operator-validation loop ships first 100 real examples.

Total patent claims after ML cortex Phase E: 18 (TREX.Art's #17 was most recent; #18 reserved).

Consequences

Positive

  • Closes the strategic memo's "ML cortex embedded (8 functions)" deferred item with a clear baseline + measurable upgrade path
  • Operator opt-in + heuristic fallback means zero risk of breaking existing enrollments
  • Patent Family E reservation is the first defensible novel-IP claim VALIDATOR.Art ships
  • Python sidecar + ONNX = trainable in any environment + inferenceable with a 50 MB wheel
  • Same-pod deployment keeps the latency budget for enrollment under control

Negative / deferred

  • No trained model yet — Phase E scaffold ships the pipeline; first real role-predictor.onnx lands in Phase F after operators accumulate validated examples
  • No operator validation workflow — Phase F adds the Dashboard "Approve / Override" surface that captures operator-validated examples
  • No automated weekly retraining job — Phase F adds a K8s CronJob that runs train.py + verify.py + auto-deploys if the threshold passes
  • No 7 other ML cortex functions — strategic memo named 8 functions (discovery method ranking, cascade topology optimization, failure prediction, auto-recovery, intelligent role assignment, anomaly detection, drift prediction, scan pattern adaptation). ML-1..ML-4 scope only function 5 (intelligent role assignment) as the proof-of-concept. Other 7 land in Phase G+

Alternatives considered

  • Same-pod gRPC instead of HTTP/JSON — rejected per #2 above; complexity not justified at this scale
  • Separate Deployment for cortex — rejected per #1 above; latency
  • Deep neural network (PyTorch + Transformer) — rejected per #3; 200 examples doesn't justify it; interpretability suffers
  • Auto-rollout opt-out instead of opt-in — rejected; breaking enrollment on a fresh customer who hasn't validated the model would be unforgivable
  • PyTorch in sidecar — rejected; 1+ GB image for inference-only adds no value
  • Patent Family E filing immediately — rejected; need ≥ 100 operator-validated examples to back the "Phase F learning loop" claim

Compliance

Per CLAUDE.md: "Dashboard is the ONLY operator interface." The opt-in --ml-cortex-url flag is operator-controlled via the K8s manifest commented line; Phase F's "Approve / Override" workflow lives on /admin/validator-art (extends the V-3 surface).

Per ZTP-prem posture: sidecar runs readOnlyRootFilesystem: true, runAsNonRoot: true, runAsUser: 2087, drop: [ALL] capabilities. Cross-container localhost only (no NetworkPolicy egress).

References

  • discuss_module_validator_tbi_2026_05_10 — strategic memo (§"ML cortex embedded (8 functions)")
  • discuss_validator_tech_stack_2026_05_10 — concrete tech-stack (§Layer 4 ML cortex Python sidecar)
  • ADR-0086 — VALIDATOR.Art receiver (§"Phase E ML cortex deferred")
  • ADR-0087 — TBI image (paired Phase I wave)
  • PRs #1051 (ML-1), #1053 (ML-2), #1055 (ML-3), this PR (ML-4)
  • docs/patents/family-e-enrollment-cortex-provisional-draft.md — claim #18 reservation
  • Patent claims after this wave: 18 (Family E #18 reserved)