ADR-0090: ML cortex Phase F — operator validation feedback loop¶
- Status: Accepted (2026-05-24) — Phase F wave (MF-1 + MF-2 + MF-3 + this) shipped in PRs #1063 / this PR
- Date: 2026-05-24
- Driver: closes ADR-0088 §"Phase F" deferred items
- Related: PRs #1063 (MF-1+2+3 consolidated) · this PR (ADR + memo) · ADR-0088 (Phase E scaffold the loop closes around) ·
docs/patents/family-e-enrollment-cortex-provisional-draft.mdclaim #18 §(d)-(g)
Context¶
ADR-0088 (Phase E) deliberately deferred two pieces of the ML cortex story:
- Operator validation workflow — a "Approve / Override" surface on
/admin/validator-artcapturing operator-validated role assignments as labelled training examples - Weekly retraining job — a K8s CronJob that pulls the operator-validated dataset, re-trains the DecisionTree, runs the
verify.pythreshold gate, and auto-deploys the new model when the gate passes
Phase F closes both. The result is the first operator-validated training loop in the project — closes the feedback gap that all earlier ML cortex sketches assumed away.
Decision¶
Ship the loop as 3 sub-PRs consolidated into a single PR (MF-1 + MF-2 + MF-3) + this ADR:
| Sub-PR | Scope |
|---|---|
| MF-1 | Drizzle 0041 validator_role_overrides table + journal entry idx 41 + POST /api/validator-art/overrides (Zod) + GET for last 200 audit |
| MF-2 | GET /api/validator-art/dataset streams up to 10k rows as JSONL TrainingExamples (source=operator_validated, weight=1.0) |
| MF-3 | k8s/oobi/99-ml-cortex-retrain-cronjob.yaml: ServiceAccount + Role (patch ml-cortex-model ConfigMap + validator-art Deployment annotation) + ml-cortex-model ConfigMap + ml-cortex-seed-dataset ConfigMap + CronJob 0 6 * * 0 running trainer that does curl dataset → append seed → train → verify ≥ 0.99 → kubectl patch ConfigMap → bump Deployment annotation |
Locked decisions¶
| # | Decision | Rationale |
|---|---|---|
| 1 | operator_role REQUIRED when action="override" | DB-level check constraint; prevents the "I clicked Override but forgot to set the role" footgun |
| 2 | Approve writes too (not just Override) | Approvals are labelled training data — label = predicted_role. Filtering only Overrides would bias the model to the operator's edge-case attention |
| 3 | JSONL export with source + weight | Same schema as TrainingExample (ADR-0088 §"Phase A datasets §source mix policy") — append-only, line-by-line, diff-friendly. Operator can grep / vim the export |
| 4 | Weekly cadence Sunday 06:00 UTC | 2 hours after TBI rebuild (Sun 04:00 UTC); low-traffic window; gives a full week of operator feedback before each retrain |
| 5 | verify.py --threshold 0.99 as the deploy gate |
Same threshold ADR-0088 locked. Phase F's combined dataset = seed + operator-validated; model MUST match accuracy on the combined set, NOT just the operator subset (protects against operator-set overfit) |
| 6 | ConfigMap holds the ONNX bytes, not a PVC | Phase F dataset stays < 1 MB (200 seed + ≤ 10k operator-validated rows × ~200 bytes each ≈ 2 MB tops). ConfigMap binaryData limit is 1 MiB but the trained ONNX DecisionTree itself stays ~50 KB — well within budget. Phase F-late switches to PVC if Phase G adds large models |
| 7 | Sidecar reload via Deployment annotation bump | K8s rolls the validator-art Deployment when an annotation changes; the ml-cortex sidecar reads the new model on container restart. Avoids inotify dependency in the Python sidecar |
| 8 | CronJob runs validator-ml-cortex:v0.1.0-training image |
Same base image with [training] extras installed; keeps the operator-facing scaffold image (without the 800 MB scikit-learn wheel) small |
| 9 | Failed verify keeps prior model | When the new model fails the threshold gate, the CronJob exits non-zero, the previous ConfigMap binaryData stays untouched, the K8s Event surfaces in the Dashboard — operator decides next step |
| 10 | No automated rollback after deploy | If the freshly-deployed model causes problems in production, operator manually reverts by re-running the previous successful retrain (or bumping the annotation to the prior model's checksum). Auto-rollback adds complexity without clear win at the Phase F scale |
Operator workflow (locked)¶
1. Day 1 Operator browses /admin/validator-art enrollments table
2. Operator clicks "Approve" (role stands) OR "Override"
+ types corrected role + reason
3. POST /api/validator-art/overrides → row in DB
4. (Repeat ~weekly until ≥ 100 rows accumulate)
5. Sunday 06:00 UTC — CronJob fires:
- curl /api/validator-art/dataset → /tmp/operator.jsonl
- cat seed + operator → /tmp/combined.jsonl
- python -m validator_ml_cortex.training.train …
- python -m validator_ml_cortex.training.verify --threshold 0.99
- On pass: kubectl patch ConfigMap ml-cortex-model + bump
validator-art annotation → sidecar restarts with new model
- On fail: exit non-zero, K8s Event, prior model unchanged
Consequences¶
Positive
- Closes ADR-0088 §"Phase F" deferred items + delivers the first operator-validated training loop in the project
- Auto-deploy gate (
verify.py --threshold 0.99) protects against model drift that would lower accuracy - ConfigMap-based ONNX delivery means no PVC dependency for the model artifact
- Operator opt-in stays intact —
--ml-cortex-urlflag still commented out in the validator-art manifest; opt-in operators get the freshest model from each retraining cycle automatically
Negative / deferred
- Approve/Override UI buttons on
/admin/validator-arttable — API + DB ready; UI buttons land in a Phase F-late PR (TS + React state for inline form per row) - Per-operator filter on dataset export — current export streams ALL rows; Phase F-late adds
?since=...&approver=...query params - Multiple-approver consensus — Phase F treats each Override as ground truth; Phase F-late could weight by approver tenure
- ml-cortex-seed-dataset ConfigMap is empty by default — operator seeds via
kubectl create configmap --from-file=(documented in the YAML)
Compliance¶
Per CLAUDE.md: operator-facing workflow on /admin/validator-art (UI buttons in Phase F-late); operator never types kubectl against the CronJob.
Per ZTP-prem posture: trainer container runs non-root UID 2087 + drop ALL caps + readWritePaths only /tmp.
References¶
- ADR-0088 (Phase E scaffold this loop closes around)
- ADR-0091 (Phase G — 7 additional ML cortex functions)
docs/patents/family-e-enrollment-cortex-provisional-draft.mdclaim #18 §(d)-(g) cover the feedback loop- PRs #1063 (MF-1+2+3 consolidated) · this PR (ADR + memo)