ADR-0080: OTel Collector promotion to base kustomization¶

Status: Accepted
Date: 2026-05-23
Driver: 2026-05-23 strategic review §P2-3 — OTel Collector was k8s/optional/ opt-in; should be base infrastructure now that the observability story is mature (15 Grafana dashboards, 9 alert files, slog JSON in 6 services).
Related: PR #964 (audit P3-11 — original OTel Collector foundation)

Context¶

The OpenTelemetry Collector landed in k8s/optional/otel-collector.yaml via PR #964 as an opt-in foundation. The original rationale ("wire-when-ready") made sense in 2026-05 when the trace backend was undecided and no Go service emitted OTLP.

Since then: - Wave-2 P2P observability shipped 11 canonical Prometheus metrics - 3 Grafana dashboards + 7 Prometheus alerts wired (PR #949) - 6 Go services migrated to slog JSON output (PR #962) - Customer-app uses @opentelemetry/sdk-node + OTLP HTTP exporter

The collector is no longer optional — it's part of the observability substrate. Keeping it under k8s/optional/ forces every operator to manually kubectl apply it after the base deploy, and to remember to re-apply it after git pull updates.

Decision¶

Move k8s/optional/otel-collector.yaml → k8s/95-otel-collector.yaml (named-numbered to fit the base manifest convention).
Add to base k8s/kustomization.yaml resources list (alongside the other base manifests).
Add k8s/96-otel-env-configmap.yaml — shared ConfigMap per-namespace exposing OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_EXPORTER_OTLP_PROTOCOL + OTEL_RESOURCE_ATTRIBUTES + trace-sampler defaults. Services consume via envFrom:.
Wire the first reference consumer (connect-art deployment) to demonstrate the pattern. Other Go services adopt the envFrom: block in follow-up PRs (no fleet-wide patch in this PR — keeps blast radius small).
Update k8s/optional/README.md to note the relocation and point readers to the new location.

Per-namespace ConfigMap (not cluster-scoped)¶

K8s ConfigMaps are namespace-scoped. The same OTel env vars are duplicated across 3 namespaces (web-agents, octopus-cloud, audit-coordinator). Trade-offs considered:

Option	Pros	Cons
Per-namespace ConfigMap ✓	Native K8s, no extra controller	3× duplication
`ClusterConfigMap` via Kyverno	One source	Adds Kyverno dependency
Mount via projected volume from one ns	Single source	Cross-namespace RBAC complexity
Inline `env:` per deployment	No ConfigMap dep	Worst — 16+ services duplicate the same env block

Per-namespace ConfigMap is the lightest-weight idiomatic solution. Adding a new namespace = add a 12-line YAML block; cluster-wide config changes mean editing 3 places (acceptable for a value that rarely changes).

Sampling defaults¶

Production sampling uses parentbased_traceidratio at 0.05 (5%). The audit-coordinator ConfigMap overrides to always_on because its event volume is intrinsically low (one event per global-root publish cycle, typically every 60s) and operators want full traces for forensic audit.

Per-service sampling overrides go in the deployment env: block (e.g. when investigating a hot path, bump OTEL_TRACES_SAMPLER_ARG to 1.0 temporarily).

Consequences¶

Positive¶

Zero "I forgot to apply otel-collector" incidents. Default deploy of the base manifests = OTel Collector is up.
Standard endpoint discovery — every Go service that adds the envFrom: otel-env line picks up the cluster's OTLP endpoint automatically. No hard-coded URLs.
First reference wiring (connect-art/10-deployment.yaml) documents the adoption pattern inline.
Trace backend wiring stays a one-line edit — the existing ConfigMap-based exporters.otlp/cloud: pattern documented in the collector manifest's comments is unchanged.

Negative¶

Three-namespace ConfigMap duplication. Mitigated by the trade-off table above. A future ADR can revisit if 5+ namespaces pile up.
Fleet-wide adoption is a follow-up. This PR wires only connect-art. The other ~10 Go services pick up the envFrom block in their next normal-cycle PR (file-size budget per ADR-0074 prevents accumulating fleet-wide patches in one mega-PR).