Skip to content

ADR 0045 — OOBI Hello / KeepAlive Mesh

  • Status: Accepted (2026-05-14)
  • Date: 2026-05-14
  • Deciders: TLSStress.Art project (operator directive 2026-05-14 — Wave 7)
  • Builds on: ADR 0038 (DSCP), ADR 0043 (OOBI VLAN), ADR 0044 (Intrusion), pkg/oobi/auth/ (Bearer L2)
  • Targets: v4.0.0
  • Companion memo: ~/.claude/projects/-Users-agallon-AI-forSE/memory/project_oobi_hello_keepalive_2026_05_14.md

Context

ADR 0043 moved OOBI from a VXLAN overlay to a single 802.1Q VLAN, and ADR 0044 added an anti-intrusion L3 filter to reject foreign hosts on the canonical /22. What is still missing is liveness observability of the mesh itself: which MÓDULOs are up, which neighbors each can actually reach, and how to detect partial L2 partitions (e.g. a switch trunk that silently drops the VLAN from one uplink while keeping it on another).

The operator captured the requirement verbatim:

Devemos implementar na rede OOBI, o envio / recebimento de pacotes do tipo KeepAlive / Hello, constantes, recorrentes, na frequencia de 1 envio a cada 30 segundos, entre todos elementos modulos/kernels/runtimes desse software. Esse pacote será enviado simultaneamente em Multicast e Broadcast, pois alguns switches que recebem pacote Multicast acabam descartando o pacote (e nao queremos ficar engenhando os comutadores do cliente apenas para termos certeza que nossa engrenagem está rodando). O envio simultâneo de Multicast e Broadcast deve ser feito por padrão. Vai dentro do pacote, o slot do modulo, dados de saúde e neighbors observed. Pode levar até 60 segundos sem receber um Hello para entender que algum vizinho está fora do ar — Será enviado uma alerta para o operador, similar a alerta do ADR 0044.

The full UI requirement (deferred to PR-OOBI-VLAN-4c) reads:

Devemos visualizar todos esses dados no Dashboard, em uma área de Observability que mostra para o operador todos elementos visualizando seus respectivos vizinhos. Será exibido também no Grafana, em um dashboard separado. Esse Dashboard deve ser similar a um mapa vivo, interativo, real-time, com cores informando status e saúde da observancia dos vizinhos.

Decision

Implement a pkg/oobi/hello/ library that every MÓDULO embeds via three lines in its main.go. The library owns:

  1. A JSON-over-UDP wire format (debuggable with tcpdump -A).
  2. A dual-transport sender — every Hello goes to BOTH the multicast group (239.127.252.1) AND the canonical directed broadcast (100.127.255.255), because field experience shows random customer L2 switches silently drop multicast.
  3. A receiver that listens on UDP/47802, deduplicates by (instance_id, sequence) so the multicast + broadcast pair counts as one logical Hello, validates each packet, and updates a neighbor table.
  4. A sweep loop (5 s ticker) that classifies neighbors as OK / Warning / Down / Unknown based on LastSeen age and dispatches hello_neighbor_timeout events to the same Alerter chain used by pkg/oobi/intrusion/ — so Hello losses and L3 intrusions surface in the same Dashboard pane and same Syslog facility.
  5. A Prometheus emitter with cardinality bounded by module × neighbor_slot ≤ ~1369 series.

Wire format

{
  "slot": 17,
  "module_name": "flow-art",
  "instance_id": "<32-hex>",
  "version": "v4.0.0",
  "started_at": "2026-05-14T00:00:00Z",
  "sent_at":    "2026-05-14T00:00:30Z",
  "sequence": 1,
  "health":  { "ok": true, "rss_bytes": 1024, "goroutines": 8, "uptime_seconds": 42 },
  "neighbors": [
    { "slot": 18, "module_name": "api-infra-art",
      "last_seen_at": "2026-05-14T00:00:25Z", "rtt_ms": 0 }
  ]
}

MaxNeighborsPerPacket = 35 keeps a worst-case packet under MTU 1500.

Canon constants (pkg/oobi/canon.go)

Constant Value
HelloMulticastIPv4 239.127.252.1
HelloMulticastIPv6 ff15::7c5e:a72:1
HelloBroadcastIPv4 100.127.255.255
HelloUDPPort 47802
HelloIntervalSeconds 30
HelloJitterMilliseconds 3000
HelloLivenessTimeoutSeconds 60
HelloWarningTimeoutSeconds 45

State machine

                 Hello received
       ┌───────────────────────────────┐
       ▼                               │
   StateOK ──── 45s without Hello ──► StateWarning
       │                               │
       │  60s without Hello            │ 60s without Hello
       ▼                               ▼
   StateDown ◄───────────────── StateDown
        (transition emits hello_neighbor_timeout alert)

StateOK → StateDown is the only transition that fires an Alerter event. StateWarning is visual-only (yellow on the map) — alerting every 45 s would generate noise on healthy mesh during ordinary GC pauses or scrape lag.

Concurrency model

Component Goroutines Lock
Sender 1 (tick) none (single-owner)
Receiver 1 (read) atomic counters
Monitor.sweep 1 (5s) shared with Table
NeighborTable n callers sync.RWMutex

The Hello hot path never blocks: alerters are fire-and-forget goroutines (inherited from intrusion.MultiAlerter semantics).

Metrics (Prometheus text)

# detected_by = MÓDULO name; neighbor_slot in [1, 254]; cardinality ≤ 37×37 = 1369

oobi_hello_sent_total{detected_by}                 counter
oobi_hello_send_errors_total{detected_by}          counter
oobi_hello_received_total{detected_by}             counter
oobi_hello_dropped_total{detected_by}              counter   (malformed)
oobi_hello_dedupe_total{detected_by}               counter   (mcast+bcast pair)
oobi_hello_neighbors_alive{detected_by}            gauge
oobi_hello_neighbors_total{detected_by}            gauge
oobi_hello_neighbor_state{detected_by,neighbor_slot,neighbor_module}  gauge (0=unknown, 1=down, 2=warning, 3=ok)
oobi_hello_neighbor_timeouts_total{detected_by}    counter

No IP or MAC labels — only slot + module_name. This bounds the series space and avoids re-introducing the cardinality issue that ADR 0044 already solved for intrusion alerts.

Consequences

Positive

  • Every MÓDULO is observed by every other MÓDULO with no central registry, scrape config, or topology assumption.
  • Dual-transport tolerates the wide range of customer L2 switches the test bench plugs into — no multicast tuning required.
  • Reuses the Alerter chain from pkg/oobi/intrusion/ — operator sees Hello + IDS in the same pane.
  • Bounded cardinality keeps the Prometheus footprint stable up to ≈37 MÓDULOs (the canonical slot range).
  • JSON wire format means tcpdump -i oobi0 udp port 47802 -A is the only debugging tool an operator needs.

Negative

  • 30 s cadence × N MÓDULOs × 2 transports = 2N/30 PPS — at 37 MÓDULOs this is ~2.5 PPS, negligible.
  • Each Hello carries up to 35 NeighborView entries → average packet ~600 B → at 2.5 PPS that's ~1.5 kB/s mesh-wide. Negligible.
  • Multicast-stripping switches still see two copies arrive (the broadcast). Counter oobi_hello_dedupe_total measures how often this fallback engages — a useful health signal in its own right.

Neutral

  • Goroutine cost per MÓDULO: +3 (sender, receiver, sweep). Each one blocks on a channel or socket; idle cost is zero.

Rollout

  1. PR-OOBI-VLAN-4a (this PR) — library + canon constants + ADR.
  2. PR-OOBI-VLAN-4b — wire hello.NewDefaultMonitor into the 8 MÓDULO main.go files (mirror of intrusion wire-up #736).
  3. PR-OOBI-VLAN-4c — Dashboard /admin/observability/oobi-mesh-map
  4. Postgres ingest + Grafana JSON with topology view.

References

  • Memo: project_oobi_hello_keepalive_2026_05_14.md
  • ADR 0043: OOBI VLAN refactor (no VXLAN)
  • ADR 0044: OOBI intrusion detection
  • ADR 0038: DSCP CS7 on control plane
  • Package: pkg/oobi/hello/