Skip to content

ADR 0072 — Dashboard Visual Regression Tooling (Wave-D4)

Status Date Author Supersedes Superseded by
Accepted 2026-05-20 André Luiz Gallon — —

Context

Wave-D of the Dashboard UI code quality audit landed:

  • D1 (PR #918) — Playwright smoke + axe-core a11y CI gate (6 public routes)
  • D2 (#920) — Admin scope (30 routes) + mobile-chrome viewport (Pixel 7)
  • D3 (#924) — Mutation flow tests (login + sink + logout) via page.route() interception

D4 is the visual regression gate: a screenshot taken on every PR is compared against a known-good baseline; any pixel-level diff fails CI. This catches the class of regressions that a smoke test misses — a button that disappeared, a layout that broke, a contrast change.

This ADR records WHY we picked Playwright's built-in toHaveScreenshot() over the alternatives, and what the rollout plan looks like.

Architectural decision

3 LOCKED decisions.

D1: Use Playwright's built-in toHaveScreenshot() — no SaaS

Options evaluated:

Tool Strengths Weaknesses Verdict
Playwright snapshots (built-in expect(page).toHaveScreenshot()) Free · already in the dependency tree · diff PNGs uploaded as CI artifacts · works air-gap · OS-stable Chromium pin Cross-OS rendering varies (need to pin runner OS) · diff is pixel-level not perceptual ✅ Adopted
Percy (BrowserStack) Industry standard · cross-browser baselines · PR comment with side-by-side diff · perceptual diff algorithm $149/mo for the cheapest team plan · SaaS-only (no self-host) · vendor lock-in · external data egress (snapshots leave the org) ❌ Conflict with ADR 0048 ZTP-prem posture + air-gap mandate
Chromatic (Storybook) Best-in-class for component-level visual diffs · paid free tier 5k snapshots/mo · perceptual diff Requires Storybook (we don't use it) · component-only (not full-page) · adds 2nd UI dev pipeline ❌ Wrong abstraction for our full-page operator console
BackstopJS OSS · self-hostable · PhantomJS/Puppeteer based Maintenance burden (separate test runner) · less polish than Playwright ❌ Redundant — we already have Playwright

The decision is forced by D1 (anti-vendor-lock-in) + D2 (air-gap mandate from ADR 0048). Sending Dashboard screenshots to a SaaS would leak operator UI state, which can contain customer tenant IDs, deployment names, and topology hints. That's a no-go for our threat model.

D2: Run snapshots only on ubuntu-latest runners

Playwright pixel diffs vary between macOS, Linux, and Windows due to font rendering, sub-pixel hinting, and ICC profiles. To keep the baseline stable:

  • CI snapshots ONLY run on ubuntu-latest (the same runner that hosts our existing dashboard-e2e.yml workflow)
  • Developer-machine snapshots are advisory — never committed
  • Snapshot updates go through a CI artifact: npm run e2e -- --update-snapshots runs on a update-visual-baselines workflow that produces a PR-attached diff
  • Baseline PNGs live under dashboard/tests/e2e/__screenshots__/ (committed to git, gzip-friendly)

D3: Coverage rollout in waves — D4.1 to D4.3

D4 ships in 3 sub-PRs to keep review scope sane:

  • D4.1 — Foundation + 3 baselines
  • playwright.config.ts adds expect: { toHaveScreenshot: { ... } } defaults
  • 3 high-signal pages get baselines: /, /dashboard, /login
  • 1 workflow: dashboard-visual.yml separate from dashboard-e2e.yml so a visual diff failure doesn't block functional CI
  • D4.2 — Admin scope baselines (30 pages)
  • Reuses the admin-smoke route list
  • Skips pages with live data (dashboard panels with polling) — those would produce non-deterministic diffs
  • D4.3 — Mobile viewport baselines
  • Pixel 7 viewport only (mobile-chrome project already exists)
  • Same 3 pages as D4.1 but in mobile

Consequences

Positive: - Pixel-level regression detection on every PR, free, no SaaS dependency - Diff PNG visible as a CI build artifact + downloadable for human review - Baseline updates are explicit (--update-snapshots flag), not implicit — no accidental drift

Negative: - Linux-only baselines — developers on macOS/Windows can't run expect(page).toHaveScreenshot() locally and trust the result. Mitigation: docs say "run with --update-snapshots --reporter=null only as advisory." - Live-data pages (any page with polling counters or timestamps) are explicitly EXCLUDED from D4.2 — they need a deterministic data layer (intercepted via page.route()) before they can have baselines. Tracked as Wave-D4.4.

Implementation order

  1. D4.1 — foundation + 3 baselines (this ADR's first follow-up)
  2. D4.2 — 30 admin baselines (excluding live-data pages)
  3. D4.3 — mobile viewport baselines
  4. D4.4 — deterministic data layer for live pages (separate ADR if scope grows)

Open questions

None. Decision is unambiguous given the air-gap + ZTP-prem constraints.

  • ADR 0048 — ZTP-prem posture (anti-SaaS-egress)
  • PR #918 — D1 foundation
  • PR #920 — D2 admin smoke
  • PR #924 — D3 mutation flows
  • dashboard/tests/e2e/README.md — operational reference