Skip to content

ADR-0097 — CI cost-cut radical (10 levers)

  • Status: Accepted
  • Date: 2026-05-25
  • Deciders: André Luiz Gallon
  • Driver: GitHub Actions billing limit exceeded; operator pediu economia drástica
  • PR: TBD (this ADR ships with the workflow edits)

Context

GitHub Actions Linux billing excedeu o limite. Forensic audit of .github/workflows/ revealed 32 workflows + 360 go.mod files (47 after vendor/dot-file exclusion) feeding matrix jobs. Major cost drivers:

Driver Impact Why
3 daily crons (stale, ztp-prem-tier-b-obfuscation, ztp-prem-sod-audit) ~21 unsupervised runs/week Most cron payloads change rarely; daily was overkill
go-security.yml matrix over 47 modules 94 jobs per cron run No tiering — every module scanned on every weekly run
ci.yml no paths: filter Every doc/changelog/memo commit triggered full Go + dashboard matrix High-volume docs PRs paid CI cost they didn't need
dashboard-visual.yml on every PR 3-5 min visual regression per dashboard PR Pixel-diff fails ≪ 1% of time but cost is per-PR
No concurrency.cancel-in-progress on 4 workflows Stale runs piled up during rapid wave-shipping Easy win
No draft-skip on 11 workflows CI ran on every WIP commit Drafts trigger full pipeline
fail-fast: false everywhere One module fail = all 46 others kept running Big waste on flaky/auth failures
Quarterly stack audit cron 4 unsupervised runs/year operator re-triggered anyway Operator workflow_dispatch is the natural cadence

Decisions (10 levers locked)

🟢 Drastic (low risk)

  1. 3 daily crons → weekly Sun: stale.yml, ztp-prem-tier-b-obfuscation.yml, ztp-prem-sod-audit.yml. 7× cheaper each. workflow_dispatch: preserved for on-demand.
  2. ci.yml paths-ignore for **.md, changelog.d/**, docs/**, LICENSE* on both push + pull_request. Skips ~40% of operator commits (memos + ADRs + changelog fragments).
  3. concurrency.cancel-in-progress: true added to 4 workflows missing it (secret-scan, module-conformance, forensic-tamper-check, dco-check). Other workflows already had it.
  4. fail-fast: true on matrix in ci.yml (go-services 3-way) + image-scan.yml (image build 3-way). Cancels remaining matrix legs when one fails; failures are cheaper to surface fast.

🟡 Medium (some judgment)

  1. Skip CI on draft PRs: added if: github.event.pull_request.draft != true to 11 workflows (ci.yml × 4 jobs + dashboard-typecheck-strict + dashboard-bundle-size + dashboard-e2e + image-scan + module-conformance + forensic-tamper-check + dco-check + check-file-size + changelog-fragment-check + customer-app-coverage + doc-audit). Operator marks PR draft while WIP → CI silent until "Ready for review".
  2. dashboard-visual.yml → opt-in only: pull_request + push triggers REMOVED. Now workflow_dispatch + weekly Sun cron only. Was 3-5min × every dashboard PR; now ~0 cost during day-to-day + drift catcher weekly.
  3. image-scan.yml paths filter: only **/Dockerfile*, **/package*.json, **/go.{mod,sum}, image source dirs trigger. Docs/changelog don't.
  4. go-security.yml tiered cadence:
  5. PR / merge_group / 1st-of-month / workflow_dispatch → FULL (all 47 modules)
  6. Weekly Sun (not month-start) → CORE (8 modules: pure-art / relay-art / validator-art / oobi / gateway-art / octopus / hyperbridge-art / cross-module-rpc — patent-critical + most-changed)
  7. ~85% reduction on the weekly scan cost. Full sweep monthly catches drift in long-tail modules.

🔴 Radical (process / out-of-scope)

  1. Quarterly stack audit → workflow_dispatch only: cron removed; operator triggers when prepping release cuts (the natural cadence in practice).
  2. Self-hosted runner (out-of-scope for this PR): operator decision — if there's an idle Linux/Mac machine, moving heavy crons (go-security, image-scan, codeql) to self-hosted is "free" recurring. Documented as carry-forward.

Expected impact

Rough back-of-envelope before/after for a typical week of operator activity:

Trigger Before After
Daily crons (3) 21 runs/wk 3 runs/wk (-86%)
Weekly go-security 47×2 = 94 jobs CORE: 8×2 = 16 jobs (-83%); FULL 1×/month
Per-PR dashboard visual 1 run × all dashboard PRs 0 + weekly cron (-100% per-PR)
Docs-only commits → full CI 100% of commits 0% of docs-only commits (-40% volume)
Sequential push CI duplicates All run to completion Only latest runs (-30% on rapid waves)
Draft-PR CI Full pipeline on WIP 0 until Ready for review
Failed-module matrix kept All 46 others continued Cancelled (-50% on failed runs)
Quarterly audit 4 unsupervised/year 0 (operator-triggered)

Estimated total Actions reduction: 60-75% (depends on operator cadence; rapid waves benefit most from cancel-in-progress + draft-skip; docs-heavy weeks benefit from paths-ignore).

Consequences

Positive

  • Drastically lower recurring cost; aligns with operator's billing constraint.
  • workflow_dispatch: preserved everywhere — operator can force any "lite" workflow back on demand without code change.
  • No coverage gap on PR-touching-code (full Go matrix + full image scan still run on real PRs).
  • Catch-up cadence preserved: weekly cron catches drift even when no PRs land; monthly full sweep catches long-tail.

Negative

  • Long-tail Go modules (39 of 47) only get gosec/govulncheck monthly instead of weekly. Acceptable: most are stable infrastructure code touched <1×/month anyway. PR-trigger remains full.
  • dashboard-visual.yml regressions caught at most weekly instead of per-PR. Mitigation: operator runs workflow_dispatch before release cuts.
  • Daily-cron security scans (ztp-prem-sod-audit, ztp-prem-tier-b-obfuscation) catch breakage 6 days later in the worst case. Mitigation: PR-trigger keeps merge-time enforcement.

Alternatives considered

  • Self-hosted runners as primary: lower steady-state cost but requires hardware + maintenance + secrets management; held as carry-forward operator decision.
  • Disable workflows entirely: too aggressive; gives up security/quality posture in exchange for cost.
  • Pay-as-you-go without limit: deferred — billing is the primary constraint per memo project_checkpoint_2026_05_25_billing_pause.
  • PR consolidation as primary (fewer larger PRs): process change, not code; partially mitigated by cancel-in-progress already added in prior waves.

References

  • Checkpoint memo: project_checkpoint_2026_05_25_billing_pause
  • CI cleanup PR #1089 (added security-events: write + continue-on-error for Code Scanning gracedown)
  • Recovery memo: project_ci_cost_cut_radical_2026_05_25
  • Foundational: GitHub docs on workflow billing https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-actions

Carry-forward

  • Self-hosted runner for heavy crons — operator hardware decision
  • PR consolidation cadence — process / habit
  • Re-enable removed triggers when budget allows — pull_request + push on dashboard-visual.yml; daily-cron on stale + ztp-prem-* if operator wants tighter feedback