Skip to content

Self-hosted runner setup runbook

Per ADR-0098 — opt-in runner pattern. Heavy workflows (go-security / image-scan / codeql / ztp-prem-* / dashboard-visual) migrate to your self-hosted box when this is wired; everything else stays on GitHub-hosted ephemeral runners. When the runner is offline OR the repo variable is unset, all workflows transparently fall back to ubuntu-latest (zero-disruption pattern).

Why

Driver: GitHub Actions Linux billing was the constraint. ADR-0097 cut ~60-75% of recurring cost; this runbook (ADR-0098) brings the heavy crons down to ~zero recurring spend by hosting them on operator hardware.

Pre-requisites

  • One Linux machine (or macOS / Windows — workflows assume Linux in actions/setup-go cache paths; macOS works for everything except the securego/gosec Docker action, which is Linux-only)
  • Recommended specs: 4+ cores, 8 GiB RAM, 50 GiB disk, stable IP
  • outbound HTTPS to *.actions.githubusercontent.com
  • Tools installed on the host (matches ubuntu-latest runner image):
  • docker (Docker Engine 24+) — gosec + image-scan + codeql + the dashboard-e2e services: postgres:16-alpine all use containers. The runner user must be in the docker group (see Step 1).
  • git, curl, jq, unzip, tar, python3 ≥ 3.10, bash ≥ 4
  • gh CLI (optional — only for ad-hoc operator debugging)
  • build-essential + pkg-config — required so setup-go@v5 builds cgo-using packages (e.g. github.com/miekg/pkcs11). Without gcc on PATH, Go defaults CGO_ENABLED=0 and cgo packages silently expose no symbols (vet: undefined: pkcs11.Ctx).
  • libpkcs11-helper1-dev — header for the same pkcs11 package.
  • Go is auto-installed per-job by actions/setup-go@v5 from each module's go.mod; no system Go needed.

Step 1 — Provision the host

Any Ubuntu 24.04 / Debian 12 box works. Quick check:

docker --version              # Docker Engine 24+
git --version                 # any 2.x
jq --version                  # any 1.x
python3 --version             # ≥ 3.10
free -h                       # ≥ 8 GiB available
df -h /                       # ≥ 50 GiB free

If Docker isn't installed:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker "$USER"
newgrp docker
docker run --rm hello-world   # smoke

Install the remaining system deps in one shot:

sudo apt-get update
sudo apt-get install -y \
  git curl jq python3 ca-certificates gnupg unzip tar \
  build-essential pkg-config libpkcs11-helper1-dev

Why build-essential? setup-go@v5 won't enable cgo unless a C compiler is on PATH. cgo-using Go packages (e.g. github.com/miekg/pkcs11 in pkg/octopus/common/hsm) silently degrade to empty exports when CGO is off, and go vet then fails with undefined: pkcs11.Ctx. Surfaced when the octopus-common-log coverage gate first ran on the self-hosted arm64 runner (PR #1093).

Step 2 — Register the runner with GitHub

  1. In the repo on GitHub: Settings → Actions → Runners → New self-hosted runner.
  2. Pick Linux x64.
  3. GitHub shows a registration token and the exact commands. Run them on your host, but add the custom label heavy in the config.sh step (the shell script GitHub's installer drops at ~/actions-runner/config.sh):
# Inside ~/actions-runner (created by GitHub's instructions):
./config.sh --url https://github.com/<owner>/<repo> \
            --token <REGISTRATION_TOKEN> \
            --labels "heavy" \
            --name "$(hostname)-heavy" \
            --work _work \
            --unattended

The runner now reports labels self-hosted, Linux, X64, heavy.

  1. Install it as a systemd service so it survives reboots:
sudo ./svc.sh install "$USER"
sudo ./svc.sh start
sudo ./svc.sh status
  1. Confirm it shows green online in Settings → Actions → Runners.

Step 3 — Flip the repo variable

This is the switch that moves heavy workflows from GitHub-hosted to your runner. Without this set, workflows stay on ubuntu-latest.

In the repo on GitHub: Settings → Secrets and variables → Actions → Variables → New repository variable:

Field Value
Name HEAVY_RUNNER
Value ["self-hosted", "Linux", "X64", "heavy"]

The JSON-array format matches what fromJSON expects in the workflow files. Do not wrap it in extra quotes.

To verify, kick off a manual workflow_dispatch run:

gh workflow run go-security.yml
gh run watch

The job log shows Runner name: <hostname>-heavy instead of Runner Image: ubuntu-24.04. Cost confirmed flipped.

Step 4 — Verify zero-disruption fallback

Test that disabling the runner does not break the workflows.

# Stop the runner
sudo ./svc.sh stop

# Trigger any heavy workflow manually
gh workflow run go-security.yml

# It should land on GitHub-hosted ubuntu-latest after the queue
# timeout (~1min), NOT fail. The fork-PR-safety condition forces
# GitHub-hosted in that fallback path.

If runs HANG instead of falling back, unset HEAVY_RUNNER:

In Settings → Variables → HEAVY_RUNNER → Delete. All heavy workflows return to GitHub-hosted immediately.

Restart the runner when you're ready.

Step 5 — Maintain

Update the runner

GitHub releases new runner versions monthly. Update with:

sudo ./svc.sh stop
# Re-download the latest runner package per GitHub Settings → Runners
# instructions (the URL changes; check the page).
sudo ./svc.sh start

Disk hygiene

The runner caches Go modules, Docker layers, and npm. Clean periodically:

# In runner home
rm -rf _work/_actions
docker system prune -af

Add to weekly cron:

0 6 * * 0  cd /home/runner/actions-runner && rm -rf _work/_actions && docker system prune -af

Monitor health

sudo systemctl status actions.runner.*  # systemd unit name varies by GitHub
journalctl -u actions.runner.* -f       # tail logs

If the runner reports offline >5min, the heavy workflows queue up. After ~10min queue, GitHub typically retries on the same runner (no auto-fallback to GitHub-hosted — operator must intervene by unsetting HEAVY_RUNNER or fixing the runner).

Scaling up: multiple instances + host tuning

Steps 1-5 set up one runner on a modest box. A powerful host (many cores, lots of RAM) is wasted running a single instance — a runner processes one job at a time. Run N instances to use the box, and tune the host so parallel CI jobs don't thrash.

How many instances? (right-sizing)

A runner instance does not reserve CPU/RAM — it only gates concurrency (1 job per instance). N instances = up to N concurrent jobs sharing the whole box via the OS scheduler. Oversubscription (more instances than the box can run at once) causes thrash → slow + flaky jobs (timeouts, OOM). A lone job already gets the whole machine; extra instances only help when several jobs land at once.

Rule of thumb: ~1 instance per 2-3 vCPUs, dialed down on boxes with tight RAM/disk or weak CPUs. Reliability beats raw parallelism for CI — the matrix wall-clock is gated by the slowest job, so a flaky oversubscribed box is a long pole.

Reference fleet (2026-06):

Box CPU RAM Instances Role
cpu-gamer i7-8700 (12t) 62 GiB 4 primary
nuc-intel i7-8809G (8t) 31 GiB 2 secondary
nuc-gigabyte i7-4500U (4t ULV) 15 GiB 1 backup
tlsstress-computer-2 Ryzen 5 4600G (6c/12t) 30 GiB 6 heavy — temporary (reverts to NGFW dataplane)

⚠️ tlsstress-computer-2 is a temporary loan. Its permanent role is firewall/NGFW load testing — it normally runs the tuned cpu-partitioning profile (isolcpus=1-5,7-11 → only 2 housekeeping threads schedulable). It was converted to a heavy runner on 2026-06-17 by switching to throughput-performance + stripping isolcpus/hugepages from GRUB (frees all 12 threads). The original tuning is backed up on the box at ~/runner-tuning-backup/ with a one-shot revert-to-dataplane.sh. To return it to dataplane duty: run that script + sudo reboot, then uninstall the 6 runner services and deregister via gh api -X DELETE repos/<owner>/<repo>/actions/runners/<id>.

Tiered routing: fast (strong-only) vs heavy (full pool)

Adding weak boxes to a flat heavy pool has limited upside — a heavy job (image-scan, dashboard-e2e/visual) landing on a weak box becomes the matrix long-pole. Split work by weight:

  • Strong boxes carry an extra fast label; repo variable FAST_RUNNER = ["self-hosted","Linux","X64","fast"].
  • The heaviest workflows (image-scan, dashboard-e2e, dashboard-visual) use a fast-first fallback chain — strong boxes only:

runs-on: ${{ github.event.pull_request.head.repo.fork && 'ubuntu-latest' || fromJSON(vars.FAST_RUNNER || vars.HEAVY_RUNNER || '"ubuntu-latest"') }}
Chain: FAST_RUNNER if set → else HEAVY_RUNNER → else ubuntu-latest (zero-disruption preserved; fork PRs stay GitHub-hosted). - Light, high-count matrices (go-security's ~47-module gosec/ govulncheck) stay on HEAVY_RUNNER (full pool) — that's where weak boxes earn their keep: many short jobs, more slots = faster drain. - codeql stays on ubuntu-latest by design (memory-bound; self-hosting doesn't help).

Add the fast label to a registered runner without reconfiguring it:

echo '{"labels":["fast"]}' | gh api --method POST \
  repos/<owner>/<repo>/actions/runners/<id>/labels --input -

Add N instances

Each instance is its own directory + systemd service; reuse the one downloaded runner tarball. Mint a fresh registration token per instance (gh api -X POST repos/<owner>/<repo>/actions/runners/registration-token --jq .token):

URL=https://github.com/<owner>/<repo>
TARBALL=$(ls ~/actions-runner/actions-runner-linux-x64-*.tar.gz | head -1)
for i in $(seq 1 N); do
  D=~/runners/$(hostname)-$i; rm -rf "$D"; mkdir -p "$D"; tar xzf "$TARBALL" -C "$D"
  ( cd "$D"
    ./config.sh --unattended --replace --url "$URL" --token "<FRESH_TOKEN_$i>" \
        --name "$(hostname)-$i" --labels "heavy" --work _work
    sudo ./svc.sh install "$USER" && sudo ./svc.sh start )
done

Right-size DOWN (remove instances)

# On the box, per instance to remove:
( cd ~/runners/<name>-<i> && sudo ./svc.sh stop && sudo ./svc.sh uninstall )
sudo rm -rf ~/runners/<name>-<i>   # sudo: job _work holds root-owned docker files
Then deregister GitHub-side — after it goes idle (a runner mid-job returns HTTP 422 "currently running a job and cannot be deleted"; wait for it to finish/timeout):

gh api -X DELETE repos/<owner>/<repo>/actions/runners/<runner_id>

Host tuning

Drain first so a docker restart doesn't corrupt a running job's containers, then restart the runner services after:

sudo systemctl stop 'actions.runner.*'    # ... apply tuning ... then:
sudo systemctl start 'actions.runner.*'

1. Disk — extend root into free LVM extents (Ubuntu installs often leave the root LV far smaller than the disk):

sudo lvextend -l +100%FREE -r "$(findmnt -no SOURCE /)"

⚠️ +100%FREE consumes all free VG extents into root. Intended for a dedicated CI box (more space for docker layers / _work). If you reserve VG space for other LVs/snapshots, extend by a fixed size instead (-L +200G). LV shrink is risky — size up deliberately.

2. sysctls — /etc/sysctl.d/99-ci-runner.conf then sudo sysctl --system:

net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr
net.ipv4.ip_local_port_range=1024 65535
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=512

3. Docker — /etc/docker/daemon.json ⚠️ critical when the runner LAN is 172.17.x (our VLAN 30): Docker's default docker0 bridge is 172.17.0.0/16, which overlaps the runner LAN → container networking and routes black-hole. Move docker off 172.17, then sudo systemctl restart docker:

{
  "bip": "10.200.0.1/24",
  "default-address-pools": [
    {"base": "10.201.0.0/16", "size": 24},
    {"base": "10.202.0.0/16", "size": 24}
  ],
  "max-concurrent-downloads": 8,
  "max-concurrent-uploads": 8,
  "log-driver": "json-file",
  "log-opts": {"max-size": "10m", "max-file": "3"},
  "builder": {"gc": {"enabled": true, "defaultKeepStorage": "10GB"}}
}

4. CPU governor → performance (persistent systemd unit):

sudo tee /etc/systemd/system/cpu-governor.service >/dev/null <<'EOF'
[Unit]
Description=Set CPU governor to performance for CI
After=multi-user.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c 'for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > "$g"; done'
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable --now cpu-governor.service

5. Weekly docker prune — /etc/cron.weekly/docker-prune (chmod +x):

#!/bin/sh
docker system prune -af --filter "until=168h" >/dev/null 2>&1

Security note on multi-instance boxes

Persistent runners reuse _work across jobs (not ephemeral). Keep the box single-repo, run under a dedicated unprivileged user, and rely on the weekly prune + the fork-PR guardrail below. For full per-job isolation you'd need ephemeral runners + an external re-registration loop (out of scope here).

Security guardrails

The workflow runs-on: clause is:

runs-on: ${{ github.event.pull_request.head.repo.fork && 'ubuntu-latest' || fromJSON(vars.HEAVY_RUNNER || '"ubuntu-latest"') }}

This guarantees:

  1. Fork PRs ALWAYS use GitHub-hosted ephemeral runners. Untrusted code from external contributors never touches the operator's self-hosted machine.
  2. HEAVY_RUNNER unset → ubuntu-latest. Zero-disruption deploy of this PR — workflows keep working until operator wires Step 3.
  3. Runner offline → GitHub queues — does NOT fall back. Operator monitors via Step 5.
  • Run the runner under a dedicated user account (e.g. runner) with no sudo, no SSH from outside.
  • Keep the runner directory on an encrypted disk (LUKS).
  • Restrict outbound network on the host to GitHub IPs only (see https://api.github.com/meta).
  • Do NOT add the runner to multiple repos — separation of concerns; if any one workflow gets compromised, blast radius stays small.
  • Run inside a VM or LXC container — easy reset if something goes wrong.

Workflows that DO use self-hosted (when HEAVY_RUNNER is set)

Workflow Heavy? Why on self-hosted
go-security.yml yes — 47-module matrix gosec + govulncheck per module ≈ 2-3min × 47 = ~2h on GitHub-hosted
image-scan.yml yes — 3 Docker builds + trivy image builds are bandwidth + disk-heavy
codeql.yml yes — JS/TS deep scan CodeQL is single-threaded + memory-bound
ztp-prem-tier-b-obfuscation.yml yes — garble + multi-module obfuscation weekly cron after ADR-0097
ztp-prem-sod-audit.yml yes — full-repo audit weekly cron after ADR-0097
dashboard-visual.yml yes — playwright + screenshots weekly cron after ADR-0097

Workflows that STAY on GitHub-hosted (always)

Workflow Why not migrated
ci.yml Primary PR gate — needs fast feedback + ephemeral isolation; runs constantly
secret-scan.yml Tiny + security-sensitive — gitleaks should run in a sterile env
dco-check.yml / changelog-fragment-check.yml / check-file-size.yml Tiny — moving them adds queueing overhead with no cost win
release.yml / tag-signature-verify.yml Signing & release events — sterile env mandatory
forensic-tamper-check.yml Reads FORENSIC_FINGERPRINTS secret — sterile env
doc-audit.yml / module-conformance.yml Tiny, PR-trigger only

Cost expectation

Scenario Monthly cost
No self-hosted (current) Whatever the heavy crons + PR runs total per ADR-0097 (60-75% lower than pre-cost-cut)
Self-hosted + everything ADR-0097 ~zero recurring cost on Linux Actions billing. The remaining GitHub-hosted minutes are fast PR gates (ci.yml + tiny checks) only.

Electricity + hardware are off-books — operator's call whether the trade is worth it.

Rollback

If the runner becomes a problem:

# Stop + uninstall service
sudo ./svc.sh stop
sudo ./svc.sh uninstall

# Unset the repo variable in Settings → Variables → HEAVY_RUNNER → Delete

# Remove the runner from GitHub Settings → Actions → Runners → click
# the runner → Remove

All workflows resume on ubuntu-latest immediately. No code change.

References

  • ADR-0097 — CI cost-cut radical (preceded this; got 60-75% reduction without hardware)
  • ADR-0098 — Self-hosted runner opt-in pattern (this runbook's source of truth)
  • GitHub docs — https://docs.github.com/en/actions/hosting-your-own-runners
  • GitHub security hardening — https://docs.github.com/en/actions/hosting-your-own-runners/security-hardening-for-self-hosted-runners