Skip to content

ci: cut Docker build time via three-layer caching + scoped make target - #97

Merged
On1x merged 14 commits into
masterfrom
ci/build-time-reduction
Apr 28, 2026
Merged

On1x merged 14 commits into
masterfrom
ci/build-time-reduction

Conversation

@chiliec

@chiliec chiliec commented Apr 28, 2026 •

Copy link
Copy Markdown
Member

Summary

Reduces GitHub Actions CI build time for the production and testnet Docker images from ~15-40 min (cold) to ~5 min on the warm path by adding three caching layers and scoping the make target. No runner-tier change, no CMake or build-graph changes.

The three caching layers:

  • GHA-backed Docker layer cache (type=gha,mode=max).
  • BuildKit ccache mount for compiled C++ objects (/root/.ccache, id=viz-ccache,sharing=locked, CCACHE_MAXSIZE=2G). Shared across testnet/production builds since they differ only in CMake flags.
  • BuildKit apt mount for package archives (/var/cache/apt, /var/lib/apt/lists).

Why apt caching needs two fixes, not one:

  • Remove /etc/apt/apt.conf.d/docker-clean (the Ubuntu hook that auto-clears caches after every apt run).
  • Set APT::Keep-Downloaded-Packages "true" (apt's own default deletes .debs after install regardless of the docker-clean hook).

Both are documented inline in the Dockerfiles.

Why a cache-dance, not just cache-to: type=gha:
The BuildKit GHA cache backend exports image layers but not the contents of --mount=type=cache directories. We use actions/cache@v4 + reproducible-containers/buildkit-cache-dance@v3 to round-trip the cache-mount filesystems through Actions Cache. Discovered after a first round of measurement showed only ~6% warm-build improvement (cold 15m57s vs warm 14m56s) — see commit history.

Scope reduction: make -j$(nproc) vizd instead of bare make. Skips cli_wallet, util programs, and tests not installed in the final images. make install afterwards still works since vizd is the only installed target.

Cache scope segregation:

  • PRs: scope=pr-${{ github.base_ref }} — PRs against master share one warm pool; PRs against release branches get isolated pools and cannot poison master's pool.
  • Master: scope=main for both jobs.

Master-branch sequencing: build_prod now has needs: build_testnet so prod inherits the warm ccache populated by testnet. Side-effect: master wall time becomes testnet + prod (sequential) instead of max(testnet, prod) (parallel). Cold master is slightly slower; warm path is the win.

Recovery: new manual Reset CI Cache workflow (Actions → Reset CI Cache → Run workflow) for poisoned-cache scenarios. Takes a substring (e.g., main, pr-master) and best-effort deletes matching cache entries via gh cache list/delete.

Measured results (this PR)

Measured on ubuntu-latest runner, Dockerfile-production. Both runs identical except for cache state.

Run Type Wall time Notes
25041089256 (initial) Cold 16m38s Empty layer cache + empty cache-mount Actions Cache
25041089256 (rerun) Warm 4m50s Layer cache hit + cache-mount restored from prior run

Warm/cold ratio: 0.29 — well under the 0.5 design target.

ccache stats on warm path: 170/170 direct hits (100%), build step (#18) completed in 24s. Cache size 28.6 MB / 2 GB (well under the 2 GB cap).

Success criteria from the design doc:

  • Cold ≤ 22 min: 16m38s ✓
  • Warm ≤ 6 min: 4m50s ✓
  • Warm/cold ratio ≤ 0.5: 0.29 ✓

Files changed

  • share/vizd/docker/Dockerfile-production — add cache mounts, scope make target, document apt cache pitfalls
  • share/vizd/docker/Dockerfile-testnet — same template, plus existing -DBUILD_TESTNET=TRUE
  • .github/workflows/docker-pr-build.yml — buildx setup, tag sanitize, cache-dance, GHA layer cache
  • .github/workflows/docker-main.yml — same, plus needs: build_testnet for shared scope
  • .github/workflows/cache-reset.yml — new manual recovery workflow
  • .gitignore — exclude local design specs (docs/superpowers/) and .claude/

Cache footprint budget

Layer cache (~3-4 GB) + ccache (≤2 GB) stays under the 10 GB GHA per-repo quota with headroom. Raising CCACHE_MAXSIZE requires re-checking this.

Rollback

Revert the merge commit on master. No migrations or external state to undo — Dockerfiles and workflows return to pre-cache state.

Notes for reviewers

  • Docker tags cannot contain /; PR workflow sanitizes via bash parameter expansion in an env: block (not direct ${{ }} interpolation in run:) to close a workflow-injection vector. Same hardening in cache-reset.yml.
  • viz-ccache is shared between production and testnet builds. The 100% direct hit rate measured on the warm path validates that -DBUILD_TESTNET=TRUE does not invalidate ccache entries — the design assumption held.
  • The make install step (#19, ~3 min on warm path) is now the dominant remaining cost. It's not cached because it has side-effects in /usr/local. A future optimization would be moving install into a separate cacheable RUN, but it's out of scope for this PR.

🤖 Generated with Claude Code

chiliec added 14 commits April 28, 2026 00:36
Holds design specs and implementation plans generated during
brainstorming/planning sessions. These are local working artifacts,
not part of the repo.
…kerfile

Adds --mount=type=cache for ccache (/root/.ccache, named id=viz-ccache)
and apt (/var/cache/apt + /var/lib/apt/lists), restricts make to the
vizd target, and emits ccache -s for hit-rate visibility.

Removes the docker-clean apt hook so cache mounts actually persist.
The final cleanup that ran apt-get clean and rm -rf /var/lib/apt/lists
on the apt step has been deleted; cache mounts are unmounted before
the next RUN executes and the builder stage is discarded anyway.

No behavior change to the produced production image. Caching only
takes effect once docker-pr-build.yml and docker-main.yml are updated
to pass cache-from/cache-to to docker/build-push-action@v6.
Comment-only follow-up to ee8efce. Addresses code-review feedback:

- CCACHE_MAXSIZE=2G now has the GHA-quota rationale inline.
- The docker-clean apt hook comment mentions both /var/cache/apt/archives
  and /var/lib/apt/lists (both are mounted, both are wiped by the hook).
- The build RUN comment explains why sharing=locked is required (ccache
  is not concurrent-safe).

No logic or directive change.
…file

Mirrors the changes to Dockerfile-production. Same cache mount IDs
(viz-ccache for ccache, default for apt) so testnet and production
builds in docker-main.yml share warm cache when run with the same
GHA scope.

Testnet-specific CMake flag (-DBUILD_TESTNET=TRUE) and runtime files
(snapshot-testnet.json, config_testnet.ini) preserved verbatim.
Mirror the line 82 'production image' → 'final image' wording fix
into the apt-cache rationale comment block (line 29) so the testnet
Dockerfile is internally consistent about the staging terminology.
Adds docker/setup-buildx-action@v3 (required for advanced cache
features) and cache-from/cache-to with type=gha,mode=max scoped per
PR base branch. mode=max persists BuildKit cache mount contents
(ccache, apt) across runs.

Cache scope pr-<base_ref> means PRs against master share warm cache;
PRs against release branches do not poison master's cache pool.
Adds docker/setup-buildx-action@v3 and cache-from/cache-to (scope=main,
mode=max) to both build_testnet and build_prod jobs. Same scope means
build_prod inherits the warm ccache populated by build_testnet.

Adds needs: build_testnet to build_prod so the two run sequentially
and the cache-to export from testnet is available for prod's
cache-from. Behavior change: a build_testnet failure now blocks
build_prod (previously they were independent).
Implements the failure-mode mitigation #1 from the build-time spec.
workflow_dispatch with a 'scope_substring' input that uses gh cache
list/delete to remove all GHA cache entries whose key contains the
substring. Common values: main, pr-master.

Input is passed through an env: block (SCOPE_PATTERN) rather than
direct ${{ inputs.* }} interpolation in the run: script, to close
a workflow-injection vector that a single-quote-containing input
could otherwise exploit.
Code-quality review of the initial cache-reset workflow flagged three
issues, all fixed here:

- --limit 100 could silently truncate the listing on busy repos. Raise
  to 1000 (gh CLI page max) and warn the operator if the listing hits
  the limit (signals possible incomplete cleanup).
- set -e aborted the delete loop on the first gh cache delete failure
  (race with auto-eviction, transient 5xx). Switch to set -uo pipefail
  and best-effort deletes; track failure count and exit non-zero at
  the end so the GHA UI surfaces the partial-failure state.
- jq contains() is substring-anywhere, so 'main' could match unrelated
  entries like 'domain-foo'. Echo id<TAB>key (not just id) before
  deleting so the operator can audit the match list in the run log.
Replaces the reference to docs/superpowers/specs/...-design.md (which
is gitignored and not committed) with a self-contained description of
when to use this workflow. External readers can now understand the
recovery path without access to the local design spec.
Claude Code creates a .claude/ directory for per-project session and
config state; it should not be tracked in the repo.
Docker tags cannot contain '/', but git branch names can (e.g.
'ci/build-time-reduction'). Replace direct ${{ github.head_ref }}
interpolation with a sanitize step that uses bash parameter
expansion (${BRANCH_NAME//\//-}) on a value passed via env: so
the substitution can't be exploited as a workflow-injection vector.

Pre-existing bug surfaced when a branch with '/' first hit the
updated workflow.
The first measured warm-build rerun showed only ~6% improvement
(15m57s cold, 14m56s warm) with ccache hit rate at 0/170 misses.
Root cause: cache-from/cache-to type=gha,mode=max exports image
LAYERS but NOT the contents of --mount=type=cache directories.
On ephemeral GHA runners the BuildKit daemon is fresh each run,
so named cache mounts (ccache, apt) start empty every time.

Add the standard workaround:
- actions/cache@v4 restores a host directory across runs.
- reproducible-containers/buildkit-cache-dance@v3 binds it into
  BuildKit's cache mounts before build and extracts the post-
  build state for the next save.

Cache key prefix is per-base-ref for PRs and 'main' for master,
hashing both Dockerfiles so testnet/production share apt cache.

Dockerfile change: explicitly enable
Binary::apt::APT::Keep-Downloaded-Packages "true". Removing
docker-clean is necessary but not sufficient — apt's own default
deletes .deb files from /var/cache/apt/archives/ after install,
which would leave the apt cache mount empty when extracted.
`github.event.pull_request.base_ref` is not a real field on the
pull_request event — it silently resolves to empty string. Switching
to `github.base_ref` (the documented top-level shortcut) restores the
intended cache-pool segregation: PRs against `master` share one warm
pool, PRs against release branches get their own.

Without this fix, all PRs shared a single `pr-` pool regardless of
base branch — functionally fine for cache reuse, but loses the
isolation property that prevents a PR against a release branch from
poisoning master's pool.

Side effect: the rename invalidates the existing warm cache on this
branch (key segment changes from empty to "master"). Next push will
be cold; subsequent reruns warm under the corrected key.
@chiliec
chiliec marked this pull request as ready for review April 28, 2026 08:26
@On1x
On1x merged commit 27fa08b into master Apr 28, 2026
2 checks passed
@chiliec
chiliec deleted the ci/build-time-reduction branch August 4, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants