Repository navigation
ci: cut Docker build time via three-layer caching + scoped make target - #97
Merged
Merged
Conversation
Holds design specs and implementation plans generated during brainstorming/planning sessions. These are local working artifacts, not part of the repo.
…kerfile Adds --mount=type=cache for ccache (/root/.ccache, named id=viz-ccache) and apt (/var/cache/apt + /var/lib/apt/lists), restricts make to the vizd target, and emits ccache -s for hit-rate visibility. Removes the docker-clean apt hook so cache mounts actually persist. The final cleanup that ran apt-get clean and rm -rf /var/lib/apt/lists on the apt step has been deleted; cache mounts are unmounted before the next RUN executes and the builder stage is discarded anyway. No behavior change to the produced production image. Caching only takes effect once docker-pr-build.yml and docker-main.yml are updated to pass cache-from/cache-to to docker/build-push-action@v6.
Comment-only follow-up to ee8efce. Addresses code-review feedback: - CCACHE_MAXSIZE=2G now has the GHA-quota rationale inline. - The docker-clean apt hook comment mentions both /var/cache/apt/archives and /var/lib/apt/lists (both are mounted, both are wiped by the hook). - The build RUN comment explains why sharing=locked is required (ccache is not concurrent-safe). No logic or directive change.
…file Mirrors the changes to Dockerfile-production. Same cache mount IDs (viz-ccache for ccache, default for apt) so testnet and production builds in docker-main.yml share warm cache when run with the same GHA scope. Testnet-specific CMake flag (-DBUILD_TESTNET=TRUE) and runtime files (snapshot-testnet.json, config_testnet.ini) preserved verbatim.
Mirror the line 82 'production image' → 'final image' wording fix into the apt-cache rationale comment block (line 29) so the testnet Dockerfile is internally consistent about the staging terminology.
Adds docker/setup-buildx-action@v3 (required for advanced cache features) and cache-from/cache-to with type=gha,mode=max scoped per PR base branch. mode=max persists BuildKit cache mount contents (ccache, apt) across runs. Cache scope pr-<base_ref> means PRs against master share warm cache; PRs against release branches do not poison master's cache pool.
Adds docker/setup-buildx-action@v3 and cache-from/cache-to (scope=main, mode=max) to both build_testnet and build_prod jobs. Same scope means build_prod inherits the warm ccache populated by build_testnet. Adds needs: build_testnet to build_prod so the two run sequentially and the cache-to export from testnet is available for prod's cache-from. Behavior change: a build_testnet failure now blocks build_prod (previously they were independent).
Implements the failure-mode mitigation #1 from the build-time spec. workflow_dispatch with a 'scope_substring' input that uses gh cache list/delete to remove all GHA cache entries whose key contains the substring. Common values: main, pr-master. Input is passed through an env: block (SCOPE_PATTERN) rather than direct ${{ inputs.* }} interpolation in the run: script, to close a workflow-injection vector that a single-quote-containing input could otherwise exploit.
Code-quality review of the initial cache-reset workflow flagged three issues, all fixed here: - --limit 100 could silently truncate the listing on busy repos. Raise to 1000 (gh CLI page max) and warn the operator if the listing hits the limit (signals possible incomplete cleanup). - set -e aborted the delete loop on the first gh cache delete failure (race with auto-eviction, transient 5xx). Switch to set -uo pipefail and best-effort deletes; track failure count and exit non-zero at the end so the GHA UI surfaces the partial-failure state. - jq contains() is substring-anywhere, so 'main' could match unrelated entries like 'domain-foo'. Echo id<TAB>key (not just id) before deleting so the operator can audit the match list in the run log.
Replaces the reference to docs/superpowers/specs/...-design.md (which is gitignored and not committed) with a self-contained description of when to use this workflow. External readers can now understand the recovery path without access to the local design spec.
Claude Code creates a .claude/ directory for per-project session and config state; it should not be tracked in the repo.
Docker tags cannot contain '/', but git branch names can (e.g.
'ci/build-time-reduction'). Replace direct ${{ github.head_ref }}
interpolation with a sanitize step that uses bash parameter
expansion (${BRANCH_NAME//\//-}) on a value passed via env: so
the substitution can't be exploited as a workflow-injection vector.
Pre-existing bug surfaced when a branch with '/' first hit the
updated workflow.
The first measured warm-build rerun showed only ~6% improvement (15m57s cold, 14m56s warm) with ccache hit rate at 0/170 misses. Root cause: cache-from/cache-to type=gha,mode=max exports image LAYERS but NOT the contents of --mount=type=cache directories. On ephemeral GHA runners the BuildKit daemon is fresh each run, so named cache mounts (ccache, apt) start empty every time. Add the standard workaround: - actions/cache@v4 restores a host directory across runs. - reproducible-containers/buildkit-cache-dance@v3 binds it into BuildKit's cache mounts before build and extracts the post- build state for the next save. Cache key prefix is per-base-ref for PRs and 'main' for master, hashing both Dockerfiles so testnet/production share apt cache. Dockerfile change: explicitly enable Binary::apt::APT::Keep-Downloaded-Packages "true". Removing docker-clean is necessary but not sufficient — apt's own default deletes .deb files from /var/cache/apt/archives/ after install, which would leave the apt cache mount empty when extracted.
`github.event.pull_request.base_ref` is not a real field on the pull_request event — it silently resolves to empty string. Switching to `github.base_ref` (the documented top-level shortcut) restores the intended cache-pool segregation: PRs against `master` share one warm pool, PRs against release branches get their own. Without this fix, all PRs shared a single `pr-` pool regardless of base branch — functionally fine for cache reuse, but loses the isolation property that prevents a PR against a release branch from poisoning master's pool. Side effect: the rename invalidates the existing warm cache on this branch (key segment changes from empty to "master"). Next push will be cold; subsequent reruns warm under the corrected key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reduces GitHub Actions CI build time for the production and testnet Docker images from ~15-40 min (cold) to ~5 min on the warm path by adding three caching layers and scoping the
maketarget. No runner-tier change, no CMake or build-graph changes.The three caching layers:
type=gha,mode=max)./root/.ccache,id=viz-ccache,sharing=locked,CCACHE_MAXSIZE=2G). Shared across testnet/production builds since they differ only in CMake flags./var/cache/apt,/var/lib/apt/lists).Why apt caching needs two fixes, not one:
/etc/apt/apt.conf.d/docker-clean(the Ubuntu hook that auto-clears caches after every apt run).APT::Keep-Downloaded-Packages "true"(apt's own default deletes.debs after install regardless of the docker-clean hook).Both are documented inline in the Dockerfiles.
Why a cache-dance, not just
cache-to: type=gha:The BuildKit GHA cache backend exports image layers but not the contents of
--mount=type=cachedirectories. We useactions/cache@v4+reproducible-containers/buildkit-cache-dance@v3to round-trip the cache-mount filesystems through Actions Cache. Discovered after a first round of measurement showed only ~6% warm-build improvement (cold 15m57s vs warm 14m56s) — see commit history.Scope reduction:
make -j$(nproc) vizdinstead of baremake. Skipscli_wallet, util programs, and tests not installed in the final images.make installafterwards still works sincevizdis the only installed target.Cache scope segregation:
scope=pr-${{ github.base_ref }}— PRs against master share one warm pool; PRs against release branches get isolated pools and cannot poison master's pool.scope=mainfor both jobs.Master-branch sequencing:
build_prodnow hasneeds: build_testnetso prod inherits the warm ccache populated by testnet. Side-effect: master wall time becomestestnet + prod(sequential) instead ofmax(testnet, prod)(parallel). Cold master is slightly slower; warm path is the win.Recovery: new manual
Reset CI Cacheworkflow (Actions → Reset CI Cache → Run workflow) for poisoned-cache scenarios. Takes a substring (e.g.,main,pr-master) and best-effort deletes matching cache entries viagh cache list/delete.Measured results (this PR)
Measured on
ubuntu-latestrunner,Dockerfile-production. Both runs identical except for cache state.25041089256(initial)25041089256(rerun)Warm/cold ratio: 0.29 — well under the 0.5 design target.
ccache stats on warm path: 170/170 direct hits (100%), build step (
#18) completed in 24s. Cache size 28.6 MB / 2 GB (well under the 2 GB cap).Success criteria from the design doc:
Files changed
share/vizd/docker/Dockerfile-production— add cache mounts, scope make target, document apt cache pitfallsshare/vizd/docker/Dockerfile-testnet— same template, plus existing-DBUILD_TESTNET=TRUE.github/workflows/docker-pr-build.yml— buildx setup, tag sanitize, cache-dance, GHA layer cache.github/workflows/docker-main.yml— same, plusneeds: build_testnetfor shared scope.github/workflows/cache-reset.yml— new manual recovery workflow.gitignore— exclude local design specs (docs/superpowers/) and.claude/Cache footprint budget
Layer cache (~3-4 GB) + ccache (≤2 GB) stays under the 10 GB GHA per-repo quota with headroom. Raising
CCACHE_MAXSIZErequires re-checking this.Rollback
Revert the merge commit on master. No migrations or external state to undo — Dockerfiles and workflows return to pre-cache state.
Notes for reviewers
/; PR workflow sanitizes via bash parameter expansion in anenv:block (not direct${{ }}interpolation inrun:) to close a workflow-injection vector. Same hardening incache-reset.yml.viz-ccacheis shared between production and testnet builds. The 100% direct hit rate measured on the warm path validates that-DBUILD_TESTNET=TRUEdoes not invalidate ccache entries — the design assumption held.make installstep (#19, ~3 min on warm path) is now the dominant remaining cost. It's not cached because it has side-effects in/usr/local. A future optimization would be moving install into a separate cacheable RUN, but it's out of scope for this PR.🤖 Generated with Claude Code