Repository navigation
[CI] Repair portable rolling ccache snapshots for Linux builds - #34163
Merged
Merged
Conversation
Repair reusable and minimal Linux ccache paths, scope rolling snapshots to build variants and toolchains, and preserve host/container access. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the Linux ccache snapshot updates.
No open issues 🎉
What changed in this PR
Repairs ccache snapshot handling for ONNX Runtime’s reusable Linux build workflow and nine cache-enabled minimal-build jobs, retaining ccache and existing compiler/build options.
Changes:
- Adds shared setup with workspace-relative archives, toolchain-scoped restore keys, and rolling snapshots published only after successful main builds.
- Routes native and container builds to the intended cache directory while preserving Python setup and supporting differing container UIDs.
- Adds regression and smoke tests for cache compatibility, publication policy, archive portability, and permissions, plus cache-contract documentation.
| File | Description |
|---|---|
tools/ci_build/test_linux_ccache.py |
Adds policy, namespace, portability, and native/container cache tests. |
.github/workflows/reusable_linux_build.yml |
Adopts shared setup, explicit container cache routing, and rolling saves. |
.github/workflows/linux_minimal_build.yml |
Updates all nine cache-enabled jobs to use portable snapshots. |
.github/actions/setup-linux-ccache/README.md |
Documents cache identity, publication rules, permissions, and smoke tests. |
.github/actions/setup-linux-ccache/action.yml |
Implements compatibility-scoped restore keys, cache environment setup, and shared access. |
Review effort: Balanced
Install the YAML parser used by the new workflow regression tests in both tooling-test CI environments. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Transfer a run-scoped compiler-cache artifact between fresh cold and warm jobs, verify compatible toolchains and meaningful warm hits, and publish measurements without PR cache writes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retain core cache regression tests and recorded CI performance evidence while removing the one-off workflow and its helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retain snapshots only for measured minimal variants with at least five minutes of cold compilation; preserve all build and test settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
eserscor
added a commit
that referenced
this pull request
Nov 13, 2026
…#34199) ### Description <!-- Describe your changes. --> Move the existing per-type `HasCudaEnvironment(is_float16 ? 530 : 0)` check to the start of `RunModelWithRandomInput`, before constructing the random generator, allocating token offsets, or generating inputs, weights, and biases. Unsupported environments now report `GTEST_SKIP()` before any `OpTester` is constructed. The shared guard covers `TestWithRandomData`, `TestWithRandomDataWithRBP`, and `TestWithRandomDataLargeSeq`. Supported GPU FP32/FP16/RBP/large-sequence dimensions, seeds, inputs, tolerances, reference comparisons, and minimum CUDA architectures are unchanged. The guard remains per invocation: lack of FP16 support does not prevent the mixed-type cases from exercising supported FP32 inputs. No CI, build optimization, cache, or image changes. **Validation — head `b8eda7e79f5c9787ae23670802a2179fc0a2418f`:** - **Linux ARM64 Debug CI: passed.** The [draft PR job](https://github.com/microsoft/onnxruntime/actions/runs/43470505744/job/132040829102) built and tested successfully on `ort-github-linux-16-arm64-eus2-ss`, the same CPU runner pool as the reference. Its provider-test command was `/onnxruntime_src/build/Debug/Debug/onnxruntime_provider_test "--gtest_output=xml:/onnxruntime_src/build/Debug/Debug/onnxruntime_provider_test.Debug.results.xml"` — passed; **7,024 tests selected**, including all three random-data cases, each explicitly reported skipped. This is not a zero-match pass. - `git diff --check` — passed. - `.\.venv\Scripts\clang-format.exe --dry-run --Werror onnxruntime\test\contrib_ops\packed_attention_op_test.cc` — passed with clang-format 20.1.8. - **Windows x64 CPU-only Debug:** in an isolated `.venv`, after `call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat"`, ran `.venv\Scripts\python.exe tools\ci_build\build.py --build_dir build\n --config Debug --update --build --parallel 12 --skip_submodule_sync --cmake_generator Ninja --target onnxruntime_provider_test --disable_memleak_checker` — passed. Build log confirms `packed_attention_op_test.cc.obj` recompiled and the executable relinked. Cache confirms Debug and CUDA OFF. - From `build\n\Debug`: `onnxruntime_provider_test.exe --gtest_filter=PackedAttentionTest.TestWithRandomData* --gtest_list_tests` — exit 0; listed exactly the three existing random-data tests. - From the same directory: `onnxruntime_provider_test.exe --gtest_filter=PackedAttentionTest.TestWithRandomData* --gtest_output=xml:C:\Users\erscor\.copilot\session-state\893b4970-454c-4884-9b9c-2c936909fed7\files\packed-attention-cpu-debug.xml` — exit 0; **3 selected, 3 skipped, 0 failures**. Local x64 XML reports 10 ms / 4 ms / 0 ms for random data / RBP / large sequence; gtest reports 16 ms suite time, 19 ms including global setup/teardown. These are separate x64 results, not the ARM64 performance comparison below. Local build notes: Visual Studio generation first hit Windows' 260-character MSBuild path limit, so validation used Ninja in this worktree. The initial default Debug executable compiled successfully and selected/skipped all three tests, but its experimental Windows memory-leak checker exited -1 at shutdown; even `--gtest_list_tests` alone reported shutdown leaks. The clean local validation above uses the repository-supported `--disable_memleak_checker` option locally only; Debug optimization settings are unchanged. No repository build settings were edited. CUDA hardware/runtime is unavailable locally. GPU execution was not performed locally; the original architecture expression and supported GPU path were inspected for unchanged behavior. CPU-only runs verify early gating, not GPU numerical correctness. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> The CUDA-only random-data tests previously generated large CPU fixtures before checking whether CUDA was available. **Measured Linux ARM64 Debug comparison**, using the completed [reference job](https://github.com/microsoft/onnxruntime/actions/runs/43458960142/job/132000464739) and successful [after-change draft PR job](https://github.com/microsoft/onnxruntime/actions/runs/43470505744/job/132040829102): | Test (`PackedAttentionTest.` prefix) | Before | After | Saved | | --- | ---: | ---: | ---: | | `TestWithRandomData` | 266,632 ms | 4 ms (skipped) | 266,628 ms | | `TestWithRandomDataWithRBP` | 133,346 ms | 3 ms (skipped) | 133,343 ms | | **Two reported slow tests** | **399,978 ms** | **7 ms** | **399,971 ms (~6m40s)** | | `TestWithRandomDataLargeSeq` | 10,943 ms | 0 ms reported (skipped) | 10,943 ms | | **All three helper callers** | **410,921 ms** | **7 ms** | **410,914 ms (~6m51s)** | The reported slow pair's runtime fell by **99.998%**, measured on the same ARM64 Debug CPU runner pool. Millisecond timings are rounded by gtest; 0 ms does not mean literally zero elapsed time. The whole provider-test CTest entry went from **2,112.21 s to 1,706.48 s**, an observed reduction of **405.73 s (19.21%)**. Treat that as descriptive, not a controlled whole-suite attribution: the reference selected **7,019** tests and the PR merge selected **7,024**, and these are different CI runs/commits. The pair alone accounted for 18.94% of the reference provider-test time; all three accounted for 19.45%. The supplied cold-run reference pair is consistent (266,686 + 133,286 ms); the supplied ARM64 Release reference pair was 2,333 + 1,170 ms. The additional ARM64 Release measurements are reported below; no GPU performance timings are claimed. Independent change based on `main`; does not modify or stack on #34163 or the ccache experiment. #### Cross-platform impact Combined gtest time for all three existing random-data tests (including the large-sequence case): | Configuration | Before | After | Saved | | --- | ---: | ---: | ---: | | Linux ARM64 Debug | 410,921 ms | 7 ms | 410,914 ms | | Linux x64 Debug + ASan | 56,309 ms | 6 ms | 56,303 ms | | macOS ARM64 Debug | 14,912 ms | 0 ms reported | 14,912 ms | | Windows x64 Release (two executions) | 6,625 ms | 2 ms | 6,623 ms | | Linux ARM64 Release | 3,598 ms | 1 ms | 3,597 ms | | Linux x64 Release | 2,442 ms | 0 ms reported | 2,442 ms | | Linux x64 Release / Python 3.14 | 2,451 ms | 0 ms reported | 2,451 ms | **Total measured aggregate test execution removed: 497,242 ms (8m17.242s)** across these jobs, not an equivalent decrease in elapsed PR/CI time. Every after-change occurrence explicitly reports skipped; all compared jobs succeeded. Windows runs the provider tests twice (regular and example-plugin executions), and both executions are included. Gtest rounds timings to milliseconds. Sources: Linux [before](https://github.com/microsoft/onnxruntime/actions/runs/43458960142) / [after](https://github.com/microsoft/onnxruntime/actions/runs/43470505744); macOS [before](https://github.com/microsoft/onnxruntime/actions/runs/43467069151/job/132028944327) / [after](https://github.com/microsoft/onnxruntime/actions/runs/43470505763/job/132040830226); Windows [before](https://github.com/microsoft/onnxruntime/actions/runs/43467069133/job/132028943543) / [after](https://github.com/microsoft/onnxruntime/actions/runs/43470505843/job/132040829952). Linux compares the earlier experiment reference with this PR; macOS/Windows use successful main runs at this branch's base commit, 6080d7c. These are separate runs, not controlled repeated benchmarks. #### Total workflow elapsed time: estimate versus observation **No net end-to-end workflow reduction was observed in these particular run pairs.** Using the first job's `started_at` through the final job's `completed_at` (excluding initial runner queue delay, but including later job scheduling gaps), GitHub reports: | Workflow | Before execution span | After execution span | Observed change | | --- | ---: | ---: | ---: | | Linux CI | 46m31s | 51m06s | 4m35s longer | | MacOS CI Pipeline | 86m42s | 94m44s | 8m02s longer | | Windows x64 Release | 33m54s | 34m04s | 10s longer | Including initial queue time, creation-to-final-job-completion was Linux 49m48s -> 54m33s, macOS 86m46s -> 94m49s, and Windows 43m14s -> 66m28s. In particular, the Windows after-run waited about 32m24s before its job started, versus 9m20s before; this is not test runtime. The fixture savings are directly visible in individual test timings, but build/configuration/cache state, scheduling, other tests, and different commits confound whole-workflow comparisons. The ARM64 Debug provider-test entry itself decreased by 405.73s (19.21%), while the entire ARM64 Debug job increased from 43m38s to 50m22s; other work more than offset the test savings in this pair. **All-else-equal critical-path estimate:** ARM64 Debug was the last-finishing Linux job in both runs. Removing its measured 410.914s of fixture work would shorten the reference Linux workflow execution span from 46m31s to about 39m40s: **approximately 6m51s, or 14.7%**, provided other timings remain fixed and no other job becomes the bottleneck. This is a modeled potential wall-clock saving, not an observed workflow speedup. Savings in parallel Linux jobs do not add to that critical-path reduction. The measured macOS CPU Debug job was not the final-finishing job (the ARM64 iPhone simulator was), so its 14.912s saving alone does not shorten that workflow's critical path. For the single-job Windows workflow, the isolated all-else-equal saving is about 6.623s (0.33% of baseline execution time). Builds that do not execute these tests receive no benefit. Supported CUDA paths still generate and exercise the original fixtures; no GPU computation speedup is claimed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tianlei Wu (tianleiwu)
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Repair portable, rolling Linux ccache snapshots without adopting sccache or changing compilers, PCH, build/security flags, or test coverage. Retain shared snapshots where cold compilation takes at least five minutes; leave shorter builds on their existing local ccache without GitHub snapshot restore/save overhead.
.cache/ccacheinstead of the reusable workflow's~/.cache/cachetypo or home-relative paths that depend on runner workspace depth.ccache-linux-v2namespace. Scope compatibility to build variant/dependency identity, architecture, and actual local Docker image ID or host compiler/ccache versions. Publish unique run-ID/attempt snapshots with only a compatibility-scoped restore prefix; no broad fallback./onnxruntime_src/.cache/ccachethrough its existing environment prefix while preserving caller Python setup. Limit cross-UID permissions and umask changes to compiler-cache contents.655926d690efbc406ba4ce5c14a4fe381315cbb6: remove both snapshot setup/restore and save from six minimal variants whose cold build phase was under five minutes: custom ops, type reduction, globally allowed types, minimal without optional features, extended minimal without optional features, and Android NNAPI. Keep snapshots for exceptions-disabled minimal, extended minimal, regular without optional features, and the unchanged reusable Linux callers. Existing local ccache tools/flags, VCPKG policy, tests, and other settings remain unchanged. The cutoff is a static policy based on cold measurements, not warm timings or a runtime threshold.requirements-dev.txt.Real GitHub cache-service cold/warm validation — November 13
Ran both existing workflows twice through normal branch-specific push triggers, preserving all six Linux CPU variants, all nine previously snapshot-enabled minimal variants, the full-build/test-data prerequisite, original compiler/PCH/security options and tests. The experiment was isolated on
eserscor-2026/11/13-branch-ccache-experiment, not added to this PR.All 32 jobs passed. All 15 cold compiler-cache restores missed. Every cold snapshot save was confirmed in job logs and the GitHub caches API before the second push. All 15 warm restores matched their corresponding cold publication exactly, with no toolchain/Docker-identity mismatch; all 15 warm saves were also confirmed. This exercised real pinned
actions/cacherestore/save, not artifact-based compiler-cache transport. The ordinarytest_dataartifact remained unchanged.Cold commit:
602870c80f0baac6e9c6bd1178dfd91a9f163dc6. Warm commit:bf9737d205e44946522a7e7b3d085f4cd98b3bcb, an intentionally empty commit. Both have identical source tree571f761d828096a281d5be672f1834ea4c90dd42. An experiment-only cache prefix kept these snapshots separate from production main keys. No cache deletion, custom maintenance, shared-image modification,actions:writegrant, manual rerun or GPU workflow expansion was used.Times below are cold → warm, in minutes:seconds. Build means the standalone Actions compilation-step wall time; starred native composite/Android rows use
build.pyCMake-build command to test-start/build-complete timestamps within a combined step. Whole job includes setup, cache operations, tests and post steps, but excludes queue/dependency waiting. Hit rates are among cacheable compiler calls, not all tool invocations. Every measured variant restored cold=miss/warm=hit, saved both times, and passed both times. The measurements precede the final six-variant snapshot-policy reduction.The unchanged full-ORT prerequisite has no compiler snapshot: its build step took 14:15 → 15:02 and whole job 16:11 → 17:27 (cold, warm). The minimal workflow was 90 seconds slower overall despite faster snapshot-enabled compilation: that prerequisite was 76 seconds slower, and warm dependent jobs waited 3:53–4:43 for runners versus 8–12 seconds cold. Compiler acceleration is not end-to-end workflow acceleration. This is one sequential pair, not a statistical benchmark; runner, Docker/setup, existing VCPKG-cache and test variability remain.
Why ARM64 Debug remains the long tail
Warm ARM64 Debug job: compilation fell from 17:54 to 3:54, but tests remained 35:18 cold / 35:17 warm, about 81% of the warm job. Ccache does not accelerate test execution; cache restore/save took roughly ten seconds each.
CTest already uses
--parallel 16, but the 7,019 provider cases execute inside one unshardedonnxruntime_provider_testprocess. That entry took 2,112.85 seconds cold / 2,112.21 seconds warm;onnxruntime_test_allfinished in 340.30 seconds warm. Eight provider cases account for 1,417 seconds, approximately 67% of its wall time.MatMulNBitscases total about 14m13s;Float16_4b_Accuracy4alone takes 4m48s and includes a large 369×6144×2048 regression case.PackedAttentionTestrandom-data cases take 6m40s combined, despite this being CPU-only.RunModelWithRandomInputgenerates inputs, weights and biases before itsHasCudaEnvironmentcheck. The two loops make 120 calls and generate approximately 212 million Gaussian weight values even when CUDA is unavailable. The same two tests take about 3.5 seconds combined in the measured ARM64 Release job.ReduceSum_batch_by_seq_by_128takes 3m24s across 90 large-shape cases;Gather_overflow_checktakes 1m49s and deliberately creates an output just over 2 GiB plus reference/output copies.This is reproducible Debug test/fixture cost, not a cache miss or a demonstrated runner stall. Moving CUDA availability checks before unused fixture generation and sharding the provider-test executable are separate potential follow-ups; this cache PR does not skip tests or change Debug optimization/coverage.
Validation
Final eligibility policy at
655926d690efbc406ba4ce5c14a4fe381315cbb6:wsl -d Ubuntu-24.04 --exec bash -c 'source /tmp/ort-ccache-ci-305f4061-venv/bin/activate && python3 -m unittest tools.ci_build.test_linux_ccache -v'— 7 tests run: 6 passed, 1 optional Docker skip.actionlint -shellcheck= -pyflakes= .github/workflows/reusable_linux_build.yml .github/workflows/linux_minimal_build.ymlwith actionlint 1.7.12, compared with the pre-cutoff baseline — no new diagnostics; the same 10 existing custom-runner-label diagnostics remain. Shellcheck and pyflakes disabled for this comparison.git diff --check— passed. Final policy committed and pushed; normal PR CI is triggered, not claimed complete here. The four successful experiment runs above validate the repair before the final eligibility-only reduction, not this new head's CI.Earlier repair validation retained as supporting evidence:
python -m pip install -r requirements-dev.txt— passed in clean Linux and Windows virtual environments, including PyYAML. Both tooling CI jobs and Python format passed after the dependency fix: https://github.com/microsoft/onnxruntime/actions/runs/43354291711 .python -m pytest -c .github/workflows/tooling_tests.pytest.ini --rootdir=. -v— 198 passed, 3 skipped, 206 subtests passed on the native Linux source copy before the eligibility reduction.ORT_CCACHE_DOCKER_IMAGE=ort-ccache-smoke:305f4061 python3 -m unittest tools.ci_build.test_linux_ccache -v— original 6 tests passed, no skips, covering differing workspace depths, external-action/direct-Docker environments, UIDs 1001/1002 and host access to container-created entries.python -m ruff check tools/ci_build/test_linux_ccache.pyandpython -m ruff format --check tools/ci_build/test_linux_ccache.py— passed on the earlier repair, including repository-pinned ruff 0.12.12 validation.Raw logs, exact per-job cache keys, compiler counters, configure/build/test/cache timing JSON and CSV were retained with the experiment report. The main-only publication policy is regression-tested; the experiment validates branch-scoped cache-service publication/consumption, not main-to-PR visibility after merge. No GPU performance claim is made, and unmeasured reusable callers retain their existing policy.
Motivation and Context
The reusable workflow saves a path that does not contain its compiler cache, while minimal builds can restore immutable historical archives containing relative traversals to an obsolete runner home/workspace layout. Static exact-match keys also prevent populated snapshots from rolling forward. Fix these root causes while preserving build behavior, and avoid spending shared snapshot resources on short builds that do not determine the workflow's long tail.
Independent fix from main, not stacked on draft #34109; that PR and its branch are unchanged. The app branch control only supported a dated kebab-case name with its automatic prefix (
eserscor-2026-11-12-linux-ccache-repair), not the requested slash-separated branch format.