ci(windows): add Windows MSVC CI jobs

[pimlock]

Summary

Run the supported Windows MSVC lint and test workload natively on x64 and ARM64 for pull requests and merge queues. Keep each architecture's Cargo and compiler caches warm on main, then build both release binaries without publishing them.

Related Issue

No issue required: localized CI infrastructure work following #2496.

Changes

• expose architecture-specific Windows lint and nextest entry points from tasks/windows.toml
• run them on windows-2025 (x64) and windows-11-arm (ARM64) for pull-request mirrors and merge queues
• run the same per-architecture workload on main and manual dispatches to seed the matching Cargo target and sccache namespaces
• build openshell-gateway.exe and openshell.exe after the cache-seed job succeeds on main or a manual dispatch
• use the official prebuilt Z3 release instead of compiling Z3 from source on Windows
• run Windows Rust tests with the same nextest CI profile and server test-support feature set used by main
• cache Cargo target artifacts and use the GitHub Actions sccache backend
• validate and build ARM64 natively on the hosted Windows ARM runner
• update the Windows build architecture notes and maintenance skill
• use enterprise-approved pinned revisions of jdx/mise-action and dtolnay/rust-toolchain

Why prebuilt Z3 on Windows

The previous bundled mode built Z3's C++ sources locally through CMake and MSBuild. That made clean Windows jobs substantially heavier and added another native build configuration, especially for x64-to-ARM64 cross-builds, without changing how OpenShell uses Z3. The z3-sys gh-release mode instead downloads the official architecture-matched Z3 4.16.0 static library and embeds it in the resulting OpenShell binary. This keeps the same pinned Z3 version and static-linking behavior while removing the Z3 source compilation from the critical CI path.

CI supplies its read-only GitHub workflow token to avoid unauthenticated API throttling, and the extracted library/build output is retained by the Cargo target cache. The existing Z3_LIBRARY_PATH_OVERRIDE path remains available for developers who need to use a locally supplied Z3 build.

Testing

• mise run pre-commit
• cargo check -p openshell-prover --features prebuilt-z3
• cargo check -p openshell-server --features prebuilt-z3
• workflow YAML parse and git diff --check
• GitHub Actionlint
• no Zizmor findings for .github/workflows/windows-msvc.yml
• enterprise policy accepted both pinned actions and started the x64 Windows job
• post-rebase lint and test completion on the hosted Windows x64 runner
• local Windows nextest run: 3,343 passed, 16 skipped
• hosted Windows x64 and native ARM64 nextest runs: 3,343 passed, 16 skipped on each architecture
• manual cache-seed to release-build validation: exact x64 (1,142 MB) and ARM64 (1,125 MB) Cargo caches restored successfully
• hosted x64 and native ARM64 optimized release builds
• E2E tests added/updated (not applicable; CI-only change)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[pimlock]

Windows MSVC cache experiments

I ran the hosted windows-2025 workflow repeatedly while changing one major cache/build variable at a time. Times below are end-to-end workflow/job times and can vary slightly with runner setup and GitHub cache transfer latency.

Configuration	Cache state	Runtime	Notes
Original workflow / bundled Z3	effectively cold	~75m48s–78m22s	Baseline; Z3 compiled from source and the Rust target tree was not providing a useful warm reuse path. run, repeat
Cargo target cache / bundled Z3	cold	~75m16s	Populated the target archive. run
Cargo target cache / bundled Z3	warm	~18m39s	Large improvement; target artifacts were the most effective cache layer. run
sccache only / bundled Z3	cold	~80m47s	Target cache disabled to isolate sccache. Remote writes made the cold run slower. run
sccache only / bundled Z3	warm	~67m55s	1,157 hits / 122 misses (90.46% hit rate), but linking, non-cacheable crate types, Z3 work, and remote reads left total time high. run
Cargo target cache + prebuilt Z3	cold	~37m28s	z3-sys downloaded the pinned official prebuilt Z3 archive instead of compiling Z3 via CMake/MSBuild. run
Cargo target cache + prebuilt Z3	warm	~16m24s	Check 1m14s, release build 6m36s, tests 4m46s, focused test 2m12s; target restore ~58s. run
Target cache + prebuilt Z3 + sccache	target warm, sccache cold	~17m09s	62 misses, 62 successful writes, 0 errors; cache writes took 32.6s. run
Target cache + prebuilt Z3 + sccache	both warm	~15m45s	62/62 sccache hits, 0 misses/errors; check 1m07s, release build 5m19s, tests 5m06s, focused test 2m25s. run
Warm hybrid cache, redundant focused test removed	both warm	~12m33s	Check 1m01s, release build 5m07s, full tests 4m58s. The full suite already includes the Windows unsupported-driver contract test. run

Conclusions

• Cargo target caching was the largest win: roughly 75 minutes cold to roughly 19 minutes warm with bundled Z3.
• Prebuilt Z3 materially improved cold builds: the cold target-cache run dropped from roughly 75 minutes to roughly 37 minutes.
• sccache alone was not competitive, despite a high hit rate. It does not cache final linking or several Rust crate types, and GitHub remote reads still cost time.
• sccache is useful as a secondary layer on top of the target cache: it reduced the warm release build from roughly 6m36s to 5m19s and produced a clean 100% hit rate for the 62 cacheable compiler requests in the measured run.
• The best measured stack is Cargo target cache + prebuilt Z3 + sccache, with the redundant focused test removed.
• The PR/main split is intended to keep PR validation focused on Windows Clippy + Rust tests, run that same workload on main to warm the exact PR cache namespaces, and build (but not upload/publish) release binaries after the seed succeeds.

Current validation note

The earlier single-job hybrid workflow is proven by the successful runs above. The new multi-job split currently receives an opaque GitHub startup_failure before any job is created, on both manual dispatch and the PR mirror. actionlint reports the workflow as valid, and a temporary ref at the last known-good commit starts normally, so I am continuing to isolate the workflow-graph difference before treating the split as ready.

[pimlock]

Validation update: the opaque pre-job startup_failure was isolated to jdx/mise-action v4.2.4 on windows-2025. Restoring the previously measured v4.2.0 pin fixes workflow startup. The PR mirror is now running PR lint and test (x64), and a manual dispatch is independently running Seed cache (x64) followed by Build binaries (x64). The newer Swatinem/rust-cache v2.9.2 pin remains in place.

[pimlock]

Pausing this PR until the compute-driver refactor stack lands:

• refactor(compute): unify gateway restart reconciliation #2743 — unify gateway restart reconciliation
• refactor(compute): support external driver parity #2744 — negotiate external driver behavior
• refactor(compute): register compiled drivers #2786 — register compiled drivers and move first-party driver composition into the new openshell-gateway crate

#2786 overlaps 8 of the 18 files changed here and establishes a cleaner Windows boundary by keeping first-party driver dependencies out of openshell-core and openshell-server. The current strict Windows Clippy fixes add 65 cfg attributes, many of which are likely to become unnecessary after that refactor. Continuing now would create avoidable code and rebase churn.

After the stack merges, the plan is to:

1. Rebase this PR again.
2. Keep the Windows PR/main/cache-seed job split, target cache, sccache, and prebuilt Z3 work.
3. Update lint/build coverage for the new openshell-gateway composition crate.
4. Remove obsolete platform-scoping changes and handle any remaining Kubernetes or UDS boundaries at module level where possible.
5. Rerun the hosted Windows timing and cache validation.

The branch is currently rebased onto main; local mise run pre-commit passes.

[pimlock]

Update to the earlier waiting note: #2786 has merged, but the stack was split before merge. The merged #2786 is only the compute-driver registry pre-factor and intentionally keeps first-party driver composition in openshell-server.

The remaining relevant stack is now:

• feat(compute): add standalone first-party drivers #2822 — add standalone first-party driver binaries
• refactor(compute): decouple gateway driver composition #2823 — move first-party composition into openshell-gateway and decouple openshell-server

#2823 is the architectural change we were originally waiting for. It overlaps 8 of this PR’s 18 files, including the core/server configuration and Kubernetes-auth files where most of the Windows cfg changes currently live. This PR is also now conflicting with main, so rebasing and simplifying it before #2823 would create another round of avoidable churn.

Plan: continue holding this PR until #2822 and #2823 land, then rebase, retain the Windows workflow/cache/prebuilt-Z3 work, update lint coverage for openshell-gateway, and recreate only the platform fixes still required by strict hosted Windows Clippy.

[pimlock]

/ok to test 5ece42d

[pimlock]

Final Windows validation

Head 5ece42d8 is green locally and in hosted CI.

• local mise run --skip-tools rust:lint: passed on Windows x64 / Rust 1.95
• local mise run --skip-tools test:rust: passed (2,279.4s cold test build)
• hosted Windows lint + test: passed
• Cargo target cache: restored successfully, 610 MB (639366429 bytes)
• identical-head cache rerun: passed in 13m35s
• sccache on identical-head rerun: 734 hits / 31 misses (95.95%), 0 read errors
• Branch Checks, Helm, DCO, Actionlint, Zizmor, and required aggregate gates: passed

The first post-fix hosted run populated the current compiler inputs; the identical-head rerun verified that both the Cargo target archive and GHA sccache namespace are reusable.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-2738.docs.buildwithfern.com/openshell

[pimlock]

Windows nextest update is validated at head 2b96238. Local native MSVC: 3,343 passed, 16 skipped. Hosted Windows: strict lint and nextest passed; the cache-population run completed in 22m29s and the follow-up cache-hit run completed in 7m58s. The follow-up restored the Cargo target cache and recorded 19 sccache hits / 2 misses (90.48%, 0 errors). The mise lockfile gate and all required CI gates are green.

[pimlock]

/ok to test ff0ca66

[pimlock]

Final Windows x64/ARM64 validation for ff0ca661:

• PR workflow: x64 lint + nextest passed in 23m31s; native ARM64 lint + nextest passed in 24m47s. Each nextest run completed 3,343 tests successfully with 16 skipped.
• Manual cache/build workflow: run 33791210292 passed completely.
  • x64 seed passed in 26m14s; its dependent release build restored the exact architecture-specific 1,142 MB Cargo cache and passed in 24m31s.
  • ARM64 seed passed in 24m01s; its dependent release build restored the exact architecture-specific 1,125 MB Cargo cache and passed in 19m18s on windows-11-arm.
• Both optimized release targets completed successfully and both sccache post-actions reported without errors.

The first release builds had low sccache hit rates because the seed phase compiles the test profile while these were the first release-profile compiles under the new per-architecture namespaces. The Cargo cache handoff itself is verified by exact-key hits and successful restores in both dependent build jobs.

[pimlock]

Warm-cache rerun results on the same commit: run 33796424002 passed completely.

Job	First run	Warm run	Improvement
Seed cache (x64)	26m14s	9m10s	17m04s (65%)
Seed cache (ARM64)	24m01s	10m04s	13m57s (58%)
Release build (x64)	24m31s	17m02s	7m29s (31%)
Release build (ARM64)	19m18s	15m07s	4m11s (22%)
End-to-end workflow	51m11s	27m29s	23m42s (46%)

Both jobs reported exact Cargo cache-key hits and successful restores. Compiler-cache evidence from the warm release builds:

• x64: 454 hits, 301 misses, 60.13% hit rate; release compile step dropped from 22m49s to 15m32s.
• ARM64: 593 hits, 439 misses, 57.46% overall hit rate (59.13% Rust); release compile step dropped from 16m51s to 12m46s.

The remaining time is setup, cache download/extraction, tests, native build scripts, cache misses, and final linking, which sccache does not eliminate.

[pimlock]

/ok to test 66c70a6

[pimlock]

/ok to test 012430a

[pimlock]

/ok to test 012430a

[pimlock]

/ok to test 3184f0f

[shailendra-nv]

General Comment - CONTRIBUTING.md says windows:build:x64 compiles Z3 from source by default and describes the local override as replacing the bundled build. After this PR, the wrapper downloads the prebuilt release by default. This misstates network, cache, and reproducibility behavior for Windows contributors.

[pimlock]

Addressed the CONTRIBUTING.md review in 47a3aab: the Windows section now states that windows:* tasks use pinned prebuilt Z3 4.16.0 by default, explains authenticated CI downloads and architecture-specific target caching, documents the optional local token, and distinguishes the system and explicit bundled-source paths.

[pimlock]

/ok to test 47a3aab

[shailendra-nv]

Looks good to me.