Fix distributed compilation for no-hash dep files and dist overlay issues

[justdoGIT]

Summary

This PR consolidates four independent fixes for sccache-dist that together resolve E0463 "can't find crate" errors on dist workers and make sccache-dist usable on distros like Void Linux. Supersedes #2847 and #2849.

Fix 1: parse_rustc_z_ls misparses deps without -HASH suffix

Modern rustc (1.75+) emits extended metadata per dependency line in rustc -Z ls=root output:

N libname[-hash] hash HASH host_hash ... kind ... public

Crates built without a content hash suffix (e.g. crc_fast, a cdylib) appear as:

N crc_fast hash 05bce6... host_hash None kind Unconditional public

parse_rustc_z_ls used splitn(2, ' ') which yields libstring as the entire trailing metadata ("crc_fast hash 05bce6... host_hash ..."). The subsequent rsplitn(2, '-') finds no - and libname becomes the whole string instead of just "crc_fast". This broken name never matches during crate_link_paths scanning, so the rmeta is not identified as a dependency.

Fix: take only the first whitespace-delimited token of libstring before rsplitn('-').

Fix 2: crate_link_paths scan skips no-hash dep files

RustInputsPackager scans crate_link_paths for dependency rmeta/rlib files, extracting the crate name from each filename via rsplitn(2, '-'). This assumes every file has a -<metadata-hash> suffix. When a file has no hash (e.g. libcrc_fast.rmeta produced by a crate with crate-type = ["lib", "cdylib", "staticlib"]), rsplitn yields a single element and the code hits continue, skipping the file entirely. The file is never packaged and sent to the dist worker, which then fails with:

error[E0463]: can't find crate for `crc_fast` which `xai_file_utils` depends on

Fix: extract the crate-name-and-extension logic into a testable helper, crate_name_and_ext_from_lib_path, that uses file_stem() to strip the extension before rsplitn, then falls back to the whole stem as the libname when no - separator is found.

Fix 3: OverlayBuilder crash on persistent toolchain dirs

When sccache-dist restarts, the in-memory toolchain_dir_map is wiped but on-disk toolchain directories survive. On the next build, prepare_overlay_dirs calls fs::create_dir on the existing directory, which returns AlreadyExists and fails the build.

Fix: guard create_dir with an exists() check. The archive_id is a content hash, so a pre-existing directory for the same id already holds the correct unpacked toolchain.

Fix 4: Package rustlib/ instead of whole sysroot lib dir

On distros where rustc --print sysroot reports /usr (e.g. Void Linux), the ToolchainPackager joins LIBS_DIR ("lib") to the sysroot and packages the entire /usr/lib tree — several GiB — blowing past the toolchain cache size.

Fix: package the rustlib/ subtree (the actual Rust stdlib) instead. Fall back to old behavior if rustlib/ doesn't exist.

Testing

• All 5 unit tests pass: test_parse_rustc_z_ls_modern_no_hash_suffix, test_parse_rustc_z_ls_pre_1_55, test_parse_rustc_z_ls_post_1_55, test_crate_name_and_ext_from_lib_path, test_can_trim_this
• Built and deployed on a 4-node LAN cluster (Void Linux, nightly 1.100)
• xai-grok-telemetry (previously failing with E0463 on crc_fast) now compiles successfully via sccache-dist
• 321+ successful distributed compiles across all 4 nodes

Related

• Bug: parse_rustc_z_ls misparses deps without -HASH suffix → E0463 on dist workers #2846
• Supersedes: Fix parse_rustc_z_ls for deps without -HASH suffix #2847, Fix dist overlay: persistent toolchain dirs and sysroot packaging #2849

[Copilot]

🟡 Changes recommended

Unresolved critical cross-platform test risk and moderate overlay reuse behavior require changes; additional regression tests are also needed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes distributed Rust compilation for no-hash dependencies and improves toolchain packaging and overlay handling.

Changes:

• Corrects Rust dependency and library-name parsing.
• Packages rustlib selectively with a fallback.
• Adjusts overlay toolchain directory handling.

File summaries

File	Review summary
src/compiler/rust.rs	Dependency parsing and sysroot packaging updates. Findings: hard-coded .so fixtures may fail cross-platform tests (critical, 1 vote); archive-layout regression coverage is missing (nit, 3 votes).
src/bin/sccache-dist/build.rs	Overlay directory handling updates. Findings: existing-directory reuse remains incomplete and cleanup is inconsistent (moderate, 3 votes); regression coverage is missing (nit, 1 vote).

Review details

Suppressed comments (1)

src/bin/sccache-dist/build.rs:197

• The new pre-existing-toolchain-directory path has no regression test. Please add a test that exercises prepare_overlay_dirs with an existing toolchain directory and verifies it no longer fails with AlreadyExists, so this restart/leftover-directory fix remains protected.

                // The directory can persist on disk from a previous server
                // process (or a toolchain evicted from the in-memory map
                // without its directory). Treat it as already-prepared.
                if !toolchain_dir.exists() {
                    fs::create_dir(&toolchain_dir)?;

• Files reviewed: 2/2 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.