[solderzzc]

Proposed changes

Possible fix for SharpAI/SwiftLM#216 (--stream-experts intermittently killed by kIOGPUCommandBufferCallbackErrorTimeout, correlated with a cold page cache).

Mechanism (from reading the code, see the issue comment): with --stream-experts the model is loaded with lazyLoad = true, so the else of if !lazyLoad { materializeModelForInference } never reads any weights. Every non-expert weight stays an unevaluated safetensors Load node. Load runs on the CPU stream (pread on io::thread_pool), and a GPU op that consumes it gets an event wait encoded into its command buffer. On a cold page cache (or a slow external disk) the first forward pass can keep a command buffer waiting longer than the GPU watchdog allows.

Change: for a streaming model, after loading, evaluate all parameters except the SwitchLinear expert weight tensors (placeholders paged from SSD on demand; evaluating them would allocate the full expert stack). Scales and biases are small and are read normally. New helper materializeNonExpertWeights in Load.swift; non-streaming loading is unchanged.

Evidence / limits:

• Qwen3.5-35B-A3B-4bit on a USB external SSD, --stream-experts, after the page cache had been evicted: startup took 86 s (the preload read), then the first 213-token request completed (368 s, very cold experts) with no GPU timeout. Before the change a comparable cold start crashed (350 s). That is one run: the crash is probabilistic, so this is encouraging but not proof.
• A warm start and second request also passed.
• No unit test: the failure needs a cold disk plus a real streamed model; I did not find a cheap deterministic test. Happy to add one if there is a good seam.
• Trade-off: startup now reads the non-expert weights up front instead of on the first request.

Checklist

• I have read the CONTRIBUTING document
• I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
• I have added tests that prove my fix is effective or that my feature works
• I have updated the necessary documentation (if needed)

AI usage

• I have read this PR description in full and approve it as my own, and it
  accurately describes the code changes.
• AI usage disclosure: prepared with Claude Code; needs human review before merging.

🤖 Generated with Claude Code