Skip to content

Report prefill progress on the VLM path - #183

Merged
solderzzc merged 1 commit into
mainfrom
fix/vlm-prefill-progress
Sep 25, 2026
Merged

solderzzc merged 1 commit into
mainfrom
fix/vlm-prefill-progress

Conversation

@solderzzc

Copy link
Copy Markdown
Member

Summary

On --vision models, n_past and fraction in the prefill progress heartbeat stayed at 0 for the whole prefill. Only the LLM prepare calls activePrefillProgressHook. The VLM prepare / prepareContinuation report through PrefillParameters.progress, which SwiftLM never set.

  • Set prefill.progress on the chat and text completion parameters to forward into activePrefillProgressHook.
  • PrefillState.update keeps the highest nPast, because updates arrive as detached Tasks and can land out of order.

Test plan

  • Qwen3.8-27B-4bit --vision, 8,413-token prompt: before, n_past stayed 0 for 26 s; after, it rose 495 → 7,920 (fraction 0.06 → 0.94), and the answer was correct
  • swift test --filter SwiftLMTests: 185 tests, 0 failures

🤖 Generated with Claude Code

Only the LLM prepare calls activePrefillProgressHook; VLM prepare reports
chunked prefill through PrefillParameters.progress, which the server never
set. On a --vision model the prefill_progress heartbeat therefore stayed at
n_past 0 for the whole prefill. Forward prefill.progress to the hook, and
keep n_past monotonic since updates arrive as unordered Tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@solderzzc

Copy link
Copy Markdown
Member Author

Cross-review (M6 session): no blocking issues. OK to merge once CI is green.

Reviewed the full diff (Server.swift, +17/−3) against main 804616e:

  • Both GenerateParameters constructions (chat and text completions) now forward prefill.progress, and each ends in a let params snapshot, so there's no captured-var warning. A clean rebuild of 499c36c on Xcode 27 / Swift 6.4 shows none from this PR.
  • On the LLM path, LLMModel.prepare still calls activePrefillProgressHook directly, and prefill.progress may forward the same (upperBound, total) again. With PrefillState.update now taking max(nPast, …), the duplicate and any out-of-order detached updates are harmless.
  • Not introduced here, noted for later: activePrefillProgressHook is a process-global, so with --parallel > 1 one request's prefill could drive another's heartbeat. That's pre-existing, and this PR doesn't make it worse.

Verified on a Mac mini M6 (32 GB): Qwen3.8-27B-4bit (VLM path), --ctx-size 16384, 4.9K-token prompt with X-SwiftLM-Prefill-Progress: true. The heartbeat's n_past goes 492 → 984 → 1476 → 1968 → 2952 → 3444 → 3936 → 4428 and fraction 0.1 → 0.9, every 2 s, monotonic, and the answer is correct. Before this PR, n_past stayed at 0 on the VLM path.

@solderzzc
solderzzc merged commit 0a2b688 into main Sep 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant