Repository navigation
[Qwen38] Enable DFlash2 fixed_single_request fast path, drop staged PLE VirtualUnlock, and add integrated GPU runtime profiles - #2965
Conversation
8d16c5b to
c0a0291
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The sanitizer policy is not persisted in the projected decoder config, leaving generated packages without padding support while fixed mode disables CUDA graph capture for 1–7-row decode steps and shortened warmup skips the 8-row graph.
1 open issue
What changed in this PR
This PR extends Qwen3.8 inference optimizations and hardware-aware package loading in ONNX Runtime GenAI, targeting faster initialization and decoding plus unified-memory GPU support.
Changes:
- Enables DFlash2 single-request mode and reduces decode warmup iterations.
- Removes per-row Windows
VirtualUnlockcalls from staged PLE gathering. - Adds integrated-GPU memory profiles, Windows plane-library fallback, and repeated provider-registration handling, with Python test updates.
| File | Description |
|---|---|
tools/python/qwen38_flash_next/qwen38_engine_util.py |
Adds Windows plane fallback and tolerates duplicate provider registration. |
tools/python/qwen38_flash_next/package_io.py |
Adds free-memory-based integrated-GPU profiles for GGUF packages. |
tools/python/qwen38_flash_next/build_package.py |
Adds integrated-GPU 128K and 256K profiles for QMoE packages. |
test/python/test_build_package_gguf.py |
Checks integrated GGUF profiles, block counts, and tier boundaries. |
test/python/builder/test_qwen38_qualification_tools.py |
Tests Windows plane fallback and repeated provider registration. |
test/python/builder/test_qwen38_package_builders.py |
Updates assertions for the additional QMoE profiles. |
src/qwen38_plane/qwen38_plane.cu |
Removes per-miss VirtualUnlock loops from staged PLE gathers. |
src/models/qwen38_graph.cpp |
Enables fixed single-request drafting and shortens decode warmup. |
Review effort: Balanced
…LE VirtualUnlock, and add integrated GPU runtime profiles - Pass fixed_single_request = (max_batch_ == 1) when constructing Dflash2Drafter in Qwen38GraphEngine::Initialize and break after r == 1 in single-slot decode warmup, consolidating 8 per-row-count decode CUDA graphs into one fixed 8-row graph (-37.1% load time, -47% per-step DFlash2 input prep + launch gap). - Remove per-miss VirtualUnlock syscall loops in host_gather_ple_rows_batch and PleGatherPool::GatherSpans on Windows staged file-mapped PLE shards, eliminating up to 98,304 VirtualUnlock calls per 8K prefill and 96 per decode step (-36.9% decode PLE host gather gap). - Add integrated-GPU (`is_integrated: true`) runtime_profiles gated on `minimum_free_device_memory_bytes` in `build_package.py` and `build_package_gguf.py` so unified-memory GPUs (e.g. NVIDIA RTX Spark / DGX Spark) automatically select 32K/64K/128K/256K (`num_blocks = 1037`) tiers. - Clear POSIX-only `.so` `plane_library` entries automatically on Windows in `qwen38_engine_util.py:load_model` when no companion `.dll` exists and tolerate duplicate EP registration in the same Python process. - Add unit tests in `test_build_package_gguf.py`, `test_qwen38_package_builders.py`, and `test_qwen38_qualification_tools.py`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
c0a0291 to
fc4c457
Compare
…lash2 decode graph on padding support - Add minimum_total_device_memory_bytes to every integrated-gpu-* runtime profile in build_package.py and package_io.py. ValidateRuntimeProfiles rejects a profile without it, so every package built with the new tiers failed to load on all devices. Use max_scheduled_tokens 8192 as in the discrete tiers; the 4096 branch had no effect on single-slot packages. - Record the DFlash drafter's session.paged_attention.skip_input_sanitizer policy in its projected config, which Dflash2Drafter reads to enable padding. Enable fixed_single_request and the one-width decode warmup only when padding is supported (single slot, windowed drafter, ORT API >= 32, sanitizer skipped); otherwise keep one captured graph per decode row count instead of running 1-7-row steps eagerly. - Add runtime_profiles_problem, a Python mirror of ValidateRuntimeProfiles, to the qwen38 tool tests and assert it on the QMoE and GGUF profiles; add C++ tests for SetQwen38DflashSanitizerPolicy and Qwen38DflashPadsDecodeSteps. - Document the integrated tiers in the README, note why the staged PLE gather no longer ages rows out of the working set, and fix ruff formatting in qwen38_engine_util.py and test_qwen38_qualification_tools.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Baiju Meswani (baijumeswani)
left a comment
There was a problem hiding this comment.
This PR improves the existing Qwen graph backend: it enables the fixed single-request DFlash drafter where padded decode is supported, removes per-row VirtualUnlock work from staged PLE gathering, and adds integrated-GPU runtime profiles. It does not introduce another backend.
The sanitizer policy now reaches both the ORT session and the drafter config. The fast path is gated on one request, a sliding window, sanitizer opt-out and ORT 1.32+, while older-runtime and batched paths retain their previous behavior. The integrated profiles have separate eligibility ranges and free-memory limits. These are sensible design and performance changes.
I found no blocking correctness or regression issue. The two inline suggestions are nonblocking follow-ups and can be addressed after approval.
Validation on cf935e9: Release/CUDA build passed without diagnostics; 758 C++ tests passed and five skipped, including a passing real CUDA padded-attention comparison on ORT 1.32. A review probe checked six generated configurations at 32 integrated-memory boundaries using the real C++ config validator. Python: 318 passed, five optional reference tests skipped. Windows and integrated-device performance results were not independently reproduced here. Two CI jobs remain pending/running at posting time.
Baiju Meswani (baijumeswani)
left a comment
There was a problem hiding this comment.
Reviewed cf935e9. Looks good for approval: no blocking correctness or regression issue found. The two suggestions in my preceding COMMENT review are nonblocking follow-ups and can be addressed after approval. The remaining CI checks should pass before merging.

Summary
Stacked on #2907 (
tlwu/20261107/qwen38-rc5-gguf-offload) and paired with microsoft/onnxruntime#34200.Key Changes
Dflash2Drafterfixed_single_requestFast Path (src/models/qwen38_graph.cpp):session.paged_attention.skip_input_sanitizerpolicy in its projected config (SetQwen38DflashSanitizerPolicy), whereDflash2Drafterreads it to enable padding, and passesfixed_single_requestwhenQwen38DflashPadsDecodeStepsholds (single slot, windowed drafter, ORT API>= 32, sanitizer skipped). The1..verify_rows_decode warmup loop then stops afterr == 1; otherwise the drafter keeps one captured graph per decode row count.r = 1..8) into a single padded 8-row CUDA graph (40 -> 5warmup runs, cutting model load time by up to-37.1%,13.77 s -> 8.67 s) and replaces 5 per-stepStoreBytesKernellaunches + 6StepTensorresizes with 1 packed 272-byteStoreBytesKernel(-47%DFlash2 input-prep + launch gap:148.1 us -> 79.1 us).VirtualUnlockSyscall Storm in Staged PLE Gather (src/qwen38_plane/qwen38_plane.cu):VirtualUnlockloops inhost_gather_ple_rows_batchandPleGatherPool::GatherSpanson un-VirtualLocked file-mapped PLE views while retaining batchedPrefetchVirtualMemory.98,304VirtualUnlockcalls per 8K prefill and96per decode step on Windows, cutting the decode PLE host gather gap by-36.9%(2,040 us -> 1,288 us,-0.75 ms/cycle).is_integrated: true)runtime_profiles(tools/python/qwen38_flash_next/build_package.py,build_package_gguf.py):integrated-gpu-128k/integrated-gpu-256k(QMoE) andintegrated-gpu-32k/64k/128k/256k(GGUF) runtime profiles gated onminimum_free_device_memory_bytesso unified-memory GPUs (SM 12.1NVIDIA RTX Spark / DGX Spark) automatically selectnum_blocks = 1037(256K) instead of falling through discrete-only (is_integrated: false) profiles.minimum_total_device_memory_bytes, which the engine'sValidateRuntimeProfilesrequires on every profile, to its free-memory minimum;runtime_profiles_problemin the qwen38 tool tests mirrors that validation for the QMoE and GGUF profiles.tools/python/qwen38_flash_next/qwen38_engine_util.py):"plane_library": ""on Windows (os.name == "nt") when a Linux-built package specifies a.soplane_librarythat exists on disk without a.dllalias, soQwen38Planeloads the bundledonnxruntime-genai-cuda.dllout-of-the-box.register_execution_provider_librarycalls across multipleload_modelinvocations in one Python process.Validation
engine_unit_tests.exe --ep_dir ... --gtest_filter=*Qwen38*:*RuntimeProfile*:*Dflash*:586 passed, 0 failed.pytest test/python/builder/test_qwen38_qualification_tools.py tools/python/qwen38_flash_next/test_ram_budget.py:190 passed.smoke.pyand2K–255K + 1Kbenchmarks on NVIDIA RTX Spark (SM 12.1, Windows 11 ARM64) across bothqwen_flash_next_rc6_2b375_fp32andqwen_flash_next_gguf_ud_iq3_xxs_q8_engram_fp32_fp16kv(bit-identical greedy token hashes).cf935e9(Linux CPU build):engine_unit_tests1721 passed, 0 failed(including the newQwen38DflashSanitizerTest.*);pytest test/python/builder/test_qwen38_qualification_tools.py test/python/builder/test_qwen38_package_builders.py test/python/test_build_package_gguf.py:318 passed; the generated QMoE and GGUFruntime_profilesload throughog.Config.