Skip to content

[Qwen38] Enable DFlash2 fixed_single_request fast path, drop staged PLE VirtualUnlock, and add integrated GPU runtime profiles - #2965

Merged
Tianlei Wu (tianleiwu) merged 2 commits into
mainfrom
tlwu/20261113/qwen38-dflash2-ple-spark-opt
Nov 14, 2026
Merged

Tianlei Wu (tianleiwu) merged 2 commits into
mainfrom
tlwu/20261113/qwen38-dflash2-ple-spark-opt

Conversation

@tianleiwu

@tianleiwu Tianlei Wu (tianleiwu) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #2907 (tlwu/20261107/qwen38-rc5-gguf-offload) and paired with microsoft/onnxruntime#34200.

Key Changes

  1. Enable Dflash2Drafter fixed_single_request Fast Path (src/models/qwen38_graph.cpp):
    • Records the drafter's session.paged_attention.skip_input_sanitizer policy in its projected config (SetQwen38DflashSanitizerPolicy), where Dflash2Drafter reads it to enable padding, and passes fixed_single_request when Qwen38DflashPadsDecodeSteps holds (single slot, windowed drafter, ORT API >= 32, sanitizer skipped). The 1..verify_rows_ decode warmup loop then stops after r == 1; otherwise the drafter keeps one captured graph per decode row count.
    • Consolidates 8 per-row-count decode DFlash2 CUDA graphs (r = 1..8) into a single padded 8-row CUDA graph (40 -> 5 warmup runs, cutting model load time by up to -37.1%, 13.77 s -> 8.67 s) and replaces 5 per-step StoreBytesKernel launches + 6 StepTensor resizes with 1 packed 272-byte StoreBytesKernel (-47% DFlash2 input-prep + launch gap: 148.1 us -> 79.1 us).
  2. Eliminate Per-Row VirtualUnlock Syscall Storm in Staged PLE Gather (src/qwen38_plane/qwen38_plane.cu):
    • Removes the per-miss VirtualUnlock loops in host_gather_ple_rows_batch and PleGatherPool::GatherSpans on un-VirtualLocked file-mapped PLE views while retaining batched PrefetchVirtualMemory.
    • Eliminates up to 98,304 VirtualUnlock calls per 8K prefill and 96 per decode step on Windows, cutting the decode PLE host gather gap by -36.9% (2,040 us -> 1,288 us, -0.75 ms/cycle).
  3. Unified-Memory (is_integrated: true) runtime_profiles (tools/python/qwen38_flash_next/build_package.py, build_package_gguf.py):
    • Adds integrated-gpu-128k / integrated-gpu-256k (QMoE) and integrated-gpu-32k / 64k / 128k / 256k (GGUF) runtime profiles gated on minimum_free_device_memory_bytes so unified-memory GPUs (SM 12.1 NVIDIA RTX Spark / DGX Spark) automatically select num_blocks = 1037 (256K) instead of falling through discrete-only (is_integrated: false) profiles.
    • Every integrated tier also sets minimum_total_device_memory_bytes, which the engine's ValidateRuntimeProfiles requires on every profile, to its free-memory minimum; runtime_profiles_problem in the qwen38 tool tests mirrors that validation for the QMoE and GGUF profiles.
  4. Windows Qualification Tool Portability (tools/python/qwen38_flash_next/qwen38_engine_util.py):
    • Automatically overlays "plane_library": "" on Windows (os.name == "nt") when a Linux-built package specifies a .so plane_library that exists on disk without a .dll alias, so Qwen38Plane loads the bundled onnxruntime-genai-cuda.dll out-of-the-box.
    • Tolerates duplicate register_execution_provider_library calls across multiple load_model invocations in one Python process.

Validation

  • engine_unit_tests.exe --ep_dir ... --gtest_filter=*Qwen38*:*RuntimeProfile*:*Dflash*: 586 passed, 0 failed.
  • pytest test/python/builder/test_qwen38_qualification_tools.py tools/python/qwen38_flash_next/test_ram_budget.py: 190 passed.
  • Verified out-of-the-box smoke.py and 2K–255K + 1K benchmarks on NVIDIA RTX Spark (SM 12.1, Windows 11 ARM64) across both qwen_flash_next_rc6_2b375_fp32 and qwen_flash_next_gguf_ud_iq3_xxs_q8_engram_fp32_fp16kv (bit-identical greedy token hashes).
  • Follow-up cf935e9 (Linux CPU build): engine_unit_tests 1721 passed, 0 failed (including the new Qwen38DflashSanitizerTest.*); pytest test/python/builder/test_qwen38_qualification_tools.py test/python/builder/test_qwen38_package_builders.py test/python/test_build_package_gguf.py: 318 passed; the generated QMoE and GGUF runtime_profiles load through og.Config.

@tianleiwu
Tianlei Wu (tianleiwu) force-pushed the tlwu/20261113/qwen38-dflash2-ple-spark-opt branch from 8d16c5b to c0a0291 Compare November 13, 2026 18:52
Copilot AI balanced review requested due to automatic review settings November 13, 2026 18:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The sanitizer policy is not persisted in the projected decoder config, leaving generated packages without padding support while fixed mode disables CUDA graph capture for 1–7-row decode steps and shortened warmup skips the 8-row graph.

1 open issue
What changed in this PR

This PR extends Qwen3.8 inference optimizations and hardware-aware package loading in ONNX Runtime GenAI, targeting faster initialization and decoding plus unified-memory GPU support.

Changes:

  • Enables DFlash2 single-request mode and reduces decode warmup iterations.
  • Removes per-row Windows VirtualUnlock calls from staged PLE gathering.
  • Adds integrated-GPU memory profiles, Windows plane-library fallback, and repeated provider-registration handling, with Python test updates.
File Description
tools/​python/​qwen38_flash_next/​qwen38_engine_util.py Adds Windows plane fallback and tolerates duplicate provider registration.
tools/​python/​qwen38_flash_next/​package_io.py Adds free-memory-based integrated-GPU profiles for GGUF packages.
tools/​python/​qwen38_flash_next/​build_package.py Adds integrated-GPU 128K and 256K profiles for QMoE packages.
test/​python/​test_build_package_gguf.py Checks integrated GGUF profiles, block counts, and tier boundaries.
test/​python/​builder/​test_qwen38_qualification_tools.py Tests Windows plane fallback and repeated provider registration.
test/​python/​builder/​test_qwen38_package_builders.py Updates assertions for the additional QMoE profiles.
src/​qwen38_plane/​qwen38_plane.cu Removes per-miss VirtualUnlock loops from staged PLE gathers.
src/​models/​qwen38_graph.cpp Enables fixed single-request drafting and shortens decode warmup.

Review effort: Balanced

Comment thread src/models/qwen38_graph.cpp Resolved
Base automatically changed from tlwu/20261107/qwen38-rc5-gguf-offload to main November 13, 2026 21:47
…LE VirtualUnlock, and add integrated GPU runtime profiles

- Pass fixed_single_request = (max_batch_ == 1) when constructing Dflash2Drafter in Qwen38GraphEngine::Initialize and break after r == 1 in single-slot decode warmup, consolidating 8 per-row-count decode CUDA graphs into one fixed 8-row graph (-37.1% load time, -47% per-step DFlash2 input prep + launch gap).
- Remove per-miss VirtualUnlock syscall loops in host_gather_ple_rows_batch and PleGatherPool::GatherSpans on Windows staged file-mapped PLE shards, eliminating up to 98,304 VirtualUnlock calls per 8K prefill and 96 per decode step (-36.9% decode PLE host gather gap).
- Add integrated-GPU (`is_integrated: true`) runtime_profiles gated on `minimum_free_device_memory_bytes` in `build_package.py` and `build_package_gguf.py` so unified-memory GPUs (e.g. NVIDIA RTX Spark / DGX Spark) automatically select 32K/64K/128K/256K (`num_blocks = 1037`) tiers.
- Clear POSIX-only `.so` `plane_library` entries automatically on Windows in `qwen38_engine_util.py:load_model` when no companion `.dll` exists and tolerate duplicate EP registration in the same Python process.
- Add unit tests in `test_build_package_gguf.py`, `test_qwen38_package_builders.py`, and `test_qwen38_qualification_tools.py`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
Tianlei Wu (tianleiwu) force-pushed the tlwu/20261113/qwen38-dflash2-ple-spark-opt branch from c0a0291 to fc4c457 Compare November 13, 2026 21:48
…lash2 decode graph on padding support

- Add minimum_total_device_memory_bytes to every integrated-gpu-* runtime profile in build_package.py and
  package_io.py. ValidateRuntimeProfiles rejects a profile without it, so every package built with the new tiers
  failed to load on all devices. Use max_scheduled_tokens 8192 as in the discrete tiers; the 4096 branch had no
  effect on single-slot packages.
- Record the DFlash drafter's session.paged_attention.skip_input_sanitizer policy in its projected config, which
  Dflash2Drafter reads to enable padding. Enable fixed_single_request and the one-width decode warmup only when
  padding is supported (single slot, windowed drafter, ORT API >= 32, sanitizer skipped); otherwise keep one
  captured graph per decode row count instead of running 1-7-row steps eagerly.
- Add runtime_profiles_problem, a Python mirror of ValidateRuntimeProfiles, to the qwen38 tool tests and assert it
  on the QMoE and GGUF profiles; add C++ tests for SetQwen38DflashSanitizerPolicy and Qwen38DflashPadsDecodeSteps.
- Document the integrated tiers in the README, note why the staged PLE gather no longer ages rows out of the
  working set, and fix ruff formatting in qwen38_engine_util.py and test_qwen38_qualification_tools.py.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
Tianlei Wu (tianleiwu) enabled stacked auto-merge (squash) November 14, 2026 01:06

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR improves the existing Qwen graph backend: it enables the fixed single-request DFlash drafter where padded decode is supported, removes per-row VirtualUnlock work from staged PLE gathering, and adds integrated-GPU runtime profiles. It does not introduce another backend.

The sanitizer policy now reaches both the ORT session and the drafter config. The fast path is gated on one request, a sliding window, sanitizer opt-out and ORT 1.32+, while older-runtime and batched paths retain their previous behavior. The integrated profiles have separate eligibility ranges and free-memory limits. These are sensible design and performance changes.

I found no blocking correctness or regression issue. The two inline suggestions are nonblocking follow-ups and can be addressed after approval.

Validation on cf935e9: Release/CUDA build passed without diagnostics; 758 C++ tests passed and five skipped, including a passing real CUDA padded-attention comparison on ORT 1.32. A review probe checked six generated configurations at 32 integrated-memory boundaries using the real C++ config validator. Python: 318 passed, five optional reference tests skipped. Windows and integrated-device performance results were not independently reproduced here. Two CI jobs remain pending/running at posting time.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed cf935e9. Looks good for approval: no blocking correctness or regression issue found. The two suggestions in my preceding COMMENT review are nonblocking follow-ups and can be addressed after approval. The remaining CI checks should pass before merging.

@tianleiwu
Tianlei Wu (tianleiwu) merged commit f76eed5 into main Nov 14, 2026
67 of 68 checks passed
@tianleiwu
Tianlei Wu (tianleiwu) deleted the tlwu/20261113/qwen38-dflash2-ple-spark-opt branch November 14, 2026 03:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants