Skip to content

[CUDA] Optimize Q6_K/IQ3_XXS/IQ4_NL GGUF, FP16-KV DSA, M=8 HC, and FP32 GDN kernels - #34200

Merged
Tianlei Wu (tianleiwu) merged 2 commits into
mainfrom
tlwu/20261113/qwen38-gguf-rc6-cuda-opt
Nov 13, 2026
Merged

Tianlei Wu (tianleiwu) merged 2 commits into
mainfrom
tlwu/20261113/qwen38-gguf-rc6-cuda-opt

Conversation

@tianleiwu

Copy link
Copy Markdown
Contributor

Summary

This PR optimizes CUDA prefill and speculative-decode (M = 8 verify) kernels across MatMulPacked, MoEPacked, DynamicSparseAttention (fp16_kv), HyperConnectionLowRankMix, GatedDeltaNet (fp32 state), and QMoE helper launches for Qwen3.8-Flash-Next GGUF (UD-IQ3_XXS) and QMoE (2b375_fp32) models:

  1. MatMulPacked Q6_K Contiguous Half-Superblock (kChunk = 128) & Aligned SWAR Unpack (gguf_quant_cuda.cuh, matmul_packed.cu):
    • Assigns contiguous 128-element Q6_K half-superblocks per warp (LoadQ6KHalf128Stage + DecodeQ6KHalf128GroupBf16x2 SWAR unpack) in MatMulPackedBf16TcWarpKSplitKernel (KW=10, MT=1 for N<=1024; KW=8 for K=6144, N<=2560; KW=4, MT=2 for N>2560) and enables Q6_K in MatMulPackedBf16DequantFastKernel.
    • Cuts MatMulPacked decode time from 19.60 -> 13.48 ms/cycle (-6.12 ms/cyc; N=512 Q6_K latency drops 5.5x from 33.8 us -> ~6.1 us).
  2. MoEPacked Contiguous kChunk = 128 (IQ3_XXS, IQ2_S, IQ4_XS) FC1 Decode & Aligned 36B Pair (IQ4_NL) FC2 Decode (gguf_quant_cuda.cuh, moe_packed.cu):
    • Hoists superblock scale/sign loads across 4 sub-blocks per warp in MoEPackedFc1DecodeWarpKSplitKernel with 4-way bank-replicated iq3xxs_grid (s_lut[1024]), and decodes 4-byte-aligned 36B (2 x block_iq4_nl) pairs (DecodeIq4NlPair64AlignedBf16x2) in MoEPackedFc2Bf16TcKernel<1, kGgmlTypeIQ4_NL> (21.47 -> 19.76 ms/cyc, reaching 295 GB/s vs 301 GB/s peak on RTX Spark).
  3. FP16-KV Tensor-Core DynamicSparseAttentionFp16TcKernel<kSplitK, kCandTile> (dynamic_sparse_attention_prefill.cuh, dynamic_sparse_attention_impl.cu):
    • Replaces the scalar GroupedDecodeDynamicSparseAttentionKernel fallback on fp16_kv with a split-K cp.async Tensor-Core m16n8k16 FP16-KV kernel (kCandTile = 32 on 100 KB/SM GPUs, P_hi + P_residual 22-bit probability split) for decode and two-buffer <false, 64> for prefill.
    • Speeds up FP16-KV attention decode by 5.4x (2.71 -> 0.50 ms/cyc) and 8K prefill by 1.49x (486.0 -> 325.3 ms).
  4. Single-Wave m16n8k16 Tensor-Core HcDownTc8Kernel & HcUpPreTc8Kernel (rows = 8) + NormVec8H2560Kernel (hyper_connection_low_rank_mix_impl.cu):
    • Replaces 2-wave scalar LaunchFusedRows<T, 4, 2> (3.81 GB/cyc weight reads) on rows=8, branches=4, hidden=2560, rank=320 (BF16) with single-wave BF16 Tensor-Core kernels (grid=(6,8,1) and grid=(80,1,1)) with PDL weight prefetch (1.27 GB/cyc = 3.0x less weight DRAM traffic, saving ~3.5–4.5 ms/cyc), plus 320-thread NormVec8H2560Kernel for prefill.
  5. Fused Dual-A 3-Slice FP32-State GatedDeltaNetChunkedKernel (gated_delta_net_mma.cuh, gated_delta_net_impl.cu):
    • Hoists LoadFragA across slices and fuses (k @ S0, Qg @ S0, R epilogue) (SmemGemmFp32StateDualS0Bf16) and (P @ U, Kd^T @ U) (SmemGemmFp32StateDualUBf16), reducing 3-slice FP32->BF16 passes per chunk from 5 -> 3 (35 -> 12 barriers) while preserving exact 24-bit IEEE-754 precision (427.0 -> 343.3 ms at 8K prefill, -19.6%).
  6. Column-Matched CTA Widths in QMoE expandInputRowsKernel & doActivationKernel (moe_kernels.cu, moe_gemm_activation_kernels.cuh):
    • Matches CTA block widths to column vector counts (320 for hidden_size=2560; 64/96 for inter_size=512/640), eliminating 69%–75% idle tail threads (-31 ms at 8K prefill).

End-to-End Performance & Accuracy (2K–255K + 1K on NVIDIA RTX Spark SM 12.1)

1. ORT gguf_ud_iq3_xxs_q8_engram (DFlash2 K=7) vs. Baseline & llama.cpp + MTP (UD-IQ3_XXS + Q8_0)

Context (Prompt + Gen) ORT Baseline TTFT / Decode (ms/cyc) ORT Optimized TTFT / Decode (ms/cyc) llama.cpp + MTP TTFT / Decode ORT Opt vs. llama.cpp + MTP
2K + 1K 1.509 s / 52.71 tok/s (67.62 ms) 1.468 s / 66.70 tok/s (52.53 ms) 2.582 s / 57.81 tok/s 1.76x prefill / +15.4% decode
8K + 1K 5.468 s / 46.42 tok/s (67.81 ms) 5.231 s / 66.49 tok/s (53.80 ms) 9.616 s / 54.22 tok/s 1.84x prefill / +22.6% decode
16K + 1K 10.822 s / 51.75 tok/s (67.24 ms) 10.425 s / 61.24 tok/s (54.06 ms) 19.331 s / 58.48 tok/s 1.85x prefill / +4.7% decode
32K + 1K 21.663 s / 48.44 tok/s (68.34 ms) 20.929 s / 63.54 tok/s (54.76 ms) 40.822 s / 54.69 tok/s 1.95x prefill / +16.2% decode
64K + 1K 44.858 s / 53.26 tok/s (70.09 ms) 42.684 s / 58.04 tok/s (57.23 ms) 81.592 s / 50.67 tok/s 1.91x prefill / +14.5% decode
128K + 1K 94.444 s / 42.02 tok/s (72.02 ms) 91.288 s / 59.12 tok/s (58.06 ms) 172.053 s / 45.89 tok/s 1.88x prefill / +28.8% decode
255K + 1K 206.564 s / 42.15 tok/s (81.71 ms) 201.523 s / 45.41 tok/s (67.65 ms) 378.801 s / 33.42 tok/s 1.88x prefill / +35.9% decode

2. ORT rc6_2b375_fp32 (DFlash2 K=7, fp16_kv + FP32 GDN) Before vs. After

Context (Prompt + Gen) Baseline TTFT (Prefill tok/s) Optimized TTFT (Prefill tok/s) Baseline Decode (ms/cyc) Optimized Decode (ms/cyc)
2K + 1K 1.232 s (1,661.9) 1.204 s (1,700.5 tok/s) 62.78 tok/s (54.68 ms) 81.20 tok/s (47.91 ms, -12.4%)
8K + 1K 4.607 s (1,778.1) 4.352 s (1,882.3 tok/s) 63.91 tok/s (54.44 ms) 76.47 tok/s (48.65 ms, -10.6%)
32K + 1K 18.642 s (1,757.8) 17.589 s (1,862.9 tok/s) 58.34 tok/s (56.02 ms) 65.65 tok/s (50.11 ms, -10.5%)
128K + 1K 81.111 s (1,616.0) 77.106 s (1,699.9 tok/s) 48.82 tok/s (58.37 ms) 51.38 tok/s (53.52 ms, -8.3%)
255K + 1K 179.976 s (1,450.9) 172.099 s (1,517.3 tok/s) 51.17 tok/s (68.93 ms) 50.52 tok/s (63.08 ms, -8.5%)

Validation

  • Multi-arch CUDA compilation verified cleanly across sm_75, sm_80, sm_86, sm_90, and sm_120f with -Werror all-warnings --Werror default-stream-launch.
  • Speculative (M = 8) vs. --force-reject (M = 1) greedy equivalence verified 128/128 (100% bit-identical) on both rc6_2b375_fp32 and gguf_ud_iq3_xxs_q8_engram.
  • MMLU-Pro shard 0/80 (10 items): 9/10 (90%, matching baseline item-for-item) on rc6_2b375_fp32 and 10/10 (100%) on gguf_ud_iq3_xxs_q8_engram with 0 truncations.
  • Added unit tests in matmul_packed_precision_test.cc, gguf_quant_packed_test.cc, and dynamic_sparse_attention_op_test.cc.

…32 GDN kernels

- Add contiguous half-superblock (kChunk=128) SWAR unpack and prefill dequantization for Q6_K in MatMulPacked, eliminating 4-warp superblock L1 thrashing and unaligned 16-bit loads (-6.12 ms/cycle on Qwen3.8-Flash-Next GGUF decode).
- Add contiguous kChunk=128 FC1 decode (IQ3_XXS with 4-way bank-replicated LUT, IQ2_S, IQ4_XS) and aligned 36-byte 2-block IQ4_NL FC2 decode in MoEPacked (-1.71 ms/cycle).
- Add DynamicSparseAttentionFp16TcKernel<kSplitK, kCandTile> (split-K cp.async Tensor-Core FP16-KV decode with P_hi + P_residual probability split and two-buffer prefill), accelerating FP16-KV attention decode by 5.4x (2.71 -> 0.50 ms/cycle) and 8K prefill by 1.49x (486 -> 325 ms).
- Add single-wave m16n8k16 Tensor-Core HcDownTc8Kernel and HcUpPreTc8Kernel for rows=8, branches=4, hidden=2560, rank=320 BF16 decode (cutting weight DRAM traffic by 3.0x and saving ~3.5-4.5 ms/cycle) plus 320-thread NormVec8H2560Kernel for prefill.
- Fuse dual-A 3-slice FP32-state conversions (SmemGemmFp32StateDualS0Bf16 and SmemGemmFp32StateDualUBf16) and hoist LoadFragA in GatedDeltaNetChunkedKernel, reducing 3-slice passes per chunk from 5 to 3 and cutting 8K FP32 GDN prefill from 427 -> 343 ms (-19.6%).
- Match CTA block widths to column vector counts in QMoE expandInputRowsKernel and doActivationKernel.
- Add unit tests covering Q6_K contiguous/prefill MatMulPacked, kChunk=128 + aligned IQ4_NL MoEPacked, and FP16-KV Tensor-Core DSA decode/prefill.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 17:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical lane-local sum check can reset valid attention contributions, producing outputs of 0.5 instead of 1; both affected paths need correction and sparse-validity regression coverage for decode and prefill.

1 open issue
What changed in this PR

This PR optimizes ONNX Runtime’s CUDA kernels for prefill and speculative decoding, targeting packed-weight models, FP16 attention caches, and FP32 recurrent state.

Changes:

  • Improves GGUF weight decoding and adjusts QMoE launch sizes to reduce redundant work.
  • Adds specialized Tensor Core attention and hyper-connection kernels, and fuses FP32-state GatedDeltaNet computations.
  • Adds parity tests for packed-weight decoding and FP16-cache attention paths.
File Description
onnxruntime/​test/​contrib_ops/​matmul_packed_precision_test.cc Adds Q6_K decode and prefill parity coverage.
onnxruntime/​test/​contrib_ops/​gguf_quant_packed_test.cc Tests optimized MoEPacked weight decoding.
onnxruntime/​test/​contrib_ops/​dynamic_sparse_attention_op_test.cc Adds FP16-cache decode and prefill coverage.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_packed.cu Adds contiguous Q6_K decoding and faster prefill dequantization.
onnxruntime/​contrib_ops/​cuda/​quantization/​gguf_quant_cuda.cuh Adds optimized Q6_K and aligned IQ4_NL unpacking helpers.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_packed.cu Optimizes expert-weight decoding and lookup-table access.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_kernels.cu Matches input-expansion thread counts to row width.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_gemm_activation_kernels.cuh Matches activation thread counts to intermediate width.
onnxruntime/​contrib_ops/​cuda/​bert/​hyper_connection_low_rank_mix_impl.cu Adds eight-row Tensor Core kernels and specialized normalization.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_mma.cuh Adds fused dual-product FP32-state helpers and fragment reuse.
onnxruntime/​contrib_ops/​cuda/​bert/​gated_delta_net_impl.cu Integrates fused helpers into chunked GatedDeltaNet execution.
onnxruntime/​contrib_ops/​cuda/​bert/​dynamic_sparse_attention_prefill.cuh Adds tiled FP16-cache Tensor Core attention kernels.
onnxruntime/​contrib_ops/​cuda/​bert/​dynamic_sparse_attention_impl.cu Dispatches FP16-cache kernels with resource-aware tile and split selection.

Review effort: Balanced

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can commit the suggested changes from lintrunner.

- DynamicSparseAttentionFp16TcKernel: derive the online-softmax rescale from the
  row-uniform running max instead of the lane-local row sum. A lane without a
  valid (or non-underflowed) candidate in a tile reset the shared accumulator.
  Add a sparse-validity regression test for decode and prefill.
- NormVec8H2560Kernel: use __launch_bounds__(320, 4); (320, 6) exceeds the
  1536 threads/SM of SM 8.6/8.9/12.x and failed ptxas in the CUDA CI jobs.
- LaunchDynamicSparseAttention: take the cudaDeviceProp from the kernel and drop
  the per-launch cudaGetDevice/cudaDeviceGetAttribute calls.
- HC TC8 dispatch: also require 8 splits and 16-byte aligned scratch partials.
- Apply clang-format to the TC8 block (lintrunner suggestions).
- Document the FP16-cache tensor-core kernels and the switches that gate them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
Tianlei Wu (tianleiwu) merged commit 0da4e68 into main Nov 13, 2026
106 of 109 checks passed
@tianleiwu
Tianlei Wu (tianleiwu) deleted the tlwu/20261113/qwen38-gguf-rc6-cuda-opt branch November 13, 2026 23:30
Tianlei Wu (tianleiwu) added a commit to microsoft/onnxruntime-genai that referenced this pull request Nov 14, 2026
…LE VirtualUnlock, and add integrated GPU runtime profiles (#2965)

## Summary
Stacked on #2907 (`tlwu/20261107/qwen38-rc5-gguf-offload`) and paired
with
[microsoft/onnxruntime#34200](microsoft/onnxruntime#34200).

### Key Changes
1. **Enable `Dflash2Drafter` `fixed_single_request` Fast Path
(`src/models/qwen38_graph.cpp`)**:
- Records the drafter's `session.paged_attention.skip_input_sanitizer`
policy in its projected config (`SetQwen38DflashSanitizerPolicy`), where
`Dflash2Drafter` reads it to enable padding, and passes
`fixed_single_request` when `Qwen38DflashPadsDecodeSteps` holds (single
slot, windowed drafter, ORT API `>= 32`, sanitizer skipped). The
`1..verify_rows_` decode warmup loop then stops after `r == 1`;
otherwise the drafter keeps one captured graph per decode row count.
- Consolidates 8 per-row-count decode DFlash2 CUDA graphs (`r = 1..8`)
into a single padded 8-row CUDA graph (`40 -> 5` warmup runs, cutting
model load time by up to **`-37.1%`, `13.77 s -> 8.67 s`**) and replaces
5 per-step `StoreBytesKernel` launches + 6 `StepTensor` resizes with 1
packed 272-byte `StoreBytesKernel` (**`-47%`** DFlash2 input-prep +
launch gap: `148.1 us -> 79.1 us`).
2. **Eliminate Per-Row `VirtualUnlock` Syscall Storm in Staged PLE
Gather (`src/qwen38_plane/qwen38_plane.cu`)**:
- Removes the per-miss `VirtualUnlock` loops in
`host_gather_ple_rows_batch` and `PleGatherPool::GatherSpans` on
un-`VirtualLock`ed file-mapped PLE views while retaining batched
`PrefetchVirtualMemory`.
- Eliminates up to `98,304` `VirtualUnlock` calls per 8K prefill and
`96` per decode step on Windows, cutting the decode PLE host gather gap
by **`-36.9%` (`2,040 us -> 1,288 us`, `-0.75 ms/cycle`)**.
3. **Unified-Memory (`is_integrated: true`) `runtime_profiles`
(`tools/python/qwen38_flash_next/build_package.py`,
`build_package_gguf.py`)**:
- Adds `integrated-gpu-128k` / `integrated-gpu-256k` (`QMoE`) and
`integrated-gpu-32k` / `64k` / `128k` / `256k` (`GGUF`) runtime profiles
gated on `minimum_free_device_memory_bytes` so unified-memory GPUs (`SM
12.1` NVIDIA RTX Spark / DGX Spark) automatically select `num_blocks =
1037` (`256K`) instead of falling through discrete-only (`is_integrated:
false`) profiles.
- Every integrated tier also sets `minimum_total_device_memory_bytes`,
which the engine's `ValidateRuntimeProfiles` requires on every profile,
to its free-memory minimum; `runtime_profiles_problem` in the qwen38
tool tests mirrors that validation for the QMoE and GGUF profiles.
4. **Windows Qualification Tool Portability
(`tools/python/qwen38_flash_next/qwen38_engine_util.py`)**:
- Automatically overlays `"plane_library": ""` on Windows (`os.name ==
"nt"`) when a Linux-built package specifies a `.so` `plane_library` that
exists on disk without a `.dll` alias, so `Qwen38Plane` loads the
bundled `onnxruntime-genai-cuda.dll` out-of-the-box.
- Tolerates duplicate `register_execution_provider_library` calls across
multiple `load_model` invocations in one Python process.

---

## Validation
- `engine_unit_tests.exe --ep_dir ...
--gtest_filter=*Qwen38*:*RuntimeProfile*:*Dflash*`: **`586 passed, 0
failed`**.
- `pytest test/python/builder/test_qwen38_qualification_tools.py
tools/python/qwen38_flash_next/test_ram_budget.py`: **`190 passed`**.
- Verified out-of-the-box `smoke.py` and `2K–255K + 1K` benchmarks on
NVIDIA RTX Spark (`SM 12.1`, Windows 11 ARM64) across both
`qwen_flash_next_rc6_2b375_fp32` and
`qwen_flash_next_gguf_ud_iq3_xxs_q8_engram_fp32_fp16kv` (bit-identical
greedy token hashes).
- Follow-up `cf935e9` (Linux CPU build): `engine_unit_tests` **`1721
passed, 0 failed`** (including the new `Qwen38DflashSanitizerTest.*`);
`pytest test/python/builder/test_qwen38_qualification_tools.py
test/python/builder/test_qwen38_package_builders.py
test/python/test_build_package_gguf.py`: **`318 passed`**; the generated
QMoE and GGUF `runtime_profiles` load through `og.Config`.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants