Repository navigation
[CUDA] Optimize Q6_K/IQ3_XXS/IQ4_NL GGUF, FP16-KV DSA, M=8 HC, and FP32 GDN kernels - #34200
Merged
Tianlei Wu (tianleiwu) merged 2 commits intoNov 13, 2026
Merged
Conversation
…32 GDN kernels - Add contiguous half-superblock (kChunk=128) SWAR unpack and prefill dequantization for Q6_K in MatMulPacked, eliminating 4-warp superblock L1 thrashing and unaligned 16-bit loads (-6.12 ms/cycle on Qwen3.8-Flash-Next GGUF decode). - Add contiguous kChunk=128 FC1 decode (IQ3_XXS with 4-way bank-replicated LUT, IQ2_S, IQ4_XS) and aligned 36-byte 2-block IQ4_NL FC2 decode in MoEPacked (-1.71 ms/cycle). - Add DynamicSparseAttentionFp16TcKernel<kSplitK, kCandTile> (split-K cp.async Tensor-Core FP16-KV decode with P_hi + P_residual probability split and two-buffer prefill), accelerating FP16-KV attention decode by 5.4x (2.71 -> 0.50 ms/cycle) and 8K prefill by 1.49x (486 -> 325 ms). - Add single-wave m16n8k16 Tensor-Core HcDownTc8Kernel and HcUpPreTc8Kernel for rows=8, branches=4, hidden=2560, rank=320 BF16 decode (cutting weight DRAM traffic by 3.0x and saving ~3.5-4.5 ms/cycle) plus 320-thread NormVec8H2560Kernel for prefill. - Fuse dual-A 3-slice FP32-state conversions (SmemGemmFp32StateDualS0Bf16 and SmemGemmFp32StateDualUBf16) and hoist LoadFragA in GatedDeltaNetChunkedKernel, reducing 3-slice passes per chunk from 5 to 3 and cutting 8K FP32 GDN prefill from 427 -> 343 ms (-19.6%). - Match CTA block widths to column vector counts in QMoE expandInputRowsKernel and doActivationKernel. - Add unit tests covering Q6_K contiguous/prefill MatMulPacked, kChunk=128 + aligned IQ4_NL MoEPacked, and FP16-KV Tensor-Core DSA decode/prefill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A critical lane-local sum check can reset valid attention contributions, producing outputs of 0.5 instead of 1; both affected paths need correction and sparse-validity regression coverage for decode and prefill.
1 open issue
What changed in this PR
This PR optimizes ONNX Runtime’s CUDA kernels for prefill and speculative decoding, targeting packed-weight models, FP16 attention caches, and FP32 recurrent state.
Changes:
- Improves GGUF weight decoding and adjusts QMoE launch sizes to reduce redundant work.
- Adds specialized Tensor Core attention and hyper-connection kernels, and fuses FP32-state GatedDeltaNet computations.
- Adds parity tests for packed-weight decoding and FP16-cache attention paths.
| File | Description |
|---|---|
onnxruntime/test/contrib_ops/matmul_packed_precision_test.cc |
Adds Q6_K decode and prefill parity coverage. |
onnxruntime/test/contrib_ops/gguf_quant_packed_test.cc |
Tests optimized MoEPacked weight decoding. |
onnxruntime/test/contrib_ops/dynamic_sparse_attention_op_test.cc |
Adds FP16-cache decode and prefill coverage. |
onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cu |
Adds contiguous Q6_K decoding and faster prefill dequantization. |
onnxruntime/contrib_ops/cuda/quantization/gguf_quant_cuda.cuh |
Adds optimized Q6_K and aligned IQ4_NL unpacking helpers. |
onnxruntime/contrib_ops/cuda/moe/moe_packed.cu |
Optimizes expert-weight decoding and lookup-table access. |
onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_kernels.cu |
Matches input-expansion thread counts to row width. |
onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemm_activation_kernels.cuh |
Matches activation thread counts to intermediate width. |
onnxruntime/contrib_ops/cuda/bert/hyper_connection_low_rank_mix_impl.cu |
Adds eight-row Tensor Core kernels and specialized normalization. |
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_mma.cuh |
Adds fused dual-product FP32-state helpers and fragment reuse. |
onnxruntime/contrib_ops/cuda/bert/gated_delta_net_impl.cu |
Integrates fused helpers into chunked GatedDeltaNet execution. |
onnxruntime/contrib_ops/cuda/bert/dynamic_sparse_attention_prefill.cuh |
Adds tiled FP16-cache Tensor Core attention kernels. |
onnxruntime/contrib_ops/cuda/bert/dynamic_sparse_attention_impl.cu |
Dispatches FP16-cache kernels with resource-aware tile and split selection. |
Review effort: Balanced
- DynamicSparseAttentionFp16TcKernel: derive the online-softmax rescale from the row-uniform running max instead of the lane-local row sum. A lane without a valid (or non-underflowed) candidate in a tile reset the shared accumulator. Add a sparse-validity regression test for decode and prefill. - NormVec8H2560Kernel: use __launch_bounds__(320, 4); (320, 6) exceeds the 1536 threads/SM of SM 8.6/8.9/12.x and failed ptxas in the CUDA CI jobs. - LaunchDynamicSparseAttention: take the cudaDeviceProp from the kernel and drop the per-launch cudaGetDevice/cudaDeviceGetAttribute calls. - HC TC8 dispatch: also require 8 splits and 16-byte aligned scratch partials. - Apply clang-format to the TC8 block (lintrunner suggestions). - Document the FP16-cache tensor-core kernels and the switches that gate them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Baiju Meswani (baijumeswani)
approved these changes
Nov 13, 2026
Tianlei Wu (tianleiwu)
deleted the
tlwu/20261113/qwen38-gguf-rc6-cuda-opt
branch
November 13, 2026 23:30
Tianlei Wu (tianleiwu)
added a commit
to microsoft/onnxruntime-genai
that referenced
this pull request
Nov 14, 2026
…LE VirtualUnlock, and add integrated GPU runtime profiles (#2965) ## Summary Stacked on #2907 (`tlwu/20261107/qwen38-rc5-gguf-offload`) and paired with [microsoft/onnxruntime#34200](microsoft/onnxruntime#34200). ### Key Changes 1. **Enable `Dflash2Drafter` `fixed_single_request` Fast Path (`src/models/qwen38_graph.cpp`)**: - Records the drafter's `session.paged_attention.skip_input_sanitizer` policy in its projected config (`SetQwen38DflashSanitizerPolicy`), where `Dflash2Drafter` reads it to enable padding, and passes `fixed_single_request` when `Qwen38DflashPadsDecodeSteps` holds (single slot, windowed drafter, ORT API `>= 32`, sanitizer skipped). The `1..verify_rows_` decode warmup loop then stops after `r == 1`; otherwise the drafter keeps one captured graph per decode row count. - Consolidates 8 per-row-count decode DFlash2 CUDA graphs (`r = 1..8`) into a single padded 8-row CUDA graph (`40 -> 5` warmup runs, cutting model load time by up to **`-37.1%`, `13.77 s -> 8.67 s`**) and replaces 5 per-step `StoreBytesKernel` launches + 6 `StepTensor` resizes with 1 packed 272-byte `StoreBytesKernel` (**`-47%`** DFlash2 input-prep + launch gap: `148.1 us -> 79.1 us`). 2. **Eliminate Per-Row `VirtualUnlock` Syscall Storm in Staged PLE Gather (`src/qwen38_plane/qwen38_plane.cu`)**: - Removes the per-miss `VirtualUnlock` loops in `host_gather_ple_rows_batch` and `PleGatherPool::GatherSpans` on un-`VirtualLock`ed file-mapped PLE views while retaining batched `PrefetchVirtualMemory`. - Eliminates up to `98,304` `VirtualUnlock` calls per 8K prefill and `96` per decode step on Windows, cutting the decode PLE host gather gap by **`-36.9%` (`2,040 us -> 1,288 us`, `-0.75 ms/cycle`)**. 3. **Unified-Memory (`is_integrated: true`) `runtime_profiles` (`tools/python/qwen38_flash_next/build_package.py`, `build_package_gguf.py`)**: - Adds `integrated-gpu-128k` / `integrated-gpu-256k` (`QMoE`) and `integrated-gpu-32k` / `64k` / `128k` / `256k` (`GGUF`) runtime profiles gated on `minimum_free_device_memory_bytes` so unified-memory GPUs (`SM 12.1` NVIDIA RTX Spark / DGX Spark) automatically select `num_blocks = 1037` (`256K`) instead of falling through discrete-only (`is_integrated: false`) profiles. - Every integrated tier also sets `minimum_total_device_memory_bytes`, which the engine's `ValidateRuntimeProfiles` requires on every profile, to its free-memory minimum; `runtime_profiles_problem` in the qwen38 tool tests mirrors that validation for the QMoE and GGUF profiles. 4. **Windows Qualification Tool Portability (`tools/python/qwen38_flash_next/qwen38_engine_util.py`)**: - Automatically overlays `"plane_library": ""` on Windows (`os.name == "nt"`) when a Linux-built package specifies a `.so` `plane_library` that exists on disk without a `.dll` alias, so `Qwen38Plane` loads the bundled `onnxruntime-genai-cuda.dll` out-of-the-box. - Tolerates duplicate `register_execution_provider_library` calls across multiple `load_model` invocations in one Python process. --- ## Validation - `engine_unit_tests.exe --ep_dir ... --gtest_filter=*Qwen38*:*RuntimeProfile*:*Dflash*`: **`586 passed, 0 failed`**. - `pytest test/python/builder/test_qwen38_qualification_tools.py tools/python/qwen38_flash_next/test_ram_budget.py`: **`190 passed`**. - Verified out-of-the-box `smoke.py` and `2K–255K + 1K` benchmarks on NVIDIA RTX Spark (`SM 12.1`, Windows 11 ARM64) across both `qwen_flash_next_rc6_2b375_fp32` and `qwen_flash_next_gguf_ud_iq3_xxs_q8_engram_fp32_fp16kv` (bit-identical greedy token hashes). - Follow-up `cf935e9` (Linux CPU build): `engine_unit_tests` **`1721 passed, 0 failed`** (including the new `Qwen38DflashSanitizerTest.*`); `pytest test/python/builder/test_qwen38_qualification_tools.py test/python/builder/test_qwen38_package_builders.py test/python/test_build_package_gguf.py`: **`318 passed`**; the generated QMoE and GGUF `runtime_profiles` load through `og.Config`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
This PR optimizes CUDA prefill and speculative-decode (
M = 8verify) kernels acrossMatMulPacked,MoEPacked,DynamicSparseAttention(fp16_kv),HyperConnectionLowRankMix,GatedDeltaNet(fp32state), andQMoEhelper launches for Qwen3.8-Flash-Next GGUF (UD-IQ3_XXS) andQMoE(2b375_fp32) models:MatMulPackedQ6_KContiguous Half-Superblock (kChunk = 128) & Aligned SWAR Unpack (gguf_quant_cuda.cuh,matmul_packed.cu):Q6_Khalf-superblocks per warp (LoadQ6KHalf128Stage+DecodeQ6KHalf128GroupBf16x2SWAR unpack) inMatMulPackedBf16TcWarpKSplitKernel(KW=10, MT=1forN<=1024;KW=8forK=6144, N<=2560;KW=4, MT=2forN>2560) and enablesQ6_KinMatMulPackedBf16DequantFastKernel.MatMulPackeddecode time from19.60 -> 13.48 ms/cycle(-6.12 ms/cyc;N=512Q6_Klatency drops5.5xfrom33.8 us -> ~6.1 us).MoEPackedContiguouskChunk = 128(IQ3_XXS,IQ2_S,IQ4_XS) FC1 Decode & Aligned 36B Pair (IQ4_NL) FC2 Decode (gguf_quant_cuda.cuh,moe_packed.cu):MoEPackedFc1DecodeWarpKSplitKernelwith 4-way bank-replicatediq3xxs_grid(s_lut[1024]), and decodes 4-byte-aligned 36B (2 x block_iq4_nl) pairs (DecodeIq4NlPair64AlignedBf16x2) inMoEPackedFc2Bf16TcKernel<1, kGgmlTypeIQ4_NL>(21.47 -> 19.76 ms/cyc, reaching295 GB/svs301 GB/speak on RTX Spark).DynamicSparseAttentionFp16TcKernel<kSplitK, kCandTile>(dynamic_sparse_attention_prefill.cuh,dynamic_sparse_attention_impl.cu):GroupedDecodeDynamicSparseAttentionKernelfallback onfp16_kvwith a split-Kcp.asyncTensor-Corem16n8k16FP16-KV kernel (kCandTile = 32on 100 KB/SM GPUs,P_hi + P_residual22-bit probability split) for decode and two-buffer<false, 64>for prefill.5.4x(2.71 -> 0.50 ms/cyc) and 8K prefill by1.49x(486.0 -> 325.3 ms).m16n8k16Tensor-CoreHcDownTc8Kernel&HcUpPreTc8Kernel(rows = 8) +NormVec8H2560Kernel(hyper_connection_low_rank_mix_impl.cu):LaunchFusedRows<T, 4, 2>(3.81 GB/cycweight reads) onrows=8, branches=4, hidden=2560, rank=320(BF16) with single-wave BF16 Tensor-Core kernels (grid=(6,8,1)andgrid=(80,1,1)) with PDL weight prefetch (1.27 GB/cyc=3.0xless weight DRAM traffic, saving~3.5–4.5 ms/cyc), plus 320-threadNormVec8H2560Kernelfor prefill.GatedDeltaNetChunkedKernel(gated_delta_net_mma.cuh,gated_delta_net_impl.cu):LoadFragAacross slices and fuses(k @ S0, Qg @ S0, R epilogue)(SmemGemmFp32StateDualS0Bf16) and(P @ U, Kd^T @ U)(SmemGemmFp32StateDualUBf16), reducing 3-slice FP32->BF16 passes per chunk from5 -> 3(35 -> 12barriers) while preserving exact 24-bit IEEE-754 precision (427.0 -> 343.3 msat 8K prefill,-19.6%).QMoEexpandInputRowsKernel&doActivationKernel(moe_kernels.cu,moe_gemm_activation_kernels.cuh):320forhidden_size=2560;64/96forinter_size=512/640), eliminating 69%–75% idle tail threads (-31 msat 8K prefill).End-to-End Performance & Accuracy (
2K–255K + 1Kon NVIDIA RTX SparkSM 12.1)1. ORT
gguf_ud_iq3_xxs_q8_engram(DFlash2 K=7) vs. Baseline &llama.cpp + MTP(UD-IQ3_XXS + Q8_0)Prompt + Gen)ms/cyc)ms/cyc)llama.cpp + MTPTTFT / Decodellama.cpp + MTP2K + 1K1.509 s/52.71 tok/s(67.62 ms)1.468 s/66.70 tok/s(52.53 ms)2.582 s/57.81 tok/s1.76xprefill /+15.4%decode8K + 1K5.468 s/46.42 tok/s(67.81 ms)5.231 s/66.49 tok/s(53.80 ms)9.616 s/54.22 tok/s1.84xprefill /+22.6%decode16K + 1K10.822 s/51.75 tok/s(67.24 ms)10.425 s/61.24 tok/s(54.06 ms)19.331 s/58.48 tok/s1.85xprefill /+4.7%decode32K + 1K21.663 s/48.44 tok/s(68.34 ms)20.929 s/63.54 tok/s(54.76 ms)40.822 s/54.69 tok/s1.95xprefill /+16.2%decode64K + 1K44.858 s/53.26 tok/s(70.09 ms)42.684 s/58.04 tok/s(57.23 ms)81.592 s/50.67 tok/s1.91xprefill /+14.5%decode128K + 1K94.444 s/42.02 tok/s(72.02 ms)91.288 s/59.12 tok/s(58.06 ms)172.053 s/45.89 tok/s1.88xprefill /+28.8%decode255K + 1K206.564 s/42.15 tok/s(81.71 ms)201.523 s/45.41 tok/s(67.65 ms)378.801 s/33.42 tok/s1.88xprefill /+35.9%decode2. ORT
rc6_2b375_fp32(DFlash2 K=7,fp16_kv + FP32 GDN) Before vs. AfterPrompt + Gen)Prefill tok/s)Prefill tok/s)ms/cyc)ms/cyc)2K + 1K1.232 s(1,661.9)1.204 s(1,700.5 tok/s)62.78 tok/s(54.68 ms)81.20 tok/s(47.91 ms,-12.4%)8K + 1K4.607 s(1,778.1)4.352 s(1,882.3 tok/s)63.91 tok/s(54.44 ms)76.47 tok/s(48.65 ms,-10.6%)32K + 1K18.642 s(1,757.8)17.589 s(1,862.9 tok/s)58.34 tok/s(56.02 ms)65.65 tok/s(50.11 ms,-10.5%)128K + 1K81.111 s(1,616.0)77.106 s(1,699.9 tok/s)48.82 tok/s(58.37 ms)51.38 tok/s(53.52 ms,-8.3%)255K + 1K179.976 s(1,450.9)172.099 s(1,517.3 tok/s)51.17 tok/s(68.93 ms)50.52 tok/s(63.08 ms,-8.5%)Validation
sm_75,sm_80,sm_86,sm_90, andsm_120fwith-Werror all-warnings --Werror default-stream-launch.M = 8) vs.--force-reject(M = 1) greedy equivalence verified128/128(100%bit-identical) on bothrc6_2b375_fp32andgguf_ud_iq3_xxs_q8_engram.0/80(10items):9/10(90%, matching baseline item-for-item) onrc6_2b375_fp32and10/10(100%) ongguf_ud_iq3_xxs_q8_engramwith0truncations.matmul_packed_precision_test.cc,gguf_quant_packed_test.cc, anddynamic_sparse_attention_op_test.cc.