Repository navigation
[CUDA] Add opt-in single-warp INT4 GEMV for FP16 decode - #33952
Sunghoon Choi (hanbitmyths) wants to merge 4 commits into
Conversation
Extract the independent GEMV optimization, controls, documentation, numeric differentials, and native dispatch tests from the CUDA graph-ownership PR. Existing kernels remain the default and unsupported shapes or metadata keep their existing fallbacks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Sunghoon Choi <sunghcho@microsoft.com>
Use only ORT_ENABLE_SINGLE_WARP_INT4_GEMV for operator-scoped opt-in. Preserve tensor-shape dispatch and default behavior; cover generic enablement, disablement, global override, and invalid controls without model-named aliases. Signed-off-by: Sunghoon Choi <sunghcho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The CUDA numerical regression leaves CPU fallback enabled, allowing tests to pass even if FP16 MatMulNBits nodes stop running on CUDA.
1 open issue
What changed in this PR
Adds an opt-in single-warp INT4 GEMV path to CUDA MatMulNBits for three FP16 decode shapes. This separates the math optimization from the companion CUDA graph lifecycle work; consumer-GPU performance validation remains pending.
Changes:
- Parameterizes kernel tiling and reduction unrolling while preserving defaults for existing callers.
- Adds shape-restricted dispatch controlled by
ORT_ENABLE_SINGLE_WARP_INT4_GEMV, with the existing global disable override. - Documents the controls and adds eligibility, launch-geometry, fallback, and numerical tests.
| File | Description |
|---|---|
onnxruntime/test/providers/cuda/test_cases/matmul_nbits_gemv_test.cc |
Adds eligibility and captured launch-geometry tests. |
onnxruntime/test/contrib_ops/matmul_4bits_test.cc |
Adds numerical tests across enablement settings and projection shapes. |
onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.cuh |
Defines shape eligibility and declares the single-warp launcher. |
onnxruntime/contrib_ops/cuda/quantization/matmul_4bits.cu |
Adds opt-in dispatch with the global disable override. |
onnxruntime/contrib_ops/cuda/quantization/matmul_4bits_m1_impl.cuh |
Parameterizes columns per block and reduction unrolling. |
onnxruntime/contrib_ops/cuda/quantization/matmul_4bits_half.cu |
Implements the FP16 single-warp kernel launcher. |
docs/contrib_ops/cuda/matmul_nbits.md |
Documents enablement, supported shapes, and disable behavior. |
Review effort: Balanced
Signed-off-by: Sunghoon Choi <sunghcho@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CUDA numerical-routing review follow-upAddressed the CPU-fallback comment in be5ed6f2. The new numerical regression now sets the existing Validation:
The published change is test-only: no GPU kernel, eligibility, runtime control, or model-specific switch was added or changed. No fresh throughput/quality result is claimed. The consumer-GPU benchmarking request is still hardware-blocked: the available devices are A100s, not consumer GPUs. This PR remains draft, and its existing historical performance/provenance caveats are unchanged. |
Exclude 2048x2048 from the single-warp whitelist to avoid its H200 hot-cache regression. Preserve the other opt-in shapes and update eligibility, dispatch coverage, and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Akshay Sonawane (apsonawane)
left a comment
Akshay Sonawane (apsonawane)
left a comment
There was a problem hiding this comment.
AI-assisted source review at 8fe60f4e9b1b50fd6be38acf2616fc275de35978.
Reviewed the single-warp mapping, packed-weight/scale access, reductions, alignment, launch geometry, opt-in/global-disable precedence, fallback preservation, and test controls. Eligibility is restricted to FP16 M=1/block-32/no-zero-point/no-bias and the two whitelisted (N,K) shapes. The earlier CPU-fallback concern is addressed.
No new actionable findings in this pass. Independent host mapping probes covered every K element exactly once for K=2048 and K=6144, with corresponding packed bytes/scales in bounds. Reported head checks are 102 successful and one neutral. No CUDA compilation, GPU numerical execution, or performance benchmark was run; this review does not independently substantiate the claimed speedup.
Summary
(N,K)=(1024,2048)and(2048,6144).ORT_ENABLE_SINGLE_WARP_INT4_GEMV=1; the global specialization-disable control wins. Other dispatch and metadata fallbacks are unchanged.(2048,2048)remains eight-warp because its one-warp hot-cache path regresses; historical gains for that shape do not describe current dispatch.Companion: ORT #33932 handles capture/lifecycle.
Reported validation: 10 eligibility/graph + 2 numerical CUDA cases; 448 CUDA-event cells verify the excluded-shape fallback and bit-identical nonzero outputs. No CPU fallback.
Draft: consumer GPUs and the narrowed whitelist's model gain are unmeasured. No Qwen2 model speedup or default-enablement recommendation.
Performance: Qwen3-VL-2B
Combined package/runtime results, not isolated gains from this PR. Production uses decoder INT4/group32; optimized uses 66 INT8 + 131 INT4 projections, FP16 vision/embeddings/KV, pruned prefill and decoder capture.
A100-SXM4-80GB
Historical production: Oct 28 (
b679913a2310); optimized: Nov 5 frozen clean gate.H200
Baseline: official
qwen3-vl-2b-instruct-cuda-gpu:2on published ORT 1.31.1 / GenAI 0.19.0, CUDA 13. Optimized: retained Nov 8 FPA1 results, not rerun.Controls: batch 1, context 8192, FP16 KV, chunk 2048, 128 outputs, 8 inference threads, no prefix reuse. TTFT is HTTP first-token latency; decode uses 127 first-to-last intervals. A100: 7 production / 14 optimized samples. H200 baseline: 14 samples/row after warmups, GPU 1 / CPU cores 8-15; pre-rendered prompts, tokenization/image processing inside HTTP.
Limits: H200 is unpaired: prompts/harness differ; the new 448x448 image has 244 input tokens versus 356 historically. Ratios are descriptive, not causal speedups. A100 is not matched to the new H200 runtime; no cross-GPU hardware claim. Optimized ORT
5481f20b/ GenAI4e51307d+41c96365are frozen, not current PR-head benchmarks.Review required: historical optimized export loses 9 AI2D answers versus the accepted native recipe; padded/batched capture limitations remain. No default promotion.