Repository navigation
[CUDA] Route short shared-buffer GQA verify steps to speculative-decoding XQA - #34198
Rui Ren (rui-ren) wants to merge 1 commit into
Conversation
…ding XQA Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The parity tests do not exercise the changed long-context split policy, leaving split-count behavior and workspace/launch agreement unverified without fixed actual past lengths of at least 8K in FP16 and BF16 cases.
1 open issue
What changed in this PR
This PR routes eligible short, multi-token GroupQueryAttention verification steps through CUDA XQA’s speculative-decoding kernel, reducing repeated KV-cache reads at long context.
Changes:
- Adds FP16/BF16 speculative XQA kernels for head size 128 and group sizes 4, 8 and 16.
- Integrates guarded dispatch, workspace allocation and per-token-tile KV splitting.
- Adds backend-selection parity tests and documents eligibility and environment controls.
| File | Description |
|---|---|
onnxruntime/test/python/transformers/test_gqa.py |
Adds speculative XQA parity and environment-disable tests. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_spec_dec_loader.cu |
Implements support checks and launch/resource dispatch. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_spec_dec_impl.cuh |
Instantiates speculative kernels for supported group sizes. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_spec_dec_impl_gen.cuh |
Implements causal-mask preparation, workspace sizing and launch. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_spec_dec_fp16_128.cu |
Adds FP16 head-size-128 instantiations. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_spec_dec_bf16_128.cu |
Adds BF16 head-size-128 instantiations. |
onnxruntime/contrib_ops/cuda/bert/xqa/xqa_loader.h |
Declares speculative XQA interfaces and contracts. |
onnxruntime/contrib_ops/cuda/bert/xqa/mha_impl.cuh |
Adds opt-in per-token-tile split sizing. |
onnxruntime/contrib_ops/cuda/bert/group_query_attention.h |
Adds enablement and cached shared-memory eligibility state. |
onnxruntime/contrib_ops/cuda/bert/group_query_attention.cc |
Selects eligible speculative steps and allocates workspace. |
onnxruntime/contrib_ops/cuda/bert/group_query_attention_impl.cu |
Launches speculative XQA after existing preprocessing. |
onnxruntime/contrib_ops/cuda/bert/attention_data.h |
Tracks speculative XQA selection. |
docs/contrib_ops/cuda/gqa.md |
Documents routing, constraints, controls and tests. |
Review effort: Balanced
| const uint32_t nbTokenBlocksPerGrp = divUp(qSeqLen * headGrpSize, rowsPerBlock); | ||
| #if defined(XQA_SPEC_DEC_SPLIT_PER_TOKEN_BLOCK) | ||
| // Every token block of a head group is an independent CTA row, so size the KV split for all of them. | ||
| const uint32_t nbSubSeqPerSeq = computeNbSubSeqPerSeqMHA(prop, batchSize, nbKHeads * nbTokenBlocksPerGrp, maxSeqLen); |
Tianlei Wu (tianleiwu)
left a comment
There was a problem hiding this comment.
Review Summary
The contiguous-cache speculative-XQA (SPEC_DEC) integration, per-token-tile KV split policy (XQA_SPEC_DEC_SPLIT_PER_TOKEN_BLOCK), CUDA Graph capture handling, and UnpackRoPEAppend reuse are clean and well-motivated. (Note: the three failing CI checks on this PR — Kernel Documentation Generation, Windows GPU Kernel Documentation Validation, and TestQMoESharedDirect::test_cuda_int3_tensor_core_compatibility_shadow — are unrelated main failures.)
Two correctness/soundness issues should be addressed before merging:
- Cache element type guard (
std::is_same_v<T, U>):GroupQueryAttention<T, U>is registered forUin{T, int8_t, Float8E4M3FN, uint8_t}, whileis_inputs_quantizedonly checksk_quant_type_ != NONE || v_quant_type_ != NONE. Addingstd::is_same_v<T, U>inComputeInternalandExtremeDecoding(matchingCudnnSdpaAttentionandUnfusedGqaAttention) prevents 1-byte/4-bitT_CACHEinstantiations with defaultk_quant_type="NONE"from dereferencing null scales inUnpackRoPEAppendand reading out-of-bounds inLaunchXQASpecDecKernel. - Level-1 workspace estimator soundness:
EstimateGroupQueryAttentionWorkspaceis already used as a provenLevel1MemoryEstimateupper bound inCUDAExecutionProvider::GetCapabilityunderkOrtSessionOptionsResourceCudaPartitioningSettings(which fails closed when an unmodeled route like cuDNN SDPA is reachable). Leaving speculative XQA unmodeled allows runtime workspace (M_TILESIZE = 32across token blocks + mask) to exceed the reported Level-1 bound.
See inline comments for details and a few smaller suggestions.
| // head as in FlashAttention, which dominates long-context verify cost. | ||
| if (!data.use_xqa && enable_xqa_spec_dec_ && query_seqlens_data == nullptr && !parameters.is_first_prompt && | ||
| parameters.sequence_length > 1 && parameters.kv_sequence_length == parameters.sequence_length && | ||
| parameters.past_present_share_buffer && !parameters.is_windowed_kv_cache && !is_inputs_quantized && |
There was a problem hiding this comment.
!is_inputs_quantized only checks k_quant_type_ != NONE || v_quant_type_ != NONE (where omitted or unrecognized attribute strings map to KVQuantizationType::NONE), while GroupQueryAttention<T, U> is registered for U in {T, int8_t, Float8E4M3FN, uint8_t}.
Every other non-quantized backend (IsGQACudnnSdpaCoreEligibleSeqFree<T, U>, CudnnSdpaAttention, and UnfusedGqaAttention) additionally requires std::is_same_v<T, U>. Without that check here (and an if constexpr (std::is_same_v<T, U>) guard around LaunchXQASpecDecKernel in ExtremeDecoding at group_query_attention_impl.cu:1102), a T_CACHE = int8 / Float8E4M3FN / uint8 node with k_quant_type="NONE" will enter use_xqa_spec_dec, dereference null k_scale/v_scale in UnpackRoPEAppend<T, U>, and reinterpret 1-byte/4-bit present_key/present_value buffers as 2-byte FP16/BF16 heads inside type-erased LaunchXQASpecDecKernel (reading 2x-4x past the cache allocation).
| parameters.past_present_share_buffer && !parameters.is_windowed_kv_cache && !is_inputs_quantized && | |
| std::is_same_v<T, U> && parameters.past_present_share_buffer && !parameters.is_windowed_kv_cache && !is_inputs_quantized && |
| device_prop, kIsBf16, parameters.batch_size, parameters.sequence_length, parameters.num_heads, | ||
| parameters.kv_num_heads, parameters.head_size, parameters.seqlen_present_kv_cache); | ||
| ORT_RETURN_IF_NOT(workspace_bytes > 0, "Speculative XQA workspace sizing failed."); | ||
| xqa_scratch_buffer = this->GetScratchBuffer<void>(workspace_bytes, GetComputeStream(context)); |
There was a problem hiding this comment.
Regarding the PR note that EstimateGroupQueryAttentionWorkspace is advisory today: while Level-2 DeclareWorkspaceRequirements is not yet enforced in an offset plan, Level-1 CUDAExecutionProvider::GetCapability (onnxruntime/core/providers/cuda/cuda_execution_provider.cc:4098-4136) already uses EstimateGroupQueryAttentionWorkspace as a proven upper bound (Level1MemoryEstimate) under kOrtSessionOptionsResourceCudaPartitioningSettings, failing closed (std::nullopt) whenever a reachable backend such as cuDNN SDPA has no graph-free bound.
Because speculative XQA uses M_TILESIZE = 32 for all group sizes (8,192 B/subseq vs 2,048 B/subseq in single-token XqaBackendEnvelope for group_size = 4), token_blocks = ceil(S * G / 32), MaskBytes, and S-token qkv_preprocess_bytes, this allocation can exceed the reported Level-1 bound (for example, sdpa_kernel = MATH, B = 1, S = 2, num_heads = 512, kv_num_heads = 128, head_size = 128, C = 2 on SM80 allocates 1,081,984 B at runtime vs 864,512 B estimated). Please either include the speculative-XQA envelope in GetGQAWorkspaceAggregateForBounds / GQAWorkspaceEstimateConfig or fail closed (Unavailable) when speculative XQA is reachable until modeled.
| device_prop, static_cast<uint32_t>(batch_size), static_cast<uint32_t>(kv_num_heads) * token_blocks, | ||
| static_cast<uint32_t>(max_seq_len)); | ||
| return MaskBytes(batch_size, sequence_length) + SemaphoreBytes(batch_size, kv_num_heads, sequence_length) + | ||
| GetScratchSize(nb_seq, nb_sub_seq_per_seq); |
There was a problem hiding this comment.
In mha_impl.cuh:1301, isMultiBlock is allowMultiBlockMode && (gridDim.x != 1), so when nb_sub_seq_per_seq == 1 (max_seq_len <= 256 or batch_size * kv_num_heads * token_blocks >= device_prop.multiProcessorCount), kernel_mha never accesses semaphores or scratch. Returning MaskBytes(batch_size, sequence_length) when nb_sub_seq_per_seq <= 1 and skipping the Fill<int32_t> launch on semaphores (passing nullptr for semaphores and scratch in Launch below) avoids hundreds of KB of unused scratch and saves a redundant kernel launch per GQA layer on single-split steps.
Also note that launchMHA throws std::runtime_error via checkCuda(...) on CUDA errors (making return CUDA_CALL(cudaGetLastError()); at line 154 unreachable for launch/symbol failures); catching const std::exception& and checking CUDA_RETURN_IF_ERROR(cudaGetLastError()) after XqaSpecDecCausalMaskKernel keeps errors inside ORT's Status contract.
| size_t GetXQASpecDecWorkspaceSize(const cudaDeviceProp& device_prop, bool is_bf16, int batch_size, | ||
| int sequence_length, int num_heads, int kv_num_heads, int head_size, | ||
| int max_seq_len) { | ||
| if (!IsXQASpecDecSupported(head_size, num_heads, kv_num_heads, sequence_length)) { |
There was a problem hiding this comment.
Unlike LaunchXQAKernel and GetXQAScratchSize in xqa_loader_fp16.cu, LaunchXQASpecDecKernel and GetXQASpecDecWorkspaceSize do not guard device_prop.major >= 8, batch_size > 0, or max_seq_len > 0. In multi-arch builds (HAS_SM80_OR_LATER defined alongside SM75), passing batch_size <= 0 divides by zero in computeNbSubSeqPerSeqMHA, and calling LaunchXQASpecDecKernel on pre-SM80 hardware hits checkCuda(cudaMemcpyFromSymbol(...)) and throws. Adding device_prop.major >= 8 && batch_size > 0 && max_seq_len > 0 checks in both functions makes these public loader APIs self-contained.
| rotary=q_len == 7, | ||
| rotary_interleaved=False, | ||
| packed=group_size == 8, | ||
| share_buffer=True, |
There was a problem hiding this comment.
docs/contrib_ops/cuda/gqa.md §7.1 documents support for QK-Norm and rotary embeddings on this speculative XQA route, but all 42 cases here keep has_qk_norm=False, rotary_interleaved=False, and has_position_ids=False. Consider toggling those flags on a subset of existing cases (e.g., has_qk_norm=(q_len == 2), rotary_interleaved=(group_size == 16 and q_len == 7), has_position_ids=(group_size == 4 and q_len == 7)) so those prologue paths are covered without adding extra test cases.

Description
Short causal multi-token steps over a shared KV buffer (
2 <= sequence_length <= 16, e.g. speculative-decoding verify) now run through XQA's speculative-decoding (SPEC_DEC) kernel instead of FlashAttention.FlashAttention handles each query head separately. In a GQA verify step every KV tile is therefore read once per query head: 16 times for Nemotron Bolt's 32 query / 2 KV heads. XQA's
SPEC_DECmode packsM_TILESIZE = 32(query head, query token) rows of one KV head group into each CTA, so a KV tile is shared by the whole tile. That KV re-read dominates verify cost at long context.Changes
xqa/xqa_spec_dec_impl_gen.cuh,xqa/xqa_spec_dec_impl.cuh,xqa/xqa_spec_dec_{bf16,fp16}_128.cuSPEC_DECinstantiations: FP16/BF16, head size 128, group sizes 4/8/16, linear causal mask (IS_SPEC_DEC_TREE 0). The launcher builds the causal mask and clears semaphores in its own workspace, both with kernels (graph-capture friendly).xqa/xqa_spec_dec_loader.cu,xqa/xqa_loader.hIsXQASpecDecSupported,LaunchXQASpecDecKernel,GetXQASpecDecWorkspaceSize,GetXQASpecDecRequiredSharedMemoryBytes.xqa/mha_impl.cuhXQA_SPEC_DEC_SPLIT_PER_TOKEN_BLOCK: size the KV split count for every token tile (SMs / (batch * kv_heads * token_tiles)). The default sizes it for the head groups only, which oversubscribes the GPU 4x for R=8 / group 16. Only the new TUs define it; paged spec-dec is unchanged.group_query_attention.{h,cc},attention_data.h,group_query_attention_impl.cuquery_seqlens, shared buffer, non-windowed cache, causal, non-quantized, no bias/softcap/smooth-softmax/head_sink, supported shape, shared memory fits (cached per node, resolved outside graph capture). They reuseExtremeDecoding'sUnpackRoPEAppendpreprocess, so rotary, packed QKV, QK-Norm and local windows work unchanged.ORT_ENABLE_XQA_SPEC_DEC=0(orORT_ENABLE_XQA=0) disables the route.docs/contrib_ops/cuda/gqa.mdtest/python/transformers/test_gqa.pyTestXQASpecDecParity: 42 PyTorch-reference parity cases (FP16/BF16 × group 4/8/16 × R 2/7/16, global and local window, past 1/300, rotary, packed QKV) that assertSdpaKernel=XQA, plus an env-disable case.Graph: source commit
6b9a1efa5f. Scope: the GQA dispatch and launch path plus the new XQA spec-dec entry points. Edges come from local static source-reference analysis, not a runtime trace.grpN::functions are macro-instantiated per group size and dtype. The test edge is runtime dispatch through an ORT session. Omitted: XQA kernel internals, Flash/cuDNN routes, paged XQA.Motivation and Context
On Nemotron Bolt (BF16, 32 query / 2 KV heads, head 128, R = 8 verify), the target verify forward is about 82–90% of each speculative round. At long context, its FlashAttention GQA calls are the part that grows with context length:
Kernel: one GroupQueryAttention call, Bolt shape, H100
One-node ORT session with IOBinding; median of 5 × 200 runs.
main(FlashAttention)The KV split change matters: with the stock split count, R = 8 at 252K takes 285.5 µs; with the per-token-tile split it takes 224.9 µs.
End to end: NVIDIA-Nemotron-3.5-Bolt-74B-A9B (RC9 package), ORT GenAI k = 7, H100
main=d2f77cd96d; this PR is the same build plus this change. Real prompts, 256 generated tokens.main→ PRmain→ PRmain→ PRPer-round cost falls at every length, by 29% at 252K. Single-prompt tok/s also depends on acceptance, which moves whenever rounding changes the greedy path at a near-tie token. So the fair throughput comparison is across many prompts:
mainAcceptance moved in both directions across the slices (5 higher, 5 lower).
Short real prompts (30-prompt set, 178–1,663 tokens, k = 7) show no regression: pooled decode is 396.7 / 330.2 / 371.9 tok/s vs 395.8 / 326.7 / 363.8 on
main.Numerics. Like the existing single-token XQA kernel, XQA merges split-KV partial outputs in the input precision; FlashAttention merges in FP32. In the Bolt verify shape, output relative L2 error against an FP64 reference is about 0.003 (XQA) vs 0.0017 (Flash). That is the same level as the single-token XQA path every R = 1 decode step already uses. Errors stay bounded from 1K to 252K context.
Validation (H100, SM90; incremental build of
d2f77cd96d+ this change,sm_90a; then rebased cleanly ontoe3a1153163)pytest test_gqa.py -k XQASpecDec: 43 passed. Onmainthe same 42 parity cases fail with'XQA' != 'FLASH_ATTENTION', so the tests exercise the new route.pytest test_gqa.py(whole file): 1102 passed, 2 failed, 2 skipped. The two failures (test_gqa_decode_bidirectional,test_windowed_cache_attention_bias_is_rejected) fail identically without this change. They come from skew between this newer test file and the olderd2f77cd96dbinary.onnxruntime_provider_test --gtest_filter='*GroupQueryAttention*:*PagedAttention*': 221 passed, 1 failed. The failure isSharedKV_ReadOnlyCache_CUDA, the existing H100 failure fixed by [CUDA] Pin attention backend in GQA SharedKV_ReadOnlyCache_CUDA test #34176; it uses a borrowed cache that cannot take this route.requirements-lintrunner.txt): clean.mainand Windows/MSVC builds. CI covers these.Follow-ups (not in this PR)
group_query_attention_workspace_estimate.cc). The estimate is advisory today and does not constrain runtime allocation.head_sink, and quantized caches for the speculative path.