Skip to content

[CUDA] Route short shared-buffer GQA verify steps to speculative-decoding XQA - #34198

Open
Rui Ren (rui-ren) wants to merge 1 commit into
mainfrom
rui-ren/gqa-xqa-spec-dec-verify
Open

Rui Ren (rui-ren) wants to merge 1 commit into
mainfrom
rui-ren/gqa-xqa-spec-dec-verify

Conversation

@rui-ren

Copy link
Copy Markdown
Contributor

Description

Short causal multi-token steps over a shared KV buffer (2 <= sequence_length <= 16, e.g. speculative-decoding verify) now run through XQA's speculative-decoding (SPEC_DEC) kernel instead of FlashAttention.

FlashAttention handles each query head separately. In a GQA verify step every KV tile is therefore read once per query head: 16 times for Nemotron Bolt's 32 query / 2 KV heads. XQA's SPEC_DEC mode packs M_TILESIZE = 32 (query head, query token) rows of one KV head group into each CTA, so a KV tile is shared by the whole tile. That KV re-read dominates verify cost at long context.

Changes

File Change
xqa/xqa_spec_dec_impl_gen.cuh, xqa/xqa_spec_dec_impl.cuh, xqa/xqa_spec_dec_{bf16,fp16}_128.cu New contiguous-cache (non-paged) SPEC_DEC instantiations: FP16/BF16, head size 128, group sizes 4/8/16, linear causal mask (IS_SPEC_DEC_TREE 0). The launcher builds the causal mask and clears semaphores in its own workspace, both with kernels (graph-capture friendly).
xqa/xqa_spec_dec_loader.cu, xqa/xqa_loader.h IsXQASpecDecSupported, LaunchXQASpecDecKernel, GetXQASpecDecWorkspaceSize, GetXQASpecDecRequiredSharedMemoryBytes.
xqa/mha_impl.cuh Opt-in XQA_SPEC_DEC_SPLIT_PER_TOKEN_BLOCK: size the KV split count for every token tile (SMs / (batch * kv_heads * token_tiles)). The default sizes it for the head groups only, which oversubscribes the GPU 4x for R=8 / group 16. Only the new TUs define it; paged spec-dec is unchanged.
group_query_attention.{h,cc}, attention_data.h, group_query_attention_impl.cu Route eligible steps to the new kernel: decode phase, no query_seqlens, shared buffer, non-windowed cache, causal, non-quantized, no bias/softcap/smooth-softmax/head_sink, supported shape, shared memory fits (cached per node, resolved outside graph capture). They reuse ExtremeDecoding's UnpackRoPEAppend preprocess, so rotary, packed QKV, QK-Norm and local windows work unchanged. ORT_ENABLE_XQA_SPEC_DEC=0 (or ORT_ENABLE_XQA=0) disables the route.
docs/contrib_ops/cuda/gqa.md Backend table, new §7.1, env var, tests.
test/python/transformers/test_gqa.py TestXQASpecDecParity: 42 PyTorch-reference parity cases (FP16/BF16 × group 4/8/16 × R 2/7/16, global and local window, past 1/300, rotary, packed QKV) that assert SdpaKernel=XQA, plus an env-disable case.

GQA speculative XQA code relationship graph

Graph: source commit 6b9a1efa5f. Scope: the GQA dispatch and launch path plus the new XQA spec-dec entry points. Edges come from local static source-reference analysis, not a runtime trace. grpN:: functions are macro-instantiated per group size and dtype. The test edge is runtime dispatch through an ORT session. Omitted: XQA kernel internals, Flash/cuDNN routes, paged XQA.

Motivation and Context

On Nemotron Bolt (BF16, 32 query / 2 KV heads, head 128, R = 8 verify), the target verify forward is about 82–90% of each speculative round. At long context, its FlashAttention GQA calls are the part that grows with context length:

Past KV 1-row XQA forward 8-row Flash forward
8K 9.1 ms 12.6 ms
252K 9.8 ms 19.5 ms

Kernel: one GroupQueryAttention call, Bolt shape, H100

One-node ORT session with IOBinding; median of 5 × 200 runs.

Rows Past KV main (FlashAttention) This PR (XQA spec-dec) Speedup
8 8,191 64.5 µs 43.5 µs 1.48×
8 32,767 161.1 µs 64.4 µs 2.50×
8 131,071 528.4 µs 136.3 µs 3.88×
8 252,421 982.5 µs 224.9 µs 4.37×
1 8K–252K 37–122 µs 38–123 µs unchanged (same single-token XQA)

The KV split change matters: with the stock split count, R = 8 at 252K takes 285.5 µs; with the per-token-tile split it takes 224.9 µs.

End to end: NVIDIA-Nemotron-3.5-Bolt-74B-A9B (RC9 package), ORT GenAI k = 7, H100

main = d2f77cd96d; this PR is the same build plus this change. Real prompts, 256 generated tokens.

Context TTFT main → PR ms per verify round main → PR Decode tok/s main → PR
8K 0.73 → 0.72 s 13.94 → 13.57 291.5 → 273.7
16K 1.46 → 1.46 s 14.25 → 13.84 299.7 → 318.9
32K 2.98 → 2.98 s 14.75 → 13.86 280.0 → 288.5
64K 6.18 → 6.17 s 15.92 → 14.03 268.0 → 257.0
128K 13.26 → 13.23 s 18.11 → 14.69 294.2 → 272.2
252K 29.01 → 28.82 s 21.59 → 15.33 171.8 → 249.3

Per-round cost falls at every length, by 29% at 252K. Single-prompt tok/s also depends on acceptance, which moves whenever rounding changes the greedy path at a near-tie token. So the fair throughput comparison is across many prompts:

10 long prompts (131K and 205K slices, 512 tokens each) main This PR llama.cpp DFlash k = 7
Mean ms per round 19.22 15.55 (−19%) —
Mean decode tok/s 304 357 (+17%) 255
Mean acceptance 0.687 0.651 0.658

Acceptance moved in both directions across the slices (5 higher, 5 lower).

Short real prompts (30-prompt set, 178–1,663 tokens, k = 7) show no regression: pooled decode is 396.7 / 330.2 / 371.9 tok/s vs 395.8 / 326.7 / 363.8 on main.

Numerics. Like the existing single-token XQA kernel, XQA merges split-KV partial outputs in the input precision; FlashAttention merges in FP32. In the Bolt verify shape, output relative L2 error against an FP64 reference is about 0.003 (XQA) vs 0.0017 (Flash). That is the same level as the single-token XQA path every R = 1 decode step already uses. Errors stay bounded from 1K to 252K context.

Validation (H100, SM90; incremental build of d2f77cd96d + this change, sm_90a; then rebased cleanly onto e3a1153163)

  • pytest test_gqa.py -k XQASpecDec: 43 passed. On main the same 42 parity cases fail with 'XQA' != 'FLASH_ATTENTION', so the tests exercise the new route.
  • pytest test_gqa.py (whole file): 1102 passed, 2 failed, 2 skipped. The two failures (test_gqa_decode_bidirectional, test_windowed_cache_attention_bias_is_rejected) fail identically without this change. They come from skew between this newer test file and the older d2f77cd96d binary.
  • onnxruntime_provider_test --gtest_filter='*GroupQueryAttention*:*PagedAttention*': 221 passed, 1 failed. The failure is SharedKV_ReadOnlyCache_CUDA, the existing H100 failure fixed by [CUDA] Pin attention backend in GQA SharedKV_ReadOnlyCache_CUDA test #34176; it uses a borrowed cache that cannot take this route.
  • clang-format 20.1.8 and ruff 0.12.12 (pinned in requirements-lintrunner.txt): clean.
  • Not run locally: a full build on main and Windows/MSVC builds. CI covers these.

Follow-ups (not in this PR)

  • Count this route in the GQA workspace estimator (group_query_attention_workspace_estimate.cc). The estimate is advisory today and does not constrain runtime allocation.
  • Head sizes 64/256, head_sink, and quantized caches for the speculative path.
  • Apply the per-token-tile split count to the paged spec-dec path.

…ding XQA

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 16:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The parity tests do not exercise the changed long-context split policy, leaving split-count behavior and workspace/launch agreement unverified without fixed actual past lengths of at least 8K in FP16 and BF16 cases.

1 open issue
What changed in this PR

This PR routes eligible short, multi-token GroupQueryAttention verification steps through CUDA XQA’s speculative-decoding kernel, reducing repeated KV-cache reads at long context.

Changes:

  • Adds FP16/BF16 speculative XQA kernels for head size 128 and group sizes 4, 8 and 16.
  • Integrates guarded dispatch, workspace allocation and per-token-tile KV splitting.
  • Adds backend-selection parity tests and documents eligibility and environment controls.
File Description
onnxruntime/​test/​python/​transformers/​test_gqa.py Adds speculative XQA parity and environment-disable tests.
onnxruntime/​contrib_ops/​cuda/​bert/​xqa/​xqa_spec_dec_loader.cu Implements support checks and launch/resource dispatch.
onnxruntime/​contrib_ops/​cuda/​bert/​xqa/​xqa_spec_dec_impl.cuh Instantiates speculative kernels for supported group sizes.
onnxruntime/​contrib_ops/​cuda/​bert/​xqa/​xqa_spec_dec_impl_gen.cuh Implements causal-mask preparation, workspace sizing and launch.
onnxruntime/​contrib_ops/​cuda/​bert/​xqa/​xqa_spec_dec_fp16_128.cu Adds FP16 head-size-128 instantiations.
onnxruntime/​contrib_ops/​cuda/​bert/​xqa/​xqa_spec_dec_bf16_128.cu Adds BF16 head-size-128 instantiations.
onnxruntime/​contrib_ops/​cuda/​bert/​xqa/​xqa_loader.h Declares speculative XQA interfaces and contracts.
onnxruntime/​contrib_ops/​cuda/​bert/​xqa/​mha_impl.cuh Adds opt-in per-token-tile split sizing.
onnxruntime/​contrib_ops/​cuda/​bert/​group_query_attention.h Adds enablement and cached shared-memory eligibility state.
onnxruntime/​contrib_ops/​cuda/​bert/​group_query_attention.cc Selects eligible speculative steps and allocates workspace.
onnxruntime/​contrib_ops/​cuda/​bert/​group_query_attention_impl.cu Launches speculative XQA after existing preprocessing.
onnxruntime/​contrib_ops/​cuda/​bert/​attention_data.h Tracks speculative XQA selection.
docs/​contrib_ops/​cuda/​gqa.md Documents routing, constraints, controls and tests.

Review effort: Balanced

const uint32_t nbTokenBlocksPerGrp = divUp(qSeqLen * headGrpSize, rowsPerBlock);
#if defined(XQA_SPEC_DEC_SPLIT_PER_TOKEN_BLOCK)
// Every token block of a head group is an independent CTA row, so size the KV split for all of them.
const uint32_t nbSubSeqPerSeq = computeNbSubSeqPerSeqMHA(prop, batchSize, nbKHeads * nbTokenBlocksPerGrp, maxSeqLen);

@tianleiwu Tianlei Wu (tianleiwu) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

The contiguous-cache speculative-XQA (SPEC_DEC) integration, per-token-tile KV split policy (XQA_SPEC_DEC_SPLIT_PER_TOKEN_BLOCK), CUDA Graph capture handling, and UnpackRoPEAppend reuse are clean and well-motivated. (Note: the three failing CI checks on this PR — Kernel Documentation Generation, Windows GPU Kernel Documentation Validation, and TestQMoESharedDirect::test_cuda_int3_tensor_core_compatibility_shadow — are unrelated main failures.)

Two correctness/soundness issues should be addressed before merging:

  1. Cache element type guard (std::is_same_v<T, U>): GroupQueryAttention<T, U> is registered for U in {T, int8_t, Float8E4M3FN, uint8_t}, while is_inputs_quantized only checks k_quant_type_ != NONE || v_quant_type_ != NONE. Adding std::is_same_v<T, U> in ComputeInternal and ExtremeDecoding (matching CudnnSdpaAttention and UnfusedGqaAttention) prevents 1-byte/4-bit T_CACHE instantiations with default k_quant_type="NONE" from dereferencing null scales in UnpackRoPEAppend and reading out-of-bounds in LaunchXQASpecDecKernel.
  2. Level-1 workspace estimator soundness: EstimateGroupQueryAttentionWorkspace is already used as a proven Level1MemoryEstimate upper bound in CUDAExecutionProvider::GetCapability under kOrtSessionOptionsResourceCudaPartitioningSettings (which fails closed when an unmodeled route like cuDNN SDPA is reachable). Leaving speculative XQA unmodeled allows runtime workspace (M_TILESIZE = 32 across token blocks + mask) to exceed the reported Level-1 bound.

See inline comments for details and a few smaller suggestions.

// head as in FlashAttention, which dominates long-context verify cost.
if (!data.use_xqa && enable_xqa_spec_dec_ && query_seqlens_data == nullptr && !parameters.is_first_prompt &&
parameters.sequence_length > 1 && parameters.kv_sequence_length == parameters.sequence_length &&
parameters.past_present_share_buffer && !parameters.is_windowed_kv_cache && !is_inputs_quantized &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!is_inputs_quantized only checks k_quant_type_ != NONE || v_quant_type_ != NONE (where omitted or unrecognized attribute strings map to KVQuantizationType::NONE), while GroupQueryAttention<T, U> is registered for U in {T, int8_t, Float8E4M3FN, uint8_t}.

Every other non-quantized backend (IsGQACudnnSdpaCoreEligibleSeqFree<T, U>, CudnnSdpaAttention, and UnfusedGqaAttention) additionally requires std::is_same_v<T, U>. Without that check here (and an if constexpr (std::is_same_v<T, U>) guard around LaunchXQASpecDecKernel in ExtremeDecoding at group_query_attention_impl.cu:1102), a T_CACHE = int8 / Float8E4M3FN / uint8 node with k_quant_type="NONE" will enter use_xqa_spec_dec, dereference null k_scale/v_scale in UnpackRoPEAppend<T, U>, and reinterpret 1-byte/4-bit present_key/present_value buffers as 2-byte FP16/BF16 heads inside type-erased LaunchXQASpecDecKernel (reading 2x-4x past the cache allocation).

Suggested change
parameters.past_present_share_buffer && !parameters.is_windowed_kv_cache && !is_inputs_quantized &&
std::is_same_v<T, U> && parameters.past_present_share_buffer && !parameters.is_windowed_kv_cache && !is_inputs_quantized &&

device_prop, kIsBf16, parameters.batch_size, parameters.sequence_length, parameters.num_heads,
parameters.kv_num_heads, parameters.head_size, parameters.seqlen_present_kv_cache);
ORT_RETURN_IF_NOT(workspace_bytes > 0, "Speculative XQA workspace sizing failed.");
xqa_scratch_buffer = this->GetScratchBuffer<void>(workspace_bytes, GetComputeStream(context));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the PR note that EstimateGroupQueryAttentionWorkspace is advisory today: while Level-2 DeclareWorkspaceRequirements is not yet enforced in an offset plan, Level-1 CUDAExecutionProvider::GetCapability (onnxruntime/core/providers/cuda/cuda_execution_provider.cc:4098-4136) already uses EstimateGroupQueryAttentionWorkspace as a proven upper bound (Level1MemoryEstimate) under kOrtSessionOptionsResourceCudaPartitioningSettings, failing closed (std::nullopt) whenever a reachable backend such as cuDNN SDPA has no graph-free bound.

Because speculative XQA uses M_TILESIZE = 32 for all group sizes (8,192 B/subseq vs 2,048 B/subseq in single-token XqaBackendEnvelope for group_size = 4), token_blocks = ceil(S * G / 32), MaskBytes, and S-token qkv_preprocess_bytes, this allocation can exceed the reported Level-1 bound (for example, sdpa_kernel = MATH, B = 1, S = 2, num_heads = 512, kv_num_heads = 128, head_size = 128, C = 2 on SM80 allocates 1,081,984 B at runtime vs 864,512 B estimated). Please either include the speculative-XQA envelope in GetGQAWorkspaceAggregateForBounds / GQAWorkspaceEstimateConfig or fail closed (Unavailable) when speculative XQA is reachable until modeled.

device_prop, static_cast<uint32_t>(batch_size), static_cast<uint32_t>(kv_num_heads) * token_blocks,
static_cast<uint32_t>(max_seq_len));
return MaskBytes(batch_size, sequence_length) + SemaphoreBytes(batch_size, kv_num_heads, sequence_length) +
GetScratchSize(nb_seq, nb_sub_seq_per_seq);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In mha_impl.cuh:1301, isMultiBlock is allowMultiBlockMode && (gridDim.x != 1), so when nb_sub_seq_per_seq == 1 (max_seq_len <= 256 or batch_size * kv_num_heads * token_blocks >= device_prop.multiProcessorCount), kernel_mha never accesses semaphores or scratch. Returning MaskBytes(batch_size, sequence_length) when nb_sub_seq_per_seq <= 1 and skipping the Fill<int32_t> launch on semaphores (passing nullptr for semaphores and scratch in Launch below) avoids hundreds of KB of unused scratch and saves a redundant kernel launch per GQA layer on single-split steps.

Also note that launchMHA throws std::runtime_error via checkCuda(...) on CUDA errors (making return CUDA_CALL(cudaGetLastError()); at line 154 unreachable for launch/symbol failures); catching const std::exception& and checking CUDA_RETURN_IF_ERROR(cudaGetLastError()) after XqaSpecDecCausalMaskKernel keeps errors inside ORT's Status contract.

size_t GetXQASpecDecWorkspaceSize(const cudaDeviceProp& device_prop, bool is_bf16, int batch_size,
int sequence_length, int num_heads, int kv_num_heads, int head_size,
int max_seq_len) {
if (!IsXQASpecDecSupported(head_size, num_heads, kv_num_heads, sequence_length)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike LaunchXQAKernel and GetXQAScratchSize in xqa_loader_fp16.cu, LaunchXQASpecDecKernel and GetXQASpecDecWorkspaceSize do not guard device_prop.major >= 8, batch_size > 0, or max_seq_len > 0. In multi-arch builds (HAS_SM80_OR_LATER defined alongside SM75), passing batch_size <= 0 divides by zero in computeNbSubSeqPerSeqMHA, and calling LaunchXQASpecDecKernel on pre-SM80 hardware hits checkCuda(cudaMemcpyFromSymbol(...)) and throws. Adding device_prop.major >= 8 && batch_size > 0 && max_seq_len > 0 checks in both functions makes these public loader APIs self-contained.

rotary=q_len == 7,
rotary_interleaved=False,
packed=group_size == 8,
share_buffer=True,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs/contrib_ops/cuda/gqa.md §7.1 documents support for QK-Norm and rotary embeddings on this speculative XQA route, but all 42 cases here keep has_qk_norm=False, rotary_interleaved=False, and has_position_ids=False. Consider toggling those flags on a subset of existing cases (e.g., has_qk_norm=(q_len == 2), rotary_interleaved=(group_size == 16 and q_len == 7), has_position_ids=(group_size == 4 and q_len == 7)) so those prologue paths are covered without adding extra test cases.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants