Skip to content

[CUDA] Pin attention backend in GQA SharedKV_ReadOnlyCache_CUDA test - #34176

Open
Rui Ren (rui-ren) wants to merge 2 commits into
mainfrom
rui-ren/gqa-sharedkv-test-pin-backend
Open

Rui Ren (rui-ren) wants to merge 2 commits into
mainfrom
rui-ren/gqa-sharedkv-test-pin-backend

Conversation

@rui-ren

@rui-ren Rui Ren (rui-ren) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

GroupQueryAttentionTest.SharedKV_ReadOnlyCache_CUDA now disables cuDNN SDPA with ORT_ENABLE_CUDNN_FLASH_ATTENTION=0, using ScopedEnvironmentVariables, for its eight cache-layout configurations. No kernel or library code changes. Also refresh the generated WebGPU INT8 kernel documentation from the failed documentation job's verified updated-docs artifact.

Why it failed: the test compares two runs that should give identical outputs:

  • the borrowed read-only cache run (omit_present_outputs = true)
  • the normal present-output run

Since #34164, auto mode prefers FlashAttention only for short (R ≤ 16) shared-buffer steps. On Hopper the two runs therefore go to different backends: the read-only-cache run goes to Flash, and the present-output run goes to cuDNN SDPA. EXPECT_EQ then compares two different kernels, so the test fails on H100. Disabling cuDNN restores the test's cache-borrowing comparison without changing runtime dispatch.

The initial revision also forced cuDNN on. CI showed that cuDNN 9.5.1 (Windows) and 9.8.0 (Linux TensorRT) cannot build the SDPA graph for the first forced configuration (head 256, query length 1, no window). The revised test retains every original cache-layout configuration and the exact-equality assertion, but no longer forces an unsupported cuDNN configuration.

GQA SharedKV test code relationship graph

Graph: source commit d95d472f96 (initial revision). The CI follow-up retains these call/type relationships but fixes the environment value to 0; the graph is not evidence that the revised test runs cuDNN. Scope: the modified test, its helper, and the static path from ORT_ENABLE_CUDNN_FLASH_ATTENTION to the GQA backend flags. Most edges come from local static source-reference analysis, not a runtime trace. Two exceptions:

  • RunGQASharedKVFp16 → ComputeInternal is OpTester runtime dispatch, added by hand.
  • The constructor's GetAttentionKernelOptions call is matched by name to the CUDA EP accessor.

Omitted: Flash/cuDNN/XQA kernel internals and other GQA tests.

Motivation and Context

Fixes a test failure on H100 (sm_90) introduced by #34164.

CI follow-up validation

  • git diff --check: passed.
  • python -c "import clang_format; clang_format.clang_format()" --dry-run --Werror --lines=2533:2568 onnxruntime\test\contrib_ops\group_query_attention_op_test.cc: passed with clang-format 20.1.8.
  • onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.SharedKV_ReadOnlyCache_CUDA: not run locally for this revision; this Windows ARM64 environment has no CMake, C++/CUDA compiler, CUDA device, or built provider-test binary.
  • Both generated documents were hash-verified against generation-metadata.json from run 43381792455, source merge 0b49f0ff886c6854440dd7ead49c1eb92a62e535, then copied unchanged. Only docs/OperatorKernels.md has a content diff: the seven missing WebGPU INT8 registration rows from [WebGPU] Support signed 8-bit and 64-bit unary math #33958.
  • Fresh CI is required to confirm the revised head. The existing failed GPU logs report only the forced =1 configuration, after the =0 cases completed.

Historical test results, H100 (initial revision, not a rerun of the CI follow-up)

Built from main d2f77cd96d with only this test file changed. The single commit between that and this branch's base fd249b559c is WebGPU-only (#33958).

Binary Result (3 runs) Time per run
main FAILED 3/3 16.0–17.6 s
This PR OK 3/3 11.8–16.8 s (16 configurations vs 8; timings are noisy because other GPUs were running benchmarks)

Every GQA SharedKV/QuerySeqlens CUDA test (12 tests) passes with the default environment and with ORT_ENABLE_CUDNN_FLASH_ATTENTION set to 0 or 1.

Performance vs main: unchanged by construction. Only the test file is compiled, and the fixed test binary links the same libonnxruntime.so.1.33.0 (573c54f81f8f14b3…) and libonnxruntime_providers_cuda.so (ff0674bb5086b2e4…) as main.

This PR has no performance gain. The Flash numbers below come from #34164's routing, which is already on main. They are included to explain why this PR keeps that routing and fixes the test instead of changing the dispatch.

GroupQueryAttention, Nemotron Bolt shape (32 Q heads, 2 KV heads, head 128, BF16, shared KV buffer of 262,152, R = 8 verify rows). One-node ORT session with IOBinding, H100, median of 5 × 200 runs:

Past KV length FlashAttention (auto) cuDNN SDPA (=1) Flash speedup
8,191 63.6 µs 159.4 µs 2.51×
32,767 160.7 µs 496.6 µs 3.09×
131,071 729.3 µs 1,902.0 µs 2.61×
252,421 974.0 µs 3,583.6 µs 3.68×

R = 1 decode uses XQA under both settings (bit-identical outputs), so it is not affected.

End to end (Nemotron Bolt 74B, GenAI, 252,422-token prompt, k = 7): Flash decodes at 171.8 tok/s vs 102.2 tok/s with cuDNN forced. That matches the kernel gap: (3,583.6 − 974.0) µs × 8 GQA layers ≈ 20.9 ms per verify round.

Against an FP64 reference in the same shape (peaked attention, R = 8), Flash outputs are also closer: 88–91% correctly rounded BF16 vs 81–86% for cuDNN, with p99 error 2.4–3.3 vs 4.6–6.1 ULP.

Validation commands

  • onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.SharedKV_ReadOnlyCache_CUDA on main: failed (3/3). On this PR: passed (3/3).
  • onnxruntime_provider_test --gtest_filter='GroupQueryAttentionTest.SharedKV*CUDA*:GroupQueryAttentionTest.QuerySeqlens*CUDA*' with the default environment, ORT_ENABLE_CUDNN_FLASH_ATTENTION=1 and ORT_ENABLE_CUDNN_FLASH_ATTENTION=0: passed (12/12 each).
  • clang-format 20.1.8 on the changed lines: clean.

Auto mode routes short shared-buffer GQA steps to FlashAttention but non-shared
steps to cuDNN SDPA on Hopper (#34164), so the bit-exact borrowed-vs-present
comparison compared two different backends and failed on H100. Run the
comparison once with cuDNN SDPA disabled and once with it forced.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 06:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the test-only change covering both cuDNN attention settings in cache-equivalence comparisons.

No open issues 🎉

What changed in this PR

This test-only change addresses the H100 failure introduced by #34164 by controlling the cuDNN attention setting during bit-exact comparisons of borrowed read-only caches and normal present-output caches. Production dispatch and kernels remain unchanged.

Changes:

  • Run the existing test cases with ORT_ENABLE_CUDNN_FLASH_ATTENTION set to both 0 and 1, restoring the environment afterward.
  • Include the setting in failure traces and explain why automatic backend selection is unsuitable for this comparison.

No actionable findings were identified.

File Description
onnxruntime/​test/​contrib_ops/​group_query_attention_op_test.cc Exercise both cuDNN settings in the read-only cache equivalence test and improve failure diagnostics.

Review effort: Balanced

Disable cuDNN for the bit-exact cache-borrowing comparison while retaining all original cache-layout cases. Forcing cuDNN fails SDPA graph construction with the 9.5.1 and 9.8.0 versions used by CI.

Refresh WebGPU INT8 kernel documentation from the verified updated-docs artifact produced by run 43381792455.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants