Repository navigation
[CUDA] Pin attention backend in GQA SharedKV_ReadOnlyCache_CUDA test - #34176
Open
Rui Ren (rui-ren) wants to merge 2 commits into
Open
Rui Ren (rui-ren) wants to merge 2 commits into
Rui Ren (rui-ren) wants to merge 2 commits into
Conversation
Auto mode routes short shared-buffer GQA steps to FlashAttention but non-shared steps to cuDNN SDPA on Hopper (#34164), so the bit-exact borrowed-vs-present comparison compared two different backends and failed on H100. Run the comparison once with cuDNN SDPA disabled and once with it forced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the test-only change covering both cuDNN attention settings in cache-equivalence comparisons.
No open issues 🎉
What changed in this PR
This test-only change addresses the H100 failure introduced by #34164 by controlling the cuDNN attention setting during bit-exact comparisons of borrowed read-only caches and normal present-output caches. Production dispatch and kernels remain unchanged.
Changes:
- Run the existing test cases with
ORT_ENABLE_CUDNN_FLASH_ATTENTIONset to both0and1, restoring the environment afterward. - Include the setting in failure traces and explain why automatic backend selection is unsuitable for this comparison.
No actionable findings were identified.
| File | Description |
|---|---|
onnxruntime/test/contrib_ops/group_query_attention_op_test.cc |
Exercise both cuDNN settings in the read-only cache equivalence test and improve failure diagnostics. |
Review effort: Balanced
Disable cuDNN for the bit-exact cache-borrowing comparison while retaining all original cache-layout cases. Forcing cuDNN fails SDPA graph construction with the 9.5.1 and 9.8.0 versions used by CI. Refresh WebGPU INT8 kernel documentation from the verified updated-docs artifact produced by run 43381792455. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GroupQueryAttentionTest.SharedKV_ReadOnlyCache_CUDAnow disables cuDNN SDPA withORT_ENABLE_CUDNN_FLASH_ATTENTION=0, usingScopedEnvironmentVariables, for its eight cache-layout configurations. No kernel or library code changes. Also refresh the generated WebGPU INT8 kernel documentation from the failed documentation job's verifiedupdated-docsartifact.Why it failed: the test compares two runs that should give identical outputs:
omit_present_outputs = true)Since #34164, auto mode prefers FlashAttention only for short (R ≤ 16) shared-buffer steps. On Hopper the two runs therefore go to different backends: the read-only-cache run goes to Flash, and the present-output run goes to cuDNN SDPA.
EXPECT_EQthen compares two different kernels, so the test fails on H100. Disabling cuDNN restores the test's cache-borrowing comparison without changing runtime dispatch.The initial revision also forced cuDNN on. CI showed that cuDNN 9.5.1 (Windows) and 9.8.0 (Linux TensorRT) cannot build the SDPA graph for the first forced configuration (head 256, query length 1, no window). The revised test retains every original cache-layout configuration and the exact-equality assertion, but no longer forces an unsupported cuDNN configuration.
Graph: source commit
d95d472f96(initial revision). The CI follow-up retains these call/type relationships but fixes the environment value to0; the graph is not evidence that the revised test runs cuDNN. Scope: the modified test, its helper, and the static path fromORT_ENABLE_CUDNN_FLASH_ATTENTIONto the GQA backend flags. Most edges come from local static source-reference analysis, not a runtime trace. Two exceptions:RunGQASharedKVFp16 → ComputeInternalis OpTester runtime dispatch, added by hand.GetAttentionKernelOptionscall is matched by name to the CUDA EP accessor.Omitted: Flash/cuDNN/XQA kernel internals and other GQA tests.
Motivation and Context
Fixes a test failure on H100 (
sm_90) introduced by #34164.CI follow-up validation
git diff --check: passed.python -c "import clang_format; clang_format.clang_format()" --dry-run --Werror --lines=2533:2568 onnxruntime\test\contrib_ops\group_query_attention_op_test.cc: passed with clang-format 20.1.8.onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.SharedKV_ReadOnlyCache_CUDA: not run locally for this revision; this Windows ARM64 environment has no CMake, C++/CUDA compiler, CUDA device, or built provider-test binary.generation-metadata.jsonfrom run43381792455, source merge0b49f0ff886c6854440dd7ead49c1eb92a62e535, then copied unchanged. Onlydocs/OperatorKernels.mdhas a content diff: the seven missing WebGPU INT8 registration rows from [WebGPU] Support signed 8-bit and 64-bit unary math #33958.=1configuration, after the=0cases completed.Historical test results, H100 (initial revision, not a rerun of the CI follow-up)
Built from
maind2f77cd96dwith only this test file changed. The single commit between that and this branch's basefd249b559cis WebGPU-only (#33958).mainEvery GQA SharedKV/QuerySeqlens CUDA test (12 tests) passes with the default environment and with
ORT_ENABLE_CUDNN_FLASH_ATTENTIONset to0or1.Performance vs
main: unchanged by construction. Only the test file is compiled, and the fixed test binary links the samelibonnxruntime.so.1.33.0(573c54f81f8f14b3…) andlibonnxruntime_providers_cuda.so(ff0674bb5086b2e4…) asmain.This PR has no performance gain. The Flash numbers below come from #34164's routing, which is already on
main. They are included to explain why this PR keeps that routing and fixes the test instead of changing the dispatch.GroupQueryAttention, Nemotron Bolt shape (32 Q heads, 2 KV heads, head 128, BF16, shared KV buffer of 262,152, R = 8 verify rows). One-node ORT session with IOBinding, H100, median of 5 × 200 runs:
=1)R = 1 decode uses XQA under both settings (bit-identical outputs), so it is not affected.
End to end (Nemotron Bolt 74B, GenAI, 252,422-token prompt, k = 7): Flash decodes at 171.8 tok/s vs 102.2 tok/s with cuDNN forced. That matches the kernel gap: (3,583.6 − 974.0) µs × 8 GQA layers ≈ 20.9 ms per verify round.
Against an FP64 reference in the same shape (peaked attention, R = 8), Flash outputs are also closer: 88–91% correctly rounded BF16 vs 81–86% for cuDNN, with p99 error 2.4–3.3 vs 4.6–6.1 ULP.
Validation commands
onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.SharedKV_ReadOnlyCache_CUDAonmain: failed (3/3). On this PR: passed (3/3).onnxruntime_provider_test --gtest_filter='GroupQueryAttentionTest.SharedKV*CUDA*:GroupQueryAttentionTest.QuerySeqlens*CUDA*'with the default environment,ORT_ENABLE_CUDNN_FLASH_ATTENTION=1andORT_ENABLE_CUDNN_FLASH_ATTENTION=0: passed (12/12 each).clang-format20.1.8 on the changed lines: clean.