Skip to content

Avoid repeated CUDA device-property queries during convolution plan rebuilds - #33776

Draft
Tianlei Wu (tianleiwu) with Copilot wants to merge 6 commits into
mainfrom
copilot/performance-fix-cuda-conv
Draft

Avoid repeated CUDA device-property queries during convolution plan rebuilds#33776
Tianlei Wu (tianleiwu) with Copilot wants to merge 6 commits into
mainfrom
copilot/performance-fix-cuda-conv

Conversation

Copilot AI commented Oct 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

  • Convolution: Initialize Conv and ConvTranspose frontend graphs with the SM version from cached kernel properties: 
    const auto& device_prop = GetDeviceProp();
s_.cudnn_fe_graph->set_sm_version(device_prop.major * 10 + device_prop.minor);
  • Other queries: Pass cached properties to multinomial sampling; replace CUTLASS’s full-property query with the required shared-memory attribute query.
  • Coverage: Add dynamic-shape convolution cases with bias fusion enabled/disabled and deterministic sampling cases.

Motivation and Context

cuDNN frontend ≥1.24 queries full device properties during support checks on rebuilt graphs. Dynamic-shape convolution models incur ~1 ms per node per shape change, causing a reported 22–42% end-to-end slowdown; reusing the EP’s cached properties avoids this overhead.

@azure-pipelines

azure-pipelines Bot commented Oct 30, 2026

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI linked an issue Oct 30, 2026 that may be closed by this pull request
[Performance] CUDA Conv: cuDNN frontend >= 1.24 calls cudaGetDeviceProperties on every plan rebuild (~1 ms per Conv node per input-shape change) #33769
Closed
Copilot AI and others added 3 commits October 30, 2026 20:25
@tianleiwu
Reuse cached CUDA SM version when rebuilding convolution frontend graphs
01ffa24 
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
@tianleiwu
Pass cached CUDA properties to generation helpers and avoid full occu…
d328bed 
…pancy queries

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
@tianleiwu
Fix dynamic CUDA convolution test output-name argument
c0e505b 
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix performance issue with CUDA Conv on input-shape change Avoid repeated CUDA device-property queries during convolution plan rebuilds Oct 30, 2026
Tianlei Wu (tianleiwu) and others added 2 commits November 10, 2026 05:12
@tianleiwu
Merge origin/main into copilot/performance-fix-cuda-conv
0d2f60c 
Resolve the conflicts in cuda/nn/conv.cc and cuda/nn/conv_transpose.cc by taking main's versions. Main already passes the cached SM version to the cuDNN frontend graphs (#33770) on top of the Conv plan cache refactor (#33123), which is the same change this branch made to these two files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
test(cuda): drop ConvTranspose fused-bias case
9c12f94 
The dynamic-shape Conv test paired ConvTranspose with fuse_conv_bias, but cuDNN frontend cannot build the fused dgrad + bias graph: validation fails because the dgrad output shape cannot be inferred, and once the shape is set there is no valid engine config (A100, cuDNN 9.23). The case failed on GPU independently of the SM version change, so cover ConvTranspose without bias fusion only.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu

Tianlei Wu (tianleiwu) commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

/azp run Linux Android Emulator QNN CI Pipeline

@azure-pipelines

azure-pipelines Bot commented Nov 10, 2026

Copy link
Copy Markdown
Azure Pipelines:
No pipelines were found matching this branch/path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tianleiwu Tianlei Wu (tianleiwu) Awaiting requested review from tianleiwu
Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

@tianleiwu Tianlei Wu (tianleiwu)

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Performance] CUDA Conv: cuDNN frontend >= 1.24 calls cudaGetDeviceProperties on every plan rebuild (~1 ms per Conv node per input-shape change)

2 participants

@tianleiwu