Skip to content

[Stateful - M12] standalone CUDA plugin greedy sampling - #34051

Draft
devang-ml wants to merge 3 commits into
stateful-M11-cuda-plugin-cache-integrationfrom
stateful-M12-cuda-plugin-greedy
Draft

devang-ml wants to merge 3 commits into
stateful-M11-cuda-plugin-cache-integrationfrom
stateful-M12-cuda-plugin-greedy

Conversation

@devang-ml

@devang-ml devang-ml commented Nov 10, 2026 •

Copy link
Copy Markdown

Description

Upstream draft stack 12/21. Original milestone: M09a.

The [Stateful - MNN] title/branch numbers identify positions in this 21-PR stack, not original milestone numbering. M16-M23 language-binding milestones remain deferred.

Historical review base: 1988019b6e23ac4def4628cddd603c1bfb504fe8, preserved as state/stateful-base-1988019b. The first PR targets that historical review anchor; each later PR targets its predecessor. These PRs have not been rebased onto current main. The anchor is for incremental review; integration/retargeting requires a later explicit decision.

Validation provenance: Recorded milestone validation below is retained. The appended reference-cleanup/merge commits change Markdown references only; executable files are byte-for-byte unchanged. No new build/test or current-main qualification is claimed. Historical commits are preserved, while current documentation and PR links use this Microsoft draft stack. Upstream CI results are separate.

M09a, stacked on #34050 (stateful-M11-cuda-plugin-cache-integration). Implements standalone CUDA-plugin greedy sampling. M09b loop/chain GPU sampling remains unimplemented and requires separate authorization.

  • Perform selected-row argmax and whole-tensor NaN validation on GPU for contiguous float32/float16 [1,V] and [1,S,V].
  • Preserve CPU-exact lowest-ID ties, infinities, all-negative-infinity, signed-zero/subnormal handling and NaN rejection in unselected rows.
  • Use a bounded two-stage CUB BlockReduce path with only a 16-byte token/status result read back. No host argmax, CPU reference delegation or full-logits readback.
  • Append a size-checked borrowed scratch-allocator field to the callback request. Native binding retains/resolves Session resources and provides a synchronous C adapter without importing native Session headers into generate/.
  • Keep authored host launch code in an LLVM-instrumented C++ unit using standard cudaLaunchKernel; keep actual device code in the NVCC unit with source information. Registered launch entry signatures match the device definitions.
  • Advertise the new greedy bit/types only in the CUDA plugin, preserve six-cache behavior and CPU defaults, and leave legacy/shared CUDA unextended.
  • Preserve out-parameters on errors, stream completion, caller-device restoration and sampling after bound-Session release/library unregistration, subject to caller-owned allocator/stream/Env obligations.

No public API table/version change or new owning C++ wrapper is needed: this is an appended borrowed provider-request resource field; existing public C/C++ sampler handles expose the feature.

Approved specification, execution plan, SDK example and exact evidence are in SPEC.md, TASKS.md and SCRATCHPAD.md.

Validation

Linux/A100 SM80; matching CUDA 13.1/cuDNN; Clang18 host LLVM and NVCC13.1/GCC13. Every reported functional suite has nonzero counts and zero skips.

Configuration Exact command, from its build output directory Result
CPU ./onnxruntime_session_loop_test --gtest_output=xml:coverage/m09a/final/native-tests.xml 63/63 passed, unfiltered
CPU ./onnxruntime_autoep_test --gtest_output=xml:coverage/m09a/final/autoep-tests.xml 139/139 passed, unfiltered
CUDA plugin ./onnxruntime_session_loop_test --gtest_output=xml:coverage/m09a/final/native-tests.xml 117/117 passed, unfiltered
CUDA plugin ./onnxruntime_autoep_test --gtest_output=xml:coverage/m09a/final/autoep-tests.xml 184/184 passed, unfiltered
Legacy/shared CUDA ./onnxruntime_session_loop_test --gtest_output=xml:coverage/m09a/final/native-tests.xml 110/110 passed
Legacy/shared CUDA ./onnxruntime_autoep_test --gtest_filter='AutoEpSelection.CudaEP:*Stateful*' --gtest_output=xml:coverage/m09a/final/stateful-tests.xml 43/43 passed; no native sampler/cache capability

The instrumented CUDA-plugin build passed:

cmake --compile-no-warning-as-error -S cmake -B build/state-cuda-plugin-13.1/RelWithDebInfo
cmake --build build/state-cuda-plugin-13.1/RelWithDebInfo --parallel 24 \
  --target onnxruntime_session_loop_test onnxruntime_autoep_test

The existing supported nonfatal-warning configuration handles baseline mixed-compiler diagnostics; changed implementation/test units pass strict type checks. No dependency was installed. Final public-header comment-only edits describe CPU-default/explicit provider behavior without changing the validated runtime code/ABI.

Measured host/glue coverage: 83/83 (100%), from actual exercised plugin/core artifacts:

  • CUDA callback: 52/52.
  • C++ kernel launcher: 15/15.
  • Native resource binding: 14/14.
  • Sampler dispatch: 2/2.

Fresh matching profiles were merged with llvm-profdata merge -sparse coverage/m09a/final/*.profraw -o coverage/m09a/final/m09a.profdata. Module-specific LLVM source lines were intersected against M08b 2b6388fe5b. All 589 plugin/174 runtime map warnings are dormant zero-hash entries, with no new-function mismatch.

Measured device coverage: 25/25 (100%) of emitted new CUDA-source locations, including helper bodies and mapped entry/epilogue braces. No device source location is excluded. Actual float/half/final kernels were observed across 44 profiled launches, with source text/SHA256 matched to the exercised artifact. The declarations-only helper header has no additional device instruction mappings.

Before production implementation, a distinct executed/unexecuted-path probe demonstrated reciprocal source execution counters. The initial unprivileged profiler attempt failed with ERR_NVGPUCTRPERM; the user explicitly authorized administrator privileges for profiling. Only the profiler process was elevated; no global driver setting/permission was changed.

# From build/state-cuda-plugin-13.1/RelWithDebInfo:
export CUDA_HOME=/datadisks/disk4/bmeswani/cuda/cuda13.1
sudo -n -- env CUDA_VISIBLE_DEVICES=0 LLVM_PROFILE_FILE=/dev/null \
  LD_LIBRARY_PATH="$PWD:$CUDA_HOME/lib64:$CUDA_HOME/targets/x86_64-linux/lib" \
  "$CUDA_HOME/bin/ncu" --devices 0 --section SourceCounters --import-source yes \
  --kernel-name-base function --kernel-name 'regex:.*Greedy.*' --force-overwrite \
  --export /home/devpatel/.copilot/session-state/819a29ec-8ea8-4b2f-9938-e40236ffd239/files/m09a-greedy-device-coverage \
  ./onnxruntime_autoep_test --gtest_filter='CudaStatefulTest.GreedyDeviceCoverage'

Result: passed, with 25/25 source-mapped executed-instruction locations. Host counters, launch success and sanitizer output are not substituted for device coverage.

A separate Nsight Systems CUDA trace of CudaStatefulTest.GreedyTransferAndLatency passed and measured 33 Device-to-Host transfers, each 16 bytes, total 528 bytes—exactly the 33 GPU sampler calls. No full-logits readback is present. The explicit input upload/setup traffic is recorded separately. Representative instrumented normal-suite mean times over 32 measured calls at vocabulary 50257 were 67.33us GPU / 98.54us CPU; these are observations, not release benchmark claims or performance gates.

These CUDA 13.1 Compute Sanitizer commands passed with zero errors and zero leaked bytes/allocations:

compute-sanitizer --tool memcheck --leak-check full --error-exitcode 99 \
  ./onnxruntime_autoep_test --gtest_filter='CudaStatefulTest.Greedy*'
# 5/5 passed.

compute-sanitizer --tool memcheck --leak-check full --error-exitcode 99 \
  ./onnxruntime_session_loop_test --gtest_filter='StatefulCudaProviderBoundaryTest.*'
# 2/2 passed.

compute-sanitizer --tool racecheck --error-exitcode 99 \
  ./onnxruntime_autoep_test --gtest_filter='CudaStatefulTest.GreedyDeviceCoverage'
# 1/1 passed; zero hazards/errors/warnings.

compute-sanitizer --tool synccheck --error-exitcode 99 \
  ./onnxruntime_autoep_test --gtest_filter='CudaStatefulTest.GreedyDeviceCoverage'
# 1/1 passed; zero errors.

The documented C++17 GPU SDK example also passed normal execution and memcheck after Session release/library unregistration while the caller-owned allocator Session remains alive. Genuine C11 compilation/execution, CPU C++17 normal/no-exceptions client examples, formatting/120 columns, whitespace, 93-requirement/24-milestone traceability and CPU-library CUDA-dependency checks passed. No new sampler sanitizer case is excluded; M08's separately documented timing-only limitation is not claimed to be resolved or extended.

Motivation and Context

Supply real GPU selection through the existing stateful plugin contract without undisclosed CPU sampling. Existing argmax NaN behavior alone cannot meet the approved whole-logits validation rule; combining the scan and selection with suitable CUB primitives preserves exact semantics and bounded scratch/result movement.

M09a is independently reviewable and verified. Macro M09 remains incomplete until separately authorized M09b wires GPU logits/sampling through loops/chains. No predecessor is automatically merged.

devang-ml and others added 2 commits October 19, 2026 18:53
Implement GPU whole-logits NaN validation and selected-row greedy reduction with lowest-ID ties, retained Session scratch allocation and synchronous16-byte token/status observation. Keep CPU defaults and legacy CUDA behavior unchanged; loop/chain GPU sampling remains M09b.

Verify CPU63/139, CUDA117/184 and legacy110/43 tests,83/83 host and25/25 actual device source coverage, measured scalar transfers, sanitizer checks and public C/C++ resource lifecycles.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document PR #22,83/83 host and25/25 measured device coverage, scalar-transfer evidence and complete standalone validation. Keep M09b unauthorized and macro M09 incomplete.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace repository, PR and branch references with this upstream draft stack. Integrate the predecessor's documentation-only cleanup without rebasing or changing executable code. Existing implementation commits remain intact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant