Repository navigation
Conversation
Implement GPU whole-logits NaN validation and selected-row greedy reduction with lowest-ID ties, retained Session scratch allocation and synchronous16-byte token/status observation. Keep CPU defaults and legacy CUDA behavior unchanged; loop/chain GPU sampling remains M09b. Verify CPU63/139, CUDA117/184 and legacy110/43 tests,83/83 host and25/25 actual device source coverage, measured scalar transfers, sanitizer checks and public C/C++ resource lifecycles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document PR #22,83/83 host and25/25 measured device coverage, scalar-transfer evidence and complete standalone validation. Keep M09b unauthorized and macro M09 incomplete. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 10, 2026
Replace repository, PR and branch references with this upstream draft stack. Integrate the predecessor's documentation-only cleanup without rebasing or changing executable code. Existing implementation commits remain intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Upstream draft stack 12/21. Original milestone: M09a.
stateful-M12-cuda-plugin-greedyatd2021826513493a6a44181112399c7a2dd616d73stateful-M11-cuda-plugin-cache-integrationat89e8fef5b093a20344b15e2188589578ddbe8ce5The
[Stateful - MNN]title/branch numbers identify positions in this 21-PR stack, not original milestone numbering. M16-M23 language-binding milestones remain deferred.Historical review base:
1988019b6e23ac4def4628cddd603c1bfb504fe8, preserved asstate/stateful-base-1988019b. The first PR targets that historical review anchor; each later PR targets its predecessor. These PRs have not been rebased onto currentmain. The anchor is for incremental review; integration/retargeting requires a later explicit decision.Validation provenance: Recorded milestone validation below is retained. The appended reference-cleanup/merge commits change Markdown references only; executable files are byte-for-byte unchanged. No new build/test or current-main qualification is claimed. Historical commits are preserved, while current documentation and PR links use this Microsoft draft stack. Upstream CI results are separate.
M09a, stacked on #34050 (
stateful-M11-cuda-plugin-cache-integration). Implements standalone CUDA-plugin greedy sampling. M09b loop/chain GPU sampling remains unimplemented and requires separate authorization.[1,V]and[1,S,V].generate/.cudaLaunchKernel; keep actual device code in the NVCC unit with source information. Registered launch entry signatures match the device definitions.No public API table/version change or new owning C++ wrapper is needed: this is an appended borrowed provider-request resource field; existing public C/C++ sampler handles expose the feature.
Approved specification, execution plan, SDK example and exact evidence are in SPEC.md, TASKS.md and SCRATCHPAD.md.
Validation
Linux/A100 SM80; matching CUDA 13.1/cuDNN; Clang18 host LLVM and NVCC13.1/GCC13. Every reported functional suite has nonzero counts and zero skips.
./onnxruntime_session_loop_test --gtest_output=xml:coverage/m09a/final/native-tests.xml./onnxruntime_autoep_test --gtest_output=xml:coverage/m09a/final/autoep-tests.xml./onnxruntime_session_loop_test --gtest_output=xml:coverage/m09a/final/native-tests.xml./onnxruntime_autoep_test --gtest_output=xml:coverage/m09a/final/autoep-tests.xml./onnxruntime_session_loop_test --gtest_output=xml:coverage/m09a/final/native-tests.xml./onnxruntime_autoep_test --gtest_filter='AutoEpSelection.CudaEP:*Stateful*' --gtest_output=xml:coverage/m09a/final/stateful-tests.xmlThe instrumented CUDA-plugin build passed:
The existing supported nonfatal-warning configuration handles baseline mixed-compiler diagnostics; changed implementation/test units pass strict type checks. No dependency was installed. Final public-header comment-only edits describe CPU-default/explicit provider behavior without changing the validated runtime code/ABI.
Measured host/glue coverage: 83/83 (100%), from actual exercised plugin/core artifacts:
Fresh matching profiles were merged with
llvm-profdata merge -sparse coverage/m09a/final/*.profraw -o coverage/m09a/final/m09a.profdata. Module-specific LLVM source lines were intersected against M08b2b6388fe5b. All 589 plugin/174 runtime map warnings are dormant zero-hash entries, with no new-function mismatch.Measured device coverage: 25/25 (100%) of emitted new CUDA-source locations, including helper bodies and mapped entry/epilogue braces. No device source location is excluded. Actual float/half/final kernels were observed across 44 profiled launches, with source text/SHA256 matched to the exercised artifact. The declarations-only helper header has no additional device instruction mappings.
Before production implementation, a distinct executed/unexecuted-path probe demonstrated reciprocal source execution counters. The initial unprivileged profiler attempt failed with
ERR_NVGPUCTRPERM; the user explicitly authorized administrator privileges for profiling. Only the profiler process was elevated; no global driver setting/permission was changed.Result: passed, with 25/25 source-mapped executed-instruction locations. Host counters, launch success and sanitizer output are not substituted for device coverage.
A separate Nsight Systems CUDA trace of
CudaStatefulTest.GreedyTransferAndLatencypassed and measured 33 Device-to-Host transfers, each 16 bytes, total 528 bytes—exactly the 33 GPU sampler calls. No full-logits readback is present. The explicit input upload/setup traffic is recorded separately. Representative instrumented normal-suite mean times over 32 measured calls at vocabulary 50257 were 67.33us GPU / 98.54us CPU; these are observations, not release benchmark claims or performance gates.These CUDA 13.1 Compute Sanitizer commands passed with zero errors and zero leaked bytes/allocations:
The documented C++17 GPU SDK example also passed normal execution and memcheck after Session release/library unregistration while the caller-owned allocator Session remains alive. Genuine C11 compilation/execution, CPU C++17 normal/no-exceptions client examples, formatting/120 columns, whitespace, 93-requirement/24-milestone traceability and CPU-library CUDA-dependency checks passed. No new sampler sanitizer case is excluded; M08's separately documented timing-only limitation is not claimed to be resolved or extended.
Motivation and Context
Supply real GPU selection through the existing stateful plugin contract without undisclosed CPU sampling. Existing argmax NaN behavior alone cannot meet the approved whole-logits validation rule; combining the scan and selection with suitable CUB primitives preserves exact semantics and bounded scratch/result movement.
M09a is independently reviewable and verified. Macro M09 remains incomplete until separately authorized M09b wires GPU logits/sampling through loops/chains. No predecessor is automatically merged.