Skip to content

[Stateful - M11] GPU-resident CUDA plugin caches in loops and chains - #34050

Draft
devang-ml wants to merge 3 commits into
stateful-M10-cuda-plugin-kv-cachefrom
stateful-M11-cuda-plugin-cache-integration
Draft

devang-ml wants to merge 3 commits into
stateful-M10-cuda-plugin-kv-cachefrom
stateful-M11-cuda-plugin-cache-integration

Conversation

@devang-ml

@devang-ml devang-ml commented Nov 10, 2026 •

Copy link
Copy Markdown

Description

Upstream draft stack 11/21. Original milestone: M08b.

The [Stateful - MNN] title/branch numbers identify positions in this 21-PR stack, not original milestone numbering. M16-M23 language-binding milestones remain deferred.

Historical review base: 1988019b6e23ac4def4628cddd603c1bfb504fe8, preserved as state/stateful-base-1988019b. The first PR targets that historical review anchor; each later PR targets its predecessor. These PRs have not been rebased onto current main. The anchor is for incremental review; integration/retargeting requires a later explicit decision.

Validation provenance: Recorded milestone validation below is retained. The appended reference-cleanup/merge commits change Markdown references only; executable files are byte-for-byte unchanged. No new build/test or current-main qualification is claimed. Historical commits are preserved, while current documentation and PR links use this Microsoft draft stack. Upstream CI results are separate.

M08b, stacked on #34049 (stateful-M10-cuda-plugin-kv-cache). Completes the approved CUDA-plugin-only M08 cache milestone. GPU greedy sampling remains M09 and is not included.

  • Resolve the actual initialized Session GPU allocator, including chains with a CPU-only first stage.
  • Validate declared cache producers/consumers against their planned GPU; reject incompatible devices/host consumers instead of silently staging caches.
  • Bind managed outputs to GPU, retain their allocator/provider leases, and follow declared component-output/named-state cache edges, including finite/feedback inputs and plain Session cache-state writers.
  • Keep selected/component/state cache tensors resident on GPU. Ordinary outputs/logits retain stable CPU snapshots; CPU greedy and staging diagnostics remain unchanged.
  • Rely on synchronous Run's actual task-stream completion instead of adding IOBinding provider-wide Sync calls.
  • Preserve snapshots across update/reset, cancellation/model failure, context/Session release and library unregistration.
  • Add native, genuine C11 and public C++ conformance tests. No public ABI fields/API-table entries or CUDA kernels are added.

The approved contract and exact milestone evidence are in SPEC.md, TASKS.md and SCRATCHPAD.md.

Validation

Linux/A100 SM80, matching CUDA 13.1/cuDNN. GPU checks expose devices 0 and 1. Every reported functional suite has zero skips.

Configuration Exact command, from the corresponding build output directory Result
CPU ./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/final/native-tests.xml 63/63 passed
CPU ./onnxruntime_autoep_test --gtest_output=xml:coverage/m08b/final/autoep-tests.xml 139/139 passed, unfiltered
CUDA plugin ./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/complete/native-tests.xml 116/116 passed, unfiltered
CUDA plugin ./onnxruntime_autoep_test --gtest_output=xml:coverage/m08b/complete/autoep-tests.xml 179/179 passed, unfiltered
Legacy/shared CUDA ./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/final/native-tests.xml 110/110 passed
Legacy/shared CUDA ./onnxruntime_autoep_test --gtest_filter='AutoEpSelection.CudaEP:*Stateful*' --gtest_output=xml:coverage/m08b/final/stateful-tests.xml 43/43 passed; native cache capability remains unavailable

The new five native and five public integration cases exercise all four Session/Loop chain forms, actual bound device ports, Share identity versus Copy/Append storage, GPU feedback/finite inputs, named-state/component aliases, a leading CPU stage, prefill/decode budgets 0/1/3, EOS, CPU-reference token/cache parity, independent contexts, cross-Session user streams, errors/reset and retained C/C++ results. XML confirms actual cross-device cache rejection.

Final builds passed for both test targets in the CPU, CUDA-plugin and legacy CUDA trees. The instrumented plugin build uses the existing supported nonfatal-warning configuration for baseline mixed-compiler diagnostics:

cmake --compile-no-warning-as-error -S cmake -B build/state-cuda-plugin-13.1/RelWithDebInfo
cmake --build build/state-cuda-plugin-13.1/RelWithDebInfo --parallel 16 \
  --target onnxruntime_session_loop_test onnxruntime_autoep_test

Measured new executable runtime-line coverage: 130/130 (100%), from actual plugin-enabled unit execution:

  • Loop integration: 88/88.
  • Chain integration: 42/42.
  • New reusable fixture helper, separately: 12/12.

Fresh profiles use coverage/m08b/complete and were merged with llvm-profdata merge -sparse coverage/m08b/complete/*.profraw -o coverage/m08b/complete/m08b.profdata. LLVM line maps from the exercised native/shared/API images were intersected with added lines against M08a 6959554e55. All 398 map warnings were dormant zero-hash entries, with no new-function mismatch.

New device functions/lines: 0/0, N/A, not a claimed 100% device-code result.

These Compute Sanitizer commands passed with zero errors and zero leaked bytes/allocations:

compute-sanitizer --tool memcheck --leak-check full --error-exitcode 99 \
  ./onnxruntime_session_loop_test \
  --gtest_filter='CudaCacheIntegrationTest.*-CudaCacheIntegrationTest.SynchronousTaskStreamWithoutGlobalDeviceWait'
# 4/4 native memory cases passed.

compute-sanitizer --tool memcheck --leak-check full --error-exitcode 99 \
  ./onnxruntime_autoep_test --gtest_filter='CudaCacheApiIntegrationTest.*:*CudaStateful*'
# 22/22 public integration and standalone cache cases passed.

Explicit sanitizer timing limitation: SynchronousTaskStreamWithoutGlobalDeviceWait passes the unfiltered normal suite and 10 consecutive normal repetitions:

./onnxruntime_session_loop_test \
  --gtest_filter='CudaCacheIntegrationTest.SynchronousTaskStreamWithoutGlobalDeviceWait' \
  --gtest_repeat=10 --gtest_break_on_failure

Its wall-clock assertion times out inside instrumented model Run while an unrelated host callback is held blocked. It is therefore excluded only from the native memcheck memory selector, not from functional testing or coverage, and is not claimed to pass memcheck. Same-thread warmup, ready device feeds and prepared outputs did not remove that instrumented timeout. A separate CUDA-only memset calibration passed under memcheck, so no universal serialization claim is made. No memory error is suppressed.

Genuine C11 compilation/execution, the documented C++17 CPU example with normal/no-exceptions client flags, strict changed-source type checks, formatting/120-column limits, whitespace, 93-requirement/24-milestone traceability and the CPU library's lack of a CUDA dependency all passed.

Motivation and Context

M08a supplied real standalone CUDA-plugin cache callbacks. This milestone connects them end-to-end to retained execution state and prefill/decode sharing without cache host round-trips or changing ordinary stateless Session behavior.

Only declared internal cache edges receive device routing; arbitrary external/initial-state chain inputs remain host-only. Legacy/shared CUDA native-cache implementation and GPU sampling remain out of scope. The PR is independently reviewable and ready for human review; no predecessor is automatically merged.

devang-ml and others added 2 commits October 19, 2026 16:24
Keep declared cache ports, outputs and state device-resident using initialized Session allocators and retained provider leases. Preserve CPU ordinary outputs/sampling and validate compatible cache routes across Sessions without extra provider-wide synchronization.

Verify CPU63/139, legacy110/43, plugin116/179 tests and130/130 new runtime lines. Document zero-error scoped sanitizer results and the instrumented timing-test limitation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document PR #21, final CPU/CUDA evidence and130/130 runtime-line coverage. Preserve the explicit memcheck timing-test limitation and leave M09 pending.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace repository, PR and branch references with this upstream draft stack. Integrate the predecessor's documentation-only cleanup without rebasing or changing executable code. Existing implementation commits remain intact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@chilo-ms

Copy link
Copy Markdown
Contributor

Declared State for CPU/GPU Placement

For context, the K1 contract and workflow (draft #34113) supports the VRAM-budgeted CPU/GPU placement proposal in #33372. The proposed workflow is: owner declares KV requirements → ORT plans placement under the budget → owner allocates each cache at its committed location.

I'd like to align the GPU-cache integration in #34050 with this workflow. Currently, the loop-wide cache_provider and single allocator/provider per KVCacheManager effectively select one device for all caches. With GPU selection, check_ports rejects CPU-layer cache ports in a CPU-prefix/CUDA-suffix plan. With CPU selection, GPU-layer caches require per-step staging.

For budget-aware mixed placement, please consider per-cache placement: follow_consumer, selecting the allocator/provider from each cache's planned memory location. An optional explicit pin should be checked against the plan; a conflicting pin should fail unless an explicitly priced copy-through mode is supported.

Cache-operation capabilities can remain independent of node-placement capabilities, but cache storage location must respect the consuming kernel's planned input-memory requirements. The existing fail-closed validation is valuable—I'm suggesting applying it per cache/device, not relaxing it or allowing hidden transfers. This could be a follow-up extension rather than expanding this milestone's scope.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants