Repository navigation
Conversation
Keep declared cache ports, outputs and state device-resident using initialized Session allocators and retained provider leases. Preserve CPU ordinary outputs/sampling and validate compatible cache routes across Sessions without extra provider-wide synchronization. Verify CPU63/139, legacy110/43, plugin116/179 tests and130/130 new runtime lines. Document zero-error scoped sanitizer results and the instrumented timing-test limitation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document PR #21, final CPU/CUDA evidence and130/130 runtime-line coverage. Preserve the explicit memcheck timing-test limitation and leave M09 pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace repository, PR and branch references with this upstream draft stack. Integrate the predecessor's documentation-only cleanup without rebasing or changing executable code. Existing implementation commits remain intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Declared State for CPU/GPU Placement For context, the K1 contract and workflow (draft #34113) supports the VRAM-budgeted CPU/GPU placement proposal in #33372. The proposed workflow is: owner declares KV requirements → ORT plans placement under the budget → owner allocates each cache at its committed location. I'd like to align the GPU-cache integration in #34050 with this workflow. Currently, the loop-wide For budget-aware mixed placement, please consider per-cache Cache-operation capabilities can remain independent of node-placement capabilities, but cache storage location must respect the consuming kernel's planned input-memory requirements. The existing fail-closed validation is valuable—I'm suggesting applying it per cache/device, not relaxing it or allowing hidden transfers. This could be a follow-up extension rather than expanding this milestone's scope. |
Description
Upstream draft stack 11/21. Original milestone: M08b.
stateful-M11-cuda-plugin-cache-integrationat89e8fef5b093a20344b15e2188589578ddbe8ce5stateful-M10-cuda-plugin-kv-cacheat30798bfccfc711ff3d49a6d55d647d2ce8476108The
[Stateful - MNN]title/branch numbers identify positions in this 21-PR stack, not original milestone numbering. M16-M23 language-binding milestones remain deferred.Historical review base:
1988019b6e23ac4def4628cddd603c1bfb504fe8, preserved asstate/stateful-base-1988019b. The first PR targets that historical review anchor; each later PR targets its predecessor. These PRs have not been rebased onto currentmain. The anchor is for incremental review; integration/retargeting requires a later explicit decision.Validation provenance: Recorded milestone validation below is retained. The appended reference-cleanup/merge commits change Markdown references only; executable files are byte-for-byte unchanged. No new build/test or current-main qualification is claimed. Historical commits are preserved, while current documentation and PR links use this Microsoft draft stack. Upstream CI results are separate.
M08b, stacked on #34049 (
stateful-M10-cuda-plugin-kv-cache). Completes the approved CUDA-plugin-only M08 cache milestone. GPU greedy sampling remains M09 and is not included.The approved contract and exact milestone evidence are in SPEC.md, TASKS.md and SCRATCHPAD.md.
Validation
Linux/A100 SM80, matching CUDA 13.1/cuDNN. GPU checks expose devices 0 and 1. Every reported functional suite has zero skips.
./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/final/native-tests.xml./onnxruntime_autoep_test --gtest_output=xml:coverage/m08b/final/autoep-tests.xml./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/complete/native-tests.xml./onnxruntime_autoep_test --gtest_output=xml:coverage/m08b/complete/autoep-tests.xml./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/final/native-tests.xml./onnxruntime_autoep_test --gtest_filter='AutoEpSelection.CudaEP:*Stateful*' --gtest_output=xml:coverage/m08b/final/stateful-tests.xmlThe new five native and five public integration cases exercise all four Session/Loop chain forms, actual bound device ports, Share identity versus Copy/Append storage, GPU feedback/finite inputs, named-state/component aliases, a leading CPU stage, prefill/decode budgets 0/1/3, EOS, CPU-reference token/cache parity, independent contexts, cross-Session user streams, errors/reset and retained C/C++ results. XML confirms actual cross-device cache rejection.
Final builds passed for both test targets in the CPU, CUDA-plugin and legacy CUDA trees. The instrumented plugin build uses the existing supported nonfatal-warning configuration for baseline mixed-compiler diagnostics:
Measured new executable runtime-line coverage: 130/130 (100%), from actual plugin-enabled unit execution:
Fresh profiles use
coverage/m08b/completeand were merged withllvm-profdata merge -sparse coverage/m08b/complete/*.profraw -o coverage/m08b/complete/m08b.profdata. LLVM line maps from the exercised native/shared/API images were intersected with added lines against M08a6959554e55. All 398 map warnings were dormant zero-hash entries, with no new-function mismatch.New device functions/lines: 0/0, N/A, not a claimed 100% device-code result.
These Compute Sanitizer commands passed with zero errors and zero leaked bytes/allocations:
Explicit sanitizer timing limitation:
SynchronousTaskStreamWithoutGlobalDeviceWaitpasses the unfiltered normal suite and 10 consecutive normal repetitions:./onnxruntime_session_loop_test \ --gtest_filter='CudaCacheIntegrationTest.SynchronousTaskStreamWithoutGlobalDeviceWait' \ --gtest_repeat=10 --gtest_break_on_failureIts wall-clock assertion times out inside instrumented model Run while an unrelated host callback is held blocked. It is therefore excluded only from the native memcheck memory selector, not from functional testing or coverage, and is not claimed to pass memcheck. Same-thread warmup, ready device feeds and prepared outputs did not remove that instrumented timeout. A separate CUDA-only memset calibration passed under memcheck, so no universal serialization claim is made. No memory error is suppressed.
Genuine C11 compilation/execution, the documented C++17 CPU example with normal/no-exceptions client flags, strict changed-source type checks, formatting/120-column limits, whitespace, 93-requirement/24-milestone traceability and the CPU library's lack of a CUDA dependency all passed.
Motivation and Context
M08a supplied real standalone CUDA-plugin cache callbacks. This milestone connects them end-to-end to retained execution state and prefill/decode sharing without cache host round-trips or changing ordinary stateless Session behavior.
Only declared internal cache edges receive device routing; arbitrary external/initial-state chain inputs remain host-only. Legacy/shared CUDA native-cache implementation and GPU sampling remain out of scope. The PR is independently reviewable and ready for human review; no predecessor is automatically merged.