From 249f60acf837ca690b33c57e827480c7b96135f4 Mon Sep 17 00:00:00 2001 From: Devang Patel <47577486+devang-ml@users.noreply.github.com> Date: Mon, 19 Oct 2026 16:24:57 +0000 Subject: [PATCH 1/2] Integrate CUDA plugin caches with stateful loops and chains (M08b) Keep declared cache ports, outputs and state device-resident using initialized Session allocators and retained provider leases. Preserve CPU ordinary outputs/sampling and validate compatible cache routes across Sessions without extra provider-wide synchronization. Verify CPU63/139, legacy110/43, plugin116/179 tests and130/130 new runtime lines. Document zero-error scoped sanitizer results and the instrumented timing-test limitation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- SCRATCHPAD.md | 46 +++ SPEC.md | 33 +- TASKS.md | 130 +++++- cmake/onnxruntime_unittests.cmake | 1 + .../core/session/onnxruntime_c_api.h | 7 +- onnxruntime/core/session/session_chain.cc | 45 +- onnxruntime/core/session/session_loop.cc | 103 ++++- onnxruntime/core/session/session_loop.h | 11 +- onnxruntime/test/autoep/stateful_api_c_test.c | 53 +++ .../autoep/test_cuda_stateful_integration.cc | 391 ++++++++++++++++++ .../test/framework/session_loop_cuda_test.cc | 319 ++++++++++++++ .../include/stateful_execution_test_utils.h | 13 + 12 files changed, 1129 insertions(+), 23 deletions(-) create mode 100644 onnxruntime/test/autoep/test_cuda_stateful_integration.cc create mode 100644 onnxruntime/test/framework/session_loop_cuda_test.cc diff --git a/SCRATCHPAD.md b/SCRATCHPAD.md index 2a0a1478021fb..be1553a53f155 100644 --- a/SCRATCHPAD.md +++ b/SCRATCHPAD.md @@ -706,3 +706,49 @@ Compressed project memory. This is not the execution plan. published ready for review as PR #20, stacked on #19: https://github.com/onnxruntime/ort-state/pull/20 M08b has NOT started and still needs user authorization. No PRs merged. + +## M08b + +- User approved M08a and authorized the next milestone on 2026-10-19. + Existing approved Appendix H/TASKS B1-B7 apply. Created + state/m08b-cuda-plugin-cache-integration from M08a6959554e55. +- Scope remains CUDA plugin only; GPU sampling stays M09. Implement all + managed/selected/component/named-state cache surfaces, retain GPU results and + local Session allocators/provider leases. CPU ordinary outputs stay stable. +- Existing IOBinding SynchronizeInputs/Outputs calls provider Sync(), which is + CUDA device-wide. Device-cache integration must rely instead on Run's synchronous + task-stream cleanup; caller external inputs must already be ready. +- Core implementation tracks private per-loop device-cache input/output names, + validates actual SessionState port devices and retains output values with local + provider/allocator leases. Chain performs a second, forward-only cache-edge + pass including named state/cache writers without managed port pairs. +- Arbitrary external/initial-state chain feeds remain host-only; GPU routing is + inferred ONLY from declared internal cache/state/component edges, not dtype/shape. + No public configuration fields or GPU kernels are added. +- New native/public tests cover Share/Copy/Append, stable device snapshots, + actual bound ports, CPU-only leading stage, four Session/Loop combinations, + component+named-state cache aliases, finite/feedback GPU inputs, generation + budgets/EOS against CPU, cross-Session user streams, cancellation/model errors, + C11 lifetimes/unregistration and unrelated-stream nonblocking proof. +- Initial complete GPU tests116 native+179 unfiltered AutoEP passed; native5 + +public5 new integration cases all pass. Runtime coverage129/129 plus new + model helper12/12; final matching-artifact profiles are being regenerated. +- Sanitizer public integration+standalone22 cases pass zero errors/leaks; + native memory4 pass zero errors/leaks. The unrelated-stream wall-clock + concurrency assertion times out only under memcheck, inside model Run. + It passes normal execution ten consecutive times with an unrelated stream + held blocked. Same-thread warmup, device-only feeds and prepared outputs were + tested; a CUDA-only memset calibration passes under memcheck, so do NOT claim + universal sanitizer serialization. Document the exact scoped limitation: + native timing test remains in full normal suites/coverage, excluded only from + memcheck's memory-safety selector. No memory error is suppressed. +- Corrected a test-only malformed NodeInfo injection: adding conflicting device + usage via the real AddInput mapping API rejects it before our guard and + dereferences a supplied null node for its error message. Final test instead + temporarily changes/restores the existing planned input device. +- FINAL M08b artifacts: CPU63/139, legacy110/43 and plugin116/179 all passed, + zero skips in the reported suites. Runtime coverage130/130 (loop88+chain42), + helper12/12; no devicefunctions0/N/A. Final profiles coverage/m08b/complete. + GPU cache-only diagnostic correctly retains unavailable GPU logits wording. + Full GPU suites and explicit native-memory4/public22 memcheck selectors are + separate from the documented instrumented timing limitation. No M09 code. diff --git a/SPEC.md b/SPEC.md index 96bfda8de63cd..04e3ff2b273c1 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1413,8 +1413,8 @@ with only its version number changed. **Status:** Approved by the user on 2026-10-18 without changes. Scope is CUDA plugin EP only, CUDA copy primitives where sufficient, GPU-resident cache views/results, and an M08a/M08b split, with no additional specification input. -The detailed execution plan is approved without changes. M08a implementation is -authorized; M08b requires separate authorization after M08a is complete. +The detailed execution plan is approved without changes. M08a is complete in +PR #20. The user authorized the next milestone, M08b, on 2026-10-19. ### H.1 Scope and preserved behavior @@ -1550,7 +1550,7 @@ authorized; M08b requires separate authorization after M08a is complete. ### H.8 Standalone CUDA-plugin SDK example (M08a) -M08a enables standalone caches, not device-cache loop/chain integration (M08b). +M08a enables standalone caches; M08b adds device-cache loop/chain integration. The plugin uses checked, compact per-row CUDA copies, avoiding 2D-copy pitch limits without new device functions. Operations finish on the default or caller-configured stream before returning; external inputs must already be @@ -1632,3 +1632,30 @@ int main(int argc, char** argv) { return retained.length == 2 && output == expected ? 0 : 4; } ``` + +### H.9 Device-resident loop/chain integration (M08b) + +- Select `CUDAExecutionProvider` with GPU memory and an explicit compatible + device (or the provider's unambiguous device). CPU remains the default. +- Resolve cache allocation from the initialized Session's actual GPU allocator, + including chains whose first component is CPU-only. Verify all declared cache + producers/consumers execute on the selected GPU; reject mismatches instead of + inserting an implicit cache transfer. +- Bind managed cache outputs directly to that GPU and retain their allocator/ + provider lease. Ordinary outputs and logits keep CPU snapshots; CPU sampling + is reported separately. No GPU sampling is added. +- Follow declared managed-cache output, component-output and named-state edges. + A state output writing a shared cache slot is device-bound even when its + component has no managed past/present pair. Feedback and finite loop inputs + routed from device caches keep their device values and validate placement. + Arbitrary external/initial-state chain inputs retain the host-accessible + contract; this milestone does not infer a cache from a tensor's shape or + silently treat unrelated device inputs as a declared cache edge. +- Synchronous Run cleans up/synchronizes its actual execution streams before + publishing values. The explicit device-cache path does not additionally call + IOBinding's provider-wide Sync, which would wait on unrelated CUDA work. + Independent caller streams/allocators keep their ordinary readiness/lifetime + obligations. +- Retained GPU result/state tensors survive execution/reset and originating + Session release. Definitions/executions still require their caller Sessions + to remain alive. GPU tensor-data getters never perform implicit readback. diff --git a/TASKS.md b/TASKS.md index 8021f1bf6b4a0..263c9b9f7362f 100644 --- a/TASKS.md +++ b/TASKS.md @@ -1408,7 +1408,7 @@ Implementation uses `state/m08a-cuda-plugin-kv-cache`, based on M07b - [x] Implement CUDA allocation/retention/update/growth/reset using ORT allocator, device, and stream abstractions; advertise only supported features. -- [ ] Wire stage sharing and required movement without illegal aliasing. +- [x] Wire stage sharing and required movement without illegal aliasing. - [x] Expose dispatch/copy/allocation evidence for tests and later measurement. **CPU test plan:** Rerun CPU cache and EP-contract regressions. @@ -1494,7 +1494,7 @@ Publish ready for review; loop/chain GPU-cache integration is not yet claimed. **M08a complete:** Implementation `e6ce58508e`, published ready for human review as [PR #20](https://github.com/onnxruntime/ort-state/pull/20), stacked on M07b -PR #19. No PR has been merged. M08b remains unauthorized. +PR #19. No PR has been merged. The user authorized M08b on 2026-10-19. - Six cache operations execute in the real CUDA plugin with float32/float16/ bfloat16 support and no greedy capability or CPU-reference delegation. @@ -1613,31 +1613,34 @@ M08b remains unimplemented and requires user authorization. #### M08b - Device-resident loop/chain integration +**Status:** Authorized 2026-10-19. Implementation branch +`state/m08b-cuda-plugin-cache-integration`, based on M08a `6959554e55`. + **Intended outcome:** Programmatic loops/chains use the selected GPU cache without cache host round-trips, while stable ordinary outputs and CPU greedy sampling retain their existing behavior. Prefill/decode share one GPU backend across compatible CUDA-plugin Sessions, including distinct streams. **Dependency:** M08a. Separate branch/PR stacked on its verified/published head. -- [ ] B1: Resolve the selected GPU allocator/device from initialized Session +- [x] B1: Resolve the selected GPU allocator/device from initialized Session resources without hard-coded allocator names or hidden CPU substitution. Remove only the implemented CUDA-plugin cache integration rejection; keep device sampling and legacy/shared native-cache requests unsupported. -- [ ] B2: Bind managed cache outputs to GPU memory and feed device-resident +- [x] B2: Bind managed cache outputs to GPU memory and feed device-resident retained cache values back to Sessions. Process cache callback results without converting their state through CPU snapshots. -- [ ] B3: Wire selected/component/state/downstream cache tensor surfaces +- [x] B3: Wire selected/component/state/downstream cache tensor surfaces consistently: managed cache values remain GPU-resident under explicit GPU selection, ordinary non-cache outputs stay stable CPU values, and CPU logits/ token sampling remain explicitly observable until M09. -- [ ] B4: Preserve compatible prefill/decode sharing across Sessions and +- [x] B4: Preserve compatible prefill/decode sharing across Sessions and nondefault streams on the same GPU. Validate shared backend/layout/device and cache-port compatibility; reject unsupported heterogeneous/cross-device paths. -- [ ] B5: Preserve retained GPU views/results across update/reset/release and +- [x] B5: Preserve retained GPU views/results across update/reset/release and unregistration; propagate callback, model, cancellation and reset failures through the existing invalidation/C/C++ error rules. Respect caller-owned streams/allocators/OrtEnv obligations. -- [ ] B6: Add end-to-end native/public C/C++ parity, device/copy/dispatch/staging +- [x] B6: Add end-to-end native/public C/C++ parity, device/copy/dispatch/staging evidence and stream/lifetime tests. Exercise relevant named-state/component handoff paths, not just one preferred cache binding. - [ ] B7: Measure every introduced core/plugin host path and any device code, @@ -1680,6 +1683,117 @@ All new core/plugin host lines and any new device lines meet their measured coverage gates. CPU/CUDA-plugin correctness, memory/order/lifetime, documentation and publication pass; mark M08 complete only after both sub-PRs. +**M08b verification complete; publication pending:** + +- Resolve actual Session GPU allocators and validate the declared cache ports' + planned device. Managed and declared cache-alias outputs are bound directly to + GPU and retain local allocator/provider leases rather than CPU snapshots. + Chains use the first cache-managing Session's allocator, not an assumed first + component allocator. A leading CPU-only component is explicitly covered. +- Track declared internal component/named-state cache edges, including a plain + Session state writer with no managed cache pair, finite inputs and feedback. + Arbitrary external/initial-state chain feeds remain host-only. Reject + cross-device/incompatible providers and host cache consumers before invocation. +- Preserve CPU logits/greedy, ordinary output snapshots and separate staging + counters. Shared GPU updates have zero CPU reference delegation. Native tests + inspect actual bound GPU inputs/outputs and Share pointer identity versus + Copy/Append fresh storage; public tests cover all four Session/Loop chain forms, + prefill/decode budgets 0/1/3, EOS, reset, independent contexts and retained results. +- Synchronous Run finishes the actual execution streams before result + publication. Device-cache execution avoids IOBinding's additional provider-wide + Sync. The normal unrelated-stream test uses warmed same-thread execution, + ready device inputs and prepared distinct outputs to isolate cache work from + synchronous host transfers/allocator effects; **10 consecutive repetitions + passed** before releasing the unrelated blocked stream. +- Final CPU: **63/63 native +139/139 unfiltered AutoEP** passed. +- Final CUDA plugin: **116/116 native +179/179 unfiltered AutoEP** passed, + **zero skips/exclusions**, with CUDA devices 0 and 1 visible. The new cases + are five native and five public GPU integration tests; parameter loops cover + phase/budget/EOS/chain/alias variations. XML confirms the cross-device + rejection test actually ran on two GPUs. +- Final legacy/shared CUDA: **110/110 native +43/43 selected public/stateful** + cases passed. Native cache capabilities remain absent; no legacy hook is added. +- Final new runtime executable-line coverage: **130/130**, comprising loop + integration **88/88** and chain integration **42/42**. The shared dynamic + decoder fixture helper is separately **12/12**. Fresh matching GPU profiles + use `coverage/m08b/complete`; the 398 LLVM map warnings are exclusively + dormant zero-hash maps, with no new-function mismatch. +- New device functions/lines: **0/0, N/A**, not a 100% device-coverage claim. + M08b adds no CUDA kernel or backend device code. +- Compute Sanitizer memcheck/leak-check: **4/4 native memory cases +22/22 public + integration/standalone cache cases** passed with **zero errors and zero leaked + bytes/allocations**. The genuine C11 loop consumer retains a GPU result through + reset, execution/definition/Session release and library unregistration. +- **Sanitizer timing limitation:** The fifth native case's wall-clock assertion + times out inside instrumented model Run while the unrelated host callback is + blocked. It remains in every full normal suite and the coverage denominator, + and passed ten repeated normal runs. It is excluded **only** from the explicit + native memcheck memory selector below; it is not claimed to pass memcheck. + Same-thread warmup/device-only feeds/prepared outputs did not change that + instrumented result. A separate CUDA-only memset calibration passed normally + and under memcheck, so no universal sanitizer-serialization claim is made. + No memory error is suppressed, and no runtime correctness test is excluded. +- Public C11 syntax checks, documented C++17 CPU example with normal and + no-exceptions client flags, strict changed-source type checks, formatting/ + 120-column limits, whitespace and 93-requirement/24-milestone traceability + passed. The CPU shared library has no CUDA runtime dependency. + +Passing commands (test commands run from their corresponding output directories): + +```bash +source .venv/bin/activate +cmake --compile-no-warning-as-error -S cmake \ + -B build/state-cuda-plugin-13.1/RelWithDebInfo +cmake --build build/state-cuda-plugin-13.1/RelWithDebInfo --parallel 16 \ + --target onnxruntime_session_loop_test onnxruntime_autoep_test +cd build/state-cuda-plugin-13.1/RelWithDebInfo +export CUDA_VISIBLE_DEVICES=0,1 +export CUDA_HOME=/datadisks/disk4/bmeswani/cuda/cuda13.1 +export LD_LIBRARY_PATH="$PWD:$CUDA_HOME/lib64:$CUDA_HOME/targets/x86_64-linux/lib" +mkdir -p coverage/m08b/complete +LLVM_PROFILE_FILE="$PWD/coverage/m08b/complete/%m-%p.profraw" \ + ./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/complete/native-tests.xml +# Passed: 116/116, zero skips. +LLVM_PROFILE_FILE="$PWD/coverage/m08b/complete/%m-%p.profraw" \ + ./onnxruntime_autoep_test --gtest_output=xml:coverage/m08b/complete/autoep-tests.xml +# Passed: 179/179, zero skips. +llvm-profdata merge -sparse coverage/m08b/complete/*.profraw \ + -o coverage/m08b/complete/m08b.profdata +# Passed: LLVM changed-source line exports against M08a6959554e55: +# runtime130/130; helper12/12. No device functions added. +export LLVM_PROFILE_FILE=/dev/null +./onnxruntime_session_loop_test \ + --gtest_filter='CudaCacheIntegrationTest.SynchronousTaskStreamWithoutGlobalDeviceWait' \ + --gtest_repeat=10 --gtest_break_on_failure +# Passed: all ten normal repetitions. +"$CUDA_HOME/bin/compute-sanitizer" --tool memcheck --leak-check full --error-exitcode 99 \ + ./onnxruntime_session_loop_test \ + --gtest_filter='CudaCacheIntegrationTest.*-CudaCacheIntegrationTest.SynchronousTaskStreamWithoutGlobalDeviceWait' +# Passed: 4/4, zero errors/leaks. Timing-only instrumentation limitation above. +"$CUDA_HOME/bin/compute-sanitizer" --tool memcheck --leak-check full --error-exitcode 99 \ + ./onnxruntime_autoep_test --gtest_filter='CudaCacheApiIntegrationTest.*:*CudaStateful*' +# Passed: 22/22, zero errors/leaks. +cd ../../state-cpu/RelWithDebInfo +./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/final/native-tests.xml +./onnxruntime_autoep_test --gtest_output=xml:coverage/m08b/final/autoep-tests.xml +# Passed: 63/63 native and 139/139 unfiltered AutoEP. +cd ../../state-cuda-baseline-13.1/RelWithDebInfo +export LD_LIBRARY_PATH="$PWD:$CUDA_HOME/lib64:$CUDA_HOME/targets/x86_64-linux/lib" +./onnxruntime_session_loop_test --gtest_output=xml:coverage/m08b/final/native-tests.xml +./onnxruntime_autoep_test --gtest_filter='AutoEpSelection.CudaEP:*Stateful*' \ + --gtest_output=xml:coverage/m08b/final/stateful-tests.xml +# Passed: 110/110 native and 43/43 selected public/stateful cases. +``` + +Compiler/CUDA/build configuration matches M08a: Clang18 host coverage, +NVCC13.1/GCC13, SM80, matching CUDA13.1/cuDNN. Regeneration retains the supported +nonfatal-warning build configuration for pre-existing mixed-compiler diagnostics. +No dependency or unrelated backend source changes were needed. Exact evidence: +`m08b-gpu-complete-*`, `m08b-cpu-*-tests.log`, `m08b-legacy-*-tests.log`, +`m08b-sanitizer-native-memory.log`, `m08b-sanitizer-public.log`, +`m08b-stream-repeated.log` and `m08b-host-coverage-final.txt` in session files. +M09 GPU sampling is not started or claimed. + **Validation and execution procedure:** - Keep TASKS/SCRATCHPAD current; stop for review of material changes to approved diff --git a/cmake/onnxruntime_unittests.cmake b/cmake/onnxruntime_unittests.cmake index 27d43b71dd010..c02f67836e4dd 100644 --- a/cmake/onnxruntime_unittests.cmake +++ b/cmake/onnxruntime_unittests.cmake @@ -1357,6 +1357,7 @@ target_include_directories(onnxruntime_test_all PRIVATE ${ONNXRUNTIME_ROOT}/core if(NOT onnxruntime_MINIMAL_BUILD AND NOT CMAKE_CROSSCOMPILING) AddTest(TARGET onnxruntime_session_loop_test SOURCES "${TEST_SRC_DIR}/framework/session_loop_test.cc" + "${TEST_SRC_DIR}/framework/session_loop_cuda_test.cc" "${TEST_SRC_DIR}/framework/stateful_provider_test.cc" "${TEST_SRC_DIR}/framework/session_chain_test.cc" ${onnxruntime_unittest_main_src} LIBS ${onnxruntime_test_providers_libs} ${onnxruntime_test_common_libs} diff --git a/include/onnxruntime/core/session/onnxruntime_c_api.h b/include/onnxruntime/core/session/onnxruntime_c_api.h index eec47ad2d5257..86e1300cbe8ab 100644 --- a/include/onnxruntime/core/session/onnxruntime_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_c_api.h @@ -9345,6 +9345,11 @@ typedef struct OrtStatefulGenerationConfig { * Pointer/count pairs permit NULL only with count zero. Nested array-record * layouts are fixed; future extensions use appended top-level fields/new records. * A loop requires explicit outputs and at least one exit condition. + * With an explicit CUDA-plugin GPU-cache policy, managed cache outputs and + * cache values routed through chain state/components remain on GPU. Ordinary + * outputs/logits remain stable CPU values; CPU inspection of device values + * requires an explicit caller copy. Device-cache ports must already execute + * on the selected GPU; implicit cache staging is rejected. * \since Version 1.32. */ typedef struct OrtStatefulLoopConfig { @@ -9873,7 +9878,7 @@ struct OrtStatefulApi { /** \brief Share/copy a full cache or append a delta. * \param[in,out] manager CPU manager. * \param[in] name Cache name. - * \param[in] value CPU tensor; shared allocator-owned data must remain immutable. + * \param[in] value Selected CPU/provider-device tensor; shared owned data must remain immutable. * \param[in] kind Explicit update policy. * \return NULL on success, otherwise an owned OrtStatus. \since Version 1.32. */ diff --git a/onnxruntime/core/session/session_chain.cc b/onnxruntime/core/session/session_chain.cc index 433d9aa9e0380..de4218dd6e2fb 100644 --- a/onnxruntime/core/session/session_chain.cc +++ b/onnxruntime/core/session/session_chain.cc @@ -125,6 +125,44 @@ Status SessionChain::Initialize() { sessions_.push_back(session); } + if (cache_provider_ && cache_provider_->Selection().device_type != OrtMemoryInfoDeviceType_CPU) { + InlinedVector> device_outputs; + for (size_t i = 0; i < loops_.size(); ++i) { + auto outputs = loops_[i]->device_cache_outputs_; + for (const auto& binding : options_.components[i].state_outputs) { + if (std::any_of(cache_specs_.begin(), cache_specs_.end(), + [&](const auto& spec) { return spec.name == binding.state_name; })) { + outputs.insert(binding.output_name); + } + } + device_outputs.push_back(std::move(outputs)); + } + InlinedHashSet device_state; + for (const auto& spec : cache_specs_) device_state.insert(spec.name); + for (size_t i = 0; i < loops_.size(); ++i) { + auto inputs = loops_[i]->device_cache_inputs_; + for (const auto& binding : options_.components[i].inputs) { + const auto& source = binding.source; + if ((source.kind == ChainSourceKind::State && device_state.contains(source.name)) || + (source.kind == ChainSourceKind::ComponentOutput && source.component_index < i && + device_outputs[source.component_index].contains(source.name))) { + inputs.insert(binding.input_name); + } + } + for (const auto& binding : options_.components[i].state_outputs) { + if (device_outputs[i].contains(binding.output_name)) { + device_state.insert(binding.state_name); + } else { + device_state.erase(binding.state_name); + } + } + if (!inputs.empty() || !device_outputs[i].empty()) { + ORT_RETURN_IF_ERROR(loops_[i]->WithDeviceCachePorts( + *cache_provider_, std::move(inputs), device_outputs[i], loops_[i])); + } + } + } + InlinedHashMap state_types; for (size_t index = 0; index < options_.components.size(); ++index) { const auto& component = options_.components[index]; @@ -289,8 +327,11 @@ Status SessionChain::CreateExecution(SessionChainInputs inputs, std::unique_ptr< std::unique_ptr binding; ORT_RETURN_IF_ERROR(sessions_[0]->NewIOBinding(&binding)); std::unique_ptr manager; - ORT_RETURN_IF_ERROR(KVCacheManager::Create(cache_specs_, binding->GetCPUAllocator(kCpuExecutionProvider), - manager, *cache_provider_)); + const auto cache_loop = std::find_if(loops_.begin(), loops_.end(), + [](const auto& loop) { return !loop->Options().kv_caches.empty(); }); + auto allocator = (*cache_loop)->device_cache_allocator_; + if (!allocator) allocator = binding->GetCPUAllocator(kCpuExecutionProvider); + ORT_RETURN_IF_ERROR(KVCacheManager::Create(cache_specs_, std::move(allocator), manager, *cache_provider_)); cache = std::move(manager); } auto execution = std::unique_ptr( diff --git a/onnxruntime/core/session/session_loop.cc b/onnxruntime/core/session/session_loop.cc index 9cbff733fbd31..2bc59e0c0450a 100644 --- a/onnxruntime/core/session/session_loop.cc +++ b/onnxruntime/core/session/session_loop.cc @@ -195,13 +195,77 @@ Status SessionLoop::Create(InferenceSession& session, SessionLoopOptions options ORT_RETURN_IF_ERROR( sampler_dispatch.Select(ORT_STATEFUL_SAMPLE_GREEDY, element_type, true, provider_selected)); } - if (options.cache_provider.device_type != OrtMemoryInfoDeviceType_CPU || - options.sampler_provider.device_type != OrtMemoryInfoDeviceType_CPU) { + if (options.sampler_provider.device_type != OrtMemoryInfoDeviceType_CPU) { return ORT_MAKE_STATUS( - ONNXRUNTIME, NOT_IMPLEMENTED, "Device-native cache/logits Session integration is unavailable"); + ONNXRUNTIME, NOT_IMPLEMENTED, "Device-native logits Session integration is unavailable"); } - out = std::shared_ptr( + auto loop = std::shared_ptr( new SessionLoop(session, std::move(options), std::move(cache_provider), std::move(sampler_provider))); + if (loop->cache_provider_.Selection().device_type != OrtMemoryInfoDeviceType_CPU) { + InlinedHashSet device_inputs, device_outputs; + for (const auto& item : loop->options_.kv_caches) { + device_inputs.insert(item.input_name); + device_outputs.insert(item.output_name); + } + ORT_RETURN_IF_ERROR(loop->ConfigureDeviceCache( + loop->cache_provider_, std::move(device_inputs), std::move(device_outputs))); + } + out = std::move(loop); + return Status::OK(); +} + +Status SessionLoop::ConfigureDeviceCache(const StatefulProviderBinding& provider, + InlinedHashSet inputs, + InlinedHashSet outputs) { + const auto& selection = provider.Selection(); + ORT_RETURN_IF_NOT(selection.device_type == OrtMemoryInfoDeviceType_GPU && + selection.provider_name == kCudaExecutionProvider, + "Device-cache Session integration requires the CUDA plugin"); + StatefulProviderBinding local; + ORT_RETURN_IF_ERROR(StatefulProviderBinding::Resolve(&session_, selection, local)); + ORT_RETURN_IF_NOT(provider.CacheCompatible(local), "Incompatible device-cache Session provider"); + ORT_RETURN_IF(options_.generation && outputs.contains(options_.generation->logits_output), + "Device cache output cannot also supply CPU sampler logits"); + const auto& state = session_.GetSessionState(); + for (const auto& [device, allocator] : state.GetAllocators()) { + if (device.Type() == OrtDevice::GPU && device.Id() == selection.device_id && + device.MemType() == OrtDevice::MemType::DEFAULT) { + device_cache_allocator_ = allocator; + break; + } + } + ORT_RETURN_IF_NOT(device_cache_allocator_, "Selected device-cache Session allocator is unavailable"); + const auto& device = device_cache_allocator_->Info().device; + for (const auto& feedback : options_.feedback) { + if (outputs.contains(feedback.output_name)) inputs.insert(feedback.input_name); + } + auto check_ports = [&](const auto& names, bool is_input) -> Status { + for (const auto& name : names) { + InlinedVector nodes; + ORT_RETURN_IF_ERROR(is_input ? state.GetInputNodeInfo(name, nodes) : state.GetOutputNodeInfo(name, nodes)); + ORT_RETURN_IF(nodes.empty() || + std::any_of(nodes.begin(), nodes.end(), [&](const auto& node) { + return !node.device || *node.device != device; + }), + "Device-cache port requires the selected GPU without staging: ", name); + } + return Status::OK(); + }; + ORT_RETURN_IF_ERROR(check_ports(inputs, true)); + ORT_RETURN_IF_ERROR(check_ports(outputs, false)); + device_cache_provider_ = std::move(local); + device_cache_inputs_ = std::move(inputs); + device_cache_outputs_ = std::move(outputs); + return Status::OK(); +} + +Status SessionLoop::WithDeviceCachePorts(const StatefulProviderBinding& provider, + InlinedHashSet inputs, + InlinedHashSet outputs, + std::shared_ptr& out) const { + auto loop = std::shared_ptr(new SessionLoop(session_, options_, cache_provider_, sampler_provider_)); + ORT_RETURN_IF_ERROR(loop->ConfigureDeviceCache(provider, std::move(inputs), std::move(outputs))); + out = std::move(loop); return Status::OK(); } @@ -221,6 +285,10 @@ Status SessionLoop::ValidateInputs(const SessionLoopInputs& inputs) const { [&](const auto& item) { return item.input_name == name; }), "Managed KV cache inputs cannot also be supplied as invariant feeds"); ORT_RETURN_IF_ERROR(CheckInput(*argument, value)); + if (device_cache_inputs_.contains(name)) { + ORT_RETURN_IF(value.Get().Location().device != device_cache_allocator_->Info().device, + "Device-cache input requires matching GPU storage: ", name); + } } ORT_RETURN_IF(finite.empty() && !inputs.batches.empty(), "Batches require finite input names"); ORT_RETURN_IF(!options_.nonempty_input.empty() && inputs.batches.empty(), "Prefill input must not be empty"); @@ -229,6 +297,10 @@ Status SessionLoop::ValidateInputs(const SessionLoopInputs& inputs) const { for (const auto& [name, value] : batch) { ORT_RETURN_IF_NOT(finite.contains(name), "Unknown finite input: ", name); ORT_RETURN_IF_ERROR(CheckInput(*FindArgument(arguments, name), value)); + if (device_cache_inputs_.contains(name)) { + ORT_RETURN_IF(value.Get().Location().device != device_cache_allocator_->Info().device, + "Device-cache finite input requires matching GPU storage: ", name); + } ORT_RETURN_IF(name == options_.nonempty_input && value.Get().Shape().Size() == 0, "Prefill input tensor must not be empty"); } @@ -253,7 +325,8 @@ Status SessionLoop::CreateExecution(SessionLoopInputs inputs, std::unique_ptr binding; ORT_RETURN_IF_ERROR(session_.NewIOBinding(&binding)); std::unique_ptr manager; - auto allocator = binding->GetCPUAllocator(kCpuExecutionProvider); + auto allocator = + device_cache_allocator_ ? device_cache_allocator_ : binding->GetCPUAllocator(kCpuExecutionProvider); ORT_RETURN_IF_ERROR(KVCacheManager::Create(std::move(specs), std::move(allocator), manager, cache_provider_)); cache = std::move(manager); } @@ -390,12 +463,19 @@ Status SessionLoopExecution::Step(SessionLoopStepResult& out) { } } for (const auto& name : loop_->fetch_names_) { - ORT_RETURN_IF_ERROR(binding->BindOutput(name)); + ORT_RETURN_IF_ERROR(binding->BindOutput( + name, loop_->device_cache_outputs_.contains(name) + ? loop_->device_cache_allocator_->Info().device + : OrtDevice{})); + } + if (!loop_->device_cache_provider_) { + ORT_RETURN_IF_ERROR(binding->SynchronizeInputs()); } - ORT_RETURN_IF_ERROR(binding->SynchronizeInputs()); RunOptions run_options; ORT_RETURN_IF_ERROR(loop_->session_.Run(run_options, *binding)); - ORT_RETURN_IF_ERROR(binding->SynchronizeOutputs()); + if (!loop_->device_cache_provider_) { + ORT_RETURN_IF_ERROR(binding->SynchronizeOutputs()); + } ORT_RETURN_IF(cancelled_.load(), "SessionLoop execution cancelled during a step; call Reset before reuse"); InlinedVector snapshots; @@ -404,6 +484,13 @@ Status SessionLoopExecution::Step(SessionLoopStepResult& out) { CPUDataTransfer transfer; for (const auto& value : binding->GetOutputs()) { const Tensor& source = value.Get(); + if (!source.Location().device.UsesCpuMemory()) { + ORT_RETURN_IF_NOT(loop_->device_cache_provider_ && + source.Location().device == loop_->device_cache_allocator_->Info().device, + "Unexpected device output in stateful execution"); + snapshots.push_back(loop_->device_cache_provider_->RetainValue(value, loop_->device_cache_allocator_)); + continue; + } if (source.Location().device.UsesCpuMemory()) { staging_.cpu_output_bytes = SafeInt(staging_.cpu_output_bytes) + source.SizeInBytes(); } diff --git a/onnxruntime/core/session/session_loop.h b/onnxruntime/core/session/session_loop.h index ebdf98da382be..fb5a34c3347f5 100644 --- a/onnxruntime/core/session/session_loop.h +++ b/onnxruntime/core/session/session_loop.h @@ -66,7 +66,7 @@ struct SessionLoopStepResult { int64_t iterations = 0; int64_t generated_tokens = 0; InlinedVector exit_reasons; - // Owned CPU snapshots, in the definition's output_names order. + // Stable outputs in definition order; explicitly selected device caches remain on device. InlinedVector outputs; std::optional generated_token; }; @@ -87,6 +87,11 @@ class SessionLoop : public std::enable_shared_from_this { SessionLoop(InferenceSession& session, SessionLoopOptions options, StatefulProviderBinding cache_provider, StatefulProviderBinding sampler_provider); Status ValidateInputs(const SessionLoopInputs& inputs) const; + Status ConfigureDeviceCache(const StatefulProviderBinding& provider, + InlinedHashSet inputs, InlinedHashSet outputs); + Status WithDeviceCachePorts(const StatefulProviderBinding& provider, + InlinedHashSet inputs, InlinedHashSet outputs, + std::shared_ptr& out) const; Status CreateExecutionWithCache(SessionLoopInputs inputs, std::shared_ptr cache, std::shared_ptr generation, std::shared_ptr sampler, @@ -97,6 +102,10 @@ class SessionLoop : public std::enable_shared_from_this { InlinedVector fetch_names_; const StatefulProviderBinding cache_provider_; const StatefulProviderBinding sampler_provider_; + std::optional device_cache_provider_; + AllocatorPtr device_cache_allocator_; + InlinedHashSet device_cache_inputs_; + InlinedHashSet device_cache_outputs_; }; // Only Cancel may run concurrently with Step/Run. Reset and execution are sequential. diff --git a/onnxruntime/test/autoep/stateful_api_c_test.c b/onnxruntime/test/autoep/stateful_api_c_test.c index d794d8226f522..15a805859dbd0 100644 --- a/onnxruntime/test/autoep/stateful_api_c_test.c +++ b/onnxruntime/test/autoep/stateful_api_c_test.c @@ -154,4 +154,57 @@ int StatefulApiCDeviceConsumerTest(OrtSession* session, OrtAllocator* allocator, return code; } +int StatefulApiCGpuLoopIntegrationTest(OrtSession* session, const OrtValue* tokens, + OrtStatefulStepResult** retained_out) { + const OrtApi* api = OrtGetApiBase()->GetApi(ORT_API_VERSION); + const OrtStatefulApi* stateful = api->GetStatefulApi(); + OrtStatus* status = NULL; + OrtSessionLoopOptions* options = NULL; + OrtSessionLoop* loop = NULL; + OrtStatefulExecution* execution = NULL; + OrtStatefulStepResult* result = NULL; + int code = 0; + const int64_t shape[] = {1, 1, 0, 1}; + const OrtStatefulKVCacheBinding cache = { + {"key", ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, shape, 4, 2, 0}, + "past_key", + "present_key", + ORT_KV_CACHE_SHARE}; + const char* finite[] = {"token"}; + const char* output[] = {"present_key"}; + const OrtStatefulExitCondition exit = {ORT_STATEFUL_END_OF_INPUTS, 0, NULL, NULL, 0}; + const OrtStatefulProviderSelection gpu = { + sizeof(gpu), ORT_STATEFUL_PROVIDER_REQUIRE, "CUDAExecutionProvider", OrtMemoryInfoDeviceType_GPU, 0}; + const OrtStatefulNamedValue token = {"token", tokens}; + const OrtStatefulInputBatch batch = {&token, 1}; + OrtStatefulLoopConfig config = {0}; + OrtStatefulInputs inputs = {0}; + config.struct_size = sizeof(config); + config.finite_input_names = finite; + config.num_finite_inputs = 1; + config.output_names = output; + config.num_outputs = 1; + config.exit_conditions = &exit; + config.num_exit_conditions = 1; + config.kv_caches = &cache; + config.num_kv_caches = 1; + inputs.struct_size = sizeof(inputs); + inputs.batches = &batch; + inputs.num_batches = 1; + CHECK(stateful->CreateSessionLoopOptions(&config, &options)); + CHECK(stateful->SessionLoopOptionsSetProvider(options, ORT_STATEFUL_FEATURE_CACHE, &gpu)); + CHECK(stateful->CreateSessionLoop(session, options, &loop)); + CHECK(stateful->CreateLoopExecution(loop, &inputs, &execution)); + CHECK(stateful->StatefulExecutionRun(execution, &result)); + CHECK(stateful->StatefulExecutionReset(execution)); + *retained_out = result; + result = NULL; +cleanup: + stateful->ReleaseStatefulStepResult(result); + stateful->ReleaseStatefulExecution(execution); + stateful->ReleaseSessionLoop(loop); + stateful->ReleaseSessionLoopOptions(options); + return code; +} + #undef CHECK diff --git a/onnxruntime/test/autoep/test_cuda_stateful_integration.cc b/onnxruntime/test/autoep/test_cuda_stateful_integration.cc new file mode 100644 index 0000000000000..773633510f393 --- /dev/null +++ b/onnxruntime/test/autoep/test_cuda_stateful_integration.cc @@ -0,0 +1,391 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#if defined(ORT_UNIT_TEST_HAS_CUDA_PLUGIN_EP) + +#include +#include +#include +#include + +#include +#include "gtest/gtest.h" +#include "core/session/onnxruntime_cxx_api.h" +#include "core/session/onnxruntime_session_options_config_keys.h" +#include "test/util/include/stateful_execution_test_utils.h" + +extern std::unique_ptr ort_env; +extern "C" int StatefulApiCGpuLoopIntegrationTest(OrtSession*, const OrtValue*, OrtStatefulStepResult**); + +namespace onnxruntime::test { +namespace { + +OrtStatefulProviderSelection GpuCache(int id = 0) { + return {sizeof(OrtStatefulProviderSelection), ORT_STATEFUL_PROVIDER_REQUIRE, "CUDAExecutionProvider", + OrtMemoryInfoDeviceType_GPU, id}; +} + +Ort::StatefulValue Named(const Ort::StatefulStepResult& result, const char* name, + OrtStatefulOutputGroup group = ORT_STATEFUL_SELECTED_OUTPUTS) { + for (const auto& item : result.GetOutputs(group)) { + if (item.name == name) return item.value; + } + ORT_CXX_API_THROW(std::string("Missing stateful output: ") + name, ORT_FAIL); +} + +class CudaCacheApiIntegrationTest : public testing::Test { + protected: + void SetUp() override { + ASSERT_EQ(cudaGetDevice(&previous_device_), cudaSuccess); + ASSERT_EQ(cudaSetDevice(0), cudaSuccess); + } + void TearDown() override { EXPECT_EQ(cudaSetDevice(previous_device_), cudaSuccess); } + + Ort::Session Session(bool gpu = true, cudaStream_t stream = nullptr, + ONNX_NAMESPACE::ModelProto model = stateful::MakeDynamicDecoderModel(), int id = 0) { + Ort::SessionOptions options; + options.SetIntraOpNumThreads(1).SetInterOpNumThreads(1).SetGraphOptimizationLevel(ORT_DISABLE_ALL); + options.AddConfigEntry(kOrtSessionOptionsRecordEpGraphAssignmentInfo, "1"); + if (gpu) { + Ort::ConstEpDevice selected(nullptr); + for (auto device : ort_env->GetEpDevices()) { + if (std::strcmp(device.EpName(), "CUDAExecutionProvider") == 0 && + device.GetMemoryInfo(OrtDeviceMemoryType_DEFAULT).GetDeviceId() == id) selected = device; + } + if (!selected) ORT_CXX_API_THROW("Required CUDA plugin test device is unavailable", ORT_FAIL); + if (stream) { + options.AddConfigEntry("ep.cuda.user_compute_stream", + std::to_string(reinterpret_cast(stream)).c_str()); + } + options.AppendExecutionProvider_V2(*ort_env, {selected}, Ort::KeyValuePairs{}); + options.AddConfigEntry(kOrtSessionOptionsDisableCPUEPFallback, "1"); + } + const auto data = model.SerializeAsString(); + Ort::Session session(*ort_env, data.data(), data.size(), options); + for (const auto& graph : session.GetEpGraphAssignmentInfo()) { + EXPECT_EQ(graph.GetEpName(), gpu ? "CUDAExecutionProvider" : "CPUExecutionProvider"); + } + return session; + } + + Ort::Value Tokens(std::initializer_list tokens = {0, 1}) { + Ort::AllocatorWithDefaultOptions allocator; + const int64_t shape[]{1, static_cast(tokens.size())}; + auto value = Ort::Value::CreateTensor(allocator, shape, 2); + std::copy(tokens.begin(), tokens.end(), value.GetTensorMutableData()); + return value; + } + + static OrtStatefulKVCacheBinding CacheBinding() { + static const int64_t shape[]{1, 1, 0, 1}; + return {{"key", ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT, shape, 4, 2, 0}, + "past_key", + "present_key", + ORT_KV_CACHE_SHARE}; + } + + Ort::SessionLoop Loop(Ort::Session& session, bool gpu = true, int phase = 0, int64_t budget = 3, + bool eos = false) { + const char* finite[]{"token"}; + const char* outputs[]{"logits", "present_key"}; + const int64_t eos_id = 3; + const std::array exits{{{ORT_STATEFUL_END_OF_INPUTS, 0, nullptr, nullptr, 0}, + {ORT_STATEFUL_MAX_GENERATED_TOKENS, budget, + nullptr, nullptr, 0}, + {ORT_STATEFUL_EOS, 0, nullptr, &eos_id, 1}}}; + const auto cache = CacheBinding(); + const OrtStatefulGenerationConfig generation{ + sizeof(generation), phase == 1 ? ORT_STATEFUL_PREFILL : ORT_STATEFUL_DECODE, + "token", "logits", nullptr, nullptr, 0, 0}; + OrtStatefulLoopConfig config{}; + config.struct_size = sizeof(config); + config.finite_input_names = phase == 2 ? nullptr : finite; + config.num_finite_inputs = phase == 2 ? 0 : 1; + config.output_names = outputs; + config.num_outputs = 2; + config.exit_conditions = phase == 2 ? &exits[1] : exits.data(); + config.num_exit_conditions = phase == 0 ? 1 : phase == 2 ? (eos ? 2 : 1) + : (eos ? 3 : 2); + config.kv_caches = &cache; + config.num_kv_caches = 1; + config.generation = phase ? &generation : nullptr; + Ort::SessionLoopOptions options(config); + if (gpu) options.SetProvider(ORT_STATEFUL_FEATURE_CACHE, GpuCache()); + return Ort::SessionLoop(session, options); + } + + Ort::SessionChain GenerationChain(Ort::Session& prefill, Ort::Session& decode, bool gpu, int64_t budget, + bool eos = false) { + auto first = Loop(prefill, gpu, 1, budget, eos), second = Loop(decode, gpu, 2, budget, eos); + const OrtStatefulInputBinding prompt{"token", {ORT_STATEFUL_EXTERNAL_INPUT, "prompt", 0}}; + const OrtStatefulChainComponent stages[]{ + {"prefill", nullptr, first, &prompt, 1, nullptr, 0, nullptr, 0}, + {"decode", nullptr, second, nullptr, 0, nullptr, 0, nullptr, 0}}; + const OrtStatefulChainOutput output{"tokens", 0, nullptr, ORT_STATEFUL_GENERATED_TOKEN_IDS}; + const OrtStatefulChainConfig config{sizeof(config), stages, 2, &output, 1}; + return Ort::SessionChain(Ort::SessionChainOptions(config)); + } + + static std::vector Read(const Ort::StatefulValue& value) { + std::vector data(value.GetTensorTypeAndShapeInfo().GetElementCount()); + if (!data.empty()) { + if (value.GetTensorMemoryInfo().GetDeviceType() == OrtMemoryInfoDeviceType_GPU) { + EXPECT_EQ(cudaMemcpy(data.data(), value.GetTensorRawData(), data.size() * sizeof(float), + cudaMemcpyDeviceToHost), + cudaSuccess); + } else { + std::copy_n(value.GetTensorData(), data.size(), data.data()); + } + } + return data; + } + int previous_device_ = 0; +}; + +TEST_F(CudaCacheApiIntegrationTest, PrefillDecodeBudgetEosCpuParityAndResidentState) { + cudaStream_t a = nullptr, b = nullptr; + ASSERT_EQ(cudaStreamCreateWithFlags(&a, cudaStreamNonBlocking), cudaSuccess); + ASSERT_EQ(cudaStreamCreateWithFlags(&b, cudaStreamNonBlocking), cudaSuccess); + { + auto prefill = Session(true, a), decode = Session(true, b); + auto cpu_prefill = Session(false), cpu_decode = Session(false); + auto prompt = Tokens(); + const OrtStatefulNamedValue feed{"prompt", prompt}; + OrtStatefulInputs inputs{}; + inputs.struct_size = sizeof(inputs); + inputs.inputs = &feed; + inputs.num_inputs = 1; + for (const int64_t budget : {0, 1, 3}) { + for (const bool eos : {false, true}) { + auto chain = GenerationChain(prefill, decode, true, budget, eos); + auto reference = GenerationChain(cpu_prefill, cpu_decode, false, budget, eos); + auto execution = chain.CreateExecution(inputs), cpu = reference.CreateExecution(inputs); + auto independent = chain.CreateExecution(inputs); + std::vector retained; + Ort::StatefulStepResult result(nullptr); + do { + result = execution.Step(); + ASSERT_EQ(cudaStreamQuery(a), cudaSuccess); + ASSERT_EQ(cudaStreamQuery(b), cudaSuccess); + if (result.GetInfo().executed) { + const auto cache = Named(result, "present_key", ORT_STATEFUL_COMPONENT_OUTPUTS); + EXPECT_EQ(cache.GetTensorMemoryInfo().GetDeviceType(), OrtMemoryInfoDeviceType_GPU); + EXPECT_EQ(Named(result, "logits", ORT_STATEFUL_COMPONENT_OUTPUTS) + .GetTensorMemoryInfo() + .GetDeviceType(), + OrtMemoryInfoDeviceType_CPU); + retained.push_back(result); + } + } while (!result.GetInfo().completed); + auto expected = cpu.Run(); + const auto actual_tokens = Named(result, "tokens"); + const auto expected_tokens = Named(expected, "tokens"); + EXPECT_EQ(actual_tokens.GetTensorTypeAndShapeInfo().GetShape(), + expected_tokens.GetTensorTypeAndShapeInfo().GetShape()); + const size_t count = actual_tokens.GetTensorTypeAndShapeInfo().GetElementCount(); + if (count) { + EXPECT_TRUE(std::equal(actual_tokens.GetTensorData(), + actual_tokens.GetTensorData() + count, + expected_tokens.GetTensorData())); + } + auto state = execution.GetState(); + const auto value = Named(state, "key", ORT_STATEFUL_STATE_VALUES); + EXPECT_EQ(value.GetTensorMemoryInfo().GetDeviceType(), OrtMemoryInfoDeviceType_GPU); + EXPECT_EQ(Read(value), Read(Named(cpu.GetState(), "key", ORT_STATEFUL_STATE_VALUES))); + EXPECT_TRUE(Read(Named(independent.GetState(), "key", ORT_STATEFUL_STATE_VALUES)).empty()); + if (budget) { + const auto dispatch = execution.GetDispatchInfo(ORT_STATEFUL_FEATURE_CACHE, ORT_STATEFUL_CACHE_SHARE); + EXPECT_GT(dispatch.provider_calls, 0u); + EXPECT_EQ(dispatch.cpu_calls, 0u); + EXPECT_EQ(dispatch.cpu_delegations, 0u); + const auto sampler = execution.GetDispatchInfo(ORT_STATEFUL_FEATURE_SAMPLER, ORT_STATEFUL_SAMPLE_GREEDY); + EXPECT_GT(sampler.cpu_calls, 0u); + EXPECT_EQ(sampler.provider_calls, 0u); + EXPECT_EQ(execution.GetStagingInfo(0).cpu_output_bytes, 32u); + EXPECT_EQ(execution.GetStagingInfo(0).cpu_snapshot_copy_bytes, 32u); + } + const auto saved = Read(value); + execution.Reset(); + execution.Run(); + EXPECT_EQ(Read(value), saved); + if (!retained.empty()) EXPECT_EQ(Read(Named(retained.front(), "present_key", + ORT_STATEFUL_COMPONENT_OUTPUTS)), + (std::vector{0, 1})); + } + } + } + ASSERT_EQ(cudaStreamDestroy(b), cudaSuccess); + ASSERT_EQ(cudaStreamDestroy(a), cudaSuccess); +} + +TEST_F(CudaCacheApiIntegrationTest, AllSessionLoopChainFormsRetainUnselectedGpuOutputs) { + auto first = Session(), second = Session(); + auto first_loop = Loop(first), second_loop = Loop(second); + auto prompt = Tokens(), next = Tokens({2}); + const auto cache = CacheBinding(); + const OrtStatefulInputBinding initial{"token", {ORT_STATEFUL_EXTERNAL_INPUT, "prompt", 0}}; + const OrtStatefulInputBinding subsequent{"token", {ORT_STATEFUL_EXTERNAL_INPUT, "next", 0}}; + const OrtStatefulNamedValue feeds[]{{"prompt", prompt}, {"next", next}}; + OrtStatefulInputs inputs{}; + inputs.struct_size = sizeof(inputs); + inputs.inputs = feeds; + inputs.num_inputs = 2; + for (const bool a : {false, true}) { + for (const bool b : {false, true}) { + const OrtStatefulChainComponent components[]{ + {"first", a ? nullptr : first, a ? first_loop : nullptr, &initial, 1, nullptr, 0, + a ? nullptr : &cache, a ? 0u : 1u}, + {"second", b ? nullptr : second, b ? second_loop : nullptr, &subsequent, 1, nullptr, 0, + b ? nullptr : &cache, b ? 0u : 1u}}; + const OrtStatefulChainOutput outputs[]{ + {"first_key", 0, "present_key", ORT_STATEFUL_MODEL_OUTPUT}, + {"last_key", 1, "present_key", ORT_STATEFUL_MODEL_OUTPUT}, + {"logits", 1, "logits", ORT_STATEFUL_MODEL_OUTPUT}}; + const OrtStatefulChainConfig config{sizeof(config), components, 2, outputs, 3}; + Ort::SessionChainOptions options(config); + if (!a) options.SetComponentProvider(0, ORT_STATEFUL_FEATURE_CACHE, GpuCache()); + if (!b) options.SetComponentProvider(1, ORT_STATEFUL_FEATURE_CACHE, GpuCache()); + Ort::SessionChain chain(options); + auto execution = chain.CreateExecution(inputs); + auto result = execution.Run(); + EXPECT_EQ(Read(Named(result, "first_key")), (std::vector{0, 1})); + EXPECT_EQ(Read(Named(result, "last_key")), (std::vector{0, 1, 2})); + EXPECT_EQ(Named(result, "last_key").GetTensorMemoryInfo().GetDeviceType(), OrtMemoryInfoDeviceType_GPU); + EXPECT_EQ(execution.GetStagingInfo(0).cpu_snapshot_copy_bytes, 32u); + EXPECT_EQ(execution.GetStagingInfo(1).cpu_snapshot_copy_bytes, 16u); + } + } +} + +TEST_F(CudaCacheApiIntegrationTest, CConsumerResultsSurviveAllExecutionAndSessionOwners) { + auto session = Session(); + auto tokens = Tokens(); + OrtStatefulStepResult* raw = nullptr; + ASSERT_EQ(StatefulApiCGpuLoopIntegrationTest(session, tokens, &raw), 0); + Ort::StatefulStepResult result(raw); + auto cache = Named(result, "present_key"); + result = Ort::StatefulStepResult(nullptr); + session = Ort::Session(nullptr); + EXPECT_EQ(cache.GetTensorMemoryInfo().GetDeviceType(), OrtMemoryInfoDeviceType_GPU); + EXPECT_EQ(Read(cache), (std::vector{0, 1})); + ort_env->UnregisterExecutionProviderLibrary("CUDAExecutionProvider"); + EXPECT_EQ(Read(cache), (std::vector{0, 1})); + cache = Ort::StatefulValue(nullptr); + ort_env->RegisterExecutionProviderLibrary( + "CUDAExecutionProvider", std::filesystem::path(ORT_UNIT_TEST_CUDA_PLUGIN_EP_LIBRARY_PATH).native()); +} + +TEST_F(CudaCacheApiIntegrationTest, LeadingCpuStageGpuComponentAndNamedStateCacheAliases) { + auto seed_model = stateful::MakeModel("leading_cpu"); + auto& seed_graph = *seed_model.mutable_graph(); + auto* seed = seed_graph.add_initializer(); + seed->set_name("seed"); + seed->set_data_type(ONNX_NAMESPACE::TensorProto_DataType_FLOAT); + seed->add_float_data(1.f); + stateful::SetTensorType(*seed_graph.add_output(), "seed_output", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {}); + stateful::AddNode(seed_graph, "Identity", {"seed"}, "seed_output"); + auto identity_model = stateful::MakeModel("cache_alias"); + auto& identity = *identity_model.mutable_graph(); + stateful::SetTensorType(*identity.add_input(), "incoming", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {1, 1, -1, 1}); + stateful::SetTensorType( + *identity.add_output(), "forwarded", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {1, 1, -1, 1}); + stateful::AddNode(identity, "Identity", {"incoming"}, "forwarded"); + auto sum_model = stateful::MakeModel("cache_consumer"); + auto& sum = *sum_model.mutable_graph(); + stateful::SetTensorType(*sum.add_input(), "incoming", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {1, 1, -1, 1}); + stateful::SetTensorType(*sum.add_output(), "summary", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {}); + auto& reduce = stateful::AddNode(sum, "ReduceSum", {"incoming"}, "summary"); + auto* keepdims = reduce.add_attribute(); + keepdims->set_name("keepdims"); + keepdims->set_type(ONNX_NAMESPACE::AttributeProto_AttributeType_INT); + keepdims->set_i(0); + auto leading = Session(false, nullptr, seed_model); + auto producer = Session(); + auto reader = Session(true, nullptr, identity_model); + auto consumer = Session(true, nullptr, sum_model); + auto wrong_reader = Session(false, nullptr, identity_model); + auto prompt = Tokens(); + const OrtStatefulNamedValue feed{"prompt", prompt}; + OrtStatefulInputs inputs{}; + inputs.struct_size = sizeof(inputs); + inputs.inputs = &feed; + inputs.num_inputs = 1; + const auto cache = CacheBinding(); + const OrtStatefulInputBinding prompt_binding{"token", {ORT_STATEFUL_EXTERNAL_INPUT, "prompt", 0}}; + const OrtStatefulStateBinding aliases[]{{"alias", "present_key"}, {"ordinary", "logits"}}; + const OrtStatefulStateBinding write_cache{"key", "forwarded"}; + const OrtStatefulInputBinding consume{"incoming", {ORT_STATEFUL_STATE, "key", 0}}; + const char* finite[]{"incoming"}; + const char* selected[]{"forwarded"}; + const OrtStatefulExitCondition end{ORT_STATEFUL_END_OF_INPUTS, 0, nullptr, nullptr, 0}; + OrtStatefulLoopConfig reader_config{}; + reader_config.struct_size = sizeof(reader_config); + reader_config.finite_input_names = finite; + reader_config.num_finite_inputs = 1; + reader_config.output_names = selected; + reader_config.num_outputs = 1; + reader_config.exit_conditions = &end; + reader_config.num_exit_conditions = 1; + Ort::SessionLoop reader_loop(reader, Ort::SessionLoopOptions(reader_config)); + for (const bool from_state : {false, true}) { + const auto kind = from_state ? ORT_STATEFUL_STATE : ORT_STATEFUL_COMPONENT_OUTPUT; + const OrtStatefulInputBinding route{"incoming", {kind, from_state ? "alias" : "present_key", 1}}; + OrtStatefulChainComponent stages[]{ + {"leading", leading, nullptr, nullptr, 0, nullptr, 0, nullptr, 0}, + {"producer", producer, nullptr, &prompt_binding, 1, aliases, 2, &cache, 1}, + {"reader", from_state ? nullptr : reader, from_state ? reader_loop : nullptr, + &route, 1, &write_cache, 1, nullptr, 0}, + {"consumer", consumer, nullptr, &consume, 1, nullptr, 0, nullptr, 0}}; + const OrtStatefulChainOutput outputs[]{ + {"cache", 2, "forwarded", ORT_STATEFUL_MODEL_OUTPUT}, + {"sum", 3, "summary", ORT_STATEFUL_MODEL_OUTPUT}}; + const OrtStatefulChainConfig config{sizeof(config), stages, 4, outputs, 2}; + Ort::SessionChainOptions options(config); + options.SetComponentProvider(1, ORT_STATEFUL_FEATURE_CACHE, GpuCache()); + Ort::SessionChain chain(options); + auto execution = chain.CreateExecution(inputs); + auto result = execution.Run(); + auto state = execution.GetState(); + EXPECT_EQ(Read(Named(result, "cache")), (std::vector{0, 1})); + EXPECT_EQ(Named(result, "cache").GetTensorMemoryInfo().GetDeviceType(), OrtMemoryInfoDeviceType_GPU); + EXPECT_EQ(Named(result, "sum").GetTensorData()[0], 1.f); + EXPECT_EQ(Named(state, "alias", ORT_STATEFUL_STATE_VALUES).GetTensorMemoryInfo().GetDeviceType(), + OrtMemoryInfoDeviceType_GPU); + EXPECT_EQ(execution.GetStagingInfo(2).cpu_input_request_bytes, 0u); + EXPECT_EQ(execution.GetStagingInfo(2).cpu_output_bytes, 0u); + EXPECT_EQ(execution.GetStagingInfo(2).cpu_snapshot_copy_bytes, 0u); + EXPECT_EQ(execution.GetStagingInfo(3).cpu_input_request_bytes, 0u); + EXPECT_EQ(execution.GetStagingInfo(3).cpu_snapshot_copy_bytes, 4u); + stages[2].session = wrong_reader; + stages[2].loop = nullptr; + Ort::SessionChainOptions invalid(config); + invalid.SetComponentProvider(1, ORT_STATEFUL_FEATURE_CACHE, GpuCache()); + EXPECT_THROW(Ort::SessionChain{invalid}, Ort::Exception); + } +} + +TEST_F(CudaCacheApiIntegrationTest, SharedCachesRejectConflictingDevicesBeforeExecution) { + int count = 0; + ASSERT_EQ(cudaGetDeviceCount(&count), cudaSuccess); + auto first = Session(); + auto second = Session(true, nullptr, stateful::MakeDynamicDecoderModel(), count > 1 ? 1 : 0); + const auto cache = CacheBinding(); + const OrtStatefulInputBinding input{"token", {ORT_STATEFUL_EXTERNAL_INPUT, "prompt", 0}}; + const OrtStatefulChainComponent stages[]{ + {"first", first, nullptr, &input, 1, nullptr, 0, &cache, 1}, + {"second", second, nullptr, &input, 1, nullptr, 0, &cache, 1}}; + const OrtStatefulChainOutput output{"cache", 1, "present_key", ORT_STATEFUL_MODEL_OUTPUT}; + const OrtStatefulChainConfig config{sizeof(config), stages, 2, &output, 1}; + Ort::SessionChainOptions options(config); + options.SetComponentProvider(0, ORT_STATEFUL_FEATURE_CACHE, GpuCache()); + if (count > 1) { + options.SetComponentProvider(1, ORT_STATEFUL_FEATURE_CACHE, GpuCache(1)); + } + EXPECT_THROW(Ort::SessionChain{options}, Ort::Exception); + RecordProperty("cross_device_cache_tested", count > 1); +} + +} // namespace +} // namespace onnxruntime::test + +#endif diff --git a/onnxruntime/test/framework/session_loop_cuda_test.cc b/onnxruntime/test/framework/session_loop_cuda_test.cc new file mode 100644 index 0000000000000..be9c384bf5687 --- /dev/null +++ b/onnxruntime/test/framework/session_loop_cuda_test.cc @@ -0,0 +1,319 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#if defined(ORT_UNIT_TEST_HAS_CUDA_PLUGIN_EP) + +#include +#include +#include + +#include +#include "gtest/gtest.h" +#include "core/framework/tensor.h" +#include "core/graph/graph_viewer.h" +#include "core/providers/cuda/cuda_provider_options.h" +#include "core/session/IOBinding.h" +#include "core/session/inference_session.h" +#include "core/session/session_chain.h" +#include "test/test_environment.h" +#include "test/unittest_util/framework_test_utils.h" +#include "test/util/include/asserts.h" +#include "test/util/include/default_providers.h" +#include "test/util/include/stateful_execution_test_utils.h" + +namespace onnxruntime::test { +namespace { + +class ObservedCacheSession : public InferenceSession { + public: + using InferenceSession::InferenceSession; + std::function before, after; + size_t calls = 0; + Status Run(const RunOptions& options, IOBinding& io) override { + ++calls; + if (before) ORT_RETURN_IF_ERROR(before(io)); + ORT_RETURN_IF_ERROR(InferenceSession::Run(options, io)); + return after ? after(io) : Status::OK(); + } +}; + +class CudaCacheIntegrationTest : public testing::Test { + protected: + void SetUp() override { + ASSERT_EQ(cudaGetDevice(&previous_device_), cudaSuccess); + ASSERT_EQ(cudaSetDevice(0), cudaSuccess); + } + void TearDown() override { EXPECT_EQ(cudaSetDevice(previous_device_), cudaSuccess); } + + std::unique_ptr Session(bool gpu = true, cudaStream_t stream = nullptr, + ONNX_NAMESPACE::ModelProto model = + stateful::MakeDynamicDecoderModel()) { + SessionOptions options; + options.intra_op_param.thread_pool_size = 1; + options.inter_op_param.thread_pool_size = 1; + options.graph_optimization_level = TransformerLevel::Default; + auto session = std::make_unique(options, GetEnvironment()); + if (gpu) { + OrtCUDAProviderOptionsV2 cuda{}; + cuda.user_compute_stream = stream; + cuda.has_user_compute_stream = stream != nullptr; + ORT_THROW_IF_ERROR(session->RegisterExecutionProvider(CudaExecutionProviderWithOptions(&cuda))); + } + const auto data = model.SerializeAsString(); + ORT_THROW_IF_ERROR(session->Load(data.data(), static_cast(data.size()))); + ORT_THROW_IF_ERROR(session->Initialize()); + for (const auto& node : session->GetSessionState().GetGraphViewer().Nodes()) { + EXPECT_EQ(node.GetExecutionProviderType(), gpu ? kCudaExecutionProvider : kCpuExecutionProvider); + } + return session; + } + + OrtValue Tokens(std::initializer_list tokens = {0, 1}) { + OrtValue value; + CreateMLValue(TestCPUExecutionProvider()->CreatePreferredAllocators()[0], + {1, static_cast(tokens.size())}, tokens, &value); + return value; + } + + SessionLoopOptions Options(bool gpu = true, KVCacheUpdateKind kind = KVCacheUpdateKind::Share) { + SessionLoopOptions options; + options.finite_input_names = {"token"}; + options.output_names = {"logits", "present_key"}; + options.exit_conditions = {{LoopExitKind::EndOfInputs}}; + options.kv_caches = {{{"key", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {1, 1, 0, 1}, 2, 0}, + "past_key", + kind == KVCacheUpdateKind::Append ? "new_key" : "present_key", + kind}}; + if (gpu) { + options.cache_provider = {ORT_STATEFUL_PROVIDER_REQUIRE, kCudaExecutionProvider, OrtMemoryInfoDeviceType_GPU, 0}; + } + return options; + } + + std::shared_ptr Loop(InferenceSession& session, SessionLoopOptions options) { + std::shared_ptr loop; + ORT_THROW_IF_ERROR(SessionLoop::Create(session, std::move(options), loop)); + return loop; + } + + std::unique_ptr Execution(InferenceSession& session, SessionLoopOptions options) { + auto loop = Loop(session, std::move(options)); + std::unique_ptr execution; + ORT_THROW_IF_ERROR(loop->CreateExecution({{}, {{{"token", Tokens()}}}}, execution)); + return execution; + } + + static InlinedVector Read(const OrtValue& value) { + const auto& tensor = value.Get(); + InlinedVector data(static_cast(tensor.Shape().Size())); + if (!data.empty()) { + if (tensor.Location().device.Type() == OrtDevice::GPU) { + EXPECT_EQ(cudaMemcpy(data.data(), tensor.DataRaw(), tensor.SizeInBytes(), cudaMemcpyDeviceToHost), cudaSuccess); + } else { + std::copy_n(tensor.Data(), data.size(), data.data()); + } + } + return data; + } + int previous_device_ = 0; +}; + +TEST_F(CudaCacheIntegrationTest, NativeCachePortsResultsStateAndRetainedLifetimes) { + for (const auto kind : {KVCacheUpdateKind::Share, KVCacheUpdateKind::Copy, KVCacheUpdateKind::Append}) { + auto model = stateful::MakeDynamicDecoderModel(); + if (kind == KVCacheUpdateKind::Append) { + stateful::SetTensorType(*model.mutable_graph()->add_output(), "new_key", + ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {1, 1, -1, 1}); + } + auto session = Session(true, nullptr, model); + session->before = [&](IOBinding& io) { + const auto index = std::find(io.GetInputNames().begin(), io.GetInputNames().end(), "past_key") - + io.GetInputNames().begin(); + EXPECT_EQ(io.GetInputs()[index].Get().Location().device.Type(), OrtDevice::GPU); + return Status::OK(); + }; + session->after = [&](IOBinding& io) { + for (size_t i = 0; i < io.GetOutputNames().size(); ++i) { + const auto& name = io.GetOutputNames()[i]; + if (name == (kind == KVCacheUpdateKind::Append ? "new_key" : "present_key")) { + EXPECT_EQ(io.GetOutputs()[i].Get().Location().device.Type(), OrtDevice::GPU); + } + } + return Status::OK(); + }; + auto options = Options(true, kind); + if (kind == KVCacheUpdateKind::Append) options.output_names = {"logits", "new_key"}; + auto execution = Execution(*session, options); + auto independent = Execution(*session, options); + SessionLoopStepResult result; + ASSERT_STATUS_OK(execution->Run(result)); + ASSERT_EQ(result.outputs.size(), 2u); + EXPECT_EQ(result.outputs[0].Get().Location().device.Type(), OrtDevice::CPU); + EXPECT_EQ(result.outputs[1].Get().Location().device.Type(), OrtDevice::GPU); + EXPECT_EQ(Read(result.outputs[1]), (InlinedVector{0.f, 1.f})); + LoopFeeds state; + ASSERT_STATUS_OK(execution->GetState(state)); + EXPECT_EQ(state.at("key").Get().Location().device.Type(), OrtDevice::GPU); + EXPECT_EQ(Read(state.at("key")), (InlinedVector{0.f, 1.f})); + if (kind == KVCacheUpdateKind::Share) { + EXPECT_EQ(state.at("key").Get().DataRaw(), result.outputs[1].Get().DataRaw()); + } else { + EXPECT_NE(state.at("key").Get().DataRaw(), result.outputs[1].Get().DataRaw()); + } + KVCacheView empty; + ASSERT_STATUS_OK(independent->Cache()->Get("key", empty)); + EXPECT_EQ(empty.length, 0); + const auto staging = execution->StagingInfo(); + EXPECT_EQ(staging.cpu_output_bytes, 32u); + EXPECT_EQ(staging.cpu_snapshot_copy_bytes, 32u); + ASSERT_STATUS_OK(execution->Reset()); + ASSERT_STATUS_OK(execution->Run(result)); + execution.reset(); + independent.reset(); + session.reset(); + EXPECT_EQ(Read(state.at("key")), (InlinedVector{0.f, 1.f})); + EXPECT_EQ(Read(result.outputs[1]), (InlinedVector{0.f, 1.f})); + } +} + +TEST_F(CudaCacheIntegrationTest, ModelErrorCancellationAndResetPreservePriorResults) { + auto session = Session(); + auto execution = Execution(*session, Options()); + SessionLoopStepResult result; + ASSERT_STATUS_OK(execution->Run(result)); + const auto retained = result.outputs[1]; + ASSERT_STATUS_OK(execution->Reset()); + session->after = [&](IOBinding&) { + execution->Cancel(); + return Status::OK(); + }; + EXPECT_FALSE(execution->Run(result).IsOK()); + EXPECT_TRUE(execution->RequiresReset()); + ASSERT_STATUS_OK(execution->Reset()); + session->after = [](IOBinding&) { return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "controlled model failure"); }; + EXPECT_FALSE(execution->Run(result).IsOK()); + EXPECT_TRUE(execution->RequiresReset()); + session->after = {}; + ASSERT_STATUS_OK(execution->Reset()); + ASSERT_STATUS_OK(execution->Run(result)); + EXPECT_EQ(Read(retained), (InlinedVector{0.f, 1.f})); +} + +TEST_F(CudaCacheIntegrationTest, DeviceFeedbackValidatesInitialStorageAndPreservesSnapshots) { + auto model = stateful::MakeDynamicDecoderModel(); + auto& graph = *model.mutable_graph(); + stateful::SetTensorType( + *graph.add_input(), "mirror", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {1, 1, -1, 1}); + auto& concat = *graph.mutable_node(graph.node_size() - 1); + concat.add_input("mirror"); + auto session = Session(true, nullptr, model); + auto options = Options(); + options.feedback = {{"mirror", "present_key"}}; + auto loop = Loop(*session, options); + OrtValue empty; + const auto& allocators = session->GetSessionState().GetAllocators(); + const auto allocator = std::find_if(allocators.begin(), allocators.end(), [](const auto& item) { + return item.first.Type() == OrtDevice::GPU && + item.first.MemType() == OrtDevice::MemType::DEFAULT; + })->second; + Tensor::InitOrtValue(DataTypeImpl::GetType(), TensorShape{1, 1, 0, 1}, allocator, empty); + SessionLoopInputs inputs{{{"mirror", empty}}, {{{"token", Tokens()}}, {{"token", Tokens()}}}}; + std::unique_ptr execution; + ASSERT_STATUS_OK(loop->CreateExecution(inputs, execution)); + SessionLoopStepResult first, second; + ASSERT_STATUS_OK(execution->Step(first)); + ASSERT_STATUS_OK(execution->Step(second)); + EXPECT_EQ(Read(first.outputs[1]), (InlinedVector{0, 1})); + EXPECT_EQ(Read(second.outputs[1]), (InlinedVector{0, 1, 0, 1, 0, 1})); + OrtValue host; + Tensor::InitOrtValue(DataTypeImpl::GetType(), TensorShape{1, 1, 0, 1}, + TestCPUExecutionProvider()->CreatePreferredAllocators()[0], host); + inputs.inputs["mirror"] = host; + EXPECT_FALSE(loop->CreateExecution(std::move(inputs), execution).IsOK()); +} + +TEST_F(CudaCacheIntegrationTest, ExplicitGpuPortsRejectHostConsumersAndUnavailableProviders) { + auto cpu = Session(false); + std::shared_ptr loop; + EXPECT_FALSE(SessionLoop::Create(*cpu, Options(), loop).IsOK()); + auto gpu = Session(); + auto options = Options(); + options.cache_provider.device_id = 1; + EXPECT_FALSE(SessionLoop::Create(*gpu, options, loop).IsOK()); + options = Options(); + // Inject a declared CPU consumer; no GPU cache may be silently staged to it. + const OrtDevice cpu_device; + const auto& state = gpu->GetSessionState(); + auto& ports = const_cast(state.GetInputNodeInfoMap()); + const auto* original = ports.at("past_key").front().device; + ports.at("past_key").front().device = &cpu_device; + EXPECT_FALSE(SessionLoop::Create(*gpu, options, loop).IsOK()); + ports.at("past_key").front().device = original; +} + +TEST_F(CudaCacheIntegrationTest, SynchronousTaskStreamWithoutGlobalDeviceWait) { + cudaStream_t compute = nullptr, unrelated = nullptr; + ASSERT_EQ(cudaStreamCreateWithFlags(&compute, cudaStreamNonBlocking), cudaSuccess); + ASSERT_EQ(cudaStreamCreateWithFlags(&unrelated, cudaStreamNonBlocking), cudaSuccess); + { + auto session = Session(true, compute); + auto options = Options(); + options.output_names = {"present_key"}; + auto loop = Loop(*session, options); + const auto& allocators = session->GetSessionState().GetAllocators(); + const auto allocator = std::find_if(allocators.begin(), allocators.end(), [](const auto& item) { + const auto& device = item.first; + return device.Type() == OrtDevice::GPU && device.MemType() == OrtDevice::MemType::DEFAULT; + })->second; + OrtValue tokens; + Tensor::InitOrtValue(DataTypeImpl::GetType(), TensorShape{1, 2}, allocator, tokens); + OrtValue prepared[2]; + for (auto& output : prepared) { + Tensor::InitOrtValue(DataTypeImpl::GetType(), TensorShape{1, 1, 2, 1}, allocator, output); + } + session->before = [&](IOBinding& io) { + return io.BindOutput("present_key", prepared[session->calls - 1]); + }; + const int64_t ids[]{0, 1}; + ASSERT_EQ(cudaMemcpy(tokens.GetMutable()->MutableDataRaw(), ids, sizeof(ids), + cudaMemcpyHostToDevice), + cudaSuccess); + std::unique_ptr execution; + ASSERT_STATUS_OK(loop->CreateExecution({{}, {{{"token", tokens}}}}, execution)); + // Isolate device-cache work from ordinary synchronous host input/output transfers. + std::promise warmed, start; + auto begin = start.get_future(); + auto finished = std::async(std::launch::async, [&] { + SessionLoopStepResult step; + const auto status = execution->Run(step); + warmed.set_value(); + if (!status.IsOK()) return status; + ORT_RETURN_IF_ERROR(execution->Reset()); + step = {}; + begin.wait(); + return execution->Run(step); + }); + warmed.get_future().wait(); + std::promise entered, release; + auto ready = release.get_future(); + std::pair*, std::future*> gate{&entered, &ready}; + ASSERT_EQ(cudaLaunchHostFunc(unrelated, [](void* pointer) { + auto& gate = *static_cast*, std::future*>*>(pointer); + gate.first->set_value(); + gate.second->wait(); }, &gate), cudaSuccess); + entered.get_future().wait(); + start.set_value(); + const auto completion = finished.wait_for(std::chrono::seconds(5)); + release.set_value(); + EXPECT_EQ(completion, std::future_status::ready); + ASSERT_STATUS_OK(finished.get()); + EXPECT_EQ(cudaStreamQuery(compute), cudaSuccess); + } + EXPECT_EQ(cudaStreamDestroy(unrelated), cudaSuccess); + EXPECT_EQ(cudaStreamDestroy(compute), cudaSuccess); +} + +} // namespace +} // namespace onnxruntime::test + +#endif diff --git a/onnxruntime/test/util/include/stateful_execution_test_utils.h b/onnxruntime/test/util/include/stateful_execution_test_utils.h index e8bb858407e71..1e1cba77a97f8 100644 --- a/onnxruntime/test/util/include/stateful_execution_test_utils.h +++ b/onnxruntime/test/util/include/stateful_execution_test_utils.h @@ -135,6 +135,19 @@ inline ONNX_NAMESPACE::ModelProto MakeDenoiserModel() { return model; } +inline ONNX_NAMESPACE::ModelProto MakeDynamicDecoderModel() { + auto model = MakeDecoderModel(); + auto& graph = *model.mutable_graph(); + graph.mutable_input(0)->mutable_type()->mutable_tensor_type()->clear_shape(); + graph.mutable_output(0)->mutable_type()->mutable_tensor_type()->clear_shape(); + SetTensorType(*graph.mutable_input(0), "token", ONNX_NAMESPACE::TensorProto_DataType_INT64, {1, -1}); + SetTensorType(*graph.mutable_output(0), "logits", ONNX_NAMESPACE::TensorProto_DataType_FLOAT, {1, -1, 4}); + for (auto& initializer : *graph.mutable_initializer()) { + if (initializer.name() == "key_shape") initializer.set_int64_data(2, -1); + } + return model; +} + } // namespace stateful } // namespace test } // namespace onnxruntime From 2b6388fe5beac14ef35841370f0196c4f4796960 Mon Sep 17 00:00:00 2001 From: Devang Patel <47577486+devang-ml@users.noreply.github.com> Date: Mon, 19 Oct 2026 16:26:15 +0000 Subject: [PATCH 2/2] Record M08b publication and complete CUDA cache milestone Document PR #21, final CPU/CUDA evidence and130/130 runtime-line coverage. Preserve the explicit memcheck timing-test limitation and leave M09 pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- SCRATCHPAD.md | 11 ++++++++--- SPEC.md | 3 ++- TASKS.md | 22 ++++++++++++---------- 3 files changed, 22 insertions(+), 14 deletions(-) diff --git a/SCRATCHPAD.md b/SCRATCHPAD.md index be1553a53f155..a1e5d8ffa0875 100644 --- a/SCRATCHPAD.md +++ b/SCRATCHPAD.md @@ -13,7 +13,7 @@ Compressed project memory. This is not the execution plan. M01. `SPEC.md` and the execution plan are approved. - M01 is complete and published as ready-for-review PR #11: https://github.com/onnxruntime/ort-state/pull/11 -- No agents launched. M01-M07 are complete; M08 is in progress; M09-M24 remain pending. +- No agents launched. M01-M08 are complete; M09-M24 remain pending. Runtime production code starts with M02. - Future tracking: keep `TASKS.md` checklist current and record compressed discoveries/agent learnings here. @@ -609,7 +609,7 @@ Compressed project memory. This is not the execution plan. - User requested M08 on 2026-10-18. Baseline M07b `03d7476859`; created `state/m08a-cuda-plugin-kv-cache`. M08a implementation/validation is active; - M08b remains unauthorized. + M08b was subsequently authorized and completed; see its section below. - Interview completed: **CUDA plugin EP ONLY**, not legacy/shared CUDA. Prefer CUDA allocation/copy primitives, no custom kernels unless necessary. All new host/glue lines measured100%; if no new device code, device denominator0/N/A @@ -705,7 +705,7 @@ Compressed project memory. This is not the execution plan. - All implementation/verification gates pass. M08a implementation e6ce58508e is published ready for review as PR #20, stacked on #19: https://github.com/onnxruntime/ort-state/pull/20 - M08b has NOT started and still needs user authorization. No PRs merged. + M08b was subsequently authorized and completed in PR #21. No PRs merged. ## M08b @@ -752,3 +752,8 @@ Compressed project memory. This is not the execution plan. GPU cache-only diagnostic correctly retains unavailable GPU logits wording. Full GPU suites and explicit native-memory4/public22 memcheck selectors are separate from the documented instrumented timing limitation. No M09 code. +- M08b implementation249f60acf8 published ready-for-review as PR #21, stacked + on M08a #20: https://github.com/onnxruntime/ort-state/pull/21 + Macro M08 complete; M09 not started. No PRs merged. All reported functional + suites unfiltered where stated, no skips; exact sanitizer selector limitation + is explicitly documented in TASKS/PR and must not be omitted from summaries. diff --git a/SPEC.md b/SPEC.md index 04e3ff2b273c1..781316eddaf84 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1414,7 +1414,8 @@ with only its version number changed. plugin EP only, CUDA copy primitives where sufficient, GPU-resident cache views/results, and an M08a/M08b split, with no additional specification input. The detailed execution plan is approved without changes. M08a is complete in -PR #20. The user authorized the next milestone, M08b, on 2026-10-19. +PR #20. M08b, separately authorized on 2026-10-19, is complete in PR #21. +Both CUDA-plugin-only sub-milestones are independently verified and ready for review. ### H.1 Scope and preserved behavior diff --git a/TASKS.md b/TASKS.md index 263c9b9f7362f..6616e8f128ce7 100644 --- a/TASKS.md +++ b/TASKS.md @@ -1,6 +1,6 @@ # Stateful Session Execution: Execution Plan -**Status:** Approved; M01-M07 complete; M08 specification approved, plan under review. +**Status:** Approved; M01-M08 complete and published for review; M09-M24 pending. M05: [PR #15](https://github.com/onnxruntime/ort-state/pull/15). M05's requested generation-folder layout follow-up is complete in the same PR. **Source:** [Approved specification](SPEC.md), approved 2026-10-15. @@ -17,7 +17,7 @@ M05's requested generation-folder layout follow-up is complete in the same PR. - [x] Review this plan with the user and obtain approval before implementation. - [ ] Complete the milestone stack and its validation gates. -M01-M07 are complete; M08 is in progress and M09-M24 are pending. Check tasks when their +M01-M08 are complete and M09-M24 are pending. Check tasks when their outcome is verified, not when code has merely been written. Record blockers, exact validation commands, results, evidence locations, and PR links in the affected milestone as work proceeds. Update this checklist and `SCRATCHPAD.md` at each meaningful checkpoint. @@ -97,7 +97,7 @@ their functional milestone still requires the CPU/CUDA evidence below. Stack order is the row order. The dependency column records actual semantic prerequisites; the previous branch is also the mechanical Git base. -M01-M07 are **complete**; M08 is **in progress** and M09-M24 are **pending**. +M01-M08 are **complete**; M09-M24 are **pending**. | Done | ID | Intended PR / branch suffix | Semantic prerequisites | | --- | --- | --- | --- | @@ -108,7 +108,7 @@ M01-M07 are **complete**; M08 is **in progress** and M09-M24 are **pending**. | [x] | M05 | CPU greedy sampling and decode iteration / `cpu-greedy` | M04 | | [x] | M06 | Public C API and C++ wrappers / `public-api` | M02, M03, M04, M05 | | [x] | M07 | Built-in/plugin EP capability contracts / `ep-stateful-hooks` | M04, M05, M06 | -| [ ] | M08 | Native CUDA KV management / `cuda-kv-cache` | M07 | +| [x] | M08 | Native CUDA KV management / `cuda-kv-cache` | M07 | | [ ] | M09 | Native CUDA greedy sampling / `cuda-greedy` | M07 | | [ ] | M10 | Standalone package schema and authoring / `package-schema` | M01, M03 | | [ ] | M11 | ORT package-to-chain loading / `package-runtime` | M06, M07, M08, M09, M10 | @@ -1402,9 +1402,9 @@ from this exact stored exclusion list, never from arbitrary unrelated failures. **Outcome:** Device-resident cache lifecycle and prefill/decode handoff. **Status:** SPEC Appendix H approved 2026-10-18 without changes. User scoped M08 to CUDA plugin EP only, copy primitives/GPU cache views and an M08a/M08b split. -The user approved the detailed plan without changes and authorized M08a only. -Implementation uses `state/m08a-cuda-plugin-kv-cache`, based on M07b -`03d7476859`. Pause for authorization before implementing M08b. +The user approved the detailed plan without changes, initially authorized M08a, +and separately authorized M08b on 2026-10-19. Both are complete and published +as independently verified ready-for-review PRs #20 and #21. No PR has been merged. - [x] Implement CUDA allocation/retention/update/growth/reset using ORT allocator, device, and stream abstractions; advertise only supported features. @@ -1609,7 +1609,7 @@ zero-hash dormant maps, with no new-function mismatch; final dumps are checked again against the passing profiles. Exact logs/report are retained as `m08a-*-complete-*`, `m08a-plugin-sanitizer-*`, `m08a-cpu-verified-*`, `m08a-legacy-stateful-tests.log` and `m08a-host-coverage-final.txt` in session files. -M08b remains unimplemented and requires user authorization. +M08b was subsequently authorized and completed in PR #21, as recorded below. #### M08b - Device-resident loop/chain integration @@ -1643,7 +1643,7 @@ across compatible CUDA-plugin Sessions, including distinct streams. - [x] B6: Add end-to-end native/public C/C++ parity, device/copy/dispatch/staging evidence and stream/lifetime tests. Exercise relevant named-state/component handoff paths, not just one preferred cache binding. -- [ ] B7: Measure every introduced core/plugin host path and any device code, +- [x] B7: Measure every introduced core/plugin host path and any device code, rerun baselines/sanitizer/examples, document exact limits/evidence, and publish the ready-for-review stacked PR. Only then complete macro M08. @@ -1683,7 +1683,9 @@ All new core/plugin host lines and any new device lines meet their measured coverage gates. CPU/CUDA-plugin correctness, memory/order/lifetime, documentation and publication pass; mark M08 complete only after both sub-PRs. -**M08b verification complete; publication pending:** +**M08b complete:** Implementation `249f60acf8`, published ready for human review +as [PR #21](https://github.com/onnxruntime/ort-state/pull/21), stacked on M08a +PR #20. This completes macro M08. No PR has been merged; M09 is not started. - Resolve actual Session GPU allocators and validate the declared cache ports' planned device. Managed and declared cache-alias outputs are bound directly to