Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 54 additions & 3 deletions SCRATCHPAD.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Compressed project memory. This is not the execution plan.
M01. `SPEC.md` and the execution plan are approved.
- M01 is complete and published as ready-for-review PR #34040:
https://github.com/microsoft/onnxruntime/pull/34040
- No agents launched. M01-M07 are complete; M08 is in progress; M09-M24 remain pending.
- No agents launched. M01-M08 are complete; M09-M24 remain pending.
Runtime production code starts with M02.
- Future tracking: keep `TASKS.md` checklist current and record compressed
discoveries/agent learnings here.
Expand Down Expand Up @@ -609,7 +609,7 @@ Compressed project memory. This is not the execution plan.

- User requested M08 on 2026-10-18. Baseline M07b `03d7476859`; created
`stateful-M10-cuda-plugin-kv-cache`. M08a implementation/validation is active;
M08b remains unauthorized.
M08b was subsequently authorized and completed; see its section below.
- Interview completed: **CUDA plugin EP ONLY**, not legacy/shared CUDA. Prefer
CUDA allocation/copy primitives, no custom kernels unless necessary. All new
host/glue lines measured100%; if no new device code, device denominator0/N/A
Expand Down Expand Up @@ -705,4 +705,55 @@ Compressed project memory. This is not the execution plan.
- All implementation/verification gates pass. M08a implementation e6ce58508e is
published ready for review as PR #34049, stacked on #34048:
https://github.com/microsoft/onnxruntime/pull/34049
M08b has NOT started and still needs user authorization. No PRs merged.
M08b was subsequently authorized and completed in PR #34050. No PRs merged.

## M08b

- User approved M08a and authorized the next milestone on 2026-10-19.
Existing approved Appendix H/TASKS B1-B7 apply. Created
stateful-M11-cuda-plugin-cache-integration from M08a6959554e55.
- Scope remains CUDA plugin only; GPU sampling stays M09. Implement all
managed/selected/component/named-state cache surfaces, retain GPU results and
local Session allocators/provider leases. CPU ordinary outputs stay stable.
- Existing IOBinding SynchronizeInputs/Outputs calls provider Sync(), which is
CUDA device-wide. Device-cache integration must rely instead on Run's synchronous
task-stream cleanup; caller external inputs must already be ready.
- Core implementation tracks private per-loop device-cache input/output names,
validates actual SessionState port devices and retains output values with local
provider/allocator leases. Chain performs a second, forward-only cache-edge
pass including named state/cache writers without managed port pairs.
- Arbitrary external/initial-state chain feeds remain host-only; GPU routing is
inferred ONLY from declared internal cache/state/component edges, not dtype/shape.
No public configuration fields or GPU kernels are added.
- New native/public tests cover Share/Copy/Append, stable device snapshots,
actual bound ports, CPU-only leading stage, four Session/Loop combinations,
component+named-state cache aliases, finite/feedback GPU inputs, generation
budgets/EOS against CPU, cross-Session user streams, cancellation/model errors,
C11 lifetimes/unregistration and unrelated-stream nonblocking proof.
- Initial complete GPU tests116 native+179 unfiltered AutoEP passed; native5
+public5 new integration cases all pass. Runtime coverage129/129 plus new
model helper12/12; final matching-artifact profiles are being regenerated.
- Sanitizer public integration+standalone22 cases pass zero errors/leaks;
native memory4 pass zero errors/leaks. The unrelated-stream wall-clock
concurrency assertion times out only under memcheck, inside model Run.
It passes normal execution ten consecutive times with an unrelated stream
held blocked. Same-thread warmup, device-only feeds and prepared outputs were
tested; a CUDA-only memset calibration passes under memcheck, so do NOT claim
universal sanitizer serialization. Document the exact scoped limitation:
native timing test remains in full normal suites/coverage, excluded only from
memcheck's memory-safety selector. No memory error is suppressed.
- Corrected a test-only malformed NodeInfo injection: adding conflicting device
usage via the real AddInput mapping API rejects it before our guard and
dereferences a supplied null node for its error message. Final test instead
temporarily changes/restores the existing planned input device.
- FINAL M08b artifacts: CPU63/139, legacy110/43 and plugin116/179 all passed,
zero skips in the reported suites. Runtime coverage130/130 (loop88+chain42),
helper12/12; no devicefunctions0/N/A. Final profiles coverage/m08b/complete.
GPU cache-only diagnostic correctly retains unavailable GPU logits wording.
Full GPU suites and explicit native-memory4/public22 memcheck selectors are
separate from the documented instrumented timing limitation. No M09 code.
- M08b implementation249f60acf8 published ready-for-review as PR #34050, stacked
on M08a #34049: https://github.com/microsoft/onnxruntime/pull/34050
Macro M08 complete; M09 not started. No PRs merged. All reported functional
suites unfiltered where stated, no skips; exact sanitizer selector limitation
is explicitly documented in TASKS/PR and must not be omitted from summaries.
34 changes: 31 additions & 3 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -1413,8 +1413,9 @@ with only its version number changed.
**Status:** Approved by the user on 2026-10-18 without changes. Scope is CUDA
plugin EP only, CUDA copy primitives where sufficient, GPU-resident cache
views/results, and an M08a/M08b split, with no additional specification input.
The detailed execution plan is approved without changes. M08a implementation is
authorized; M08b requires separate authorization after M08a is complete.
The detailed execution plan is approved without changes. M08a is complete in
PR #34049. M08b, separately authorized on 2026-10-19, is complete in PR #34050.
Both CUDA-plugin-only sub-milestones are independently verified and ready for review.

### H.1 Scope and preserved behavior

Expand Down Expand Up @@ -1550,7 +1551,7 @@ authorized; M08b requires separate authorization after M08a is complete.

### H.8 Standalone CUDA-plugin SDK example (M08a)

M08a enables standalone caches, not device-cache loop/chain integration (M08b).
M08a enables standalone caches; M08b adds device-cache loop/chain integration.
The plugin uses checked, compact per-row CUDA copies, avoiding 2D-copy pitch
limits without new device functions. Operations finish on the default or
caller-configured stream before returning; external inputs must already be
Expand Down Expand Up @@ -1632,3 +1633,30 @@ int main(int argc, char** argv) {
return retained.length == 2 && output == expected ? 0 : 4;
}
```

### H.9 Device-resident loop/chain integration (M08b)

- Select `CUDAExecutionProvider` with GPU memory and an explicit compatible
device (or the provider's unambiguous device). CPU remains the default.
- Resolve cache allocation from the initialized Session's actual GPU allocator,
including chains whose first component is CPU-only. Verify all declared cache
producers/consumers execute on the selected GPU; reject mismatches instead of
inserting an implicit cache transfer.
- Bind managed cache outputs directly to that GPU and retain their allocator/
provider lease. Ordinary outputs and logits keep CPU snapshots; CPU sampling
is reported separately. No GPU sampling is added.
- Follow declared managed-cache output, component-output and named-state edges.
A state output writing a shared cache slot is device-bound even when its
component has no managed past/present pair. Feedback and finite loop inputs
routed from device caches keep their device values and validate placement.
Arbitrary external/initial-state chain inputs retain the host-accessible
contract; this milestone does not infer a cache from a tensor's shape or
silently treat unrelated device inputs as a declared cache edge.
- Synchronous Run cleans up/synchronizes its actual execution streams before
publishing values. The explicit device-cache path does not additionally call
IOBinding's provider-wide Sync, which would wait on unrelated CUDA work.
Independent caller streams/allocators keep their ordinary readiness/lifetime
obligations.
- Retained GPU result/state tensors survive execution/reset and originating
Session release. Definitions/executions still require their caller Sessions
to remain alive. GPU tensor-data getters never perform implicit readback.
Loading
Loading