[Tianlei Wu (tianleiwu)]

Description

Extend session.use_external_initializer_file_buffers_directly to AddExternalInitializers so applications can opt into borrowing eligible non-owning CPU weight buffers instead of allocating and copying redundant host storage. Borrowing requires naturally aligned buffers larger than 127 bytes on little-endian hosts; default copying behavior remains unchanged.

Device initializers are always copied into owned CPU staging memory before graph injection, optimization, or shape inference, using the registered execution provider's data-transfer implementation. This PR does not enable device-buffer borrowing or same-device GPU zero-copy reuse.

Motivation and Context

Large memory-mapped models otherwise incur a redundant host allocation and copy when their non-owning weight views are supplied through AddExternalInitializers. Keeping borrowing opt-in preserves the existing guarantee that copied caller buffers can be released after session creation. Borrowed CPU buffers must instead remain unchanged and alive until all sessions using them are released.

Summary of Changes

• Honor the direct-buffer option for eligible CPU tensors supplied through AddExternalInitializers, while preserving existing ownership retention for CPU buffers owned by their OrtValue.
• Stage device tensors on CPU before graph processing, reject graph injection of device pointers, and report missing data-transfer support explicitly.
• Remove the proposed small-device address-tag exception and device-specific allocation-planning changes.
• Document copying versus borrowing and buffer lifetime requirements in the C API, configuration keys, and model-loading guide.
• Add CUDA regressions for scalar Add, Level1 Conv/Add fusion, small and large device buffers, source-buffer overwrite/release, and CPU-planned shape initializers.
• Correct the framework test's native-endian backing storage, add a graph-level misaligned-buffer regression, and verify public API buffer reuse with a 256-byte initializer observed inside a test kernel.

Testing

Validated on Linux/H200 with the existing CUDA 13.0 / cuDNN 9.23 build configuration.

• cmake --build build/cu130/Release --target onnxruntime_test_all onnxruntime_shared_lib_test --parallel 12: passed.
• From build/cu130/Release: ./onnxruntime_test_all --gtest_filter='GraphTest.InjectExternalInitializedTensors*:InferenceSessionTests.*ExternalInitializer*:InferenceSessionTests.ExternalDeviceInitializers*': 11 passed, none skipped.
• From build/cu130/Release: ./onnxruntime_shared_lib_test --gtest_filter='*ExternalInitializer*': 8 passed, none skipped.
• lintrunner --revision HEAD before committing the feedback fixes: passed.
• git diff --check: passed.

The device-buffer aliasing regression failed against the original PR and passes after the fix. Big-endian execution was not validated locally; the original GGUF end-to-end workload was not rerun.

Checklist

• Tests added/updated
• Default CPU copying and caller-buffer lifetime behavior preserved
• Documentation updated

[Copilot]

🟡 Changes recommended

Non-owning device buffers are borrowed without opt-in, risking dangling pointers, while optimizer reads use host memcpy on device memory and can crash session initialization.

4 open issues

• Device buffers are borrowed even when direct use is disabled · New
• ConvAdd fusion reads device initializers with host memcpy · New
• External float buffers are misread on big-endian hosts · New
• Test initializer is too small to exercise buffer borrowing · New

What changed in this PR

This PR extends external-initializer loading to reduce duplicate weight allocations during session initialization, particularly for large memory-mapped models.

Changes:

• Passes the direct-buffer option into graph injection to borrow eligible caller-owned CPU buffers.
• Adds device-buffer handling and skips redundant allocation planning when initializers already reside on the planned device.
• Updates configuration documentation and adds framework and public API tests.

File	Description
onnxruntime/​test/​shared_lib/​test_model_loading.cc	Adds a public API test with direct-buffer mode enabled.
onnxruntime/​test/​framework/​inference_session_test.cc	Checks default copying, opt-in borrowing, and inference results.
onnxruntime/​core/​session/​inference_session.cc	Passes the direct-buffer configuration into graph injection.
onnxruntime/​core/​graph/​graph.cc	Adds borrowing paths for eligible CPU and device buffers.
onnxruntime/​core/​framework/​tensorprotoutils.cc	Uses in-memory references for small nonempty device tensors.
onnxruntime/​core/​framework/​session_state_utils.cc	Skips allocation planning and reuses initializers on the planned device.
include/​onnxruntime/​core/​session/​onnxruntime_session_options_config_keys.h	Expands the configuration comment to cover AddExternalInitializers.
include/​onnxruntime/​core/​graph/​graph.h	Adds the optional direct-buffer parameter.
docs/​Model_Loading_Performance.md	Documents buffer borrowing and device-initializer reuse.

_{Review effort: Balanced}

[Justin Chu (justinchuby)]

Can this be set to true by default?

[Copilot]

🟡 Changes recommended

Critical device-buffer handling can copy CUDA pointers as host memory during initialization and borrow caller-owned buffers without opt-in, risking crashes and use-after-free.

8 open issues

• Document borrowed-buffer lifetime requirements across APIs · New
• Stage device-backed initializers for optimization and shape inference · New
• ConvAdd fusion reads device initializers with host memcpy
• Device buffers are borrowed even when direct use is disabled
• Add CUDA tests for device reuse and cross-device transfers · New
• Test the misaligned-buffer fallback in direct mode · New
• Test initializer is too small to exercise buffer borrowing
• External float buffers are misread on big-endian hosts

_{Review effort: Max}

[Tianlei Wu (tianleiwu)]

Justin Chu (@justinchuby) I kept borrowing opt-in to preserve the existing caller-buffer lifetime contract. Applications can currently release non-owning initializer buffers after session creation; enabling borrowing by default would silently require those buffers to remain unchanged and alive for the lifetime of every consuming session. CPU OrtValues that own their buffers already retain ownership without requiring this option, so safe owned-buffer sharing remains available. The update in dd1ccfa also always stages device initializers into owned CPU storage before graph processing instead of introducing device-buffer borrowing.

[Copilot]

🟡 Changes recommended

Critical memory-safety issues remain: borrowed weights can trigger unaligned-access undefined behavior, and staging non-contiguous device initializers can overrun host buffers.

3 open issues

• Borrowed uint8 weights may be unaligned for MatMulNBits · New
• Non-contiguous device initializer can overrun staging allocation · New
• Rejecting owned buffers contradicts initializer sharing claims · New

8 resolved since last review

• Stage device-backed initializers for optimization and shape inference
• Document borrowed-buffer lifetime requirements across APIs
• ConvAdd fusion reads device initializers with host memcpy
• Device buffers are borrowed even when direct use is disabled
• Test the misaligned-buffer fallback in direct mode
• Add CUDA tests for device reuse and cross-device transfers
• Test initializer is too small to exercise buffer borrowing
• External float buffers are misread on big-endian hosts

_{Review effort: Max}

[Rui Ren (rui-ren)]

LGTM, thanks!