[Tianlei Wu (tianleiwu)]

Description

Adds native support for loading ONNX model initializers directly from GGUF v2/v3 weight files (.gguf) via a new gguf_initializers configuration in genai_config.json (supported on model.decoder, model.dflash2, and model.mtp), eliminating the need to duplicate weights into .onnx.data external data files.

When paired with ONNX Runtime's direct external initializer buffer support (microsoft/onnxruntime#33877) and GGUF custom operators (gguf_onnx_ops), the 88-layer NVIDIA-Nemotron-3.5-Bolt-74B-A9B target model (model.onnx = 210 KB) loads all 843 weight tensors (20.5 GiB) directly from NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf with 0 bytes of model.onnx.data, while sharing token_embd.weight and output.weight in GPU memory with the LiLiCorr dflash2 drafter ("share": true), cutting session load time by 2.34x (11.43 s -> 4.89 s) and reducing peak GPU memory by 22.3 GiB (49.4 GiB -> 27.0 GiB).

Stacked on #2761 (tlwu/20261019/nemotron-bolt-runtime).

Summary of Changes

Configuration & GGUF Initializer Loading

File	Change
src/config.h	Add GgufTensorMapping, GgufInitializers, and LoadGgufInitializersConfig declarations for decoder, dflash2, and mtp.
src/config.cpp	Parse gguf_initializers (as an external JSON file path or inline object, supporting map and array tensors syntax) and validate all GGUF paths in ValidateModelPaths and LoadGgufInitializersConfig.
src/dflash2_drafter.cpp	Forward dflash2.gguf_initializers into the drafter's Config in CreateDflash2Config.
src/models/model.h	Add AddGgufInitializers() and gguf_external_values_ storage on Model.
src/models/model.cpp	Implement GgufFileEntry (GGUF v2/v3 binary header parser with bounds/overflow checks and cached mmap), automatic ONNX shape/dtype inference for GGUF block-quantized and unquantized types, zero-copy AddExternalInitializers injection with session.use_external_initializer_file_buffers_directly = "1", and cross-session GPU tensor sharing via AddInitializer when "share": true.

Documentation & Tests

File	Change
docs/GGUFInitializers.md	Document gguf_initializers schema, supported GGUF types, zero-copy mmap loading, and cross-session GPU sharing.
docs/NemotronBolt.md	Document GGUF-backed Bolt packaging with gguf_initializers.json and dflash2_gguf_initializers.json.
test/cpp/engine/nemotron_bolt_tests.cpp	Add NemotronBolt.GgufInitializersConfigParsingAndPathValidation testing config parsing (decoder, dflash2, mtp), path traversal rejection, and synthetic GGUF v3 binary header parsing/validation.

Testing

• ./build/cuda/engine_unit_tests --ep_dir $PACKAGE/runtime/lib --gtest_filter='NemotronBolt*' (9/9 CPU + GPU tests pass)
• MMLU-Pro 200 evaluation on nemotron_bolt_74b_genai_gguf: 120 / 200 = 60.0% (matches llama.cpp GGUF and rc1 120 / 200 = 60.0%)
• 8-prompt speculative decoding benchmark on H200: 432.5 tok/s (fixed_budget_eos_disabled), 4.89 s model load time, 27,043 MiB peak GPU memory

Checklist

• Tests added/updated
• No breaking changes
• Documentation updated

[Justin Chu (justinchuby)]

I wonder onnx should support representing "externally supplied" weights in the models. A simple way to do it maybe allowing the path and offsets to be empty in the external tensor. Then allow runtimes to supply the initializers by name.

[Justin Chu (justinchuby)]

Right now, an onnx model can already index into a gguf file and use it as the external data file. So the above would support tensor lookup by name (instead of the data offsets in current onnx external data format)

[Copilot]

🟡 Changes recommended

Unresolved issues break Windows compilation, leave a Config ABI change unversioned, and allow CUDA and DML to share incompatible cached allocations, risking build failures and undefined behavior.

7 open issues

• Bump the device interface version for the Config layout change · New
• Use the custom path API to open files on Windows · New
• Include the allocator in the GGUF cache key · New
• Validate draft GGUF paths before loading · New
• Build an unambiguous, deduplicated GGUF file search list · New
• Preserve explicitly empty scalar shapes · New
• Apply GGUF mappings to Parakeet and Nemotron Parse decoder sessions · New

What changed in this PR

Adds GGUF-backed initializer loading to GenAI model sessions, avoiding duplicated .onnx.data weights and enabling optional device tensor sharing between target and drafter models.

Changes:

• Adds GGUF mapping configuration for decoder, DFlash2, and MTP models.
• Implements cached GGUF parsing, direct-buffer registration, and shared device allocations.
• Documents packaging and configuration, with tests for parsing, path validation, and synthetic GGUF loading.

File	Description
test/​cpp/​engine/​nemotron_bolt_tests.cpp	Tests configuration, path checks, and synthetic GGUF validation.
src/​models/​model.h	Declares GGUF loading and retains external tensor views.
src/​models/​model.cpp	Implements GGUF parsing, buffer injection, and device sharing.
src/​dflash2_drafter.cpp	Forwards GGUF configuration to the drafter model.
src/​config.h	Defines GGUF mapping configuration structures.
src/​config.cpp	Parses and validates mappings and forwards MTP configuration.
docs/​NemotronBolt.md	Describes GGUF-backed Bolt packaging.
docs/​GGUFInitializers.md	Documents mapping fields and runtime behavior.

_{Review effort: Max}

[Tianlei Wu (tianleiwu)]

Thanks Justin Chu (@justinchuby) — agreed that ONNX external data can already index into a GGUF file when byte offsets are known. This PR adds lookup through the GGUF tensor directory by name and optional device-buffer sharing through the existing ORT initializer APIs; it does not change the ONNX schema. An explicit externally supplied-weight convention would complement this mechanism, but belongs in a separate ONNX/exporter discussion.

Merged origin/main and addressed all seven review items in 78a268d. Local validation: CPU core and CUDA add-on builds passed; the full engine suite passed 1,644 tests with 118 hardware/opt-in skips, including all 14 new GGUF regressions. The fixtures verify real ONNX loading and tensor ranks/values without the original external data file. New Windows and Azure validation runs are now running on the updated, conflict-free PR.

[Copilot]

🟡 Changes recommended

Unresolved critical issues break supported C++17 builds and cause DML models with shared tensors to crash during teardown.

7 open issues

• Reject DML sharing until allocator teardown is safe · New
• Replace C++20 bit_cast calls in C++17 builds · New
• Reject chained mapping config_file references · New
• Use lazy Windows file mapping for GGUF files · New
• Reject Moonshine until decoder mappings are registered · New
• Test zero-copy buffers and cached mapping lifetime · New
• Document zero-copy runtime prerequisites and fallbacks · New

7 resolved since last review

• Include the allocator in the GGUF cache key
• Use the custom path API to open files on Windows
• Bump the device interface version for the Config layout change
• Apply GGUF mappings to Parakeet and Nemotron Parse decoder sessions
• Preserve explicitly empty scalar shapes
• Build an unambiguous, deduplicated GGUF file search list
• Validate draft GGUF paths before loading

_{Review effort: Max}

[Tianlei Wu (tianleiwu)]

/azp run Integration Tests

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Baiju Meswani (baijumeswani)]

Reviewed b66fd5b.

This PR lets an existing ONNX model read its weights from GGUF files by tensor name, instead of keeping another copy in .onnx.data. It can also share GPU weight allocations between the decoder and drafter. It is not a general GGUF model importer: a compatible ONNX graph and any required custom operators are still needed.

The approach has several good parts: read-only file mapping on both Windows and POSIX, model-owned references that keep mappings alive, byte-size and path checks, explicit scalar shapes, and clear rejection of unsupported model paths. The fixes for alternate decoder sessions and the device-interface version bump are also welcome.

I did not find a new heavyweight library dependency. The reader does not link GGML or llama.cpp. With matching Linux Release compiler settings, the two main changed object files grew by about 62 KiB in total sections; this is not an exact final shared-library size measurement. The code is included even when GGUF is unused. External custom-operator libraries have a separate package footprint that I did not measure.

Before approval, I would like the retained host-copy issue addressed, unused unsupported tensors kept from blocking supported mappings, and direct tests for GPU allocation reuse and actual ORT buffer borrowing. The inline comments explain these requests. The schema simplification comment is a suggestion, not an approval requirement.

The zero-copy benefit also depends on an ORT build containing microsoft/onnxruntime#33877. The documentation now explains that correctly. Some build and benchmark dependency paths still select older ORT versions, so benchmark results should record the actual runtime and should not assume the session option alone prevents copying. Name-based lookup and device sharing are the added value here; fixed packages can already reference weight-file byte ranges through ONNX external-data offsets.

Validation: an isolated CPU build passed the focused GGUF/Nemotron tests (27 passed, one GPU/package-dependent test skipped), and the GGUF tests passed a C++17 syntax check. A separate probe mapping only an F32 tensor failed when the same file contained an unused FP8 tensor, with Unsupported GGUF tensor type: 43. I did not independently verify the reported GPU memory or throughput results.

[Baiju Meswani (baijumeswani)]

This upload keeps a full-sized host copy on AMDGPU and WebGPU. Those buffers inherit DeviceBuffer::CopyFromCpu, which allocates a CPU mirror, copies into it, and uploads it. Their destructors are the only place that mirror is freed. The shared cache therefore keeps that extra host allocation alive for as long as the shared weights are used. For example, sharing 10 GiB of weights also retains about 10 GiB of private host memory, in addition to the file mapping and device allocation.

Please upload directly from the mapped CPU view without retaining a staging mirror, using the existing backend copy facilities with the required synchronization. Alternatively, limit sharing to the backends that have been validated. A test that counts host staging allocations would help keep this from coming back.

[Tianlei Wu (tianleiwu)]

Good catch, and confirmed. AMDGPU GpuMemory and WebGPU WebGPUMemory inherited the base DeviceBuffer::CopyFromCpu, which allocates p_cpu_, copies into it, and keeps it until the buffer is destroyed. Fixed in 0a25032: both now override CopyFromCpu to upload straight from the caller's buffer. They use the same CopyTensors call that their CopyCpuToDevice already makes, with a null stream, so the copy completes before the call returns. No mirror is allocated or retained.

The other backends were already fine. CUDA uploads directly, QNN/RyzenAI/OpenVINO buffers are host memory (p_cpu_ == p_device_), and DML sharing stays rejected. The existing shared_initializers path makes the same call, so it no longer keeps a host copy either.

Tests: models now record their shared GGUF device allocations (gguf_shared_device_data_). ExpectUploadedWithoutHostCopy asserts that the shared buffer keeps no separate host copy and contains the GGUF bytes. It runs in the new CUDA sharing test and in WebGpuSharedUploadsKeepNoHostCopy, which runs wherever a WebGPU EP is available. Switching the upload back to CpuSpan() + CopyCpuToDevice() makes the CUDA check fail. I couldn't run the AMDGPU override here because no AMDGPU EP is available; its change is identical to the WebGPU one.

[Baiju Meswani (baijumeswani)]

This checks every tensor's type before we know which tensors the mapping actually uses. A valid but unsupported type in an unused tensor therefore rejects the whole file. I reproduced this with a mapping that selects only an F32 tensor: adding an unused F8_E4M3 tensor makes loading fail with Unsupported GGUF tensor type: 43.

Please separate reading the tensor directory from validating the selected tensors. Unsupported selected types should still fail, but an unused tensor should not prevent a supported mapping from loading. Add a regression for this mixed-type case.

[Tianlei Wu (tianleiwu)]

Agreed. Fixed in 0a25032. Header parsing now only reads the tensor directory: name, GGML dims, type, and offset. It still rejects invalid ranks and duplicate names, since the directory can't be read safely or looked up without those. Type support, block geometry, byte-size overflow, and file bounds are now checked in GgufFileEntry::Resolve, which runs only for tensors that a mapping selects.

A selected unsupported tensor still fails, and the error now names the tensor and file: Unsupported GGUF tensor type 43 for tensor 'unsupported' in <path>. The new OnlySelectedTensorsAreValidated test covers your case. Its file contains an unused type-43 tensor and an unused tensor whose data runs past EOF. The supported F32 mapping loads and runs, and selecting either of the other tensors fails with its specific error.

[Baiju Meswani (baijumeswani)]

The non-CPU sharing test here only checks that DML is rejected. The successful cases use CPU, where share is ignored, so the new GPU allocation-sharing path has no positive regression test.

Please add a GPU test that loads two sessions using the same shared tensor, verifies that only one device allocation is used, releases one model, and runs the survivor again. Also check that different device identities or tensor layouts do not reuse an incompatible allocation. This is needed to protect the main GPU-memory benefit of the change.

[Tianlei Wu (tianleiwu)]

Added CudaSharedTensorsUseOneDeviceAllocation in 0a25032. It loads two CUDA models with the same share: true mapping, like a target and its drafter, and checks:

• Both models bind the same device allocation (SameBufferAs), and that buffer keeps no host copy.
• Both sessions produce correct outputs.
• After one model and its session are released, the surviving session still runs correctly.
• A second registration through a counting device makes no allocations.
• Mapping the same bytes with shape [8, 8], with data_type INT32, or through a device that reports another device id makes a new allocation each time.

GenAI binds one CUDA device per process, so the device-identity case uses a forwarding device that reports a different id; no session runs on that allocation. The test passes on an H200. I also checked that it catches regressions: disabling the cache lookup, or removing the layout and device id from the cache key, makes the corresponding assertions fail.

[Baiju Meswani (baijumeswani)]

This proves that the loader reuses the same mapped address, but not that ORT borrows that address. The session-output checks also pass if ORT makes a private copy. I ran this test successfully with the older ORT SDK available locally, which exercises the copying fallback.

Please keep this lifetime test and add a separate check, using a compatible ORT build, that observes the initializer address inside the session or otherwise detects an extra initializer allocation. That check should fail if the direct-buffer option is removed. It would protect the zero-copy claim rather than only the loader's file cache.

[Tianlei Wu (tianleiwu)]

Kept the lifetime test, and added SessionsReadMappedTensorsWithoutCopying in 0a25032. After creating the session, the test rewrites the tensor's bytes in large.gguf through a writable shared mapping. It then expects the next Run to return the new values, which only happens if the session reads the mapped buffer instead of a private copy.

The test first runs a probe that does the same check directly against ORT, with AddExternalInitializers and the option set explicitly. If the runtime still copies, the test skips instead of failing.

Results:

• It passes with ORT builds that include Support direct external initializer buffers in AddExternalInitializers onnxruntime#33877.
• It fails, returning the original values, when the session.use_external_initializer_file_buffers_directly line is removed from AddGgufInitializers. The lifetime test still passes in that case.
• It skips with ORT 1.31.

[Baiju Meswani (baijumeswani)]

Could we reduce the number of spellings before this becomes public configuration? We currently accept onnx_name/name, gguf_name/gguf_tensor/tensor, file/data_file, and data_type/dtype, as well as object and array forms. Each spelling becomes something we need to preserve and test.

Unless existing packages need these aliases, I suggest choosing one documented spelling for each field. This is a design suggestion, not an approval requirement.

[Tianlei Wu (tianleiwu)]

Agreed. Nothing in this repo or in the documented examples uses the aliases, so in 0a25032 I narrowed the schema before it becomes public. Each field now has one spelling:

• Top level: config_file, files, share, and tensors in object form.
• Each tensors value is either a GGUF tensor name or an object with gguf_name, file, data_type, shape, and share.

I removed onnx_name/name, gguf_tensor/tensor, data_file, dtype, the single-string file, and the array form of tensors. They now fail with Unknown value "...", which MappingFieldsHaveOneSpelling pins down. I kept the "gguf_initializers": "mapping.json" shorthand because it's the documented way to reference an external mapping file. Docs and tests are updated.

[Tianlei Wu (tianleiwu)]

Thanks Baiju Meswani (@baijumeswani). I addressed this review in 0a25032:

• Retained host copy: AMDGPU and WebGPU device buffers now upload straight from the source buffer, so shared tensors no longer keep a host staging copy alive.
• Unused unsupported tensors: type, geometry, and bounds are checked only for tensors a mapping selects. A selected unsupported tensor still fails, and the error names the tensor and file.
• GPU sharing test: added a CUDA test for allocation reuse, survivor runs after a model is released, and cache-key isolation across layouts and device identities.
• Zero-copy test: added a test that fails if the session copies the mapped GGUF data. Its runtime probe skips the test on ORT builds without Support direct external initializer buffers in AddExternalInitializers onnxruntime#33877 instead of assuming that the session option prevents copying.
• Schema: each mapping field now has one spelling, and tensors accepts only the object form.

Validation, on a Linux CUDA 13.0 build running on an H200:

• The GGUF, Nemotron Bolt GGUF, and device-span tests pass: 30 passed. The WebGPU test skipped because no WebGPU EP was available.
• In the full engine suite, 1741 tests passed, 66 skipped, and 4 failed. The same 4 tests fail on the unmodified previous head b66fd5b: three ModelPackage tests, whose setup tries to copy the external-data/ directory from Add memory-backed CPU model loading for sandboxed inference #2749, and EngineBackendFactoryTest.DeviceMemoryBudgetSupportsWindowedDflash.
• The zero-copy test passes with ORT builds that include #33877. It fails when the direct-buffer option is removed, and it skips with ORT 1.31.