[Baiju Meswani (baijumeswani)]

Summary

Adds Ling-3.0-flash-VL model export and image inputs to the CUDA dynamic-batching Engine.

• Exports the decoder, vision/projector and CPU embedding components as a complete model package.
• Supports image-and-text prompts and text continuation, including processor-owned prompt expansion and image positions.
• Uses the shared examples/python/engine/vision-qa.py example rather than a model-specific application.

Behavioral scope

• BF16 pixel conversion uses round-to-nearest-even for all Qwen-family processors that produce BF16 pixels.
• An available CUDA provider bridge takes precedence over automatic discovery of an adjacent plugin. An explicit ORT_CUDA_PLUGIN_EP_PATH still selects the caller's plugin.
• CUDA Bailing exports, including bailing_hybrid, default to use_tf32=0. Explicit runtime overrides remain caller-owned; the runtime does not enforce model-specific session policy.
• All VLM types with dynamic batching use the packed decoder path. This does not enable every VLM architecture: packages must satisfy the CPU embedding, packed vision and position-input contracts. Existing phi3v/gemma3 packages without those components are rejected at load.
• Packed-image admission tokenizes the processor's expanded prompt before pixel decoding. A cheap row-capacity check rejects oversized turns before media-cache eviction, while the exact token-count check remains authoritative.
• The CUDA nightly test pin is updated for grouped routing/per-key KDA and post-SiLU expert clipping. Fresh BF16 shared-expert exports clamp in FP32 and do not require the separate CUDA BF16 Less registration repair.
• Qwen3.8 backend details and the sampled-proposal benchmark rationale live in the model-builder guide; the general paged-attention Engine guide remains model-neutral.

Unsupported Ling draft/MTP/DFlash2 combinations and rotary-position-range rules are validated by the owning exporter. They are intentionally not duplicated as model-name checks in the runtime. Generic configured graph bindings are still validated at load.

Validation

• Native runtime, tests and Python bindings rebuilt successfully.
• 22 focused CPU/native bridge, media, processor-whitespace and BF16 conversion tests passed without skips.
• Actual Ling exact-capacity admission test passed across image/whitespace variants.
• Four actual Qwen vision tests passed: retained-media reuse after oversized admission, missing/invalid rotary bindings, abandoned-media reclamation and mixed-geometry host budgeting.
• Six clamp/order tests passed on the same ORT 1.31.0 core and CUDA plugin 0.3.0 used by CI. The previous BF16/FP16 failures were reproduced first. The clamp regression uses the provider's unclipped SiLU as its activation baseline, isolating clipping from provider-specific low-precision SiLU/QuickGelu behavior without changing tolerances or the production graph.
• Changed Python/C++ lint, formatting and whitespace checks passed.

Full-model numerical qualification, fresh full-model export, Windows execution and completed new-head CI are not claimed by this validation.

[Copilot]

🟡 Changes recommended

Critical export defects can select incorrect token embeddings, produce unloadable packages, overwrite decoder files, and break strict shape validation.

6 open issues

• Use rank-three shape for non-paged router projection · New
• Preserve explicit nested tie-word-embeddings policy · New
• Reserve component-exporter-owned filenames during preflight · New
• Reject unsupported context extensions before export · New
• Avoid overwriting shared native reference during comparison · New
• Document the BF16 CUDA Less registration requirement · New

What changed in this PR

Adds Ling-3.0-flash-VL export and image inputs to ONNX Runtime GenAI’s existing CUDA dynamic-batching Engine, reusing the Bailing loader and packed-vision infrastructure. Full-model numerical, memory, and capture qualification remains unfinished.

Changes:

• Export decoder, CPU token lookup, and vision/projector components with Ling-specific rotary, routing, clipping, and checkpoint handling.
• Integrate exact image-prompt retokenization, capacity checks, image positions, and BF16 pixel conversion.
• Add regression coverage, an image-QA example, runtime dependency guidance, and CUDA bridge-selection handling.

File	Description
test/​python/​test_onnxruntime_genai_engine_vision.py	Add Ling admission, capacity, and recovery cases.
test/​python/​models/​test_bailing_builder.py	Cover Ling components, rotary, quantization, routing, and clipping.
test/​python/​cuda/​ort-nightly/​requirements.txt	Pin nightly support for post-SiLU expert clipping.
test/​python/​builder/​test_qwen_vl.py	Verify explicit expert-width initialization.
test/​python/​builder/​test_qmoe_weights.py	Extend builder mocks and checkpoint-policy coverage.
test/​python/​builder/​test_pr2713_regressions.py	Test quantized CPU embeddings with floating outputs.
test/​python/​builder/​test_paged_block_size.py	Cover Ling packed-vision option validation.
test/​python/​builder/​test_builder_expansions.py	Register new portable expansion helpers.
test/​cpp/​engine/​varlen_decoder_io_tests.cpp	Initialize image-position layouts in fixtures.
test/​cpp/​engine/​session_options_tests.cpp	Test bridge preference without plugin auto-registration.
test/​cpp/​engine/​media_inputs_tests.cpp	Cover Ling tokenization, positions, and admission failures.
test/​cpp/​engine/​engine_run_tests.cpp	Add calibration tracing and BF16 pixel-rounding checks.
src/​python/​py/​models/​README.md	Document clipping prerequisites and signed rotary metadata.
src/​python/​py/​models/​loaders/​vision.py	Share checkpoint reading and load Ling vision/projector weights.
src/​python/​py/​models/​loaders/​bailing.py	Support direct query projections and validate untied heads.
src/​python/​py/​models/​docs/​model-specific-options.md	Document clipped-expert runtime requirements.
src/​python/​py/​models/​builders/​qwen3_8/​vision.py	Separate vision compute and boundary dtypes.
src/​python/​py/​models/​builders/​qwen3_5/​text.py	Pass the architecture’s explicit expert width.
src/​python/​py/​models/​builders/​lfm2.py	Pass the architecture’s explicit expert width.
src/​python/​py/​models/​builders/​expansions/​standard.py	Add clipped SwiGLU and packed multimodal rotary helpers.
src/​python/​py/​models/​builders/​base.py	Extend shared embedding, rotary, expert-width, and clipping support.
src/​python/​py/​models/​builders/​bailing.py	Add direct queries, FP32 routing, and per-layer clipping.
src/​python/​py/​models/​builders/​bailing_vl.py	Implement Ling component export and destination validation.
src/​python/​py/​models/​builders/​__init__.py	Expose the Ling composite builder.
src/​python/​py/​models/​builder.py	Register Ling configuration loading and export dispatch.
src/​python/​py/​models/​builder_config.py	Reject TF32 overrides for Bailing routing.
src/​models/​preprocessing/​qwen2_5_vl_image_processor.h	Declare Ling preprocessing and image-layout interfaces.
src/​models/​preprocessing/​qwen2_5_vl_image_processor.cpp	Implement Ling expansion, geometry, and BF16 rounding.
src/​models/​preprocessing/​processor.h	Define reusable image-position layout data.
src/​models/​preprocessing/​multi_modal_processor.cpp	Register the Ling image processor.
src/​models/​model.cpp	Enable target-only Ling Engine model loading.
src/​models/​model_type.h	Classify Ling vision and three-axis position support.
src/​models/​decoder_only.cpp	Validate signed rotary metadata and packed-vision inputs.
src/​ep/​cuda/​session_options.cpp	Prefer an available bridge before automatic plugin discovery.
src/​engine/​request.cpp	Use stored media-position layouts.
src/​engine/​media.h	Store per-image layouts and update position interfaces.
src/​engine/​media.cpp	Add Ling-aware admission, retokenization, and positions.
src/​engine/​engine.cpp	Reserve a generation row during image preflight.
src/​engine/​decoders/​varlen_decoder_io.cpp	Use the updated media-position interface.
examples/​python/​engine/​ling-image-qa.py	Demonstrate streaming Ling image questions through Engine.
docs/​paged_attention_engine.md	Document Ling image contracts and qualification limits.
.github/​workflows/​linux-gpu-x64-build.yml	Rename the regression step to include clipped experts.

_{Review effort: Max}

[Copilot]

🔵 Needs a closer look

The Bailing/Ling use_tf32=0 requirement can be bypassed after export because native CUDA session construction does not validate effective options, allowing reduced-precision router projections that can alter expert selection.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Validate effective CUDA TF32 settings during native session creation

src/​python/​py/​models/​builder_config.py:1622

The Bailing/Ling use_tf32=0 requirement is checked only during export; a later Config overlay or config.set_provider_option("cuda", "use_tf32", "1") bypasses it. Both native CUDA append paths forward that setting without validation, allowing reduced-precision router projections that can change expert selection. Validate the effective decoder CUDA options during native session construction as well.

6 resolved since last review

• Reject unsupported context extensions before export
• Reserve component-exporter-owned filenames during preflight
• Preserve explicit nested tie-word-embeddings policy
• Use rank-three shape for non-paged router projection
• Avoid overwriting shared native reference during comparison
• Document the BF16 CUDA Less registration requirement

_{Review effort: Balanced}

[Tianlei Wu (tianleiwu)]

Thanks for the thorough work here. I reviewed the full diff at be001af5 and cross-checked the Ling-specific behavior against the upstream reference (inclusionAI/TorchTurbo modeling_bailing_moe_v3_vl.py, the HF Ling-3.0-flash-VL processor and config.json). No blocking correctness issues found; the points below are suggestions.

What I verified

• Position layout: ported GetImagePositionLayout + MediaPositions to Python and compared with the reference get_rope_index on ~3.9k random sequences / ~23k chunk windows (odd/even grids, several images, chunk boundaries) → identical.
• mRoPE owner map ([H,W]×12 + [T]×8, pair-interleaved) matches the reference apply_video_rope / apply_rotary_pos_emb_interleave; the clipped SwiGLU order and the 42-entry limit lists match the reference MLP/MoE blocks and config.json.
• TokenizeLingImagePrompt matches the HF processor's _expand_image_tokens (vision_start + N×image_pad + vision_end + "\n", N = T·H·W // 4).
• The BF16 round-to-nearest-even lambda is bit-identical to torch.float32 → bfloat16 on 450k values (ties, ±Inf, overflow, subnormals, NaN).
• The loader/processor contract matches the real model.safetensors.index.json (model.visual.*, linear_proj.{0,2}.*, nothing left over) and preprocessor_config.json (top-level min_pixels/max_pixels).
• Locally on the PR head: test_bailing_builder.py 229 passed / 36 skipped (the skips need ORT with #33817/#33954 or CUDA); test/python/builder + test/python/models 3494 passed (the only failures are No module named 'onnxruntime_genai' in my environment).

Inline suggestions

1. Example: extra newline, qualification-harness chunk sizes (17), default media budget vs. Ling max_pixels, unhandled admission errors.
2. BF16 pixel rounding is a shared-path behavior change without a CI-run test.
3. The CUDA EP precedence change is global and relies on an ORT implementation detail.
4. enable_cuda_graph defaults to "1" although capture is documented as unqualified.
5. BF16 shared-expert Clip ties full BF16 execution to the still-open ORT #34005.
6. README lacks an export recipe; plus two nits (stale CI comment, duplicate include).

Not inline

• Please call out the cross-cutting changes in the PR description: BF16 RNE for all Qwen-family BF16 pixel inputs, bridge-over-plugin precedence for CUDA sessions, use_tf32=0 now emitted for every CUDA Bailing export (including text-only bailing_hybrid), and the nightly pin bump.
• At the time of review windows-cpu-x64-build, windows-cuda-x64-build, windows-webgpu-x64-build and linux-cuda-x64-build were still pending. Since the Ling C++ tests that need a real package are env-gated, those builds are the only coverage of the native changes. ORT #34005 (needed for full BF16 execution) is still open.
• The earlier automated note about enforcing use_tf32=0 at native session creation is reasonable hardening but not blocking in my view.

[Baiju Meswani (baijumeswani)]

Follow-up to reviews 5854563097 and 5854225513 at 269f4d3:

Each inline suggestion now has an individual explanation of the implementation and relevant checks. The dedicated Ling example and application repair overlays are removed in favor of shared vision-qa.py; prompt/position policy lives in the existing selected processors, while Engine orchestration uses their contracts. Model setup lives in the builder README, keeping the paged-attention guide general-purpose.

For clarity, the cross-cutting scope is intentional:

• Shared Qwen-family BF16 pixel conversion now uses nearest-even, with an ordinary model-independent native regression. General CPU Cast and FP16/FLOAT image conversion are unchanged; BF16 golden outputs may move.
• Default CUDA sessions prefer an available classic provider bridge over automatic adjacent-plugin discovery. Explicit plugin/device-filter selection retains precedence, and the plugin Engine CI invocation now explicitly selects the plugin. The probe updates a real option and has available/unavailable controls.
• Every CUDA Bailing decoder export, including text-only bailing_hybrid, emits use_tf32=0. Native model-name-specific enforcement was deliberately not added: the agreed design trusts a correctly built package and leaves explicit caller overrides caller-owned. The model-specific check was also removed from the shared builder validator; required defaults remain in the owning builder.
• The GPU nightly pin is 1.32.0.dev20261110001, covering routing/per-key KDA and post-SiLU clipping. Newly exported BF16 shared-expert clamps use FP32 clamp islands, eliminating their separate BF16 Less prerequisite; old exported graphs are not retroactively changed.

Local matched builds and affected native/builder/real-model checks pass, as detailed in the inline replies. This does not claim a fresh full Ling export, Windows execution, capture qualification or completed remote CI.

[Tianlei Wu (tianleiwu)]

Second pass at dc114baae6 (previous review at be001af5). I re-checked the nine threads from my earlier review against the new head, and each fix is in place:

• Ling example folded into vision-qa.py
• Float32ToBFloat16RoundNearestEven with an unconditional test
• device_id bridge probe, plus explicit ORT_CUDA_PLUGIN_EP_PATH in CI
• Capture off by default for complete vision
• BF16-only FP32 clamps
• README recipe (I verified the CLI flags and config keys exist)
• Workflow comment and duplicate include

The processor refactor (ExpandImagePrompt / GetImagePositionLayout virtuals) looks good. QwenImageProcessor::ExpandImagePrompt is the old expansion verbatim, and Qwen's position extent is unchanged. The merge of main only touched the relocated Qwen3.8 docs, and I confirmed that section is verbatim, including the #2876 text.

Inline (4): one CI-impacting test issue in session_options_tests.cpp, two suggestions (admission ordering in media.cpp, the relaxed rotary_position_ids check), and a stale requirements comment.

Not inline

• Window-cache policy (base.py). 0 < window_size < context_length fixes the failure currently breaking main's Linux/Windows CUDA CI: the phi-4-mini e2e test fails with sliding_window_cache: KV cache capacity (200) must be at least local_window_size (262144). That was also the cause of this PR's CUDA failures at be001af5. The change switches CPU/CUDA/TRT-RTX exports of Phi-3.5-mini and Phi-4-mini (sliding_window 262144 ≥ 131072) to full KV caches. It's correct but unrelated to Ling. Consider a separate PR so main goes green sooner, or at least call it out.
• PR description. It still doesn't list the cross-cutting changes:
  • BF16 RNE for all Qwen-family BF16 pixels
  • Bridge-over-adjacent-plugin precedence
  • use_tf32=0 for every CUDA Bailing export, including bailing_hybrid
  • The nightly pin bump
  • The window-cache policy
  • IsVLM && dynamic_batching routing for every VLM type (phi3v/gemma3 with dynamic batching now fail at load)
  • Admission-time tokenization for Qwen packed vision
  • Relocation of the Qwen3.8 docs
• Removed Ling guards.
  • The export-time use_tf32 != "0" rejection and its test are gone. bailing.py still calls FP32 routing a requirement, so --runtime_config with use_tf32=1 is now accepted silently.
  • The target-only check (no draft/MTP/DFlash2) and the vision_position_range check are also gone from the runtime, so those rules are now enforced only by the exporter.
  • Fine if intended; please confirm.
• Docs. The Qwen3.8-27B MMLU-Pro paragraph (lattice 5.01 vs greedy_path 4.78 vs independent 4.66 tokens per target step) justifies lattice as the default sampled_proposal. It was deleted from paged_attention_engine.md rather than moved to qwen3_8.md.
• CI. All builds on this head are still pending. The 269f4d31/fdc66c79 runs were cancelled, so the native changes haven't had a completed CUDA run since be001af5.

[Baiju Meswani (baijumeswani)]

Follow-up to review 5855923892: all four inline concerns have fix-specific replies and are addressed in the current head. The fixes were published in 9ba4aa7; 73381d2 also adds independent Sigmoid/SiLU dtype assertions to the clamp regression.

For the review-level points:

• The PR description now covers the cross-cutting BF16 pixel conversion, bridge/plugin precedence, Bailing CUDA TF32 defaults, nightly pin, packed-VLM routing implications, admission-time tokenization and documentation relocation.
• The separate Phi window-cache workaround, its local tests and fixture changes were removed in cbfbccf after [Model Builder] Treat a sliding window that covers the whole context as full attention (fixes Phi-4-mini CUDA regression from #2866) #2943 merged to main. The affected window-policy methods and three regression-test files now exactly match main.
• The removed model-specific guards are intentional. The exporter owns unsupported Ling target/draft/MTP/DFlash2 combinations and rotary-position-range rules. CUDA Bailing packages default to use_tf32=0, but explicit caller overrides are not rejected or silently rewritten by model-specific runtime policy. Generic configured graph bindings are still checked at load.
• The Qwen3.8-27B sampled-proposal benchmark paragraph is restored in src/python/py/models/docs/qwen3_8.md, keeping the general Engine guide model-neutral.
• The later BF16/FP16 clamp CI failures were reproduced with the CI pair (ORT 1.31.0 core, CUDA plugin 0.3.0). Their low-precision SiLU/QuickGelu behavior differs from PyTorch. The test now isolates clipping against the provider's unclipped activation, with unchanged tolerances and explicit activation dtype assertions; the production activation graph is unchanged. All six focused clamp/order tests pass on that pair.

The latest integrated build and 133 focused window/GQA/cache, MemoryAssets and Engine bridge/media tests passed. Actual Qwen cache/binding and Ling exact-capacity checks also passed during the fixes. At this update, no current-head checks report failure, but Linux CUDA and integration checks are still running; completed full CI and fresh full-model numerical qualification are not being claimed.