[Baiju Meswani (baijumeswani)]

Description

Add opt-in swiglu_limit_after_activation to MoE/QMoE for models that clip the gate after SiLU. Existing clipping behavior remains the default.

The change covers supported floating, integer and block-FP8 CUDA paths, rejects unsupported combinations explicitly, and adds numerical/default-behavior tests and operator documentation.

Validation: 18 clipping tests / 55 subtests passed; the full FP8 suite passed 63 tests / 66 subtests with one existing opt-in memory skip. Forced dense, GEMV and WMMA cases were verified with CUDA kernel traces.

Motivation and Context

Ling-3.0-flash-VL requires post-SiLU clipping rather than the existing pre-SiLU order.

Stacked on #33817. This PR adds only the clipping follow-up and its tests.

[Copilot]

🔵 Needs a closer look

The change spans architecture-dependent grouped GEMM and tensor-core GEMV paths, but cross-architecture runtime validation could not be performed in the review environment.

No open issues 🎉

What changed in this PR

Adds opt-in post-SiLU gate clipping to ONNX Runtime’s MoE and QMoE operators for models such as Ling-3.0-flash-VL, while preserving existing clipping behavior by default.

Changes:

• Propagates the clipping option through supported floating-point, integer, and block-FP8 CUDA execution paths.
• Validates attributes and explicitly rejects unsupported CPU, WebGPU, offload, and quantization combinations.
• Adds numerical, default-behavior, and dispatch-specific tests, plus operator documentation.

File	Description
onnxruntime/​test/​python/​transformers/​test_qmoe_fp8_cuda.py	Adds block-FP8 clipping and dispatch-precision tests.
onnxruntime/​test/​python/​transformers/​test_qmoe_cuda.py	Tests integer clipping, defaults, and invalid configurations.
onnxruntime/​test/​python/​transformers/​test_moe_cuda.py	Tests floating-point MoE clipping and default equivalence.
onnxruntime/​core/​graph/​contrib_ops/​contrib_defs.cc	Defines the new MoE and QMoE attribute.
onnxruntime/​contrib_ops/​webgpu/​moe/​moe.h	Rejects post-activation clipping on WebGPU.
onnxruntime/​contrib_ops/​cuda/​moe/​qmoe_offload.cc	Rejects post-activation clipping with offload.
onnxruntime/​contrib_ops/​cuda/​moe/​qmoe_kernels.h	Extends the FP8 activation launcher interface.
onnxruntime/​contrib_ops/​cuda/​moe/​qmoe_kernels.cu	Implements post-activation clipping in the FP8 kernel.
onnxruntime/​contrib_ops/​cuda/​moe/​moe.cc	Passes the option to floating-point MoE execution.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_quantization.cc	Validates quantization support and forwards the option.
onnxruntime/​contrib_ops/​cuda/​moe/​moe_base.h	Parses and validates the clipping attribute.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_int2_prefill.h	Adds the option to shared integer-prefill parameters.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_int2_prefill.cu	Forwards the option to prefill activation.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_gemv.cu	Applies clipping in GEMV activation paths.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_gemv_tc.cuh	Supports clipping in tensor-core INT4 GEMV.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_gemv_device.cuh	Updates the shared device SwiGLU epilogue.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​moe_gemm_activation_kernels.cuh	Applies the option in GEMM activation kernels.
onnxruntime/​contrib_ops/​cuda/​llm/​moe_gemm/​common.h	Adds the default-disabled shared activation flag.
onnxruntime/​contrib_ops/​cpu/​moe/​moe_base_cpu.h	Rejects post-activation clipping on CPU.
docs/​ContribOperators.md	Documents clipping semantics and support restrictions.

_{Review effort: Balanced}

[Tianlei Wu (tianleiwu)]

Reviewed a8a562c and rebuilt the CUDA 13.0 Release runtime with FP8 QMoE enabled on H200. The flag is propagated through the relevant activation paths, but the new tests have two reproducible fixture problems on this supported configuration; please address the inline findings before merging.

Validation:

• All 22 new parent test cases executed; 11 subtests failed in the floating/mixed-width fixtures.
• Full FP8 module: 63 parent tests passed, one opt-in skip, and two failed omitted/zero subtests. All eight forced dispatch/precision cases passed with CUDA kernel-profile assertions enabled.
• 64 supplemental numerical checks passed, covering FP32 MoE, asymmetric INT4/INT8, and block-FP8 nondefault alpha/beta with zero, finite, and infinite limits.
• Changing only the routing logits to one-hot made all four floating MoE/FP8 clipping methods pass without relaxing assertions.

These findings concern newly added test assumptions. In particular, the underlying legacy chunk-order discrepancy is already present in the base commit, not a new kernel regression introduced here.

[Tianlei Wu (tianleiwu)]

Re-reviewed the full diff at 6b4bfcd. Both earlier findings are addressed: one-hot routing removes the low-precision routing/autotuning dependence, and equal chunks make the mixed-width chunked fixtures tolerate the existing Ampere/TMA ordering difference. The original tolerances and exact omitted/zero checks remain intact. No remaining actionable findings.

Validated the current-head test files on H200:

• Focused clipping, attribute-validation, and FP8 dispatch tests: 22 passed, 115 subtests passed.
• Remaining FP8 module tests: 54 passed, 42 subtests passed; one existing opt-in 512-expert memory test skipped.
• All eight forced FP8 dispatch/precision cases passed with ORT_QMOE_REQUIRE_CUDA_PROFILE=1, including the expected CUDA kernel assertions.

Validation used the native CUDA 13.0 / cuDNN 9.23 / SM90 CUDA-plugin build from a8a562c. I verified that the only changes between that build's commit and this head are in the three Python test files; production sources are unchanged. This was a rerun of the updated tests against that build, not a rebuild of the new head.