Repository navigation
[GPU] Decode each GEMM of the fused MoE 3-GEMM with its own weight dtype - #39061
Open
Cyberpunk1210 wants to merge 5 commits into
Open
Cyberpunk1210 wants to merge 5 commits into
Cyberpunk1210 wants to merge 5 commits into
Conversation
The moe_3gemm decode kernels took one weight dtype from the gate GEMM (WEIGHT_COMPRESSEION_DT) and derived from it the GEMV variant, the per-expert weight stride and the per-expert zp stride for all three GEMMs. A layer whose GEMMs differ -- NNCF mixed precision produces e.g. gate/up u8 with down u4 -- therefore decoded at least one GEMM with the wrong dtype and the wrong stride: wrong output, no error. Emit GATE/UP/DOWN_WEIGHT_DT and use them for the GEMV dispatch, the expert weight stride and the zp stride of each GEMM. The activation interleave stays keyed on one dtype per kernel: gate and up share one SLM copy whose layout is picked from the gate dtype, so they must still agree on whether they are u4; down stages its own copy and is free. The prefill paths already took each GEMM's dtype. Also fix the host-side GEMV check: FAKE_GROUP_SIZE in the kernel is the minimum of the gate/up and the down group size, but the assert only looked at gate/up, so a smaller down group slipped through to an ELEMS_PER_LANE with no kernel branch. Tests: moe_3gemm_compressed_gpu_mixed_dtype, gate/up u8 + down u4 and gate/up u4 + down u8, for single-token decode, batched GEMV and just above the batched-GEMV threshold, against a reference built from the exact dequantized weights. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>
This was referenced Oct 21, 2026
Comment-only cleanup: keep the why, drop restated code and history. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>
Contributor
|
build_jenkins |
…e WEIGHT_COMPRESSEION_DT validate_impl now checks WEIGHT/SCALE/ZP of gate, up and down, plus the cross-input constraints the kernel already relies on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>
Prefill runs the shared expert through oneDNN with per-projection dtypes; validate_impl cannot tell decode from prefill, so the check rejected valid models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>
Lyamin-Roman
approved these changes
Oct 29, 2026
src/plugins/intel_gpu/src/graph/impls/ocl_v2/moe/moe_3gemm_swiglu_opt.hpp: upstream added u3 to the weight type list and our branch replaced the single weight/scale/zp checks with a per-GEMM loop; kept the loop, added u3 to supported_wei_type with upstream's comment, and exempted u3 (only) from the "zp bit width == weight bit width" rule: that rule is the batched GEMV's MOE_EXPERT_ZP_BYTES stride, u3 never runs the batched GEMV (decode_impl=grouped_gemm, OneDNN) and its zp is u8. add_common_consts / moe_weight_code are only reached through the GEMV stages, which are not added for u3, so no throw or wrong code applies. The grouped GEMM zp dtype is now taken per GEMM instead of from ZP_0 (a mixed u4/u8 layer has different zp dtypes). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>
Contributor
|
@Cyberpunk1210 please resolve the conflicts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
Part 5 of the split of #37335 (series: #39057 … #39063). No dependencies, and no u2: this is a fix
for existing u4/u8 models that #39062 builds on.
The
moe_3gemmdecode kernels took one weight dtype from the gate GEMM (WEIGHT_COMPRESSEION_DT)and derived from it the GEMV variant, the per-expert weight stride and the per-expert zp stride for
all three GEMMs. A layer whose GEMMs differ — NNCF mixed precision produces e.g. gate/up u8 with down
u4 — decoded at least one GEMM with the wrong dtype and stride: wrong output, no error.
GATE/UP/DOWN_WEIGHT_DTare emitted and used for the GEMV dispatch, the expert weight stride andthe zp stride of each GEMM.
layout is picked from the gate dtype, so they must still agree on whether they are u4; down stages
its own copy. (The fusion gate in [Transformations] Support u2 expert weights in MoE op fusion #39060 / [GPU] Support u2 in the fused MoE 3-GEMM path #39062 enforces this.)
FAKE_GROUP_SIZEin the kernel is the minimum of the gate/up and the downgroup size, but the assert only looked at gate/up, so a smaller down group slipped through to an
ELEMS_PER_LANEwith no kernel branch.Tests:
moe_3gemm_compressed_gpu_mixed_dtype— gate/up u8 + down u4 and gate/up u4 + down u8, forsingle-token decode, batched GEMV and just above the batched-GEMV threshold, against a reference
built from the exact dequantized weights.
Known:
smoke/moe_3gemm_compressed_gpu_random.moe_accuracy_test_random/10(u4, softmax) fails onmaster itself and is unchanged by this PR.
Tickets:
AI Assistance:
close supervision.
*moe*unit tests (192/193,the one failure being the pre-existing case above), including all 6 mixed-dtype cases.
🤖 Generated with Claude Code