Cm u2 fc gnk native by vmadananth · Pull Request #39044 · openvinotoolkit/openvino

vmadananth · 2026-10-20T21:36:14Z

Summary

Native GNK path for CM only. When cm::woq_u2_accepts_gnk passes (Xe2/Xe3 + CM JIT, u2 weights, gs = 64, N % 32 == 0, f16 per-group scale, per-group zp, f16 inference precision), the ONNX MatMulNBits reordered-B chain

u2[G,N,gs] → Convert → Subtract(zp) → Multiply(scale) → Transpose{1,0,2} → Reshape[N,K] → MatMul(transpose_b)

stays compressed. It becomes a FullyConnectedCompressed with weights_layout=GNK on the raw [G, N, gs] u2 weights, and the CM kernel runs it with WLAYOUT=0. Otherwise the chain is constant-folded to f16 exactly as on master.

The non-CM permute-to-NK fallback is #38206's scope; whichever lands second reconciles the shared pattern.

Changes

impls/cm/fully_connected_woq_u2_gnk.hpp: cm::woq_u2_accepts_gnk, the single check used by the three places below.
transformations_pipeline.cpp: is_decompression_multiply keeps Multiply → Transpose{1,0,2} → Reshape(3D→2D) → MatMul on constant weights unfolded only when the check passes. Every other chain behaves as on master.
compressed_weights_pattern.hpp: new narrow branch gnk_transpose_reshape_m (order {1,0,2}, rank 3→2, output [N, G·gs]). The existing branches are unchanged.
ConvertFullyConnectedToFullyConnectedCompressed:
- On that branch, re-checks and emits the GNK FC (transpose_b=true, transpose_b_scale/zp=false, scale/zp squeezed to [G, N]). No Transpose or Reshape is left on the weights.
- Asserts if the two checks disagree.
CM u2 FC: validate_impl accepts GNK via the same check. N/K come from fc_weight_dims. WLAYOUT=0 for GNK. Fusion reads the epilogue N from fc_weight_dims.

Dependencies / commit structure

Depends on Gpu fc weights layout #39011 (weights_layout attribute) and [GPU] CM u2 weight-compressed FullyConnected (XMX/DPAS) #38894 (CM u2 FC, NK). Draft until both merge.
76621478fb, 69559d3cc6: from Gpu fc weights layout #39011. Dropped once it merges.
35aeb25a97: [TEMP from #38894] CM u2 FC (NK), squashed. Dropped once [GPU] CM u2 weight-compressed FullyConnected (XMX/DPAS) #38894 merges.
4cea2372bd: this PR.

Validation (Xe2/Xe3 with CM JIT)

--gtest_filter=*woq_u2*: 130 run, 129 passed, 1 skipped (no_cm_device, which runs only on a device without CM). New tests:
- gnk_check.accepts_only_cm_eligible_gnk: each accept/reject condition of the check.
- gnk_pipeline.eligible_chain_becomes_gnk_fc: [80,5120,64] through the full TransformationsPipeline gives one GNK FC on the raw u2 constant.
- gnk_fold.matches_pipeline_without_gnk (gs=32, N=5104, no zp): the graph and constants are identical to a run with the GNK path disabled; no compressed FC or u2 constant remains.
- gnk.selects_cm_and_matches_nk (M=5/40, f16/f32 out): CM is selected without forcing, WLAYOUT=0 is used, and the output is bit-identical to the NK layout of the same weights.
*fully_connected*:*FullyConnected*:*ConvertFCToCompressed*:*ConvertMatMulToFullyConnected*:*dynamic_quantize*:*fc_weights_layout*: 1417 run, 1385 passed, 0 failed, 32 skipped (31 device-specific int4 GEMV + no_cm_device).

…ogue broadcast on CM u2 FC

…multiplying FC weight dimensions' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…acy padded GNK shape

# Conflicts: # src/plugins/intel_gpu/include/intel_gpu/primitives/fully_connected.hpp # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp

…gpu-fc-weights-layout # Conflicts: # src/plugins/intel_gpu/include/intel_gpu/primitives/fully_connected.hpp # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp

# Conflicts: # src/plugins/intel_gpu/include/intel_gpu/primitives/fully_connected.hpp # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp # src/plugins/intel_gpu/src/graph/registry/fully_connected_impls.cpp

# Conflicts: # src/plugins/intel_gpu/include/intel_gpu/op/fc_weights_layout.hpp # src/plugins/intel_gpu/include/intel_gpu/primitives/fully_connected.hpp # src/plugins/intel_gpu/src/graph/fully_connected.cpp # src/plugins/intel_gpu/src/graph/graph_optimizer/prepare_quantization.cpp # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp # src/plugins/intel_gpu/src/plugin/transformations/op/fully_connected.cpp # src/plugins/intel_gpu/tests/unit/test_cases/fully_connected_weights_layout_test.cpp

…f the deferred scale/zp reorder

Kotomi-Du · 2026-10-24T04:56:00Z

please remove onednn change in the PR which I think is not needed

…e the CM u2 FC test scans the FC registry

…u2 FC (clang-tidy readability-braces-around-statements)

…reorder when it already has the target format

…GEMV kernel

…ched programs keep the CAT macros

e-ddykim

After merging #38206, transpose_before_reshape_m precedes and shadows gnk_transpose_reshape_m. This means the native GNK path introduced by this PR is never reached, and the PR's own GNK tests would fail. The pattern overlap/order needs to be reconciled in this PR.

# Conflicts: # src/plugins/intel_gpu/tests/unit/transformations/convert_fc_to_compressed_test.cpp

…conversion path

…-u2-fc-gnk-native # Conflicts: # src/plugins/intel_gpu/src/plugin/transformations_pipeline.cpp

Kotomi-Du · 2026-11-03T05:27:04Z

this PR seems to cover all the changes from both #39011 and #38894. Do you want to convert rest of two into draft then?

Kotomi-Du · 2026-11-03T05:27:24Z

+        const auto G = w[0].get_length();
+        const auto gs = w[2].get_length();
+        return {w[1], w[0] * w[2], G, gs};
+    }


it is possible that weights-rank = 3 not because it is per group quantization, but because it is MOE model as I described in this PR #39268.

In MOE model, it is 3D which has first dimension as experts. Please make sure the refactor of weights layout also take this case into consideration.

…nto cm-u2-fc

…chain

vmadananth · 2026-11-04T20:26:12Z

This PR is merged with #38894

vmadananth added 2 commits October 19, 2026 19:39

[GPU] Add explicit weights_layout attribute to FullyConnected

7662147

[GPU] Add FullyConnected weights_layout tests

69559d3

vmadananth requested review from a team as code owners October 20, 2026 21:36

github-actions Bot added category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations labels Oct 20, 2026

vmadananth mentioned this pull request Oct 20, 2026

[GPU] CM u2 weight-compressed FullyConnected (XMX/DPAS) #38894

Open

vmadananth added 2 commits October 20, 2026 22:51

[GPU] Add explicit weights_layout attribute to FullyConnected

a5b338f

[GPU] Add FullyConnected weights_layout tests

d00da86

vmadananth force-pushed the cm-u2-fc-gnk-native branch from 4cea237 to e4285d4 Compare October 21, 2026 05:56

vmadananth added 2 commits October 20, 2026 23:24

[TEMP from openvinotoolkit#38894] CM u2 FullyConnected (n-major)

1dfb8a3

[GPU] CM u2 FC: consume native GNK weights

9ffdf20

vmadananth force-pushed the cm-u2-fc-gnk-native branch from e4285d4 to 9ffdf20 Compare October 21, 2026 06:25

vmadananth and others added 10 commits October 21, 2026 12:16

[TEMP from openvinotoolkit#38894] Address review: reject runtime epil…

7d9eb61

…ogue broadcast on CM u2 FC

Potential fix for pull request finding 'Prevent signed overflow when …

4bba0ad

…multiplying FC weight dimensions' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

[GPU] Validate FC weights_layout consistency in primitive, handle leg…

0f75596

…acy padded GNK shape

Merge remote-tracking branch 'origin/master' into gpu-fc-weights-layout

39ee236

# Conflicts: # src/plugins/intel_gpu/include/intel_gpu/primitives/fully_connected.hpp # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp

[GPU] Fix legacy GNK shape inference test expectation

9de8053

Merge branch 'master' into gpu-fc-weights-layout

14a3dc1

Merge remote-tracking branch 'vmadananth/gpu-fc-weights-layout' into …

b6c740c

…gpu-fc-weights-layout # Conflicts: # src/plugins/intel_gpu/include/intel_gpu/primitives/fully_connected.hpp # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp

[TEMP from openvinotoolkit#38894] Keep scalar-zero-point u2 FCs out o…

c8088b1

…f the deferred scale/zp reorder

Kotomi-Du reviewed Oct 24, 2026

View reviewed changes

vmadananth added 5 commits October 27, 2026 19:11

[GPU] Accept Parameter u2 GNK weights in FC-to-compressed conversion

d465eb3

[GPU] Restore oneDNN GPU submodule pointer to master's

85ba912

[TEMP from openvinotoolkit#38894] Initialize impl registrations befor…

3b5bafd

…e the CM u2 FC test scans the FC registry

[TEMP from openvinotoolkit#38894] Add braces around statements in CM …

331964a

…u2 FC (clang-tidy readability-braces-around-statements)

[TEMP from openvinotoolkit#38894] Skip the FC decompression scale/zp …

453e942

…reorder when it already has the target format

Prithviraj-R and others added 3 commits October 29, 2026 14:33

[TEMP from openvinotoolkit#38894] Decode optimizations for the CM u2 …

76bcb39

…GEMV kernel

[TEMP from openvinotoolkit#38894] Remove WOQ_U2_GEMV_CAT guard so bat…

0a0e382

…ched programs keep the CAT macros

[GPU] Add braces around statements in the CM u2 GNK header (clang-tidy)

e1da1c1

geunhwan added this to the 2026.5 milestone Oct 30, 2026

geunhwan added the orca_CF label Oct 30, 2026

e-ddykim reviewed Oct 31, 2026

View reviewed changes

vmadananth and others added 7 commits November 1, 2026 20:20

Merge remote-tracking branch 'origin/master' into cm-u2-fc-gnk-native

e7ebb30

# Conflicts: # src/plugins/intel_gpu/tests/unit/transformations/convert_fc_to_compressed_test.cpp

[GPU] Take native GNK u2 weights inside the transpose-before-reshape …

0ae0b39

…conversion path

Merge branch 'master' into cm-u2-fc-gnk-native

bb5d3ba

Merge remote-tracking branch 'origin/master' into cm-u2-fc-gnk-native

6611f8e

Merge remote-tracking branch 'vmadananth/cm-u2-fc-gnk-native' into cm…

138ae85

…-u2-fc-gnk-native # Conflicts: # src/plugins/intel_gpu/src/plugin/transformations_pipeline.cpp

[GPU] Select CM u2 FC for GNK weights only

0321044

[GPU] Test the NK fallback for GNK u2 weights when CM rejects the device

c4042e6

Kotomi-Du reviewed Nov 3, 2026

View reviewed changes

vmadananth added a commit to vmadananth/openvino that referenced this pull request Nov 3, 2026

Merge native GNK weights work from openvinotoolkit#39044 and master i…

ec8ec05

…nto cm-u2-fc

[GPU] Expect no GNK FC, not folded weights, when CM rejects a GNK u2 …

edf0847

…chain

vmadananth marked this pull request as draft November 4, 2026 20:25

Navigation Menu

Repository navigation

Cm u2 fc gnk native - #39044

Cm u2 fc gnk native#39044
vmadananth wants to merge 32 commits into
openvinotoolkit:masterfrom
vmadananth:cm-u2-fc-gnk-native

vmadananth commented Oct 20, 2026 •

edited

Loading

Uh oh!

Kotomi-Du Oct 24, 2026

Uh oh!

e-ddykim left a comment

Uh oh!

Kotomi-Du commented Nov 3, 2026

Uh oh!

Kotomi-Du Nov 3, 2026 •

edited

Loading

Uh oh!

vmadananth commented Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

vmadananth commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Dependencies / commit structure

Validation (Xe2/Xe3 with CM JIT)

Uh oh!

Kotomi-Du Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

e-ddykim left a comment

Choose a reason for hiding this comment

Uh oh!

Kotomi-Du commented Nov 3, 2026

Uh oh!

Kotomi-Du Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vmadananth commented Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vmadananth commented Oct 20, 2026 •

edited

Loading

Kotomi-Du Nov 3, 2026 •

edited

Loading