[GPU] Add UINT2 (2-bit) weight compression support via oneDNN fully_connected by vmadananth · Pull Request #37212 · openvinotoolkit/openvino

vmadananth · 2026-08-03T21:59:30Z

Details:

This PR enables 2-bit unsigned integer (UINT2) compressed weight support for the GPU plugin's oneDNN fully_connected implementation

UINT2 packs 4 weight values per byte, enabling extreme weight compression for LLM inference (e.g., Phi/Orca 2-bit quantized models).

oneDNN fully_connected enablement:

Add data_types::u2 to compressed weight dtype validation and allow bitwidth==2 in assertions
Map cldnn::data_types::u2 → dnnl::memory::data_type::u2 in oneDNN utils
Handle u2 offset calculation (÷4) in get_offset() and weight padding/strides for memory descriptor creation
Support u2 as a decompression zero-point dtype
Add u2 memory dump support in debug_helper (4 values per byte)

Horizontal FC fusion:

Update concat reference kernel to handle u2 in the horizontal FC fusion pass

Tickets:

ticket-id

AI Assistance:

yes

vmadananth · 2026-08-03T22:28:36Z

The oneDNN submodule is temporarily pointing to mdvoretc/u2 for CI and review. Will update it once the changes make it to oneDNN.

p-durandin · 2026-08-04T04:55:10Z

build_jenkins

…n pass

- Add data_types::u2 to compressed_case weight dtype validation - Allow bitwidth==2 in compressed weights assertion - Add u2 to weights padding/strides path for memory descriptor creation - Map cldnn::data_types::u2 to dnnl::memory::data_type::u2 in utils - Handle u2 offset calculation (divide by 4) in get_offset() - Add u2 to decompression zero-point dtype validation - Add GPU_DEBUG_TRACE logging for FC implementation selection - Add U2 memory dump support in debug_helper (4 values per byte)"

vmadananth · 2026-08-13T19:40:15Z

@Lyamin-Roman @v-Golubev : Could you please review this PR (complementary OV support needed for U2 support needed for oneDNN). The oneDNN submodule in this PR is temporarily pointing to mdvoretc/u2 for CI and review. . I will change this eventually once the PR in oneDNN is merged.

vmadananth · 2026-08-14T17:22:34Z

build_jenkins

vmadananth · 2026-08-17T16:38:14Z

build_jenkins

vmadananth · 2026-08-18T05:12:36Z

@p-durandin Could you pls review this?

vmadananth · 2026-08-18T20:34:19Z

build_jenkins

…vation support

# Conflicts: # src/core/reference/src/op/concat.cpp # src/plugins/intel_gpu/src/graph/debug_helper.cpp # src/plugins/intel_gpu/src/graph/impls/onednn/utils.cpp # src/plugins/intel_gpu/thirdparty/onednn_gpu

u2 weight-only-quantized FCs require f16 activations on both the CM and oneDNN kernels; skip per-token INT8 dynamic quantization for them so the u2 path is taken. Carried from openvinotoolkit#37212.

…d of crashing

u2 weight-only-quantized FCs require f16 activations on both the CM and oneDNN kernels; skip per-token INT8 dynamic quantization for them so the u2 path is taken. Carried from openvinotoolkit#37212.

…onger u2 FC tests

…c11d7)

# Conflicts: # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.cpp

vmadananth · 2026-10-22T21:55:54Z

Merged latest master (resolved the conflict with #38934 by dropping get_grouped_mask()) and updated oneDNN to uxlfoundation/oneDNN#6296

…u2 oneDNN tests

isanghao · 2026-10-27T12:14:57Z

@@ -23,22 +23,6 @@ struct fully_connected_onednn : typed_primitive_onednn_impl<fully_connected> {
    static constexpr int COMMON = 0;


random spot)

onednn update requires broad validation. Let's separate that.

Please attach a ticket and upload performance/accuracy result there.

Perf results attached here - https://jira.devtools.intel.com/browse/CVS-198829

# Conflicts: # src/plugins/intel_gpu/thirdparty/onednn_gpu

# Conflicts: # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.cpp # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp # src/plugins/intel_gpu/src/graph/impls/onednn/utils.cpp

isanghao

lgtm

…kit#39316) ### This PR fixes micro SDPA config selection on integrated GPUs, which picked a decode (second-token) config for every prefill chunk. ### Tickets: - [CVS-197466](https://jira.devtools.intel.com/browse/CVS-197466) This PR is one part of a set of TTFT optimizations for 2-bit LLMs on Panther Lake. The other parts are openvinotoolkit#39317 (L2 normalization into RMS), openvinotoolkit#39320 (int8 activations for u2 FullyConnected) and uxlfoundation/oneDNN#6381 (u2 weights with int8 activations). --- ### Changes Xe2/Xe3 reach `choose_config_xehpc` for head size <= 128. There were two problems: - **Check order:** the `is_integrated` check returned `xehpc_q_h128_2nd_integrated`, a second-token config, before `thin_q` was checked. With an f16 KV cache, every prefill chunk ran a decode-shaped kernel. - **Unknown key count:** a dynamic key count (StatelessKV) arrives as `seq = 0`, so a prefill over thousands of keys was sized as a short sequence. After the fix: - the integrated decode config is used for `thin_q` only; - a multi-token call with an unknown key count on an integrated GPU is sized as a long sequence. With a quantized KV cache this selects `xehpc_q_h128`, the config that a known long sequence already gets. Discrete GPUs, PagedAttention and decode (`thin_q`) take the same paths as before. --- ### Before / after LLM (2-bit weights, f16 KV cache), PTL 4Xe, ~8200-token prompt in 512-token chunks, TTFT median of 3 runs: | | TTFT | | |---|---|---| | before | 28.92 s | | | after | 24.21 s | **1.19x** | Greedy output is identical and decode speed is unchanged. Running a 2-bit model needs u2 FullyConnected support (openvinotoolkit#37212), so this was measured on a branch that has it. This change does not depend on it. --- ### Validation - The existing SDPA unit tests give the same results with and without this PR. - End to end on PTL 4Xe: greedy output identical to before on all prompts. > [!NOTE] > AI assistance was used. All changes were reviewed, built, tested and measured on hardware.

…oolkit#39469) ### Details: - **Issue 1.** In file _src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.cpp_ function _get_ifm_dim_idx()_ guessed which axis of the scale/zero-point tensor holds the number of groups. It derived the axis from the rank of the weights, after a heuristic squeeze. The guess is wrong when the scale or zero-point tensor keeps an extra group-size axis of length 1 that the weights don't have. The extra_multiply cases and 3D weights are examples. The code then picked the OC axis and got a bogus ngroups, for example 32 instead of 1. That made group_size = ifm / ngroups meaningless. oneDNN rejected it. - **Fix 1**. _get_ngroups()_ replaces the axis guess. It computes the group count from element counts, scale_total_count / (OC * batch). It uses the IFM and OC positions from the weights layout, which are reliable. create() and load() both use it, for scales and zero points. This fixes the 4 tests. - **Issue 2.** The tests failed with clWaitForEvents ... CL_INVALID_EVENT, later clEnqueueMemFillINTEL ... CL_OUT_OF_RESOURCES. These are only downstream symptoms. dmesg shows a GPU memory fault: Faulted Address, Engine memory CAT error, Engine reset. The engine reset kills the OpenCL queue, so the next command fails. Cause is OOB read in oneDNN. - **Fix 2.** In separate PR [oneDNN-6417](uxlfoundation/oneDNN#6417. This fixes other 23 tests. - **Issue 3.** The new one came from openvinotoolkit#37212. smoke_MatMulCompressedWeights_basic with u2 weights, shape [377,335], f16 activations and constant weights, one per transpose_weights value. The oneDNN rejecting the weights descriptor with bad dimension strides. In the vendored oneDNN, memory_desc_strides_check (the "generalize strides check for subbyte dts" commit) requires every non-unit stride of a sub-byte descriptor to be a whole number of bytes. For u2 that means a multiple of 4 elements. The weights row stride is the innermost weights dimension. Here it is 335 or 377, so oneDNN refuses the descriptor. - **Fix 3.** In _fully_connected_onednn.hpp_, _validate_impl()_ now rejects the oneDNN FC implementation for compressed weights narrower than 8 bits when the innermost weights dimension isn't a whole number of bytes. It then picks another FC implementation, as it already does for unsupported u2 scale grouping. This fixes the 2 tests. ### Tickets: - *[CVS-199121](https://jira.devtools.intel.com/browse/CVS-199121)* ### AI Assistance: - *AI assistance used: yes*

vmadananth requested review from a team as code owners August 3, 2026 21:59

github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin labels Aug 3, 2026

sys-openvino-ci added the ExternalPR External contributor label Aug 3, 2026

vmadananth force-pushed the u2-onednn-support branch from 2ee75c5 to cb5b4fe Compare August 4, 2026 04:46

vmadananth changed the title ~~[GPU] Add UINT2 (2-bit) weight compression support via oneDNN fully_connected and reorder kernels~~ [GPU] Add UINT2 (2-bit) weight compression support via oneDNN fully_connected Aug 4, 2026

vmadananth force-pushed the u2-onednn-support branch from 717fd16 to 604b51f Compare August 6, 2026 05:03

vmadananth added 3 commits August 10, 2026 15:20

Enable compiling dynmodels by adding path to UIN2 Horizontal FC Fusio…

7bc262d

…n pass

Code clean up

4527d82

Kotomi-Du force-pushed the u2-onednn-support branch from 604b51f to 4527d82 Compare August 10, 2026 22:29

Kotomi-Du added 3 commits August 10, 2026 15:34

code clean up

b6887bd

Merge branch 'master' into u2-onednn-support

36bdabf

Merge branch 'master' into u2-onednn-support

24cc25b

Merge branch 'master' into u2-onednn-support

7b3c131

Update oneDNN GPU submodule to latest mdvoretc/u2

63ffaa8

GPU: disable dynamic quantization for u2 weights until oneDNN i8 acti…

b683805

…vation support

Kotomi-Du reviewed Aug 26, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/plugin/transformations_pipeline.cpp Resolved

vmadananth and others added 3 commits September 18, 2026 10:48

Merge remote-tracking branch 'upstream/master' into u2-onednn-support

d080dcd

# Conflicts: # src/core/reference/src/op/concat.cpp # src/plugins/intel_gpu/src/graph/debug_helper.cpp # src/plugins/intel_gpu/src/graph/impls/onednn/utils.cpp # src/plugins/intel_gpu/thirdparty/onednn_gpu

Merge branch 'master' into u2-onednn-support

a926669

Add u2 coverage to FC transformation and primitive tests

10fe4ac

github-actions Bot added the category: transformations OpenVINO Runtime library - Transformations label Oct 19, 2026

Reject non-16-aligned u2 group size in oneDNN FC validate_impl instea…

528239e

…d of crashing

Cyberpunk1210 mentioned this pull request Oct 21, 2026

[GPU] Support u2 weights in the optimized bf_tiled FC kernel #39058

Open

v-Golubev reviewed Oct 22, 2026

View reviewed changes

vmadananth added 3 commits October 22, 2026 11:02

[GPU] Address u2 review: shared group-size helper, one_of checks, str…

8eb39df

…onger u2 FC tests

[GPU] Update oneDNN GPU submodule to uxlfoundation/oneDNN#6296 (07804…

8512af3

…c11d7)

Merge remote-tracking branch 'upstream/master' into u2-onednn-support

a927d2a

# Conflicts: # src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.cpp

v-Golubev reviewed Oct 23, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.cpp Resolved

Comment thread src/plugins/intel_gpu/tests/unit/test_cases/fully_connected_gpu_test.cpp Resolved

[GPU] Use shared weight-rank helper in oneDNN FC load(), add grouped …

2f2c66a

…u2 oneDNN tests

v-Golubev approved these changes Oct 27, 2026

View reviewed changes

isanghao reviewed Oct 27, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/master' into u2-onednn-support

fe75a61

# Conflicts: # src/plugins/intel_gpu/thirdparty/onednn_gpu

This was referenced Oct 29, 2026

[GPU] SDPA micro: use prefill config on integrated GPUs #39316

Merged

[GPU] Int8 activations for u2 FullyConnected on Xe2/Xe3 #39320

Draft

ValentinaKats added the Code Freeze label Oct 30, 2026

geunhwan added the orca_CF label Oct 30, 2026

geunhwan added this to the 2026.5 milestone Oct 30, 2026

isanghao approved these changes Nov 2, 2026

View reviewed changes

isanghao added this pull request to the merge queue Nov 2, 2026

Merged via the queue into openvinotoolkit:master with commit 23ea61f Nov 2, 2026
231 of 233 checks passed

This was referenced Nov 3, 2026

[GPU] Support u2 weights in the batch-1 GEMV fully-connected kernel #39059

Open

[GPU] Support u2 in the fused MoE 3-GEMM path #39062

Open

nshchego mentioned this pull request Nov 4, 2026

[GPU] Fix for invalid number of decompression groups in FC #39469

Merged

		@@ -23,22 +23,6 @@ struct fully_connected_onednn : typed_primitive_onednn_impl<fully_connected> {
		static constexpr int COMMON = 0;

Conversation

vmadananth commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

AI Assistance:

Uh oh!

vmadananth commented Aug 3, 2026

Uh oh!

p-durandin commented Aug 4, 2026

Uh oh!

vmadananth commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vmadananth commented Aug 14, 2026

Uh oh!

vmadananth commented Aug 17, 2026

Uh oh!

vmadananth commented Aug 18, 2026

Uh oh!

vmadananth commented Aug 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vmadananth commented Oct 22, 2026

Uh oh!

Uh oh!

Uh oh!

isanghao Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

vmadananth Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

isanghao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

vmadananth commented Aug 3, 2026 •

edited

Loading

vmadananth commented Aug 13, 2026 •

edited

Loading