[GPU] Support u2 weights in the optimized bf_tiled FC kernel by Cyberpunk1210 · Pull Request #39058 · openvinotoolkit/openvino

Cyberpunk1210 · 2026-10-21T07:34:14Z

Details:

Part 2 of the split of #37335 (series: #39057 … #39063). No dependencies. The batch-1 GEMV
kernel follows in #39059.

#36170 enabled u2 FC on the reference kernel fully_connected_gpu_bfyx_ref; #37212 adds u2 FC via
oneDNN for XMX platforms. This PR enables u2 on fully_connected_gpu_bf_tiled, so a u2 FC that does
not take the oneDNN path is not left on the reference kernel. That covers two cases: devices
without XMX (or with oneDNN disabled), and u2 models with a fractional zero point. oneDNN's
weight zero point must be an integer type (i4/u4/u8/i8), but the symmetric 4-level grid
{-1.5, -0.5, +0.5, +1.5}·scale used by some 2-bit models is (q - 1.5)·scale, i.e. a scalar zp of
1.5. #37212 already declines such a layer at implementation selection (its zero-point dtype check
admits i4/u4/u8/i8 only), so the layer falls to the OCL kernels; with this PR that is bf_tiled, which
takes the zero point in accumulator precision and decodes the grid exactly, instead of the reference
kernel. (#37212's selection code notes that "u2 has no OCL fallback"; after this PR it has one.)

Relation to #38460 (TernOCL): ternocl_int2 takes u2 FCs with a scalar zero point of exactly 1
(ternary {-1, 0, +1}) on Xe2+ with DPAS, and is selected ahead of the other implementations. Every
other u2 FC — e.g. NNCF INT2_SYM (scalar zp 2), grouped zero points, fractional zero points, or
devices without DPAS — is not taken by it and lands here (and on the GEMV kernel of #39059 at
batch 1).

bf_tiled: non-dyn-quant tilings only, IFM a multiple of 4, no swiglu fusion, TILE_IFM 2 /
TILE_K 4 (one weight byte per lane), os_iyx_osv16 and os_is_yx_osv64_isv2 layouts.
reorder_weights_int4: u2 OIYX padding, IOYX->OIYX transpose and os_iyx_osv16 repack;
matching osv16 index in fetch_weights.cl.
prepare_padding aligns u2 weights to 4 values per byte; fake alignment and the
async-compilation rule treat u2 like u4.
kernel_selector toCLType maps u2 to uchar (same two lines as [GPU] Support u2 compressed dictionaries in GatherCompressed #39057).
DynamicQuantizeFullyConnected declines u2. [GPU] Add UINT2 (2-bit) weight compression support via oneDNN fully_connected #37212 adds the same rule in the pipeline
callback; whichever lands second should drop one of them.

Tests: bf_tiled u2 FC over batch / IFM / OFM sizes and scale group sizes, with and without a grouped
zero point, plus an os_is_yx_osv64_isv2 case; u2 reorder_data cases.

Tickets:

N/A

AI Assistance:

AI assistance used: yes
Claude (Claude Code) was used to split [GPU][CPU] Support u2 (INT2) weight decompression: optimized FC kernel and fused MoE 3-GEMM #37335 into this series and to draft the change under
close supervision.
Human validation performed: Windows Release build on Arc B390; GPU
*fully_connected*:*reorder* unit tests (1575 passed, 32 skipped, 0 failed).

🤖 Generated with Claude Code

PR openvinotoolkit#36170 enabled u2 on FullyConnected_bfyx_Ref, the one-work-item-per-output reference kernel. This enables it on bf_tiled, the optimized OCL path, so a u2 FC on a device that does not take the oneDNN path (no XMX, or oneDNN disabled) is not left on the reference kernel. u2 is not dynamic-quantizable, so it takes the non-dyn-quant tilings, with constraints that all come from 4 values sharing one byte: - Validate: IFM must be a multiple of 4, and swiglu fusion is excluded (it needs os_is_yx_osv32_isv2, which cannot be produced for u2). - VerifyTuneParams: only TILE_IFM 2 / TILE_K 4, whose per-lane weight staging reads exactly one byte, matching both u2 packings; SLM only when IFM divides evenly so no partial byte is left over. - Jit: tile_k_ofm_packed /= 4; FILTER_ELEMENTS_PER_BYTE is a jit constant, not a .cl #define, because kernels are concatenated into one batched program and only jit constants get an automatic #undef. - Layouts: os_iyx_osv16 and os_is_yx_osv64_isv2. Supporting changes: - reorder_weights_int4 gains u2 variants of its three layout paths (OIYX padding, IOYX->OIYX transpose, os_iyx_osv16 repack), gathering 4 values per output byte and zero-filling the padded tail; fetch_weights.cl gets the matching osv16 index for 4-values-per-byte packing. - prepare_padding aligns u2 weights to 4 values per byte instead of 2; fake alignment and the async-compilation rule treat u2 like u4. - kernel_selector toCLType maps u2 to uchar, like u4. - DynamicQuantizeFullyConnected declines u2: no kernel implements i8 activations x u2 weights. openvinotoolkit#37212 adds the same rule in the pipeline callback; whichever lands second should drop one of them. Tests: bf_tiled u2 FC over batch / IFM / OFM sizes and scale group sizes, with and without a grouped zero point, plus an os_is_yx_osv64_isv2 case; u2 reorder_data cases (identity, u2->f16, f16->u2 clamping and partial bytes). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com> # Conflicts: # src/plugins/intel_gpu/tests/unit/test_cases/fully_connected_gpu_test.cpp

Merges current master (via the updated openvinotoolkit#39058/openvinotoolkit#39059 branches) so the combined branch stays mergeable; content is still exactly the seven split PRs. Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

p-durandin · 2026-10-21T13:28:52Z

build_jenkins

Comment-only cleanup in the u2 bf_tiled / reorder_weights_int4 changes and their tests (keep the why, drop restated code and history), plus the toCLType case order requested in the GatherCompressed review. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

p-durandin · 2026-10-22T05:37:21Z

build_jenkins

Return the conditions directly (clang-tidy readability-simplify-boolean-expr). No behaviour change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

p-durandin · 2026-10-22T09:32:49Z

build_jenkins

michal-miotk · 2026-10-22T22:05:58Z

+    // #undef'd between kernels of a batched program.
+    jit.AddConstant(MakeJitConstant("FILTER_ELEMENTS_PER_BYTE", weights_dt == WeightsType::UINT2 ? 4 : 2));
+    if (weights_dt == WeightsType::UINT2) {
+        jit.AddConstant(MakeJitConstant("COMPRESSED_WEIGHTS_INT4", 1));


nit: probably better not mix this name COMPRESSED_WEIGHTS_INT4 into u2 path

Done: bf_tiled now uses its own COMPRESSED_WEIGHTS_SUB_BYTE (i4/u4/u2) for the shared packed-weight paths, and u2 no longer sets COMPRESSED_WEIGHTS_INT4.

u2 no longer sets COMPRESSED_WEIGHTS_INT4; bf_tiled defines its own macro for i4/u4/u2 and its kernels switch to it. Rename only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

p-durandin · 2026-10-23T05:23:09Z

build_jenkins

michal-miotk · 2026-10-26T10:32:47Z

+        const uint h = output_index / OUTPUT_INNERMOST_NUM;
+        if (w < INPUT0_INNERMOST_NUM) {
+            const uint in_elem_offset = h * INPUT0_INNERMOST_NUM + w;
+            out |= ((input[in_elem_offset / 4] >> ((in_elem_offset % 4) * 2)) & 0x3) << (j * 2);


nit: please split out | expression to more vars

Done: split into in_shift / val locals.

…order expression Move the "no FC kernel takes i8 activations with u2 weights" check out of the DynamicQuantizeFullyConnected pass body into its pass_config callback, so the pass itself matches upstream and DynamicQuantizeFullyConnectedTests (u2 case) passes; plugin behaviour is unchanged. Also split the packed-u2 reorder expression in reorder_weights_int4.cl into named locals. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

…ts_int4 Use in_shift/val locals instead of the compound bit expression in the IOYX->OIYX branch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

v-Golubev · 2026-10-27T08:43:04Z

build_jenkins

michal-miotk · 2026-10-27T15:06:02Z

            auto weights_layout_dt = orig_impl_param.weights_layout.value().data_type;
-            auto is_4bit = weights_layout_dt == data_types::i4 || weights_layout_dt == data_types::u4;
+            auto is_4bit = weights_layout_dt == data_types::i4 || weights_layout_dt == data_types::u4 ||
+                           weights_layout_dt == data_types::u2;


nit: please ad is_2bit

Done: added is_2bit for u2; is_4bit is back to i4/u4.

michal-miotk · 2026-10-27T15:22:46Z

                #elif FILTER_LAYOUT_OS_IS_YX_OSV64_ISV2
                SLM_FILTER_PACKED_VEC wei_packed0 = BLOCK_READN(FILTER_TYPE, FILTER_ACTUAL_LOAD_BLOCK_SIZE, weights, weights_idx);
+                #if COMPRESSED_WEIGHTS_UINT2
+                SLM_FILTER_PACKED_VEC wei_packed1 = BLOCK_READN(FILTER_TYPE, FILTER_ACTUAL_LOAD_BLOCK_SIZE, weights, (weights_idx + (FILTER_LOAD_BLOCK_SIZE * SIMD / 2)));


please change this line to: SLM_FILTER_PACKED_VEC wei_packed1 = BLOCK_READN(FILTER_TYPE, FILTER_ACTUAL_LOAD_BLOCK_SIZE, weights, (weights_idx + U2_OSV64_ISV2_LINE_SIZE_BYTES)); where U2_OSV64_ISV2_LINE_SIZE_BYTES is 32

Done: the read now uses U2_OSV64_ISV2_LINE_SIZE_BYTES (32), defined at the top of the kernel.

michal-miotk · 2026-10-27T16:21:55Z

please add SLM test

…ests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

Cyberpunk1210 · 2026-10-28T01:44:37Z

Added u2 bf_tiled SLM tests: fully_connected_gpu_u2_bf_tiled_slm.osv16 (batch 256 and 512 with zp) and fully_connected_gpu_u2_bf_tiled_osv64.horizontal_weights_slm (batch 256). They check the selected kernel JIT has USE_SLM 1 on integrated GPUs.

prepare_padding.cpp conflicted with openvinotoolkit#39073; take upstream's version. The u2 change there only touched the oneDNN branch, and oneDNN FC does not take u2 weights, so it was never reached. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

v-Golubev · 2026-10-28T08:58:05Z

build_jenkins

michal-miotk · 2026-10-28T19:36:50Z

 #endif

-#if COMPRESSED_WEIGHTS_INT4
+#if COMPRESSED_WEIGHTS_SUB_BYTE


looks like this branch is not touched by u2 - please express that in #if

Done, added && COMPRESSED_WEIGHTS_INT4.

michal-miotk · 2026-10-28T20:23:33Z

@@ -55,7 +86,23 @@ KERNEL(reorder_weights_int4)(const __global INPUT0_TYPE* input, __global OUTPUT_



[random spot][nit] please change filename to sth like ....compressed... to not contain int4 explicitly

Done, renamed to reorder_weights_compressed.

… the osv32 path int4-only The kernel now also repacks u2, so the int4 name is misleading. The osv32_isv2 branch of the bf_tiled FC kernel is never taken for u2; say so in the #if. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

michal-miotk · 2026-10-29T09:23:06Z


+                if (root->get_input_element_type(1) == ov::element::u2) {
+                    GPU_DEBUG_TRACE << root->get_friendly_name() << "  dyn_quan is turned off:"
+                                                                    " no FC kernel takes i8 activations with u2 weights" << std::endl;


please change comment to be more broad - not contain any compressed activation not only i8

Done, reworded to "compressed activations".

v-Golubev · 2026-10-29T09:57:15Z

build_jenkins

michal-miotk · 2026-10-29T10:46:36Z

overall LGTM

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

v-Golubev · 2026-10-30T09:33:05Z

build_jenkins

v-Golubev

A question on the overall direction. oneDNN already supports u2 FC (#37212): it uses plain weights layouts, so no repacking to os_iyx_osv16 / os_is_yx_osv64_isv2 and no extra reorder kernels. This path looks preferable, since it means much less code to maintain on the OpenVINO side and it avoids extra memory consumption.

Could you please clarify which code paths this PR closes that oneDNN wouldn't? Do we have real workloads (models/devices) that hit them?

Cyberpunk1210 requested review from a team as code owners October 21, 2026 07:34

github-actions Bot added category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations labels Oct 21, 2026

sys-openvino-ci added the ExternalPR External contributor label Oct 21, 2026

This was referenced Oct 21, 2026

[GPU] Support u2 compressed dictionaries in GatherCompressed #39057

Open

[GPU] Support u2 weights in the batch-1 GEMV fully-connected kernel #39059

Open

[GPU][CPU] Support u2 (INT2) weight decompression: optimized FC kernel and fused MoE 3-GEMM #37335

Open

Merge remote-tracking branch 'upstream/master' into int2-fc

14906c3

Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com> # Conflicts: # src/plugins/intel_gpu/tests/unit/test_cases/fully_connected_gpu_test.cpp

p-durandin assigned michal-miotk Oct 21, 2026

[GPU] Simplify u2 tiling checks in bf_tiled VerifyTuneParams

a4691a8

Return the conditions directly (clang-tidy readability-simplify-boolean-expr). No behaviour change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

v-Golubev self-assigned this Oct 22, 2026

michal-miotk reviewed Oct 22, 2026

View reviewed changes

Cyberpunk1210 mentioned this pull request Oct 23, 2026

[GPU][CPU][Transformations] u2 (INT2) weight support on releases/2026/4 #38837

Closed

michal-miotk reviewed Oct 26, 2026

View reviewed changes

github-actions Bot removed the category: transformations OpenVINO Runtime library - Transformations label Oct 27, 2026

michal-miotk reviewed Oct 27, 2026

View reviewed changes

[GPU] Add is_2bit, name the u2 osv64 line size, add u2 bf_tiled SLM t…

0fb1480

…ests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

michal-miotk reviewed Oct 28, 2026

View reviewed changes

Cyberpunk1210 requested a review from a team as a code owner October 29, 2026 06:36

michal-miotk reviewed Oct 29, 2026

View reviewed changes

[GPU] Word the u2 dyn-quant trace for any compressed activation

1aa3b36

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Cyberpunk1210 <genji.1231@outlook.com>

v-Golubev requested changes Nov 3, 2026

View reviewed changes

		@@ -55,7 +86,23 @@ KERNEL(reorder_weights_int4)(const __global INPUT0_TYPE* input, __global OUTPUT_

Conversation

Cyberpunk1210 commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

AI Assistance:

Uh oh!

p-durandin commented Oct 21, 2026

Uh oh!

p-durandin commented Oct 22, 2026

Uh oh!

p-durandin commented Oct 22, 2026

Uh oh!

michal-miotk Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

p-durandin commented Oct 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v-Golubev commented Oct 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michal-miotk commented Oct 27, 2026

Uh oh!

Cyberpunk1210 commented Oct 28, 2026

Uh oh!

v-Golubev commented Oct 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v-Golubev commented Oct 29, 2026

Uh oh!

michal-miotk commented Oct 29, 2026

Uh oh!

v-Golubev commented Oct 30, 2026

Uh oh!

v-Golubev left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Cyberpunk1210 commented Oct 21, 2026 •

edited

Loading

michal-miotk Oct 22, 2026 •

edited

Loading