[GPU] support scale/zp as input by Kotomi-Du · Pull Request #38204 · openvinotoolkit/openvino

Kotomi-Du · 2026-09-16T21:56:05Z

Summary:

As a follow up of #34767 which is to recognize parameters as valid inputs for weights, this PR extends the same support to scale and zero-point (ZP) tensors. The scope doesn't cover conv1x1, because OVEP has blocked scale/zp promotion for conv operator in this related PR

Details:

Recognize scale / zero-point as model inputs in ConvertFullyConnectedToFullyConnectedCompressed (Intel GPU plugin). Previously the decompression scale and zero point had to be Constants; now they can also come from a Parameter.
- compressed_weights_pattern.hpp: scale (Multiply) and zero-point (Subtract) operands now match:
  - Constant
  - Parameter
  - Reshape(Parameter)
  - Reshape(Convert(Parameter))
- convert_fc_to_compressed.cpp:
  - Static shapes required: the callback bails out if the scale or zero-point shape is not static.
  - Reshaping: reshape_const became reshape_input. Constants are still folded into a new Constant; any other input gets a Reshape node.
  - Zero-point conversion: convert_zp_const decides on a Convert from the input's element type, so it works for constant and non-constant zero points. A u8 zero point is passed through unchanged.
  - Runtime info: it is copied onto all newly created nodes.
oneDNN FC keeps the scale / zero-point input format (fully_connected_onednn.hpp).
- query_formats now keeps the existing format of the decompression scale and zero-point inputs instead of forcing the default format.
- This preserves the layout chosen earlier, e.g. fbyx from prepare_quantization for grouped transposed scale/zp. Without it, oneDNN would force a reorder back to the default format.

Tests

convert_fc_to_compressed_test.cpp: ConvertFCToCompressedWithScaleAndZeroPointParameters. Uses a Parameter scale with Convert + Reshape and a Parameter u8 zero point with Reshape; checks the result is FullyConnectedCompressed with the reshaped parameters.

fully_connected_gpu_test.cpp: new general (not oneDNN-only) test test_compressed_int8_parameter_scale_zp.

Setup: u8 weights [N, K], scale/zp as input_layout with bfyx data.
Accuracy: output is compared with a reference computed as (w - zp) * scale.

Layout checks per case:

Test	Scale/zp shape	Expected scale/zp input
`compressed_int8_parameter_scale_zp_per_channel`	`[1, N]`	parameter, `bfyx`
`compressed_int8_parameter_scale_zp_per_channel_transposed`	`[N, 1]`	parameter, `bfyx` (already N-innermost, no reorder)
`compressed_int8_parameter_scale_zp_per_group`	`[G, N]`	parameter, `bfyx`
`compressed_int8_parameter_scale_zp_per_group_transposed`	`[N, G]`	`reorder` `bfyx` → `fbyx` inserted by `prepare_quantization`

Tickets:

CVS-194522

AI Assistance:

AI assistance used: yes
What AI did: implemented the pattern, callback and oneDNN format changes and wrote the unit tests, iterating on review feedback from the author.
Human validation (RelWithDebInfo, Windows):
- ov_gpu_unit_tests --gtest_filter=TransformationTestsF.ConvertFCToCompressed*:*FullyConnected*Compressed*: 27 passed.
- ov_gpu_unit_tests --device_suffix=0 --gtest_filter=fully_connected_gpu_tests.compressed* (iGPU, UHD 770): 88 passed.
- ov_gpu_unit_tests --device_suffix=1 --gtest_filter=fully_connected_gpu_tests.compressed* (dGPU, with hardware matmul): 41 passed, 47 skipped by existing device-capability checks, 0 failed.
- All 4 new compressed_int8_parameter_scale_zp* tests passed on both iGPU and dGPU

p-durandin · 2026-09-22T06:09:55Z

build_jenkins

Copilot

Copilot review overview

🟡 Changes recommended

The new matcher has unsupported ordering and dynamic-shape failure cases without regression coverage.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Guard against dynamic scale or zero-point shapes before get_shape() · New
Add regression tests for parameter scale and zero-point inputs · New

What changed in this PR

Extends GPU compressed fully connected conversion to accept runtime scale and zero-point tensors.

Changes:

Expands decompression patterns to Parameter inputs.
Adds runtime reshape/conversion handling.
Preserves oneDNN decompression input formats.

Risks: Dynamic shapes can throw, one scale ordering is unmatched, and regression tests are missing.

File	Description
`convert_fc_to_compressed.cpp`	Handles non-constant decompression inputs.
`compressed_weights_pattern.hpp`	Matches Parameter-based scale/ZP paths.
`fully_connected_onednn.hpp`	Preserves scale/ZP input formats.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

### Details: The reason for this refactor is to make the code cleaner and better prepare for upcoming features: - Support for [num_group, N, group_size] weights layout (openvinotoolkit#38206) - Support for scale/zp as parameters (openvinotoolkit#38204) #### Changes: 1. Use the new pattern matching operators (`| and optional`) to simplify the patterns. 2. Add `transpose_b_scale` and `transpose_b_zp` to track the actual layout of scale/zp. **Current logic:** 1. If weights and scale/zp are transposed, scale/zp will be reordered back to a non-transposed layout to satisfy kernel requirements. 2. If weights and scale/zp are not transposed, weights and scale/zp will first be reordered to a transposed layout. Since the kernel still requires scale/zp in a non-transposed layout, another reorder is added to convert scale/zp back. As a result, in case 2, scale/zp are unnecessarily transposed and then transposed back. This increases memory usage and adds extra compilation work. The impact is currently limited because the tensors are constants and the cost is paid only during compilation. **With this PR**: By tracking the actual layout of scale/zp, the plugin can skip the unnecessary transpose-and-revert steps in case 2. This simplifies the logic, reduces memory usage, and will be especially helpful when weights/scale/zp are provided as parameters instead of constants. ``` Data flow (non-transposed case) Weights [K,N] --Transpose--> FC (transpose_b=true) Scale [G,N] -------------> FC (transpose_b_scale=false) // no Transpose, no reorder ZP [G,N] -------------> FC (transpose_b_zp=false) OCL : bfyx[G,N] reinterpreted as fbyx (N=batch, G=feature) oneDNN: ngroups read from axis 0 ([G,N]) ``` ### Tickets: - *[CVS-195001](https://jira.devtools.intel.com/browse/CVS-195001)* ### AI Assistance: AI assistance used to implement fix and unit test. The generated code was verified for correctness, manually built, executed, and fixed as needed. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Take v9's fully_connected_onednn.cpp (explicit transpose_b_scale/zp flags supersede the PR's shape heuristic), fix the PR's unbalanced query_formats block, keep v9's GNK, Slice zero-point and transpose_before_reshape handling, and let the GNK branch take scale/zp Parameters.

bkaruman · 2026-11-03T20:06:43Z

Add support for Conv1x1 ->FC Compressed and update the PR with refactored implementation based on transpose_b_scale and transpose_b_zp flags introduced in #38697

bkaruman · 2026-11-04T18:46:33Z

The PR is based on older master, which likely contributed to some unintentional changes. Will update it against latest master and remove unrelated code.

This reverts commit a2a07c0.

Kotomi-Du · 2026-11-06T07:24:08Z

@isanghao Updated PR and description broadly based on previous review. Please check

Copilot

🟡 Changes recommended

Valid MatMul graphs with dynamic decompressed-weight shapes or vector scales can trigger assertion failures during compilation.

3 open issues

2 resolved since last review

_{Review effort: Balanced}

 \
-        auto mul_const_m = wrap_type<ov::op::v0::Constant>();\
+        auto mul_value_const_m = wrap_type<ov::op::v0::Constant>();\
+        auto mul_value_param_m = wrap_type<ov::op::v0::Parameter>();\


+        if (!scale_output.get_partial_shape().is_static()) {
+            return false;
+        }
+        auto scale_shape = scale_output.get_shape();


+        auto mul_value_param_convert_reshape_m =\
+            wrap_type<ov::op::v1::Reshape>({mul_value_param_convert_m, any_input()});\
+        auto mul_const_m = mul_value_const_m | mul_value_param_m | mul_value_param_reshape_m |\
+                           mul_value_param_convert_reshape_m;\


Kotomi-Du requested review from a team as code owners September 16, 2026 21:56

github-actions Bot added the category: GPU OpenVINO GPU plugin label Sep 16, 2026

MayureshV1 requested review from isanghao and p-durandin September 22, 2026 00:55

p-durandin requested a balanced review from Copilot September 22, 2026 07:17

Copilot started reviewing on behalf of p-durandin September 22, 2026 07:18

View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/plugin/transformations/convert_fc_to_compressed.cpp Resolved

Comment thread src/plugins/intel_gpu/src/plugin/transformations/compressed_weights_pattern.hpp Resolved

e-ddykim reviewed Sep 23, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/graph/impls/onednn/fully_connected_onednn.hpp Resolved

bkaruman requested a review from a team as a code owner October 7, 2026 21:30

github-actions Bot added the category: transformations OpenVINO Runtime library - Transformations label Oct 7, 2026

Kotomi-Du mentioned this pull request Oct 9, 2026

[GPU] Refactor for FC Compressed #38697

Merged

CuriousPanCake reviewed Oct 16, 2026

View reviewed changes

Comment thread ...ormations/src/transformations/op_conversions/convert_weight_compressed_conv1x1_to_matmul.cpp Resolved

geunhwan added this to the 2026.5 milestone Oct 30, 2026

geunhwan added the orca_CF label Oct 30, 2026

bkaruman force-pushed the scale_zp_parameter branch from 05d1e44 to d9e5759 Compare November 3, 2026 19:59

bkaruman requested a review from a team as a code owner November 3, 2026 20:35

github-actions Bot added category: CI OpenVINO public CI github_actions Pull requests that update GitHub Actions code labels Nov 3, 2026

Kotomi-Du commented Nov 4, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/plugin/transformations/convert_fc_to_compressed.cpp Resolved

Kotomi-Du commented Nov 4, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/src/plugin/transformations/convert_fc_to_compressed.cpp Resolved

Kotomi-Du commented Nov 5, 2026

View reviewed changes

Comment thread ...ormations/src/transformations/op_conversions/convert_weight_compressed_conv1x1_to_matmul.cpp Resolved

isanghao reviewed Nov 5, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/tests/unit/test_cases/fully_connected_gpu_test.cpp Resolved

Comment thread src/plugins/intel_gpu/tests/unit/test_cases/fully_connected_gpu_test.cpp Resolved

initial change

31d9a5c

revert changes related to conv1x1

e9ab345

Kotomi-Du force-pushed the scale_zp_parameter branch from 388f308 to e9ab345 Compare November 6, 2026 05:27

github-actions Bot removed category: CI OpenVINO public CI github_actions Pull requests that update GitHub Actions code labels Nov 6, 2026

update logic

167732b

This reverts commit a2a07c0.

Kotomi-Du force-pushed the scale_zp_parameter branch from a2a07c0 to 167732b Compare November 6, 2026 06:50

Kotomi-Du added 2 commits November 5, 2026 23:14

update unit tests

5475712

recover missing path

85feee8

Merge branch 'master' into scale_zp_parameter

cbc60ce

Kotomi-Du requested a balanced review from Copilot November 6, 2026 20:13

Copilot started reviewing on behalf of Kotomi-Du November 6, 2026 20:14

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Conversation

Kotomi-Du commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary:

Details:

Tickets:

AI Assistance:

Uh oh!

p-durandin commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkaruman commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkaruman commented Nov 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kotomi-Du commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Kotomi-Du commented Sep 16, 2026 •

edited

Loading

bkaruman commented Nov 3, 2026 •

edited

Loading

Kotomi-Du commented Nov 6, 2026 •

edited

Loading