[GPU] Transpose grouped FC decompression scales at compile time to reduce peak GPU memory by osabnis · Pull Request #38995 · openvinotoolkit/openvino

osabnis · 2026-10-19T17:17:45Z

Details:

Reduce peak GPU memory for compressed LLMs by storing grouped weight decompression scales and zero points of
FullyConnectedCompressed in the [groups, N] layout at compile time, so prepare_quantization no longer adds a
reorder that keeps a second copy of each parameter in device memory during program build.

Implementation

New transformation TransposeFCDecompressionParams (registered after the FullyConnected passes, before
EliminatePad): for FullyConnectedCompressed with transposed 2D weights [N, K], transposes [N, groups] scale and
zero-point constants to [groups, N] and sets transpose_b_scale / transpose_b_zp to false. The OCL kernels
(convert_decompression_tensor) and oneDNN (get_ifm_dim_idx) already handle this layout, and
optimize_weights_decompression_parameters then adds no reorder.
- Applies to constants with more than one group and at least 8 bits; sub-byte zero points keep the existing path.
- A constant shared with other users gets a separate transposed copy for the FC; the other users keep the original.
Weightless cache: the new constant keeps the WeightlessCacheAttribute of the original one and records the
transpose in the gpu_weightless_transpose_order rt_info entry. weightless_cache_manager stores it and replays the
transpose after loading the original data from the weights file, so the cache blob does not grow.
prepare_quantization.cpp: reorder_bfyx returns early when the parameter is already in the target format.
The earlier upload-time transpose in plugin/ops/constant.cpp is removed.

GPU memory

Measured on Windows, Intel Arc B390 (Panther Lake), OV master 1acdfcd, llama.cpp OpenVINO backend, Q4_K_M,
llama-bench -p 0 -n 128 -d 4096; before = the same build with the new pass disabled.

Duplicate parameter copies during program build (GPU plugin allocation counters in propagate_constants):

Model	Before	After	Removed
Llama-3.2 1B	74 MB	0 MB	74 MB
Gemma-4 E2B	220 MB	79 MB	141 MB
Qwen3.5 4B	226 MB	7.5 MB	218 MB
Phi-4 mini	207 MB	0 MB	207 MB

Peak process GPU memory (dedicated + shared, two runs each, within 1 MB of each other):

Model	Before	After	Reduction
Llama-3.2 1B	2,050 MB	2,042 MB	8 MB
Gemma-4 E2B	4,495 MB	4,387 MB	108 MB
Qwen3.5 4B	5,030 MB	4,800 MB	230 MB
Phi-4 mini	6,866 MB	6,801 MB	65 MB

With 4096 tokens of context, the runtime allocations of Llama-3.2 1B and Phi-4 mini exceed their compile-time peak, so
the process peak drops less than the compile-time saving for those two models.

Correctness is unchanged: KL divergence against the CPU backend (wikitext-2, 20 x 512 tokens), before / after:
Llama 0.051698 / 0.051698, Gemma 0.330667 / 0.330667, Qwen 0.573779 / 0.573851 (± 0.0126), Phi-4 0.048667 / 0.048667.

Performance is unchanged within noise. llama-bench -p 4096 -n 128 -d 0,4096 -r 3, mean of two rounds, tokens/s
(before → after):

Model	Decode, depth 0	Decode, depth 4096	Prefill pp4096
Llama-3.2 1B	98.31 → 98.41	64.21 → 64.05	7,432 → 7,146
Gemma-4 E2B	53.07 → 53.08	45.32 → 45.50	4,059 → 3,974
Qwen3.5 4B	27.22 → 27.25	24.07 → 24.07	1,861 → 1,855
Phi-4 mini	32.60 → 32.52	21.13 → 21.02	2,044 → 2,122

Decode is within 1% everywhere. Prefill differences are within run-to-run variation on this machine (the two rounds
differ by up to 9% for Llama prefill and up to 20% for Phi-4 prefill at depth 4096).

Tests

Transformation tests (transpose_fc_decompression_params_test.cpp): scale with and without zero point
(u8 / f16 / u4), single group, broadcast [1, groups], already [groups, N], non-constant scale, a scale shared by two
FCs and a Multiply, and the weightless cache attribute plus transpose record.
Weightless cache round trip (cache_serialization.weightless_transpose_is_replayed_on_load): f32 / f16 source
data, including a model constant of another rank ([N, groups, 1], as before the FC compression reshape).
Functional (CheckWeightlessCacheAccuracyLowPrecision.MatmulGroupedWeightsDecompression): grouped scale and zero
point with transposed weights for all import APIs and cache modes. On the previous version of this PR, 24 of 96 cases
gave wrong results; with the transpose replay disabled, 48 of 96 fail.
Existing suites *CheckWeightlessCacheAccuracy*:*MatMulCompressedWeights*:*MatMulSharedCompressedWeights* (Intel Arc
B580, Windows, after merging master 3d39b76): 1,839 passed; the 6 failures
(smoke_MatMulCompressedWeights_extra_multiply, u4 / i4, group size 2, and smoke_MatMulCompressedWeights_basic,
u2, f16 activations) also fail with this PR's changes disabled on this device.

Reviewer notes

propagate_constants: the root cause of the duplicate copies is that it builds all constant subgraphs in one
network with separate output buffers. A refactor to reuse one buffer would cover every constant, not only FC scales
and zero points; this PR keeps to the FC case (see the discussion below).
Overlap with [GPU] Use oneDNN static_sbzp for exact f16/bf16 decompression zero points #38696: [GPU] Use oneDNN static_sbzp for exact f16/bf16 decompression zero points #38696 adds a dep_layout.format != format::fbyx check to need_reorder. With the
same-format skip in reorder_bfyx from this PR, that check is no longer needed; I will remove it from [GPU] Use oneDNN static_sbzp for exact f16/bf16 decompression zero points #38696 once one
of them merges. The edits are in different lines, so they merge cleanly in either order.

Tickets:

CVS-196701

AI Assistance:

AI assistance used for analysis, test execution, and measurements.

…ce peak GPU memory

ZackyLake · 2026-10-20T01:04:12Z

+static bool is_grouped_fc_decompression_param(const ProgramBuilder& p, const std::shared_ptr<ov::op::v0::Constant>& op) {
+    const auto& shape = op->get_shape();
+    // Sub-byte zp (u4 / i4) packs two values per byte and keeps the plain layout
+    if (!p.use_new_shape_infer() || shape.size() != 2 || shape[1] < 2 || op->get_output_element_type(0).bitwidth() < 8)


why check shape[1] < 2, is it skipping dim==1?

Yes, on purpose. With one group ([N, 1]), bfyx and fbyx have the same memory order, so prepare_quantization doesn’t reorder it either (need_reorder requires groups_count > 1). There’s no second copy to save, so it stays on the normal upload. I will add a comment.

I think using shape[1]==1 seems better

Done, changed to shape[1] == 1

ZackyLake · 2026-10-20T01:04:36Z

+        if (!(user.get_index() == 3 ? fc->get_transpose_b_scale() : fc->get_transpose_b_zp()))
+            return false;
+        const auto& w_shape = fc->get_input_partial_shape(1);
+        if (w_shape.rank().is_dynamic() || w_shape.size() != 2 || w_shape[0].is_dynamic() || static_cast<size_t>(w_shape[0].get_length()) != shape[0])


what if broadcasting on dim[1]?

Broadcast on dim[1] ([N, 1]) is the single-group case above and is skipped. Broadcast on dim[0] ([1, groups]) fails the shape[0] == N check, so it also keeps the plain upload, and prepare_quantization reorders it as it does today. Only exact [N, groups] params take the new path. I will add a comment.

p-durandin · 2026-10-20T07:09:38Z

build_jenkins

…t cases

…n-upload-rebase # Conflicts: # src/plugins/intel_gpu/src/plugin/ops/constant.cpp

Lyamin-Roman · 2026-10-21T13:30:39Z

        auto groups_count = dep_pshape[groups_idx].get_length();
-        return groups_count > 1;
+        // grouped parameters that are already in fbyx (uploaded or computed that way) need no reorder
+        return groups_count > 1 && dep_layout.format != format::fbyx;


It appears that the fbyx format is not used in all cases, it would be better to skip adding a reorder operation inside reorder_bfyx whenever the current format is already the same as the target format

Done: reorder_bfyx now returns early when the dependency is already in the target format, and the PR-specific fbyx check in need_reorder is gone.

Lyamin-Roman · 2026-11-02T12:46:09Z




+        // fbyx of [N, groups] is [groups][N] in memory


It looks like this breaks the weightless cache path

You were right: on the previous version, 24 of 96 weightless cache cases with grouped parameters gave wrong results (new test: MatmulGroupedWeightsDecompression).

The constant.cpp change is removed. The new transformation records the transpose, and the weightless cache manager replays it on load. All 96 cases pass, and the cache blob stays the same size as on master.

One side finding: the weightless tests can't detect a failed cache import, because the core silently recompiles and the output is still correct. I hit this myself and only caught it by checking ov::loaded_from_cache. Should the shared test helper assert ov::loaded_from_cache? I left it out because it changes the common test body.

Lyamin-Roman · 2026-11-02T13:46:55Z


+// Scale/ZP [N, groups] used only by FullyConnectedCompressed with transposed 2D weights (transpose_b_scale / transpose_b_zp).
+// prepare_quantization reorders exactly these to fbyx.
+static bool is_grouped_fc_decompression_param(const ProgramBuilder& p, const std::shared_ptr<ov::op::v0::Constant>& op) {


[Globaly] If the issue of excessive memory consumption stems from redundant weight copying - caused by reordering and the associated computations within propagate_constants - then the best solution is to address the root cause of the inefficiency in propagate_constants. Currently, it creates numerous small constant subgraphs, each with its own unique buffer. Ideally, this should be refactored to reuse a single, pre-allocated, maximum-sized buffer. But this is a separate large work

Rather than the current approach, a potentially better generalized solution would be to avoid making such changes here. Instead, you can create a new transformation - to be applied after all transformations related to the FC layer. Use the logic from your is_grouped_fc_decompression_param in pattern matcher, and modify transpose_b_scale and transpose_b_zp from true to false with const data modification. prepare_quantization should not use reorder in this case, and convert_decompression_tensor should handle it w/o perf affect

Done: TransposeFCDecompressionParams, placed after DynamicQuantizeFullyConnected and before EliminatePad. It transposes [N, groups] scales and zero points (more than one group, at least 8 bits) and clears transpose_b_scale / transpose_b_zp. A shared constant gets its own transposed copy for the FC, and its other users keep the original. Updated numbers are in the description.

…transposed constants

…ro points

…n-upload-rework

osabnis · 2026-11-02T21:02:02Z

Measured on Intel Arc B390 with four Q4_K_M models (Llama-3.2 1B / Gemma-4 E2B / Qwen3.5 4B / Phi-4 mini):

This PR removes 74 / 141 / 218 / 207 MB of duplicate scale and zero-point copies during propagate_constants. Remaining duplicate copies: 0 / 79 / 7.5 / 0 MB (constants this PR does not cover, e.g. sub-byte zero points).
Why the copies exist: propagate_constants runs all constant subgraphs (160 to 401 outputs for these models) in one network with optimize_data(false), so each output has its own buffer, and all outputs and all original inputs are alive at the same time. For Llama, allocated memory goes from 993 MB to 1,067 MB in this pass.
What the refactor (one reused buffer, freeing each input once it is replaced) would add: the extra memory becomes the largest single constant instead of the sum of all of them, for every constant in every model. For these four models that is about 79 MB more saved on Gemma and 7.5 MB more on Qwen than this PR, and no change on Llama and Phi-4. It would also not need the new transformation or the transpose replay.
Effect on the process peak: with 4096 tokens of context, runtime allocations are larger than the compile-time peak for some models (OV allocation peak 1,886 MB at runtime vs 1,067 MB after compile for Llama), so both approaches lower the process peak only where compilation sets it.

@Lyamin-Roman what would be the better way?

Keep this PR (small, tested, covers most of the saving for these models)
The refactor

I am happy to help with the refactor either way.

[GPU] Upload grouped FC decompression scales directly in fbyx to redu…

ec39d2a

…ce peak GPU memory

osabnis requested review from a team as code owners October 19, 2026 17:17

github-actions Bot added category: GPU OpenVINO GPU plugin category: build OpenVINO cmake script / infra labels Oct 19, 2026

sys-openvino-ci added the ExternalIntelPR External contributor from Intel label Oct 19, 2026

This was referenced Oct 19, 2026

[GPU] SDPA: use key-partitioned single-token kernels for up to 4 query rows #38996

Open

[GPU] Use oneDNN static_sbzp for exact f16/bf16 decompression zero points #38696

Draft

osabnis added 2 commits October 19, 2026 13:26

Merge branch 'master' into gpu-fc-decompression-upload

d216eb1

[GPU] FC decompression upload: optimize the code and improve comments

37a7d50

ZackyLake reviewed Oct 20, 2026

View reviewed changes

p-durandin assigned ZackyLake Oct 20, 2026

osabnis added 3 commits October 20, 2026 08:25

[GPU] FC decompression: add comments to the single-group and broadcas…

88c012f

…t cases

Merge branch 'master' into gpu-fc-decompression-upload

f2c767f

[GPU] FC decompression: check the single-group case as shape[1] == 1

9f2179a

github-actions Bot removed the category: build OpenVINO cmake script / infra label Oct 20, 2026

osabnis added 2 commits October 20, 2026 15:21

Merge branch 'master' into gpu-fc-decompression-upload

1cfa062

Merge remote-tracking branch 'origin/master' into gpu-fc-decompressio…

a10fe44

…n-upload-rebase # Conflicts: # src/plugins/intel_gpu/src/plugin/ops/constant.cpp

osabnis mentioned this pull request Oct 22, 2026

[GPU] FullyConnectedGGUF: MatMul on GGUF Q5_K / Q6_K weights without conversion #39119

Draft

Merge branch 'master' into gpu-fc-decompression-upload

e9f82fd

Lyamin-Roman reviewed Nov 2, 2026

View reviewed changes

osabnis added 4 commits November 2, 2026 12:27

[GPU] Skip same-format reorders in reorder_bfyx instead of uploading …

fc9932d

…transposed constants

[GPU] Replay transposes of weightless-cached constants on load

f6b71f0

[GPU] Add TransposeFCDecompressionParams for grouped FC scales and ze…

6fec150

…ro points

Merge remote-tracking branch 'origin/master' into gpu-fc-decompressio…

f2fd40b

…n-upload-rework

github-actions Bot added the category: transformations OpenVINO Runtime library - Transformations label Nov 2, 2026

osabnis changed the title ~~[GPU] Upload grouped FC decompression scales directly in fbyx to reduce peak GPU memory~~ [GPU] Transpose grouped FC decompression scales at compile time to reduce peak GPU memory Nov 2, 2026

osabnis added 2 commits November 3, 2026 19:21

[GPU] Fix implicit bool conversion flagged by clang-tidy

8b66b0b

Merge branch 'master' into gpu-fc-decompression-upload

7d4468f

Conversation

osabnis commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Implementation

GPU memory

Tests

Reviewer notes

Tickets:

AI Assistance:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZackyLake Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

p-durandin commented Oct 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

osabnis Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

osabnis commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

osabnis commented Oct 19, 2026 •

edited

Loading

ZackyLake Oct 20, 2026 •

edited

Loading

osabnis Nov 2, 2026 •

edited

Loading