[osabnis]

Details:

Depends on #39118

• Add FullyConnectedGGUF, a GPU primitive that performs MatMul directly on GGUF Q5_K/Q6_K weight blocks without converting them to 8-bit weights.
• Decode uses an int8 matrix-vector kernel.
• Prefill uses a DPAS kernel that decodes each weight block once into local memory and reuses it across activation rows on Xe2+.
• Other devices decode to a pooled f16 scratch buffer and use oneDNN MatMul.
• A following residual Add is fused into the kernels.
• ConvertMatMulToFullyConnectedGGUF recognizes MatMul(A, GGUFDequantize(Constant)) and replaces it with FullyConnectedGGUF.

Only models containing GGUFDequantize are affected; the GGUF frontend changes are a follow-up.

Why a new primitive

GGUF weights are stored as byte blocks, [N, K / 256 * block_bytes], rather than [N, K] values. Existing FullyConnected and its transformations assume decoded weights with separate scales and zero points, so a dedicated primitive keeps those paths unchanged.

Transformation

Constant u8 [N, B]
    -> GGUFDequantize (Q5_K / Q6_K)
    -> [Convert]
    -> MatMul(A, ..., transpose_b=true)

becomes

FullyConnectedGGUF(A, Constant u8 [N, B])

The transformation runs after TransposeMatMul and before constant folding, when:

• the device supports int8 dot products;
• inference precision is f16;
• dynamic quantization supports groups of 32;
• the GGUFDequantize has no other consumer.

Q5_K blocks are passed in their original layout. Q6_K blocks are regrouped once at compile time into aligned planes, QL [nblk][128] | QH [nblk][64] | SC [nblk][16] | D [nblk]. This does not change their size or contents.

Implementation

FullyConnectedGGUF selects the execution path from the number of activation rows, so the same primitive can switch between decode and prefill:

Rows	Path
< 8 or f32	int8 quantization + GEMV
>= 8, f16, Xe2+	DPAS prefill
>= 8, f16, other devices	f16 dequantization + oneDNN MatMul

• GEMV: quantizes activations per 32 values and performs int8 dot products directly against Q5_K/Q6_K blocks.
• DPAS prefill: decodes each 256-value weight block into local memory and reuses it across multiple activation rows. Double buffering overlaps block loading and decoding. The tile grows with the row count: 64 x 64 up to 64 rows, 128 x 64 up to 128, 256 x 64 up to 256, 512 x 128 above.
• Scratch path: uses a pooled f16 buffer, so decoded weights are not retained per layer; oneDNN MatMuls are cached by activation-row count.
• Kernels are compiled by weight shape (N, K), allowing layers with matching shapes to share them.
• A same-shape/type residual Add is fused into the output store; on the scratch path it becomes a oneDNN binary post-op.

Internal option GPU_GGUF_PREFILL_PATH can force the scratch or gemv path for testing.

Tests

• fully_connected_gguf_gpu_test: 22 tests covering Q5_K/Q6_K, different row counts and input ranks, dynamic shapes, all execution paths and prefill tiles, model caching, compile_model, and fused residual Add.
• convert_matmul_to_fc_gguf_test: 7 tests covering both formats, Q6_K plane layout, successful conversion, and cases that must not match.

Expected values are generated from GGUFDequantize::evaluate() and compared in float.

Performance

Decode on Panther Lake with Phi-3.5-mini through llama.cpp's OpenVINO backend, as a share of llama.cpp Vulkan on the same iGPU:

• Q5_K_M decode: 71% -> 98% of Vulkan (+38%)
• Q6_K decode: 78% -> 97% of Vulkan (+25%)
• Q4_K_M decode: 79% -> 92% of Vulkan (+17%)
• Peak memory: 0.6 to 0.9 GB lower

Phi-3.5-mini-instruct on a Panther Lake iGPU through llama.cpp's OpenVINO backend, which emits GGUFDequantize for Q5_K/Q6_K weights. Both OpenVINO columns run master with our pending GPU PRs (#38696, #38995, #38996); Vulkan is llama.cpp b11236 on the same iGPU. llama-bench -fa 1; decode in tok/s at depth 0 / 4K / 8K:

	Converted	This PR	Vulkan
Q5_K_M decode	26.8 / 18.5 / 13.8	36.9 / 22.8 / 16.1	37.8 / 23.6 / 17.1
Q6_K decode	26.8 / 18.5 / 13.7	33.5 / 21.5 / 15.4	34.4 / 22.3 / 16.4
Q4_K_M decode	34.4 / 21.8 / 15.6	40.1 / 24.0 / 16.7	43.6 / 25.9 / 18.3

Peak GPU memory drops from 5.46 GB to 4.54 GB for Q5_K_M and 5.46 GB to 4.82 GB for Q6_K.

Q4_K_M also benefits because its Q5_K/Q6_K weights use this path; its Q4_K weights remain on the converted path.

The main tradeoff is prefill: at 8K tokens, this PR reaches 89–94% of the converted path because the converted weights use oneDNN's tuned GEMM, while this implementation decodes GGUF blocks in the kernel.

Short prompts are also slower, with Q5_K_M reaching about 70% of the converted path around 96–128 tokens.

Limitations

• Q5_K and Q6_K only. Q4_K remains on the converted path.
• 2D weights only; MoE expert weights are not supported yet.
• Q6_K plane regrouping creates a second host-side constant during compilation until the original weights are released.
• The native GGUF frontend path is not included in this PR.

Tickets:

• CVS-194265

AI Assistance:

• AI assistance used: yes
• AI assistance was used for code review, build and test runs.
• Human validation performed: Windows builds; ov_gpu_unit_tests --gtest_filter=*gguf* (29 tests) and all fully connected unit tests pass on Arc B580 (Xe2, this branch on master 1c169aa) and on Panther Lake (Xe3, the same code on master 38ff87c); on Panther Lake, llama.cpp test-backend-ops -o MUL_MAT passes 964/964 with the native path, and the benchmarks above.

[v-Golubev]

/ci-doctor