Repository navigation
[GPU] Int8 activations for u2 FullyConnected on Xe2/Xe3 - #39320
Draft
kwieloch-intel wants to merge 2 commits into
Draft
kwieloch-intel wants to merge 2 commits into
kwieloch-intel wants to merge 2 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. Some pipeline(s) encountered errors during trigger evaluation. |
github-actions
Bot
added
category: GPU
This was referenced Oct 29, 2026
Kotomi-Du
reviewed
Nov 2, 2026
kwieloch-intel
force-pushed
the
prefill/u2-int8-xe3
branch
from
November 2, 2026 18:30
9e9b6f6 to
b0547b2
Compare
github-actions
Bot
removed
the
category: transformations
akazakov-github
pushed a commit
to akazakov-github/openvino
that referenced
this pull request
Nov 2, 2026
…kit#39316) ### This PR fixes micro SDPA config selection on integrated GPUs, which picked a decode (second-token) config for every prefill chunk. ### Tickets: - [CVS-197466](https://jira.devtools.intel.com/browse/CVS-197466) This PR is one part of a set of TTFT optimizations for 2-bit LLMs on Panther Lake. The other parts are openvinotoolkit#39317 (L2 normalization into RMS), openvinotoolkit#39320 (int8 activations for u2 FullyConnected) and uxlfoundation/oneDNN#6381 (u2 weights with int8 activations). --- ### Changes Xe2/Xe3 reach `choose_config_xehpc` for head size <= 128. There were two problems: - **Check order:** the `is_integrated` check returned `xehpc_q_h128_2nd_integrated`, a second-token config, before `thin_q` was checked. With an f16 KV cache, every prefill chunk ran a decode-shaped kernel. - **Unknown key count:** a dynamic key count (StatelessKV) arrives as `seq = 0`, so a prefill over thousands of keys was sized as a short sequence. After the fix: - the integrated decode config is used for `thin_q` only; - a multi-token call with an unknown key count on an integrated GPU is sized as a long sequence. With a quantized KV cache this selects `xehpc_q_h128`, the config that a known long sequence already gets. Discrete GPUs, PagedAttention and decode (`thin_q`) take the same paths as before. --- ### Before / after LLM (2-bit weights, f16 KV cache), PTL 4Xe, ~8200-token prompt in 512-token chunks, TTFT median of 3 runs: | | TTFT | | |---|---|---| | before | 28.92 s | | | after | 24.21 s | **1.19x** | Greedy output is identical and decode speed is unchanged. Running a 2-bit model needs u2 FullyConnected support (openvinotoolkit#37212), so this was measured on a branch that has it. This change does not depend on it. --- ### Validation - The existing SDPA unit tests give the same results with and without this PR. - End to end on PTL 4Xe: greedy output identical to before on all prompts. > [!NOTE] > AI assistance was used. All changes were reviewed, built, tested and measured on hardware.
mkruszyx
pushed a commit
to mkruszyx/openvino
that referenced
this pull request
Nov 3, 2026
### This PR adds a GPU pass that fuses the L2-normalization form of RMS norm into a single RMS op. ### Tickets: - [CVS-197466](https://jira.devtools.intel.com/browse/CVS-197466) - [CVS-197053](https://jira.devtools.intel.com/browse/CVS-197053) This PR is one part of a set of TTFT optimizations for 2-bit LLMs on Panther Lake. The other parts are openvinotoolkit#39316 (SDPA prefill config), openvinotoolkit#39320 (int8 activations for u2 FullyConnected) and uxlfoundation/oneDNN#6381 (u2 weights with int8 activations). --- ### Changes Models exported from ONNX LpNormalization compute RMS norm as `x / sqrt(ReduceSum(x^2) + eps) * gamma`. RMSFusion only matches the ReduceMean form with a constant gamma, so this form ran as about 7 separate kernels per norm. The new `L2NormToRMS` pass rewrites it to `RMS(x, gamma / sqrt(N), eps / N)`. That is mathematically identical, because `sum(x^2) = N * mean(x^2)`. It also absorbs a trailing Convert. It applies only when all of these hold: - the reduction is over a static last axis; - `keep_dims` is set; - the row fits the RMS kernel's work group. Gamma may be a constant or a run-time input, for example weights passed as model inputs. --- ### Before / after 2-bit LLM, PTL 4Xe, ~8200-token prompt in 512-token chunks, TTFT median of 3 runs, measured with the SDPA prefill config fix applied: | | TTFT | | |---|---|---| | before | 24.21 s | | | after | 22.72 s | **1.07x** | All 81 norms of the model are fused, so each norm now runs as 1 RMS kernel instead of about 7. Output is identical and decode speed is unchanged. --- ### Validation - New unit tests `L2NormToRMS*` (`l2_norm_to_rms_test.cpp`) cover the plain pattern, f16 activations with Converts, and three negative cases (not the last axis, no keep_dims, row too long). All 5 pass. - End to end on PTL 4Xe: output identical to before on all prompts. ### AI assistance: > [!NOTE] > AI assistance was used for the analysis, the profiling and the implementation. All changes were reviewed, built, tested and measured on hardware. --------- Co-authored-by: Kim, Eddy <eddy.kim@intel.com>
kwieloch-intel
force-pushed
the
prefill/u2-int8-xe3
branch
from
November 3, 2026 12:43
b0547b2 to
c09a98d
Compare
kwieloch-intel
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR enables dynamic int8 quantization of activations for prefill FullyConnected layers with 2-bit (u2) weights on Xe3 and integrated Xe2, using the oneDNN u2 x s8 kernels.
Tickets:
This PR is one part of a set of TTFT optimizations for 2-bit LLMs on Lunar Lake and Panther Lake. The other parts are uxlfoundation/oneDNN#6381 (u2 weights with int8 activations), #39316 (SDPA prefill config, merged) and #39317 (L2 normalization into RMS, merged).
Important
Needs uxlfoundation/oneDNN#6381 in the GPU plugin's oneDNN (for
rls-v3.15-pcvia its backport, together with d05d595e7a and, for u2 decode on Xe2, d3f8eb8416). Without it, int8 u2 calls fall back to a reference kernel. Draft until the oneDNN update is in.Changes
[GPU] Enable int8 activations for u2 FullyConnected on Xe2 and Xe3:dynamic_quantization_threshold(64) keep f16 activations. The runtime skip of small batches works only for dynamic shapes, so a separate static decode model would otherwise always be quantized.Before / after
2-bit LLM, ~8200-token prompt in 512-token chunks, int8 activations off vs on, median of 3 runs in rotated order:
Measured with this commit and uxlfoundation/oneDNN#6381 on the plugin's oneDNN. Absolute PTL times depend on the power state of the machine; each row comes from one session.
Warning
Int8 activations change the arithmetic, so output is not bit-identical to f16. In 128-token answers the final results and the tool calls are the same; the wording differs, and in one prompt the model writes out the calculation before calling the tool. The int8 output is identical on LNL and PTL. It can be turned off without a rebuild with
DYNAMIC_QUANTIZATION_GROUP_SIZE=0.Validation
benchdnn --mode=Cpasses for all five projection calls at 65 to 512 tokens with the oneDNN used in the build, on LNL and PTL.AI assistance:
Note
AI assistance was used. All changes were reviewed, built, tested and measured on hardware.