Repository navigation
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. Some pipeline(s) encountered errors during trigger evaluation. |
This was referenced Oct 29, 2026
kwieloch-intel
marked this pull request as ready for review
October 29, 2026 14:17
kwieloch-intel
force-pushed
the
prefill/sdpa-igpu-prefill-config
branch
from
October 30, 2026 13:30
69cde32 to
e8826f4
Compare
Merged
via the queue into
openvinotoolkit:master
with commit Nov 2, 2026
a3cc46f
197 of 198 checks passed
mkruszyx
pushed a commit
to mkruszyx/openvino
that referenced
this pull request
Nov 3, 2026
### This PR adds a GPU pass that fuses the L2-normalization form of RMS norm into a single RMS op. ### Tickets: - [CVS-197466](https://jira.devtools.intel.com/browse/CVS-197466) - [CVS-197053](https://jira.devtools.intel.com/browse/CVS-197053) This PR is one part of a set of TTFT optimizations for 2-bit LLMs on Panther Lake. The other parts are openvinotoolkit#39316 (SDPA prefill config), openvinotoolkit#39320 (int8 activations for u2 FullyConnected) and uxlfoundation/oneDNN#6381 (u2 weights with int8 activations). --- ### Changes Models exported from ONNX LpNormalization compute RMS norm as `x / sqrt(ReduceSum(x^2) + eps) * gamma`. RMSFusion only matches the ReduceMean form with a constant gamma, so this form ran as about 7 separate kernels per norm. The new `L2NormToRMS` pass rewrites it to `RMS(x, gamma / sqrt(N), eps / N)`. That is mathematically identical, because `sum(x^2) = N * mean(x^2)`. It also absorbs a trailing Convert. It applies only when all of these hold: - the reduction is over a static last axis; - `keep_dims` is set; - the row fits the RMS kernel's work group. Gamma may be a constant or a run-time input, for example weights passed as model inputs. --- ### Before / after 2-bit LLM, PTL 4Xe, ~8200-token prompt in 512-token chunks, TTFT median of 3 runs, measured with the SDPA prefill config fix applied: | | TTFT | | |---|---|---| | before | 24.21 s | | | after | 22.72 s | **1.07x** | All 81 norms of the model are fused, so each norm now runs as 1 RMS kernel instead of about 7. Output is identical and decode speed is unchanged. --- ### Validation - New unit tests `L2NormToRMS*` (`l2_norm_to_rms_test.cpp`) cover the plain pattern, f16 activations with Converts, and three negative cases (not the last axis, no keep_dims, row too long). All 5 pass. - End to end on PTL 4Xe: output identical to before on all prompts. ### AI assistance: > [!NOTE] > AI assistance was used for the analysis, the profiling and the implementation. All changes were reviewed, built, tested and measured on hardware. --------- Co-authored-by: Kim, Eddy <eddy.kim@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes micro SDPA config selection on integrated GPUs, which picked a decode (second-token) config for every prefill chunk.
Tickets:
This PR is one part of a set of TTFT optimizations for 2-bit LLMs on Panther Lake. The other parts are #39317 (L2 normalization into RMS), #39320 (int8 activations for u2 FullyConnected) and uxlfoundation/oneDNN#6381 (u2 weights with int8 activations).
Changes
Xe2/Xe3 reach
choose_config_xehpcfor head size <= 128. There were two problems:is_integratedcheck returnedxehpc_q_h128_2nd_integrated, a second-token config, beforethin_qwas checked. With an f16 KV cache, every prefill chunk ran a decode-shaped kernel.seq = 0, so a prefill over thousands of keys was sized as a short sequence.After the fix:
thin_qonly;xehpc_q_h128, the config that a known long sequence already gets.Discrete GPUs, PagedAttention and decode (
thin_q) take the same paths as before.Before / after
LLM (2-bit weights, f16 KV cache), PTL 4Xe, ~8200-token prompt in 512-token chunks, TTFT median of 3 runs:
Greedy output is identical and decode speed is unchanged. Running a 2-bit model needs u2 FullyConnected support (#37212), so this was measured on a branch that has it. This change does not depend on it.
Validation
Note
AI assistance was used. All changes were reviewed, built, tested and measured on hardware.