Skip to content

fix: preserve K-quantized embedding weights - #1936

Open
zjn20030811 wants to merge 1 commit into
leejet:masterfrom
zjn20030811:nan/get-rows-k-quants
Open

fix: preserve K-quantized embedding weights#1936
zjn20030811 wants to merge 1 commit into
leejet:masterfrom
zjn20030811:nan/get-rows-k-quants

Conversation

@zjn20030811

@zjn20030811 zjn20030811 commented Sep 4, 2026

Copy link
Copy Markdown

Summary

Embedding::init_params currently converts K-quantized embedding weights to F32 because support_get_rows predates GGML's K-quant GET_ROWS implementations. This can multiply embedding memory by up to four times and trigger out-of-memory failures on constrained devices.

This change:

  • keeps Q2_K, Q3_K, Q4_K, Q5_K, and Q6_K embeddings in their stored quantized type;
  • retains the existing backend capability probe and CPU staging fallback for backends that cannot execute GET_ROWS for a type; and
  • adds a regression test covering the allowlist and allocation path for every K-quant type.

Fixes #1930

Verification

  • whitespace and formatting checks pass for the changed files;
  • the new regression executable is registered with the project's test runner and
    is built when SD_BUILD_TESTS=ON.

The complete build is left to the repository CI matrix because the local
environment cannot complete a native build.

@zjn20030811

Copy link
Copy Markdown
Author

Validation update: the new regression covers Q2_K through Q6_K plus all previously supported embedding types and checks that allocation preserves the stored quantized type. The change keeps backend capability probing/CPU staging intact, so unsupported accelerators retain their fallback. git diff --check passes; CI can exercise the CMake test target.

@leejet please review when convenient.

@zjn20030811

Copy link
Copy Markdown
Author

@leejet The GET_ROWS allowlist now retains Q2_K–Q6_K embeddings and preserves the backend fallback. The new type/allocation regression covers all K-quants and existing types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support_get_rows predates K-quant get_rows support, silently 4x-ing embedding RAM

1 participant