[OpenVINO] Support MiniCPM-OCR-1.0-DSpark: DSpark drafts with a pruned vocabulary by imargulis · Pull Request #2099 · huggingface/optimum-intel

imargulis · 2026-10-30T19:49:21Z

What does this PR do?

Adds export of DSpark drafts that predict only part of the target vocabulary, such as openbmb/MiniCPM-OCR-1.0-DSpark, the speculative decoding draft of MiniCPM-OCR 1.0. It predicts 114,688 of the 248,100 target tokens. Until now, the export loaded such a draft as one over the target vocabulary and failed on the shape of the Markov head's next-token table. This PR also gives MiniCPM-OCR targets the hidden-state annotation, so that OpenVINO GenAI feeds the draft the right target hidden states.

This PR builds on #2098 (DFlash v2 and DSpark drafts for continuous batching), which should land first. Until it does, the branch also carries its commits.

Changes to the export:

Pruned draft vocabulary: a draft has one when its config sets draft_vocab_size below vocab_size. Its checkpoint then holds its own lm_head over the draft vocabulary, and d2t, which maps draft token i to target token i + d2t[i], as for Eagle-3 drafts. Loading checks that the checkpoint has both tensors, which would otherwise keep their initial values.
Draft model: openvino_model.xml applies the draft lm_head and outputs logits over the draft vocabulary, with d2t as a constant output, instead of last_hidden_state.
Markov head: openvino_markov_head_model.xml still works over the target vocabulary. Its next-token table is zero for the target tokens outside the draft vocabulary, and the K of its candidate scorer is at most the draft vocabulary size: top_k = min(64, draft_vocab_size).
Weight compression: INT4 keeps the draft lm_head in INT8, as for other models with an lm_head.
DFlash v2 drafts with a pruned vocabulary are rejected.
MiniCPM-OCR targets get the hidden-state annotation that MiniCPM-V 4.7 targets have. Without it, OpenVINO GenAI falls back to a pattern match that takes the post-attention residual of their gated full-attention layers instead of the layer output, which lowers the acceptance of the draft: 4.61 instead of 5.47 tokens per step on GPU.
Docs: MiniCPM-OCR-1.0-DSpark in the supported architectures, and what a pruned-vocabulary export produces.

Runtime support: openvinotoolkit/openvino.genai#4848.

How to test

Environment (Python 3.11+). Until openvinotoolkit/openvino.genai#4848 is merged, OpenVINO GenAI is built from its branch, which needs CMake 3.26+ and a C++ compiler:

python -m venv dspark-ocr-env
source dspark-ocr-env/bin/activate  # Windows: dspark-ocr-env\Scripts\activate
python -m pip install --upgrade pip

# optimum-intel; MiniCPM-OCR needs transformers>=5.21
pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel.git"
pip install "transformers>=5.21"

# OpenVINO and OpenVINO Tokenizers nightly, and OpenVINO GenAI with pruned-vocabulary DSpark drafts
pip install --pre -U openvino openvino-tokenizers --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
pip install --pre "openvino-genai @ git+https://github.com/imargulis/openvino.genai.git@dflash-exp-pruned-vocab" --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

Export the target and the draft:

optimum-cli export openvino --model openbmb/MiniCPM-OCR-1.0 --task image-text-to-text --weight-format int8 MiniCPM-OCR-1.0-ov-int8
optimum-cli export openvino --model openbmb/MiniCPM-OCR-1.0-DSpark MiniCPM-OCR-1.0-DSpark-ov

Run with OpenVINO GenAI on a page of OmniDocBench:

import numpy as np
import openvino as ov
import openvino_genai as ov_genai
from huggingface_hub import hf_hub_download
from PIL import Image

page = hf_hub_download("opendatalab/OmniDocBench", "images/scihub_AEM.01420-09.pdf_3.jpg", repo_type="dataset")
image = ov.Tensor(np.array(Image.open(page).convert("RGB"))[None])

num_assistant_tokens = 15
scheduler_config = ov_genai.SchedulerConfig()
scheduler_config.cache_size = 2
scheduler_config.max_num_batched_tokens = 4096
# MiniCPM-OCR has linear attention: the scheduler needs at least num_assistant_tokens + 2 linear-attention blocks.
scheduler_config.num_linear_attention_blocks = num_assistant_tokens + 2

# Top-64 candidate scorer by default; markov_head_top_k=0 selects the dense Markov walk.
draft = ov_genai.draft_model("MiniCPM-OCR-1.0-DSpark-ov", "GPU")
pipe = ov_genai.VLMPipeline("MiniCPM-OCR-1.0-ov-int8", "GPU", draft_model=draft, scheduler_config=scheduler_config)

config = ov_genai.GenerationConfig()
config.max_new_tokens = 2048
config.num_assistant_tokens = num_assistant_tokens

result = pipe.generate("Text Recognition:", images=[image], generation_config=config)
print(result.texts[0])
print("accepted tokens:", result.extended_perf_metrics.get_num_accepted_tokens())

Validation

Intel Core Ultra X7 358H, Arc B390 GPU.
MiniCPM-OCR-1.0 int8 target, one OmniDocBench page per document category (8 pages), greedy, up to 2048 new tokens, num_assistant_tokens=15. Decode throughput over the 6,292 generated tokens:

Draft weights	Markov head	Decode tok/s	Speedup	Tokens per step
(target alone)		90.6	1.00x
bf16 (default)	Top-64 scorer	201.8	2.23x	5.47
int8	Top-64 scorer	203.9	2.25x	5.45
int4	Top-64 scorer	202.7	2.24x	5.30
bf16 (default)	dense walk (`markov_head_top_k=0`)	98.2	1.08x	4.83

Tests

The tests use optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, a draft for tiny-random-minicpm-ocr laid out like the published one, whose draft vocabulary is a random, unsorted subset of 48 target tokens. Tests cover:

the config: which draft_vocab_size values prune the vocabulary or are rejected, the K of the scorer, the logits and d2t outputs, and the rejection for DFlash v2;
the checkpoint validation on load: a missing lm_head or d2t;
the export package: the draft logits against PyTorch, d2t as a constant, and both Markov head computations against a DeepSpec-style reference that scores over the draft vocabulary;
the PagedAttention parity test, with the MiniCPM-OCR DSpark draft next to DFlash and DFlash v2;
the INT4 CLI export, which keeps the draft lm_head in INT8;
GenAI speculative decoding with the tiny MiniCPM-OCR target;
the hidden-state annotation of the tiny MiniCPM-OCR target, which the preview workflow now checks in its MiniCPM-OCR step.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

…olkit#2099 The DFlash and DSpark CI jobs install optimum-intel from huggingface/optimum-intel#2099, which adds the export of drafts with a pruned draft vocabulary to openvinotoolkit#2098, so the pruned vocabulary tests of test_dflash_markov_head_cb.py run instead of skipping. test_vlm_dspark_with_pruned_draft_vocabulary_matches_main_only pairs optimum-intel-internal-testing/tiny-random-minicpm-ocr with its tiny DSpark draft optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, which predicts 48 of the target's tokens: greedy speculative decoding with the Markov head's candidate scorer and with its dense step gives the target's text. The target needs transformers 5.21, so the DSpark job moves from 5.18.0 to 5.21.0. Co-authored-by: Cursor <cursoragent@cursor.com>

…olkit#2099 The DFlash and DSpark CI jobs install optimum-intel at the head commit of huggingface/optimum-intel#2099, which adds the export of drafts with a pruned draft vocabulary to openvinotoolkit#2098, so the pruned vocabulary tests of test_dflash_markov_head_cb.py run instead of skipping. test_vlm_dspark_with_pruned_draft_vocabulary_matches_main_only pairs optimum-intel-internal-testing/tiny-random-minicpm-ocr with its tiny DSpark draft optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, which predicts 48 of the target's tokens: greedy speculative decoding with the Markov head's candidate scorer and with its dense step gives the target's text. The target needs transformers 5.21, so the DSpark job moves from 5.18.0 to 5.21.0. Co-authored-by: Cursor <cursoragent@cursor.com>

DynamicCache(config=...) gives sliding-window layers a sliding cache layer, which keeps only the last window of keys. The draft applies the window through its attention mask instead, so the second cached step built a mask over more keys than the cache held. A plain DynamicCache keeps every key, as the exported graph does. Co-authored-by: Cursor <cursoragent@cursor.com>

…olkit#2099 The DFlash and DSpark CI jobs install optimum-intel at the head commit of huggingface/optimum-intel#2099, which adds the export of drafts with a pruned draft vocabulary to openvinotoolkit#2098, so the pruned vocabulary tests of test_dflash_markov_head_cb.py run instead of skipping. test_vlm_dspark_with_pruned_draft_vocabulary_matches_main_only pairs optimum-intel-internal-testing/tiny-random-minicpm-ocr with its tiny DSpark draft optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, which predicts 48 of the target's tokens: greedy speculative decoding with the Markov head's candidate scorer and with its dense step gives the target's text. The target needs transformers 5.21, so the DSpark job moves from 5.18.0 to 5.21.0. Co-authored-by: Cursor <cursoragent@cursor.com>

…v head DFlash-2 drafts now use the per-row DFlash backbone that SDPAToPagedAttention converts: every row comes out, and the runtime rolls the block rows of the KV cache back through past_lens. The grouped dynamic convolution shifts its taps along the flattened rows and masks them to the block of the row they produce, so the taps hold on the token axis after the PagedAttention conversion too. The DFlash-2 candidate selector is exported as the Markov head of the draft, openvino_markov_head_model.xml with dflash/interface=markov_head_v1 and markov_head_type=dflash2, instead of a separate selector model. It is the packed Top-K lattice of the DSpark Markov head, with transitions gated by the draft hidden states and no dense step. The backbone advertises it through markov_rank and markov_head_type, as a DSpark draft does. DSpark drafts are DFlash v1 drafts to the exporter: the same validation, backbone and RT info, with the candidate offset and the Markov head taken from their config. The tests use optimum-intel-internal-testing/tiny-random-qwen3-dflash2, a two-layer DFlash-2 draft for tiny-random-qwen3 with the tensor layout of the published checkpoints. Co-authored-by: Ofir Zafrir <ofir.zafrir@intel.com> Co-authored-by: Cursor <cursoragent@cursor.com>

main_export read architectures[0] three times, each with its own empty-list guard: to validate DFlash configs and pick the Eagle3 loader, to load native DFlash drafts, and to drop the tokenizer of DFlash drafts. It now derives the architecture once, with config_architecture() from dflash_utils, which the export config, the dummy input generator and the draft model also use instead of their own copies of the same expression. No behavior change: the architecture stays None outside the transformers library and for configs without architectures, and the Eagle3 config update does not touch architectures. Co-authored-by: Cursor <cursoragent@cursor.com>

…t dynamic Item assignment (x[block_rows] = value) strips the leading unit dimension of a one-row value with a view. When the random dummy token_type_ids hold a single block row, the trace freezes that view's shape, and the exported draft fails on any other block length. The query, attention input and output row writes of the Qwen3 and MuseGlimmer drafts now call index_put_ directly. The PagedAttention tests trace the Qwen3 variants with a single block row, so they cover this case regardless of the dummy seed. Co-authored-by: Cursor <cursoragent@cursor.com>

… all layers Since huggingface#1983, the exporter infers text-generation-with-past for the natively loaded draft architectures and compresses DFlash draft IRs to INT4 on all layers, but task inference on a local draft checkpoint still raised RuntimeError. infer_task now falls back to the config architectures for it, as for Eagle-3 drafts, so the CLI needs no --task there either. The DFlash and DSpark CLI tests run without --trust-remote-code. The Markov head package test also omits --task, and checks that no int8 weights remain in the INT4 drafts. Co-authored-by: Cursor <cursoragent@cursor.com>

The PyTorch reference of the PagedAttention tests passed its own DynamicCache, so no test reached the cache the draft builds when it gets none (db5a55c). The reference now takes the draft's cache. With the sliding layers of these tests, a cache built from the config drops past keys and fails both variants. Co-authored-by: Cursor <cursoragent@cursor.com>

The draft export config computes the Markov head spec once and hands it to the Markov head model config and to the RT-info writer, which used to rebuild it from the HF config. The Markov head config takes its gating from the spec's head type. The default Top-K helper, used once, is inlined. Co-authored-by: Cursor <cursoragent@cursor.com>

…Top-K MarkovHeadOpenVINOConfig.markov_head_top_k repeated the K of its Markov head spec and served only to tell convert.py that the graph is a Markov head. A class flag, markov_head_model, now does that, and the dflash/top_k RT info reads the spec, the same one that sizes the exported TopK. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

MarkovHeadOpenVINOConfig.inputs spelled out the three inputs that both heads take twice. It now adds draft_hidden_states for the gated head or prev_token_ids for the DSpark head to the shared ones. The IR keeps its input order, which the export takes from the head's forward signature; the DSpark head test now checks that order, as the gated head test already did. Co-authored-by: Cursor <cursoragent@cursor.com>

…ly compute Since 5be7015 the convolution runs on the packed block rows, so a masked tap reaches another sequence's block (or the zero padding), not a context row. Co-authored-by: Cursor <cursoragent@cursor.com>

- The anchor-layout test is what is left of the one that checked the backbone dropping the anchor row: the offset is checked by the config test and the row count by the PagedAttention tests. - The DFlash-2 target annotation test exported one more Qwen3 target; the decoder locator test now checks one locator per decoder layer. - The two DSpark Markov head export tests export the draft once; the full-model parity loop is covered by the split computations and the bf16 test. - The single-sequence PagedAttention test is folded into the multi-sequence one, which repeats its steps. Co-authored-by: Cursor <cursoragent@cursor.com>

The Markov head model carried dflash/interface=markov_head_v1 to version its contract with runtimes. markov_head_type already names the head, and runtimes find the computations and read their results by port name and check every port, so the tag could only catch a port whose meaning changes without a new name. Such a port gets a new name instead. Co-authored-by: Cursor <cursoragent@cursor.com>

main exports Muse-Glimmer assistants with a DFlash export config, but their config is not a DFlash, DSpark or DFlash-2 one, so the strict parser rejected them. They get the rt_info keys their config has, as on main. Co-authored-by: Cursor <cursoragent@cursor.com>

MiniCPM-OCR targets get the hidden-state annotation, as MiniCPM-V 4.7 targets do. Without it, GenAI falls back to a pattern match that takes the post-attention residual of their gated full-attention layers instead of the layer output, which lowers the acceptance of DFlash and DSpark drafts. The hidden-state locator test covers the tiny MiniCPM-OCR model on Transformers >= 5.21, and the preview workflow runs test_dflash.py in the MiniCPM-OCR step, as it does for MiniCPM-V 4.7. Co-authored-by: Cursor <cursoragent@cursor.com>

A DSpark draft can predict a subset of the target vocabulary: its config sets draft_vocab_size below vocab_size, and its checkpoint holds its own lm_head over the draft vocabulary and d2t, which maps draft token i to target token i + d2t[i], as for Eagle-3 drafts. openbmb/MiniCPM-OCR-1.0-DSpark predicts 114,688 of 248,100 tokens. The export loaded such a draft as one over the target vocabulary and failed on the shape of the Markov head's next-token table. Export the draft with its own lm_head, as logits over the draft vocabulary, and d2t as a constant output. Loading checks that the checkpoint has both tensors, which would otherwise keep their initial values. The Markov head is exported over the target vocabulary: its next-token table is zero for the target tokens outside the draft vocabulary, and its candidate scorer keeps at most one candidate per draft token, top_k = min(64, draft_vocab_size). INT4 compression keeps the draft lm_head in INT8, as for other models with an lm_head. DFlash v2 drafts with a pruned vocabulary are rejected. The tests use optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, a tiny draft for optimum-intel-internal-testing/tiny-random-minicpm-ocr laid out like the published one, which predicts 48 of the target's tokens. Co-authored-by: Cursor <cursoragent@cursor.com>

optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark now runs in the export tests and, with optimum-intel-internal-testing/tiny-random-minicpm-ocr as the target, in the GenAI speculative decoding test. GenAI has no video preprocessing for MiniCPM-OCR, so that test passes it an image. A GenAI without pruned-vocabulary support ignores d2t: the target rejects the draft's tokens, and the outputs still match. The DSpark step of the preview workflow installs transformers 5.21 or later, which the target needs. Co-authored-by: Cursor <cursoragent@cursor.com>

The CI CPUs run OpenVINO in bf16 by default, so the test that compares the pruned vocabulary draft and Markov head models with PyTorch now compiles them in f32, like the other DSpark tests. Co-authored-by: Cursor <cursoragent@cursor.com>

…olkit#2099 The DFlash and DSpark CI jobs install optimum-intel at the head commit of huggingface/optimum-intel#2099, which adds the export of drafts with a pruned draft vocabulary to openvinotoolkit#2098, so the pruned vocabulary tests of test_dflash_markov_head_cb.py run instead of skipping. test_vlm_dspark_with_pruned_draft_vocabulary_matches_main_only pairs optimum-intel-internal-testing/tiny-random-minicpm-ocr with its tiny DSpark draft optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, which predicts 48 of the target's tokens: greedy speculative decoding with the Markov head's candidate scorer and with its dense step gives the target's text. The target needs transformers 5.21, so the DSpark job moves from 5.18.0 to 5.21.0. Co-authored-by: Cursor <cursoragent@cursor.com>

rkazants added the Code Freeze label Nov 3, 2026

imargulis force-pushed the dflash2-pruned-vocab branch 5 times, most recently from eaa5baa to 2b15cf6 Compare November 4, 2026 17:47

imargulis force-pushed the dflash2-pruned-vocab branch from 2b15cf6 to a091df7 Compare November 5, 2026 00:51

imargulis mentioned this pull request Nov 5, 2026

Support DFlash-style drafts with a pruned draft vocabulary openvinotoolkit/openvino.genai#4848

Open

3 of 4 tasks

imargulis force-pushed the dflash2-pruned-vocab branch from a091df7 to ec82fa5 Compare November 7, 2026 15:56

imargulis and others added 7 commits November 9, 2026 02:53

imargulis and others added 5 commits November 9, 2026 02:59

Reword the Markov head part of the Eagle-3 dummy generator docstring

ae02558

Co-authored-by: Cursor <cursoragent@cursor.com>

imargulis force-pushed the dflash2-pruned-vocab branch from ec82fa5 to f712a28 Compare November 9, 2026 13:15

imargulis and others added 5 commits November 9, 2026 07:10

imargulis force-pushed the dflash2-pruned-vocab branch from f712a28 to c5221a2 Compare November 9, 2026 16:01

Navigation Menu

Repository navigation

[OpenVINO] Support MiniCPM-OCR-1.0-DSpark: DSpark drafts with a pruned vocabulary - #2099

[OpenVINO] Support MiniCPM-OCR-1.0-DSpark: DSpark drafts with a pruned vocabulary#2099
imargulis wants to merge 18 commits into
huggingface:mainfrom
imargulis:dflash2-pruned-vocab

imargulis commented Oct 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

imargulis commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

How to test

Validation

Tests

Before submitting

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

imargulis commented Oct 30, 2026 •

edited

Loading