Repository navigation
Conversation
Without top-k, TopPFilter built a 16-byte Token for every vocabulary entry (4 MB at a 248k vocabulary), partially sorted it with M = 16, 32, ... 1024 and fell back to a full sort when the nucleus was larger. With the full-vocabulary softmax before it, a sampled row cost about 1.9 ms on the host, about 20 ms when the nucleus exceeded 1024 tokens and about 10 ms when the tail expf results went denormal. Speculative decoding samples several such rows per emitted token, so requests with top-p but no top-k slowed down every method, target-only included. TopPFilter now copies only the tokens whose probability is at least p_max * e^-10 (then e^-20, e^-40, then every token with a probability), sorts them by probability with ties ordered by index, and walks them until the mass exceeds top_p. Each such set is a prefix of the sorted order, so the nucleus is the one a full sort gives. The temperature softmax passes p_max = 1 / sum(exp) through Logits::m_max_probability; the value only changes the number of rounds, never the nucleus. MinPFilter without top-k collects the surviving tokens straight from the probabilities instead of building the full Token vector. The temperature softmax and FullVocabLogSumExpTransform flush exponents below -64 to zero: such terms cannot change a float sum that includes the top token's 1.0, and expf is very slow when its result is denormal. Beam search partially sorts the 2 * group_size tokens it takes instead of the whole vocabulary. Sampled tokens and scores match the previous build on 2,008 CPU generations (top-p, min-p, temperature, top-k + top-p, logprobs, beam search). Per row at a 248k vocabulary, top-p goes from 1.7-26 ms to 0.4-1.6 ms, min-p from 1.9-10.8 ms to 0.6-1.0 ms and a beam from 20-23 ms to 1.6-1.8 ms. Co-authored-by: Cursor <cursoragent@cursor.com>
The logit processor's generated-token count was set once per step, so every position of a speculative validation window saw the count from the start of the window and the min_new_tokens stop-token mask stayed on too long. After a rejected draft token the count was also reset to the prompt length plus the generated length, which lifted the mask too early for the rest of the request. Set the count from the sequence's verified length right before the logits of each position are processed, which is also correct for the parallel samples of a request that share one processor, and reset it to the generated length after a rejection. Co-authored-by: Cursor <cursoragent@cursor.com>
Sequence::update_generated_log_prob replaced a token's log-probability but not the cumulative one, so after speculative validation the score used to rank and report results still counted the draft model's log-probabilities for the accepted tokens. Co-authored-by: Guy Boudoukh <guy.boudoukh@intel.com> Co-authored-by: Cursor <cursoragent@cursor.com>
are_tokenizers_equal encoded a probe string with both tokenizers but compared only the shapes of the results and the special token ids, so tokenizers that split the string into the same number of different tokens were accepted as equal. Co-authored-by: Cursor <cursoragent@cursor.com>
The FastDraft and EAGLE3 strategies each copied the request config for the draft pipeline and adjusted it by hand, and MTP did the same in its make_draft_generation_config(). Use one function so draft-only adjustments apply to every strategy; MTP builds on it and also caps an unlimited max_new_tokens. Also reuse the draft's generated requests in SpeculativeDecodingImpl::step() instead of collecting them twice and discarding the first result. Co-authored-by: Cursor <cursoragent@cursor.com>
LogitProcessor::decrease_generated_token_occurance() left a token whose last occurrence was removed in the generated history with count 0. The repetition and presence penalties apply to every token in the history whatever its count, so a speculative draft kept penalizing the tokens that update_request() removed after the main model rejected them. Erase a token from the history once its count reaches 0. Co-authored-by: Cursor <cursoragent@cursor.com>
Sampled draft tokens were checked with a ratio of the draft's log-probability of the candidate to the target's log-probability of an independently drawn token, a 101-level uniform, and a replacement taken from the next position's logits. The output followed neither model. Record the post-filter distribution q each draft token was sampled from, sparse over the logit processor's candidates or dense, in target token ids. Carry it with the token from the draft pipeline to the main pipeline and verify each window position against the target's post-filter p with the standard test: accept with probability min(1, p/q), otherwise emit a sample of norm(max(0, p - q)) and drop the later candidates. Accepted tokens report the target's log-probability. The draft sampler is seeded independently of the main one, so the uniforms that accept candidates do not reuse the ones that proposed them. Draft and main logits may differ in width, as padded vocabularies do; ids beyond a model's row have zero probability under it. Distributions are normalized and checked with sums in double, since float sums over a full vocabulary, or over a candidate list about as long (top-p or min-p without top-k), drift past the proposal tolerance. Co-authored-by: Guy Boudoukh <guy.boudoukh@intel.com> Co-authored-by: Cursor <cursoragent@cursor.com>
MTP admission and MTP/EAGLE3 streaming asserted greedy decoding even though verification now handles multinomial requests for every CB drafting strategy. Accept multinomial configs there, matching the FastDraft and prompt-lookup streaming checks. Beam search, tree search, parallel sampling and the confidence threshold remain rejected for MTP. Co-authored-by: Cursor <cursoragent@cursor.com>
After a speculative or prompt-lookup step, match_stop_string decodes the last num_tokens_to_validate + longest-stop-string tokens, starting at generated_tokens.size() - draft_generated_tokens. Two cases went wrong: - After an early rejection the sequence can be shorter than the number of validated candidates. Prompt lookup pads its candidates to num_assistant_tokens, so this happens on most first steps. The size_t subtraction wrapped and the buffer started before the generated ids, so tokens read out of bounds were decoded; a stop string found in them made remove_last_tokens remove more tokens than had been generated. - When the sequence was shorter than the window, the check returned without a match. A stop string in a first window that was accepted whole was skipped, and generation ran past it. The window now starts at the first generated token when the sequence is shorter than the window. The out-of-bounds read is pre-existing: master reads out of range in every prompt-lookup stop-string test. It was uncovered here because the draft proposals added to Sequence changed the memory layout, after which this test failed intermittently on Windows with "Cannot remove more tokens than has been generated": test_stop_strings_facebook_opt[PROMPT_LOOKUP_DECODING-multiple_stop_strings_match] The new test_stop_strings_in_first_validation_window covers the skipped check and fails on master with prompt lookup. Co-authored-by: Cursor <cursoragent@cursor.com>
The draft records q normalized in double, but _multinomial_sample
summed its total and running weights in float. A weight below half a
float ulp of the running sum was never drawn, so a draw over most of a
large vocabulary did not follow the recorded q, and verification was no
longer exact. With weights 0.39 and 0.6 followed by 400,000 weights of
2.5e-8, the tail holds 1% of q but got no draws, and against a target of
{0.5, 0.5} the emitted token 1 had probability 0.505.
Sum in double on both paths, deferred expf included. CandidateWeights
keeps a double total, which replaces the separate double normalizer of
the proposal. The uniform stays a float, so the RNG stream is
unchanged. Seeded draws over most of a large vocabulary (temperature
only, top-p near 1) can still pick a different token than before; with
top-k, top-p 0.9 or min-p the tokens were identical in CPU runs.
Co-authored-by: Cursor <cursoragent@cursor.com>
The Python MTP tests were greedy only. Generate 32 tokens with do_sample, top-k and top-p over several verification steps, and check that the streamed chunks concatenate to the final text and that the draft proposed tokens. Co-authored-by: Cursor <cursoragent@cursor.com>
Apply the ruff formatting that the lint job checks on changed lines. Co-authored-by: Cursor <cursoragent@cursor.com>
verify_draft_tree applied the logit processor at every tree depth and to the bonus token without updating its generated-token count, so all of them kept the stop-token mask of the window's first position. Set the count before each depth from the tokens generated before the window and the depths accepted so far, and before the bonus token from the sequence's generated length. Co-authored-by: Cursor <cursoragent@cursor.com>
With prefix caching, verification of models with linear-attention state supports greedy decoding only, because the checkpoint of a verification window is committed from the greedy acceptance result. A sampled MTP request on such a model was admitted and then failed at its first verification window. Reject it in add_request, before either pipeline receives it, and document the limitation. Co-authored-by: Cursor <cursoragent@cursor.com>
When a stop string that is excluded from the output starts the decoded window, nothing is left before it, and the whitespace trim called back() on an empty string. The window clamp checks first windows that were skipped before, so a stop string at the start of the generation reaches this path. An empty result still removes the whole window. Co-authored-by: Cursor <cursoragent@cursor.com>
The sampler indexed d2t with every draft logit without checking its size: when mapping the sampled token, in the EAGLE3 tree search and when recording proposals. Sampler::sample now checks once per call that d2t has an entry for every draft logit. validate_draft_proposal put every sparse proposal into a hash set to reject repeated ids. An EAGLE3 draft without top-k or top-p records its whole draft row, so each candidate paid one insert per draft token. Repeated ids can only come from a d2t that sends two draft tokens to one target id, so set_d2t_for_decoding checks that once and sampled drafting requires it. Greedy drafting accepts any mapping, as before. Validation is now one pass over the probabilities, without allocation. Co-authored-by: Cursor <cursoragent@cursor.com>
The multinomial draw scaled a float uniform by the summed weight. A float in [0.5, 1) moves in steps of 2^-24, so a token whose probability is below that step could be drawn at the wrong rate or not at all, although the weights are summed in double. The draw now takes a double uniform, as the residual draw already does, and so does the acceptance test of rejection sampling. A double uniform takes two outputs of the random engine instead of one, so a given rng_seed gives a different sequence than before. A seed still reproduces its own output. Co-authored-by: Cursor <cursoragent@cursor.com>
Sampled drafting was streamed only in the MTP test. The new test streams a sampled EAGLE3 request on Qwen3-1.7B, checks that the streamed text matches the result and that the draft has tokens accepted. Co-authored-by: Cursor <cursoragent@cursor.com>
imargulis
requested review from
Wovchena,
as-suvorov,
pavel-esir,
sbalandi and
yatarkan
as code owners
November 5, 2026 15:29
github-actions
Bot
added
category: visual language
The DFlash fixtures passed the Hub ID to optimum-cli, which then queries the Hub for every file it loads. Download a snapshot first, with retries, and export from the local path, as the other test helpers do to stay under the Hub rate limits (openvinotoolkit#3348). Co-authored-by: Cursor <cursoragent@cursor.com>
The draft runs with an unlimited max_num_batched_tokens, so a draft window is never split, and with cache rotation the pipeline sizes its rotation table from that budget, which fails to allocate. The draft windows were also not validated with evicted draft KV. Reject cache eviction on the target or the draft until it is supported. Co-authored-by: Cursor <cursoragent@cursor.com>
…he docs Drafts exported for OpenVINO GenAI 2026.4 or earlier no longer load, and with a VLM target and prefix caching, ChatHistory and image add_request() requests need num_assistant_tokens > 0. Co-authored-by: Cursor <cursoragent@cursor.com>
The draft projects the concatenated target hidden states with an FC before its layers. Running that FC on the target's last_hidden_state makes the rows passed to the draft in_width / out_width times smaller (8x for Qwen3.5-9B: 32768 to 4096 per row), which shrinks the target output, the copy into the draft, and the draft hidden_states input. The FC weight subgraph moves to the target as is, so its decompression markers stay and no weights are copied; the draft FC consumers read hidden_states, which is resized to the projected width. A draft whose hidden_states does not feed only an FC MatMul is rejected, and the width check between the target hidden states and the draft input moves into the transform. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
DFlash rejects return_omni_outputs again: the speculative result path does not fill the Omni outputs, so the request returned empty payloads. add_request again requires one request per tensor, i64 [1, seq_len] token IDs or f32 [1, rows, hidden] embeddings; a [2, N] tensor kept only its first row. A draft step that schedules nothing returns before the forward and keeps the previous inference_duration, so the draft metrics are recorded only when the step scheduled a request. Co-authored-by: Cursor <cursoragent@cursor.com>
insert_tokens_to_sequence takes the draft proposals of the inserted tokens, a parameter that rejection sampling (openvinotoolkit#4769) added; the target's tokens carry none for the draft. Co-authored-by: Cursor <cursoragent@cursor.com>
On pure attention the scheduler splits a validation window that does not fit the token budget and runs only its first rows. The sampler treated the last scheduled row as the bonus row: it validated one candidate fewer than the rows it got, appended the token sampled from that row after the last candidate, and left the candidates past the split in the sequence unvalidated. It also left the unscheduled candidates to validate after a rejection had removed them, so when no new candidates replaced them, as a DFlash draft often proposes none, the target scheduled tokens that no longer existed. Every scheduled row now validates the candidate after it, the candidates past the split wait for the next step, and a rejection leaves nothing to validate. Co-authored-by: Cursor <cursoragent@cursor.com>
Visual token pruning, echo and prompt logprobs stay allowed for a DFlash VLM request: the target runs them as it does without a draft. Omni intermediate outputs are rejected, since speculative decoding results leave them empty. Co-authored-by: Cursor <cursoragent@cursor.com>
The DFlash draft always runs on PagedAttention, but an ATTENTION_BACKEND in the draft properties reached compile_model, where the plugin rejects the property. Drop "PA", and reject any other backend with a message that names the property, before the models are transformed. Co-authored-by: Cursor <cursoragent@cursor.com>
The target compiles with ov::cache_model_path, so the OpenVINO model cache key covers the target graph and directory but not the weight values. Once the draft's hidden-state FC moved into the target, two drafts of the same architecture on one target got the same key, and the second loaded the target compiled with the first draft's FC. On an Intel Arc B390 GPU, a Qwen3-4B draft run after another Qwen3-4B draft with different FC weights dropped from 5.08 to 1.02 tokens per step. The moved FC now carries a hash of its weight constants in its rt_info, which the cache key includes. Co-authored-by: Cursor <cursoragent@cursor.com>
The DFlash draft took the target's hidden rows to end with one row per candidate the target removed, and stopped drafting for the request when there were fewer rows. That holds when the whole validation window runs. A window that does not fit the token budget now validates only its scheduled rows, so a rejection there also removes candidates that never ran. The draft then found too few rows, and the request continued on the target alone. With a budget of 3 tokens and 3 candidates, the draft proposed 3 tokens for a 20-token request instead of 51. The sequence now records the token position of its first hidden row, the DFlash target publishes its rows up to its processed length, and the draft reads its context from the end of them. Co-authored-by: Cursor <cursoragent@cursor.com>
A sequence's hidden state is a view of the output of the forward that ran it, and the next inference writes that output again. A request that the target did not run kept publishing the view. After a draft window that did not run whole, the DFlash draft took its context rows from it, which by then held rows of the requests the target ran since; the bounds check passed, and the draft KV of the request kept those rows for the rest of its generation. Before each forward, the model runner now copies the rows of the sequences that the forward does not schedule. Co-authored-by: Cursor <cursoragent@cursor.com>
The acceptance metrics took the candidates that reached the target in a step, less those rejected in that step, as accepted, and skipped a step that brought none. A window that the token budget split, or that the target did not schedule, is validated over several steps, so a candidate rejected in a later step stayed counted as accepted: with the tiny DeepSpec draft, 4 requests on a budget of 9 tokens reported 31 accepted tokens instead of 29. A request now gathers its candidates and their rejections until the target has none left to validate. Co-authored-by: Cursor <cursoragent@cursor.com>
A window of candidates counted in the acceptance metrics once the draft reported which of them the main model removed. The draft cannot report that after the main request finished or the draft request ended, so the window was dropped: every request lost its last window, and a request whose draft ran out of KV cache, or whose DFlash window could not run, lost the window in progress, with the candidates an earlier step of a split window had accepted. The window now keeps the main request and the candidates as they entered it, and counts once the main model validated all of them or the request finished: the candidates the main sequence still starts with are accepted. A tree window still counts the candidates the draft reports removed. Without dynamic split fuse and with 4 new tokens, each of 4 requests drafts one candidate; only the rejected ones were counted. Co-authored-by: Cursor <cursoragent@cursor.com>
The docs said that OpenVINO GenAI does not limit num_assistant_tokens, but GenerationConfig::validate() rejects more than 100, for DFlash requests too. Co-authored-by: Cursor <cursoragent@cursor.com>
A DFlash draft without properties of its own compiles with the target's, in which GenAI sets ov::cache_model_path to the target directory. With a model path, the OpenVINO model cache keys a compiled model on its graph and that path in place of its weights, so with CACHE_DIR set, drafts that differ only in their weights shared one cached blob: a pipeline could run with the draft of another. A draft that takes the target's properties drops the model path, so its cache key takes its weights, as it does when the draft has properties of its own. Co-authored-by: Cursor <cursoragent@cursor.com>
A tree window counted the candidates the draft reported removed, which it reports only while both requests run. The window that ended a request was dropped uncounted, as on master, where every request lost its last window, and so was a window whose draft request had ended. After it validates a tree, the main sequence keeps the path it accepted: the root, which is the token before the candidates, and the accepted candidates. The main request keeps it after the main pipeline removed the request, so a tree window counts from that path when the main model has validated it, as a linear window counts from the main tokens. The draft reported the same count: the tree nodes less that path. Co-authored-by: Cursor <cursoragent@cursor.com>
A matched stop string removes tokens from the end of a sequence after the sampler validated them: the tokens after the stop string, and the stop string's own tokens unless include_stop_str_in_output is set. The window compared its candidates with the tokens the main sequence kept, so it counted the removed candidates as rejected, and the counts depended on include_stop_str_in_output. Each sequence now counts the candidates the sampler accepts, and a window takes the count from when it began to when the main model validated the last of its candidates, also over the steps of a split window. With the target drafting for itself, a stop string in the first window of a request left 0 of its 5 candidates accepted before. Co-authored-by: Cursor <cursoragent@cursor.com>
…split fuse A DFlash draft window that starts inside the prompt carried only prompt rows, so the first block waited for the next window and every request spent one target step without candidates. Without dynamic split fuse that wait stays: a step runs either prompt or generation windows, and the vLLM prompt phase takes a window only when all of it fits the batch. Dynamic split fuse schedules prompt and generation rows together, so it runs the rest of the prompt and the extra rows of the block as one window, and a window the token budget cuts keeps the candidates of the rows it ran. There the window of the last prompt rows carries the first block. Co-authored-by: Cursor <cursoragent@cursor.com>
Speculative decoding validates a sampled request with rejection sampling, which needs the distribution each candidate was drawn from, and a DFlash request with sampling was rejected. The block sampler draws the candidate rows of a multinomial request with the request's draft RNG after its logit processors, and records the proposal of every sampled candidate for the target to verify. The draft config is the speculative decoding draft config, with its salted seed; it keeps the stop tokens and min_new_tokens, so the draft suppresses the stop tokens as the target does. The draft sync clears the proposals the target has validated. Beam search and parallel sampling are still rejected. With prefix caching, a target with linear-attention state verifies greedy requests only, so add_request rejects a sampled request that drafts, as it does for MTP, instead of failing at its first verification window. Co-authored-by: Cursor <cursoragent@cursor.com>
A DFlash-2 draft can carry a Markov head that scores the candidates of a block from the tokens drafted before them, and its rows differ from the backbone logits by the output multiplier and the soft cap. The block sampler drew every candidate from the gathered backbone logits. The sampler now takes an optional BlockRowSource with the block sampling start offset. It collects the blocks of a step, prepares the source once, and draws every block from the rows the source fills for the token before. A source that runs per position is stepped over the blocks that still draw, and a source can draw a whole block, or per position a token, itself when the request takes its tokens without the logit processors. A position the source has no row for ends the block. The DFlash candidate selector is that source for every DFlash draft. Without a head it scales and soft caps the backbone rows, and a greedy request whose logit processors keep the argmax takes the best token of every row without running them. A Top-K scorer is run once over the packed block rows, and the gated scorer also reads the draft hidden states of those rows; a greedy request whose logit processors keep the argmax walks the lattice greedily, taking the best candidate after the token before, a sampled one walks the lattice with the request's draft RNG, and any other request samples the lattice row after the token before, ending the block at a token outside the candidates. A dense Markov step is run per position and biases the scaled rows by the token drafted before; a greedy request whose logit processors keep the argmax takes the best biased token without filling the row. The selector compiles the head with the draft properties. Every block follows at least one context row, which keeps the blocks of different requests apart in the packed rows that the DFlash-2 convolutions run over. Co-authored-by: Cursor <cursoragent@cursor.com>
The draft metrics of a DFlash draft held the wall-clock time of every draft step, with its backbone inference as inference time, so a report could not tell how much of a step the backbone, the Markov head, the candidate sampling and the rest of the step took. Their batch sizes counted the candidate rows a step forwarded, also when a Markov head ended a block before them, and none of the context rows: a forward of prompt rows alone counted no processed tokens. Report the wall-clock time of every draft step of continuous batching speculative decoding in draft_step_metrics, from the start of the draft step until its candidates are submitted to the target, with the submitted candidates as its batch size. Only steps in which the draft pipeline recorded an inference are counted, so a DFlash draft records the parts of a step with one entry per step: draft_model_metrics holds the backbone forward, draft_markov_head_metrics the Markov head inference of a draft with a Markov head, and draft_sampling_metrics the rest of the draft sampler call, which draws the candidates from the rows the Markov head gives. The rest of a step is scheduling, input preparation and candidate submission. The backbone counts the rows it forwarded, context rows included, as the draft's processed tokens; the Markov head and the sampling count the candidates the step drafted. Co-authored-by: Cursor <cursoragent@cursor.com>
DFlash speculative decoding runs on the continuous batching pipeline only, but with an NPU target, LLMPipeline and VLMPipeline chose the stateful speculative decoding pipeline, which does not support DFlash drafts, and so did LLMPipeline with an NPU draft. Reject a DFlash draft in both pipelines when the target or the draft runs on NPU, with an error that names the supported backends. Co-authored-by: Cursor <cursoragent@cursor.com>
Describe the DFlash, DSpark and DFlash-2 drafts, their export and the markov_head_top_k modes of the Markov head, greedy and sampled drafting, the draft KV cache, the first block under dynamic split fuse, the draft step metrics and the runtime limits of DFlash speculative decoding. Co-authored-by: Cursor <cursoragent@cursor.com>
The DFlash CI job installs optimum-intel at the head commit of huggingface/optimum-intel#2098, which adds the DFlash v2 and DSpark exports to openvinotoolkit#2096, and also runs test_dflash_markov_head_cb.py, which skips its module without them. openvinotoolkit#2098 exports Qwen3.5 with transformers 5.15 or later, so the job moves from 5.2.0 to 5.15.1. test_vlm_dspark_matches_main_only pairs optimum-intel-internal-testing/tiny-random-minicpmv47 with its tiny DSpark draft optimum-intel-internal-testing/tiny-random-minicpmv47-dspark: greedy speculative decoding with the Markov head's candidate scorer and with its dense step (markov_head_top_k=0) gives the target's text. It runs in a new job with transformers 5.18, which the target needs. The job is commented out on Windows, as the other VLM jobs are there (ticket 198295). Co-authored-by: Cursor <cursoragent@cursor.com>
A draft with a pruned vocabulary predicts a subset of the target vocabulary: optimum-intel exports its own lm_head as logits over the draft vocabulary and a constant d2t output, which maps draft token i to target token i + d2t[i], as for Eagle-3 drafts. The DFlash pipeline took the draft logits for target-vocabulary rows and kept d2t as a draft output. Read d2t before transforming the draft and remove the d2t output. The draft must be a DFlash v1 draft whose d2t maps its tokens to distinct target tokens, and a Markov head must score target-vocabulary logits with a candidate scorer K no larger than the draft vocabulary. After the gather before the lm_head, widen the draft logits to the target vocabulary, with -inf for the target tokens outside the draft vocabulary, so the Markov head, the draft sampler and the proposals work in target token ids as for any other draft. The candidate selector's soft cap keeps those -inf logits, which tanh would have turned into the lowest capped logit, so those tokens stay undrawable. Document the pruned vocabulary of DSpark and DFlash v1 drafts. The tests of the pruned DSpark draft skip with optimum-intel versions that cannot export it. Co-authored-by: Cursor <cursoragent@cursor.com>
…olkit#2099 The DFlash and DSpark CI jobs install optimum-intel at the head commit of huggingface/optimum-intel#2099, which adds the export of drafts with a pruned draft vocabulary to openvinotoolkit#2098, so the pruned vocabulary tests of test_dflash_markov_head_cb.py run instead of skipping. test_vlm_dspark_with_pruned_draft_vocabulary_matches_main_only pairs optimum-intel-internal-testing/tiny-random-minicpm-ocr with its tiny DSpark draft optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, which predicts 48 of the target's tokens: greedy speculative decoding with the Markov head's candidate scorer and with its dense step gives the target's text. The target needs transformers 5.21, so the DSpark job moves from 5.18.0 to 5.21.0. Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds DSpark and DFlash v1 drafts with a pruned draft vocabulary to DFlash speculative decoding, such as openbmb/MiniCPM-OCR-1.0-DSpark, the draft of MiniCPM-OCR 1.0, which predicts 114,688 of the target's 248,100 tokens.
Stacking
This PR builds on #4847 (DSpark and DFlash v2 drafts, sampled DFlash drafting), and through it on #4790 and #4769. Until they merge, the branch also carries their commits. Only the last 2 commits need review (their diff).
Change
lm_head, as logits over the draft vocabulary, and a constantd2toutput that maps draft tokenito target tokeni + d2t[i], as for EAGLE3 drafts. Before, the DFlash pipeline read those logits as target-vocabulary rows and keptd2tas a draft output.d2tbefore it transforms the draft, checks it and then removes the output. Only DFlash v1 drafts (DSpark included) can have a pruned vocabulary, andd2tmust map the draft tokens to distinct target tokens. A Markov head must score target-vocabulary logits, with a candidate scorer K no larger than the draft vocabulary.lm_head, the draft logits are widened to the target vocabulary, with -inf for the target tokens outside the draft vocabulary. The Markov head, the draft sampler and the proposals then work in target token ids, as for any other draft. The candidate selector's soft cap keeps those -inf logits, so those tokens stay undrawable.speculative-decoding.mdx.Dependencies
optimum-intel-internal-testing/tiny-random-minicpm-ocr-dspark, which is not on the Hub yet, so the job fails until the model is uploaded. Add DSpark and DFlash v2 drafts and sampled drafting to DFlash speculative decoding #4847 'stiny-random-minicpmv47-dsparkis in the same state. Locally the tests ran on a Hub cache that holds both.Validation
MiniCPM-OCR-1.0 int8 target with the MiniCPM-OCR-1.0-DSpark draft, greedy, one request at a time, on PagedAttention. The pages are the 30 OmniDocBench pages with the longest reference text, up to 2048 new tokens each, with
num_assistant_tokens=15. Speedup is target TPOT over speculative TPOT. B390: Intel Core Ultra X7 358H with Arc B390 iGPU, Windows. B70: Intel Arc Pro B70, Ubuntu 24.04.AI assistance used: yes
If yes: Cursor agents helped write the implementation and tests, and ran the builds, tests and benchmarks; I reviewed every change.
Human validation performed: C++ unit tests, the DFlash and DSpark Python CI jobs locally, and the benchmarks above.
Checklist:
tests/cpp/test_dflash_cb_strategy.cpp: the widened draft logits, the malformedd2tand Markov head rejections, the soft cap of the -inf logits, and drafts that stay in the draft vocabulary.tests/python_tests/test_dflash_markov_head_cb.py: a pruned DSpark variant of the tiny draft, with both Markov head modes, in every test of the module. It skips with optimum-intel versions that cannot export it.tests/python_tests/test_vlm_pipeline.py:test_vlm_dspark_with_pruned_draft_vocabulary_matches_main_only, with the tiny MiniCPM-OCR target and its tiny DSpark draft, which predicts 48 of the target's tokens, in both modes.speculative-decoding.mdx, DFlash section. Docs preview: