Repository navigation
Conversation
Without top-k, TopPFilter built a 16-byte Token for every vocabulary entry (4 MB at a 248k vocabulary), partially sorted it with M = 16, 32, ... 1024 and fell back to a full sort when the nucleus was larger. With the full-vocabulary softmax before it, a sampled row cost about 1.9 ms on the host, about 20 ms when the nucleus exceeded 1024 tokens and about 10 ms when the tail expf results went denormal. Speculative decoding samples several such rows per emitted token, so requests with top-p but no top-k slowed down every method, target-only included. TopPFilter now copies only the tokens whose probability is at least p_max * e^-10 (then e^-20, e^-40, then every token with a probability), sorts them by probability with ties ordered by index, and walks them until the mass exceeds top_p. Each such set is a prefix of the sorted order, so the nucleus is the one a full sort gives. The temperature softmax passes p_max = 1 / sum(exp) through Logits::m_max_probability; the value only changes the number of rounds, never the nucleus. MinPFilter without top-k collects the surviving tokens straight from the probabilities instead of building the full Token vector. The temperature softmax and FullVocabLogSumExpTransform flush exponents below -64 to zero: such terms cannot change a float sum that includes the top token's 1.0, and expf is very slow when its result is denormal. Beam search partially sorts the 2 * group_size tokens it takes instead of the whole vocabulary. Sampled tokens and scores match the previous build on 2,008 CPU generations (top-p, min-p, temperature, top-k + top-p, logprobs, beam search). Per row at a 248k vocabulary, top-p goes from 1.7-26 ms to 0.4-1.6 ms, min-p from 1.9-10.8 ms to 0.6-1.0 ms and a beam from 20-23 ms to 1.6-1.8 ms. Co-authored-by: Cursor <cursoragent@cursor.com>
The logit processor's generated-token count was set once per step, so every position of a speculative validation window saw the count from the start of the window and the min_new_tokens stop-token mask stayed on too long. After a rejected draft token the count was also reset to the prompt length plus the generated length, which lifted the mask too early for the rest of the request. Set the count from the sequence's verified length right before the logits of each position are processed, which is also correct for the parallel samples of a request that share one processor, and reset it to the generated length after a rejection. Co-authored-by: Cursor <cursoragent@cursor.com>
Sequence::update_generated_log_prob replaced a token's log-probability but not the cumulative one, so after speculative validation the score used to rank and report results still counted the draft model's log-probabilities for the accepted tokens. Co-authored-by: Guy Boudoukh <guy.boudoukh@intel.com> Co-authored-by: Cursor <cursoragent@cursor.com>
are_tokenizers_equal encoded a probe string with both tokenizers but compared only the shapes of the results and the special token ids, so tokenizers that split the string into the same number of different tokens were accepted as equal. Co-authored-by: Cursor <cursoragent@cursor.com>
The FastDraft and EAGLE3 strategies each copied the request config for the draft pipeline and adjusted it by hand, and MTP did the same in its make_draft_generation_config(). Use one function so draft-only adjustments apply to every strategy; MTP builds on it and also caps an unlimited max_new_tokens. Also reuse the draft's generated requests in SpeculativeDecodingImpl::step() instead of collecting them twice and discarding the first result. Co-authored-by: Cursor <cursoragent@cursor.com>
LogitProcessor::decrease_generated_token_occurance() left a token whose last occurrence was removed in the generated history with count 0. The repetition and presence penalties apply to every token in the history whatever its count, so a speculative draft kept penalizing the tokens that update_request() removed after the main model rejected them. Erase a token from the history once its count reaches 0. Co-authored-by: Cursor <cursoragent@cursor.com>
Sampled draft tokens were checked with a ratio of the draft's log-probability of the candidate to the target's log-probability of an independently drawn token, a 101-level uniform, and a replacement taken from the next position's logits. The output followed neither model. Record the post-filter distribution q each draft token was sampled from, sparse over the logit processor's candidates or dense, in target token ids. Carry it with the token from the draft pipeline to the main pipeline and verify each window position against the target's post-filter p with the standard test: accept with probability min(1, p/q), otherwise emit a sample of norm(max(0, p - q)) and drop the later candidates. Accepted tokens report the target's log-probability. The draft sampler is seeded independently of the main one, so the uniforms that accept candidates do not reuse the ones that proposed them. Draft and main logits may differ in width, as padded vocabularies do; ids beyond a model's row have zero probability under it. Distributions are normalized and checked with sums in double, since float sums over a full vocabulary, or over a candidate list about as long (top-p or min-p without top-k), drift past the proposal tolerance. Co-authored-by: Guy Boudoukh <guy.boudoukh@intel.com> Co-authored-by: Cursor <cursoragent@cursor.com>
MTP admission and MTP/EAGLE3 streaming asserted greedy decoding even though verification now handles multinomial requests for every CB drafting strategy. Accept multinomial configs there, matching the FastDraft and prompt-lookup streaming checks. Beam search, tree search, parallel sampling and the confidence threshold remain rejected for MTP. Co-authored-by: Cursor <cursoragent@cursor.com>
After a speculative or prompt-lookup step, match_stop_string decodes the last num_tokens_to_validate + longest-stop-string tokens, starting at generated_tokens.size() - draft_generated_tokens. Two cases went wrong: - After an early rejection the sequence can be shorter than the number of validated candidates. Prompt lookup pads its candidates to num_assistant_tokens, so this happens on most first steps. The size_t subtraction wrapped and the buffer started before the generated ids, so tokens read out of bounds were decoded; a stop string found in them made remove_last_tokens remove more tokens than had been generated. - When the sequence was shorter than the window, the check returned without a match. A stop string in a first window that was accepted whole was skipped, and generation ran past it. The window now starts at the first generated token when the sequence is shorter than the window. The out-of-bounds read is pre-existing: master reads out of range in every prompt-lookup stop-string test. It was uncovered here because the draft proposals added to Sequence changed the memory layout, after which this test failed intermittently on Windows with "Cannot remove more tokens than has been generated": test_stop_strings_facebook_opt[PROMPT_LOOKUP_DECODING-multiple_stop_strings_match] The new test_stop_strings_in_first_validation_window covers the skipped check and fails on master with prompt lookup. Co-authored-by: Cursor <cursoragent@cursor.com>
The draft records q normalized in double, but _multinomial_sample
summed its total and running weights in float. A weight below half a
float ulp of the running sum was never drawn, so a draw over most of a
large vocabulary did not follow the recorded q, and verification was no
longer exact. With weights 0.39 and 0.6 followed by 400,000 weights of
2.5e-8, the tail holds 1% of q but got no draws, and against a target of
{0.5, 0.5} the emitted token 1 had probability 0.505.
Sum in double on both paths, deferred expf included. CandidateWeights
keeps a double total, which replaces the separate double normalizer of
the proposal. The uniform stays a float, so the RNG stream is
unchanged. Seeded draws over most of a large vocabulary (temperature
only, top-p near 1) can still pick a different token than before; with
top-k, top-p 0.9 or min-p the tokens were identical in CPU runs.
Co-authored-by: Cursor <cursoragent@cursor.com>
The Python MTP tests were greedy only. Generate 32 tokens with do_sample, top-k and top-p over several verification steps, and check that the streamed chunks concatenate to the final text and that the draft proposed tokens. Co-authored-by: Cursor <cursoragent@cursor.com>
Apply the ruff formatting that the lint job checks on changed lines. Co-authored-by: Cursor <cursoragent@cursor.com>
verify_draft_tree applied the logit processor at every tree depth and to the bonus token without updating its generated-token count, so all of them kept the stop-token mask of the window's first position. Set the count before each depth from the tokens generated before the window and the depths accepted so far, and before the bonus token from the sequence's generated length. Co-authored-by: Cursor <cursoragent@cursor.com>
With prefix caching, verification of models with linear-attention state supports greedy decoding only, because the checkpoint of a verification window is committed from the greedy acceptance result. A sampled MTP request on such a model was admitted and then failed at its first verification window. Reject it in add_request, before either pipeline receives it, and document the limitation. Co-authored-by: Cursor <cursoragent@cursor.com>
When a stop string that is excluded from the output starts the decoded window, nothing is left before it, and the whitespace trim called back() on an empty string. The window clamp checks first windows that were skipped before, so a stop string at the start of the generation reaches this path. An empty result still removes the whole window. Co-authored-by: Cursor <cursoragent@cursor.com>
The sampler indexed d2t with every draft logit without checking its size: when mapping the sampled token, in the EAGLE3 tree search and when recording proposals. Sampler::sample now checks once per call that d2t has an entry for every draft logit. validate_draft_proposal put every sparse proposal into a hash set to reject repeated ids. An EAGLE3 draft without top-k or top-p records its whole draft row, so each candidate paid one insert per draft token. Repeated ids can only come from a d2t that sends two draft tokens to one target id, so set_d2t_for_decoding checks that once and sampled drafting requires it. Greedy drafting accepts any mapping, as before. Validation is now one pass over the probabilities, without allocation. Co-authored-by: Cursor <cursoragent@cursor.com>
The multinomial draw scaled a float uniform by the summed weight. A float in [0.5, 1) moves in steps of 2^-24, so a token whose probability is below that step could be drawn at the wrong rate or not at all, although the weights are summed in double. The draw now takes a double uniform, as the residual draw already does, and so does the acceptance test of rejection sampling. A double uniform takes two outputs of the random engine instead of one, so a given rng_seed gives a different sequence than before. A seed still reproduces its own output. Co-authored-by: Cursor <cursoragent@cursor.com>
Sampled drafting was streamed only in the MTP test. The new test streams a sampled EAGLE3 request on Qwen3-1.7B, checks that the streamed text matches the result and that the draft has tokens accepted. Co-authored-by: Cursor <cursoragent@cursor.com>
imargulis
requested review from
Wovchena,
as-suvorov,
pavel-esir,
sbalandi and
yatarkan
as code owners
November 5, 2026 15:19
github-actions
Bot
added
category: visual language
The draft runs with an unlimited max_num_batched_tokens, so a draft window is never split, and with cache rotation the pipeline sizes its rotation table from that budget, which fails to allocate. The draft windows were also not validated with evicted draft KV. Reject cache eviction on the target or the draft until it is supported. Co-authored-by: Cursor <cursoragent@cursor.com>
…he docs Drafts exported for OpenVINO GenAI 2026.4 or earlier no longer load, and with a VLM target and prefix caching, ChatHistory and image add_request() requests need num_assistant_tokens > 0. Co-authored-by: Cursor <cursoragent@cursor.com>
The draft projects the concatenated target hidden states with an FC before its layers. Running that FC on the target's last_hidden_state makes the rows passed to the draft in_width / out_width times smaller (8x for Qwen3.5-9B: 32768 to 4096 per row), which shrinks the target output, the copy into the draft, and the draft hidden_states input. The FC weight subgraph moves to the target as is, so its decompression markers stay and no weights are copied; the draft FC consumers read hidden_states, which is resized to the projected width. A draft whose hidden_states does not feed only an FC MatMul is rejected, and the width check between the target hidden states and the draft input moves into the transform. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
DFlash rejects return_omni_outputs again: the speculative result path does not fill the Omni outputs, so the request returned empty payloads. add_request again requires one request per tensor, i64 [1, seq_len] token IDs or f32 [1, rows, hidden] embeddings; a [2, N] tensor kept only its first row. A draft step that schedules nothing returns before the forward and keeps the previous inference_duration, so the draft metrics are recorded only when the step scheduled a request. Co-authored-by: Cursor <cursoragent@cursor.com>
insert_tokens_to_sequence takes the draft proposals of the inserted tokens, a parameter that rejection sampling (openvinotoolkit#4769) added; the target's tokens carry none for the draft. Co-authored-by: Cursor <cursoragent@cursor.com>
On pure attention the scheduler splits a validation window that does not fit the token budget and runs only its first rows. The sampler treated the last scheduled row as the bonus row: it validated one candidate fewer than the rows it got, appended the token sampled from that row after the last candidate, and left the candidates past the split in the sequence unvalidated. It also left the unscheduled candidates to validate after a rejection had removed them, so when no new candidates replaced them, as a DFlash draft often proposes none, the target scheduled tokens that no longer existed. Every scheduled row now validates the candidate after it, the candidates past the split wait for the next step, and a rejection leaves nothing to validate. Co-authored-by: Cursor <cursoragent@cursor.com>
Visual token pruning, echo and prompt logprobs stay allowed for a DFlash VLM request: the target runs them as it does without a draft. Omni intermediate outputs are rejected, since speculative decoding results leave them empty. Co-authored-by: Cursor <cursoragent@cursor.com>
The DFlash draft always runs on PagedAttention, but an ATTENTION_BACKEND in the draft properties reached compile_model, where the plugin rejects the property. Drop "PA", and reject any other backend with a message that names the property, before the models are transformed. Co-authored-by: Cursor <cursoragent@cursor.com>
The target compiles with ov::cache_model_path, so the OpenVINO model cache key covers the target graph and directory but not the weight values. Once the draft's hidden-state FC moved into the target, two drafts of the same architecture on one target got the same key, and the second loaded the target compiled with the first draft's FC. On an Intel Arc B390 GPU, a Qwen3-4B draft run after another Qwen3-4B draft with different FC weights dropped from 5.08 to 1.02 tokens per step. The moved FC now carries a hash of its weight constants in its rt_info, which the cache key includes. Co-authored-by: Cursor <cursoragent@cursor.com>
The DFlash draft took the target's hidden rows to end with one row per candidate the target removed, and stopped drafting for the request when there were fewer rows. That holds when the whole validation window runs. A window that does not fit the token budget now validates only its scheduled rows, so a rejection there also removes candidates that never ran. The draft then found too few rows, and the request continued on the target alone. With a budget of 3 tokens and 3 candidates, the draft proposed 3 tokens for a 20-token request instead of 51. The sequence now records the token position of its first hidden row, the DFlash target publishes its rows up to its processed length, and the draft reads its context from the end of them. Co-authored-by: Cursor <cursoragent@cursor.com>
A sequence's hidden state is a view of the output of the forward that ran it, and the next inference writes that output again. A request that the target did not run kept publishing the view. After a draft window that did not run whole, the DFlash draft took its context rows from it, which by then held rows of the requests the target ran since; the bounds check passed, and the draft KV of the request kept those rows for the rest of its generation. Before each forward, the model runner now copies the rows of the sequences that the forward does not schedule. Co-authored-by: Cursor <cursoragent@cursor.com>
The acceptance metrics took the candidates that reached the target in a step, less those rejected in that step, as accepted, and skipped a step that brought none. A window that the token budget split, or that the target did not schedule, is validated over several steps, so a candidate rejected in a later step stayed counted as accepted: with the tiny DeepSpec draft, 4 requests on a budget of 9 tokens reported 31 accepted tokens instead of 29. A request now gathers its candidates and their rejections until the target has none left to validate. Co-authored-by: Cursor <cursoragent@cursor.com>
A window of candidates counted in the acceptance metrics once the draft reported which of them the main model removed. The draft cannot report that after the main request finished or the draft request ended, so the window was dropped: every request lost its last window, and a request whose draft ran out of KV cache, or whose DFlash window could not run, lost the window in progress, with the candidates an earlier step of a split window had accepted. The window now keeps the main request and the candidates as they entered it, and counts once the main model validated all of them or the request finished: the candidates the main sequence still starts with are accepted. A tree window still counts the candidates the draft reports removed. Without dynamic split fuse and with 4 new tokens, each of 4 requests drafts one candidate; only the rejected ones were counted. Co-authored-by: Cursor <cursoragent@cursor.com>
The docs said that OpenVINO GenAI does not limit num_assistant_tokens, but GenerationConfig::validate() rejects more than 100, for DFlash requests too. Co-authored-by: Cursor <cursoragent@cursor.com>
A DFlash draft without properties of its own compiles with the target's, in which GenAI sets ov::cache_model_path to the target directory. With a model path, the OpenVINO model cache keys a compiled model on its graph and that path in place of its weights, so with CACHE_DIR set, drafts that differ only in their weights shared one cached blob: a pipeline could run with the draft of another. A draft that takes the target's properties drops the model path, so its cache key takes its weights, as it does when the draft has properties of its own. Co-authored-by: Cursor <cursoragent@cursor.com>
A tree window counted the candidates the draft reported removed, which it reports only while both requests run. The window that ended a request was dropped uncounted, as on master, where every request lost its last window, and so was a window whose draft request had ended. After it validates a tree, the main sequence keeps the path it accepted: the root, which is the token before the candidates, and the accepted candidates. The main request keeps it after the main pipeline removed the request, so a tree window counts from that path when the main model has validated it, as a linear window counts from the main tokens. The draft reported the same count: the tree nodes less that path. Co-authored-by: Cursor <cursoragent@cursor.com>
A matched stop string removes tokens from the end of a sequence after the sampler validated them: the tokens after the stop string, and the stop string's own tokens unless include_stop_str_in_output is set. The window compared its candidates with the tokens the main sequence kept, so it counted the removed candidates as rejected, and the counts depended on include_stop_str_in_output. Each sequence now counts the candidates the sampler accepts, and a window takes the count from when it began to when the main model validated the last of its candidates, also over the steps of a split window. With the target drafting for itself, a stop string in the first window of a request left 0 of its 5 candidates accepted before. Co-authored-by: Cursor <cursoragent@cursor.com>
…split fuse A DFlash draft window that starts inside the prompt carried only prompt rows, so the first block waited for the next window and every request spent one target step without candidates. Without dynamic split fuse that wait stays: a step runs either prompt or generation windows, and the vLLM prompt phase takes a window only when all of it fits the batch. Dynamic split fuse schedules prompt and generation rows together, so it runs the rest of the prompt and the extra rows of the block as one window, and a window the token budget cuts keeps the candidates of the rows it ran. There the window of the last prompt rows carries the first block. Co-authored-by: Cursor <cursoragent@cursor.com>
Speculative decoding validates a sampled request with rejection sampling, which needs the distribution each candidate was drawn from, and a DFlash request with sampling was rejected. The block sampler draws the candidate rows of a multinomial request with the request's draft RNG after its logit processors, and records the proposal of every sampled candidate for the target to verify. The draft config is the speculative decoding draft config, with its salted seed; it keeps the stop tokens and min_new_tokens, so the draft suppresses the stop tokens as the target does. The draft sync clears the proposals the target has validated. Beam search and parallel sampling are still rejected. With prefix caching, a target with linear-attention state verifies greedy requests only, so add_request rejects a sampled request that drafts, as it does for MTP, instead of failing at its first verification window. Co-authored-by: Cursor <cursoragent@cursor.com>
A DFlash-2 draft can carry a Markov head that scores the candidates of a block from the tokens drafted before them, and its rows differ from the backbone logits by the output multiplier and the soft cap. The block sampler drew every candidate from the gathered backbone logits. The sampler now takes an optional BlockRowSource with the block sampling start offset. It collects the blocks of a step, prepares the source once, and draws every block from the rows the source fills for the token before. A source that runs per position is stepped over the blocks that still draw, and a source can draw a whole block, or per position a token, itself when the request takes its tokens without the logit processors. A position the source has no row for ends the block. The DFlash candidate selector is that source for every DFlash draft. Without a head it scales and soft caps the backbone rows, and a greedy request whose logit processors keep the argmax takes the best token of every row without running them. A Top-K scorer is run once over the packed block rows, and the gated scorer also reads the draft hidden states of those rows; a greedy request whose logit processors keep the argmax walks the lattice greedily, taking the best candidate after the token before, a sampled one walks the lattice with the request's draft RNG, and any other request samples the lattice row after the token before, ending the block at a token outside the candidates. A dense Markov step is run per position and biases the scaled rows by the token drafted before; a greedy request whose logit processors keep the argmax takes the best biased token without filling the row. The selector compiles the head with the draft properties. Every block follows at least one context row, which keeps the blocks of different requests apart in the packed rows that the DFlash-2 convolutions run over. Co-authored-by: Cursor <cursoragent@cursor.com>
The draft metrics of a DFlash draft held the wall-clock time of every draft step, with its backbone inference as inference time, so a report could not tell how much of a step the backbone, the Markov head, the candidate sampling and the rest of the step took. Their batch sizes counted the candidate rows a step forwarded, also when a Markov head ended a block before them, and none of the context rows: a forward of prompt rows alone counted no processed tokens. Report the wall-clock time of every draft step of continuous batching speculative decoding in draft_step_metrics, from the start of the draft step until its candidates are submitted to the target, with the submitted candidates as its batch size. Only steps in which the draft pipeline recorded an inference are counted, so a DFlash draft records the parts of a step with one entry per step: draft_model_metrics holds the backbone forward, draft_markov_head_metrics the Markov head inference of a draft with a Markov head, and draft_sampling_metrics the rest of the draft sampler call, which draws the candidates from the rows the Markov head gives. The rest of a step is scheduling, input preparation and candidate submission. The backbone counts the rows it forwarded, context rows included, as the draft's processed tokens; the Markov head and the sampling count the candidates the step drafted. Co-authored-by: Cursor <cursoragent@cursor.com>
DFlash speculative decoding runs on the continuous batching pipeline only, but with an NPU target, LLMPipeline and VLMPipeline chose the stateful speculative decoding pipeline, which does not support DFlash drafts, and so did LLMPipeline with an NPU draft. Reject a DFlash draft in both pipelines when the target or the draft runs on NPU, with an error that names the supported backends. Co-authored-by: Cursor <cursoragent@cursor.com>
Describe the DFlash, DSpark and DFlash-2 drafts, their export and the markov_head_top_k modes of the Markov head, greedy and sampled drafting, the draft KV cache, the first block under dynamic split fuse, the draft step metrics and the runtime limits of DFlash speculative decoding. Co-authored-by: Cursor <cursoragent@cursor.com>
The DFlash CI job installs optimum-intel at the head commit of huggingface/optimum-intel#2098, which adds the DFlash v2 and DSpark exports to openvinotoolkit#2096, and also runs test_dflash_markov_head_cb.py, which skips its module without them. openvinotoolkit#2098 exports Qwen3.5 with transformers 5.15 or later, so the job moves from 5.2.0 to 5.15.1. test_vlm_dspark_matches_main_only pairs optimum-intel-internal-testing/tiny-random-minicpmv47 with its tiny DSpark draft optimum-intel-internal-testing/tiny-random-minicpmv47-dspark: greedy speculative decoding with the Markov head's candidate scorer and with its dense step (markov_head_top_k=0) gives the target's text. It runs in a new job with transformers 5.18, which the target needs. The job is commented out on Windows, as the other VLM jobs are there (ticket 198295). Co-authored-by: Cursor <cursoragent@cursor.com>
imargulis
force-pushed
the
dflash-exp-markov
branch
from
November 10, 2026 10:55
3c49819 to
3808b07
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Outstanding repository requirements include a tracking issue for the prefix-caching xfail, a dedicated DFlash/DSpark test module with reusable fixtures and CI updates, and completion of the documentation preview and PR checklist prerequisites.
3 open issues
Review effort: Balanced
| return Path(_get_ov_model(main_model_id)), draft_model_path | ||
|
|
||
|
|
||
| def test_vlm_dflash_prefix_caching(cat_tensor): |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds DSpark and DFlash v2 drafts, whose Markov heads make every drafted token depend on the one before it, to DFlash speculative decoding in the continuous batching pipeline. It also adds sampled (multinomial) DFlash drafting, verified with the rejection sampling of #4769.
Stacking
This PR builds on two open PRs, which should land first:
Until they merge, the branch also carries #4769's 19 commits and #4790's 19 commits, rebased onto #4769. Only the 12 commits after "Run the DFlash draft's hidden-state FC in the target" need review (their diff):
The follow-up PR for drafts with a pruned draft vocabulary (MiniCPM-OCR-1.0-DSpark) builds on this one.
Change
Markov heads.
draft_modelloads the Markov head (openvino_markov_head_model.xml) next to the draft backbone and compiles it with the draft properties. The block sampler takes an optionalBlockRowSource, and the new DFlash candidate selector (dflash_candidate_selector.cpp) is that source for every DFlash draft. With a head, it runs in one of two modes, selected by themarkov_head_top_kdraft property:markov_head_top_k=0, DSpark only) corrects every drafted position, one inference per position.The gated Markov head of DFlash v2 also reads the draft hidden states. A DFlash v2 draft on GPU compiles with at least the
ACTIVATIONS_SCALE_FACTORstored in its metadata.Sampled drafting. A multinomial request draws its candidates with the draft's RNG after its logit processors, and records the distribution of every candidate for the target's rejection sampling (Make sampled speculative decoding exact for FastDraft, EAGLE3 and MTP #4769). Beam search and parallel sampling stay rejected. With prefix caching, a target with linear-attention state verifies greedy requests only, so a sampled request is rejected when it is added, as Make sampled speculative decoding exact for FastDraft, EAGLE3 and MTP #4769 does for MTP.
First block under dynamic split fuse. The draft step that runs the last prompt rows of a request also drafts its first block, so a request no longer spends one target step without candidates.
Draft step metrics.
draft_step_metricsholds the wall-clock time of every draft step.draft_model_metrics,draft_markov_head_metricsanddraft_sampling_metricshold its backbone, Markov head and sampling parts, one entry per step.NPU.
LLMPipelineandVLMPipelinereject a DFlash draft when the target or the draft runs on NPU. NPU pipelines use the stateful backend, and DFlash runs on the continuous batching backend only.Docs. The DFlash section of
speculative-decoding.mdxdescribes DSpark and DFlash v2 drafts, the Markov head modes, greedy and sampled drafting, the draft KV cache, the draft step metrics and the runtime limits.Fixes to #4790:
insert_tokens_to_sequence; the target's tokens carry none for the draft.ATTENTION_BACKEND=PAin the draft properties is accepted and dropped beforecompile_model, and any other backend is rejected.Dependencies
Validation
Greedy speculative decoding, one request at a time, on PagedAttention. HumanEval and MT-Bench (first turn), 20 prompts x 2048 new tokens. Speedup is target TPOT over speculative TPOT. B390: Intel Core Ultra X7 358H with Arc B390 iGPU, Windows. B70: Intel Arc Pro B70, Ubuntu 24.04.
num_assistant_tokensOn the B70, DFlash is faster than MTP on the same targets: 1.83x (v1) and 1.78x (v2) by tok/s on HumanEval, and 1.11x (v1) and 1.49x (v2) on MT-Bench.
Tokens per step agree between the two devices within 2.5% in 11 of these 12 rows, and within 5% in all of them. Where a speculative output differs from the target's, it diverges at a near-tie of the FP16 GPU inference. The benchmarks ran on the 2026-11-03 revision of this branch. Since then, the stack was rebased onto newer master and #4769 revisions, and this PR got review fixes; none of them changes greedy drafting.
Known issues
test_target_and_draft_preemption_completexfails when that changes the output, as Support DFlash and DSpark drafts through continuous batching on PagedAttention #4790's preemption test does. Locally it does so for the DSpark drafts on the random tiny models.AI assistance used: yes
If yes: Cursor agents helped write the implementation and tests, and ran the builds, tests and benchmarks; I reviewed every change.
Human validation performed: C++ unit tests, the DFlash and DSpark Python CI jobs locally, and the benchmarks above.
CVS-###
Checklist:
tests/cpp/test_dflash_cb_strategy.cppcovers:markov_head_top_kmodes and their rejections, and the DFlash v2 activation scale;tests/cpp/sampler.cppcovers the block sampler with a row source; for the Support DFlash and DSpark drafts through continuous batching on PagedAttention #4790 fixes, it also covers split validation windows.tests/cpp/scheduler.cppcovers the draft window of the prompt end;tests/cpp/speculative_decoding.cppcoversdraft_step_metrics.tests/python_tests/test_dflash_markov_head_cb.py: DFlash v1, DSpark (both modes) and DFlash v2 drafts, built from a tiny Qwen3, in continuous batching with several requests. Greedy output must match the target alone: with and without dynamic split fuse, staggered requests, chunked prefill, small draft and target KV caches and prefix caching. It also covers reproducible sampled requests, penalties, the draft step metrics,LLMPipelinedrafts without properties, and the NPU, missing-input and missing-Markov-head rejections.tests/python_tests/test_vlm_pipeline.py:test_vlm_dspark_matches_main_only(tiny MiniCPM-V-4.7 with its tiny DSpark draft, both modes) and sampled VLM drafting with prefix caching on a linear-attention target.tests/python_tests/test_continuous_batching.py: reproducible sampled DFlash requests, greedy requests with penalties, and the draft attention backend.test_dflash_markov_head_cb.py, with transformers 5.15.1 for fix llm_bench and wwb parameters for new transformers #2098's export. A new DSpark job runs the MiniCPM-V-4.7 test with transformers 5.18, on Linux only: VLM jobs are commented out on Windows (ticket 198295).speculative-decoding.mdx