October 26, 2026 · written in one session on deepseek-v4.1-flash (custom provider, reasoning mode unspecified) · one DGX Station GB300 with an RTX PRO 6000 sidecar · explained from the Card BR1 and BR2 receipts on DeepSeek-V4.1-Flash-1017

MTP, DSpark and Block Verification, Explained: Six Tokens per Step

A decoder normally writes one token per pass through the model. Three ideas stack up to write more: MTP gives the model opinions about the next few tokens, DSpark drafts a whole block of them in one parallel pass, and block verification checks that block jointly instead of one token at a time. This is the plain-language version, with the numbers we measured on our live lane this week — including the one config key that bought 2% and the nine-token setting that gave it back.

Created · Updated

THE SHORT VERSION

MTP is extra prediction stages trained into the checkpoint. Each one is a single decoder layer that takes the model's hidden state plus the last token's embedding and guesses a token further ahead. At serving time they are the draft tokens.

DSpark is how this checkpoint drafts a block: one forward pass over N slots (an anchor plus N−1 noise positions), then a cheap left-to-right Markov fix-up so the slots know about each other. The block it was trained on is five.

Block verification is the target model checking all five drafts with one joint test instead of five sequential ones. The output distribution does not change. On our lane it moved us from 3.204 to 3.262 tokens per verify step (+2.0%) and we promoted it as v27.

The same experiment also showed the limit: drafting nine tokens instead of five cost 8–19%, because acceptance falls off a cliff past the block the drafter was trained on. More drafts is not more throughput.

Tokens per verify step3.204 → 3.262+2.0%, block verification on
Best class, one user+5.4%tool-JSON, 347.6 vs 329.9 tok/s
Trained block5dspark_block_size in the checkpoint
k=9 vs k=52.60 vs 2.85tokens per step, prose, one user
Output distributionunchangedflips at the A-vs-A2 floor

1. One token per step

A decoder writes text one token at a time, and every token costs one full pass through the model. At one user, most of that pass is spent waiting on memory: the GPU reads weights — and on this box, the routed expert rows — to produce one column of probabilities. The arithmetic units are mostly idle. The bandwidth is the binding constraint.

That is the opening for speculative decoding. If we can guess the next few tokens cheaply and then have the real model check the guesses, we might get several tokens out of a pass we were going to make anyway.

The shape of it:

  1. A drafter proposes k tokens. Cheap: it is a small model, or a few layers of the big one.
  2. The target model runs one forward pass over all k proposals at once and produces its own distribution at every one of those positions.
  3. The two are compared, and some prefix of the drafts is accepted. The target also gets to emit one token of its own at the first position the draft got wrong.

Two things make this worth doing. One forward pass over k draft positions costs almost the same as one forward pass over one position at one user, because the batch is tiny and the step is memory-bound. And the check is lossless: done right, the text you emit is distributed exactly as if the target had sampled it token by token, with no drafter in the loop at all. You are not trading quality for speed. You are trading idle bandwidth for speculative work.

Everything in this post is one of those three parts. MTP is where the drafts come from in this checkpoint. DSpark is how the drafter proposes a whole block in one pass. Block verification is how the target checks the block.

PieceWho does itOne-line version
MTPThe checkpointExtra prediction stages, trained to guess tokens further ahead. They become the drafter.
DSparkThe drafter's serving modePropose N tokens in one parallel pass, then bias them left-to-right with a cheap Markov head.
Block verificationThe target's samplerAccept the longest prefix with one joint probability test instead of N sequential ones.

2. MTP: drafts in the checkpoint

Multi-token prediction started as a training trick. A normal decoder is supervised on one next token per position. MTP adds small extra stages on top of the main stack and supervises each one on a token further ahead.

Mechanically, a stage takes two inputs: the hidden state the main model produced, and the embedding of the last token. Both are RMS-normed, concatenated, and projected back down to hidden size. That vector runs through one decoder layer, then through the model's shared output head. Stage 0 is trained to predict the token after next, stage 1 two after that, and so on.

This checkpoint ships three of them — mtp.0, mtp.1, mtp.2 — inside the same weight files as the main model. At serving time you have a choice: ignore them and serve a normal model, or use them as the drafter. We use them.

The catch is that the chain is serial. Stage 0 needs the embedding of the token the target just produced. Stage 1 needs the token stage 0 produced. So k drafts cost k small forward passes, one after another. Each is cheap — one layer, not forty — but there is no parallelism in it.

That is a fine trade when the target pass dwarfs a draft pass. It stops being fine when you want ten drafts: ten serial small passes start to look like a second model. Width is the thing MTP cannot give you. That is the gap DSpark fills.

Three ways to spend one decode step: target only, MTP chain, DSpark block Top row: a target-only decoder runs one forward pass per token. Middle row: an MTP chain runs three small serial draft passes then one target pass, producing up to four tokens for four passes. Bottom row: DSpark runs one parallel draft pass over five slots then one target pass, producing up to six tokens for two passes. Same user, same step budget · what one pass buys you Target only 1 pass → 1 token target fwd target fwd target fwd 3 tokens, 3 passes MTP chain (k=3) 4 passes → up to 4 d0 d1 d2 verify serial: 3 small passes, one after another DSpark (k=5) 2 passes → up to 6 anchor 4 noise slots one parallel draft pass verify 5 draft tokens + 1 target token target forward drafter forward verify forward draft token · target’s own token (at least one per step) Counts are passes, not milliseconds. Whether the drafts survive is a separate question: section 6.

Figure 1. The three designs. MTP shortens each draft pass but keeps the chain serial, so k drafts cost k passes. DSpark packs all k drafts into one pass; the price is that the slots are drafted in parallel and cannot see each other, which section 3 fixes at sampling time.

3. DSpark: draft a block, then fix it

DSpark keeps the same weights and removes the serial chain. One draft pass produces the whole block.

One pass, N slots

The drafter is handed N query positions at once: an anchor — the position of the last token the target accepted — plus N−1 "noise" slots after it. Context KV is precomputed for the existing tokens, so the drafter runs a single forward pass over that query block and returns N distributions. Slot 0 predicts the token after the anchor; slot 1 the token after that; and so on. Every slot is a prediction, which is why a k=5 round hands the verifier five drafts and not four. In the engine this is the sample_from_anchor layout.

The price of parallelism

Slots cannot see each other. Slot 3 predicts "the token after anchor+3" without knowing what slot 2 sampled. If slot 2's token is wrong, slot 3 was never predicting the right distribution — it was conditioning on the anchor plus noise. This is the structural weakness of every parallel drafter: accuracy decays as the block grows, not because the model is bad but because the conditioning degrades.

The Markov fix-up

DSpark patches this at sampling time. Instead of sampling all five slots independently, it walks them left to right and biases each slot's logits with a prefix-dependent Markov term derived from the token sampled one step earlier. It is sampling-time arithmetic, not another model forward, and it runs inside the same CUDA graph as the draft pass. The block keeps its one-pass cost and recovers most of the intra-block dependency.

One more serving detail that matters for correctness: the drafter scores a reduced vocabulary (dspark_draft_topk) and scatters those logits into target-vocabulary columns. That is only valid on the probabilistic path — the drafter has to know probabilities for the target's rejection test, not just an argmax.

The number to remember: five

The checkpoint says dspark_block_size = 5. That is the block shape the drafter was trained on. It is not the number of MTP stages (three), and it is not a limit the engine enforces: vLLM will run k=9 if you ask it to, which is exactly how we broke it. Five is the width the model knows. Section 6 is what happens outside it.

Inside one DSpark round: one parallel draft pass, then a serial Markov sample The last accepted hidden state is projected, then copied into five query slots, an anchor and four noise positions. One forward pass over those slots produces five distributions. A sequential Markov head then walks the slots left to right, biasing each distribution with the previously sampled token, and emits five draft tokens for the target to verify. last accepted token + target hidden state h + context KV (precomputed) main_proj A anchor n1 n2 n3 n4 noise ×4 N = k = 5 query slots anchor predicts draft 1 draft backbone · one forward, all 5 slots slots do not see each other 5 distributions reduced draft vocab slot 0 no prior slot 1 bias(t₀) slot 2 bias(t₁) slot 3 bias(t₂) slot 4 bias(t₃) Markov head · serial, sampling only (no model forward) 5 draft tokens → one target verify pass (+1 token, section 4) trained block size = 5 · k=9 leaves the trained shape Parallel where it is expensive (the backbone), serial where it is nearly free (one categorical sample at a time).

Figure 2. One DSpark round. The expensive part — the transformer pass — runs once over all N slots. The cheap part — turning distributions into tokens — runs left to right so slot i knows what slot i−1 picked. The draft vocabulary is reduced, which is why the drafter has to produce probabilities and not just a token.

4. Verification: the guarantee

The drafter is a proposal engine. The target still decides, and the decision has to be exact: the emitted tokens must be distributed as if the target had sampled them itself, one at a time, with no drafter in the loop. If that is not true, you are not serving a faster version of the model. You are serving a different model.

Token by token

The classic rule (Leviathan et al., 2023) walks the drafts in order. For draft token xi, let pi be the target's distribution at that position and qi the drafter's. Accept it with probability

min(1, p_i(x_i) / q_i(x_i))

On the first rejection, sample from the normalized residual max(0, p_i − q_i) and stop: that token is the target's own, and nothing after it can be used. This is exact, and it is inherently sequential — the test at position i only makes sense if position i−1 passed.

Block verification

Block verification (Sun et al., 2024) verifies the block jointly instead of one token at a time. The kernel computes a cumulative joint ratio of the draft prefix and a residual mass, then a threshold per position:

prefix_joint = exp(cumulative_log_p[i])                 # joint p/q of the drafted prefix
h = residual_mass / (residual_mass + 1 - prefix_joint)  # last slot: h = prefix_joint
accepted_len = last i such that u_i <= h_i

The paper proves the resulting accept-length distribution is optimal in expected tokens per iteration, is never worse than the token-level test, and leaves the output distribution unchanged. Nothing about the model changes. One accept rule replaces another, and the verifier stops throwing away a whole block because the first independent coin came up tails.

Greedy is a different code path

Here is the detail that cost us a measurement window. In the vLLM image we run, the block branch sits after the greedy branch:

if is_greedy:                        # temperature == 0
    accepted = target_argmax == draft_sampled
elif USE_BLOCK_VERIFICATION:
    ...                              # joint test above
else:
    accepted = target_logprob > log(u) + draft_logprob

So at temperature 0 the flag is not a small effect. It is zero: the greedy path never reaches the block code. Our first window was a stock temperature-0 benchmark and it showed nothing, correctly. The primary window had to be temperature 1.0 with the seeded sampler, because that is the path the feature actually occupies — and, on this lane, the path our agent traffic takes.

Two ways to verify the same five drafts: token by token versus jointly Left: five draft tokens tested in order with a probability ratio each; the first failure ends the block and the target resamples from the residual distribution. Right: the same five drafts tested with one joint ratio and residual mass per position, giving a threshold for each; the accepted length is the last position whose uniform draw passes. Token by token sequential: each test needs the one before it Block verification joint: one ratio per position, decided together x1 x2 x3 x4 x5 min(1,p/q) min(1,p/q) fail — — resample from residual max(0, p − q), then stop two drafts kept, one target token, three passes thrown away the block is lost on the first independent miss joint prefix ratio + residual mass → h₁ … h₅ computed over the whole block, in one reduction h₁ h₂ h₃ h₄ h₅ u₁ ≤ h₁ u₂ ≤ h₂ u₃ ≤ h₃ miss miss accepted length = 3 optimal in expected tokens, same output distribution a later position can survive an earlier miss, because the test is on the block Schematic, not our data: the point is the grain of the decision, per-token versus per-block. Section 5 has what it changed on our lane.

Figure 3. The two verifiers. Both are lossless. The difference is whether a bad draft at position 3 destroys the whole block (left) or is absorbed into a joint threshold that can still accept a longer prefix (right).

5. What we measured

The lane: :30006 on our DGX Station GB300 with the RTX PRO 6000 sidecar, serving DeepSeek-V4.1-Flash-1017 on image nightly-af7f9488, KV pool 4,400,013 tokens, DSpark with the production k-schedule (k=5 at 1–4 streams, k=3 above that).

Card BR1 is a bracketed A → B → A2 test with exactly one key added:

# production spec config + one key
{"method":"dspark","num_speculative_tokens":5,
 "num_speculative_tokens_per_batch_size":[[1,4,5],[5,8,3],[9,32,3]],
 "draft_sample_method":"probabilistic",
 "rejection_sample_method":"block"}      <-- the only change

All three arms booted as autotune hash hits (528–529 s), because rejection_sample_method is not one of the inputs to the compute hash. The flag was verified in the container args on B only (0 / 1 / 0). Every arm came up with the same KV pool and zero foreign clients: the lane was firewalled to loopback while dark.

The primary window is what matters: 40 real transcripts per class, 256 forced tokens, temperature 1.0, top_p 1.0, seeded, two rounds. The stock temperature-0 window was kept as a parity gate, not as evidence.

Decode tok/s, temp 1.0C1 AC1 A2C1 BB ÷ ctrlC8 AC8 A2C8 BB ÷ ctrl
prose266.7270.2278.31.037972.4978.7983.01.008
structured302.2300.8305.31.0131,059.81,024.81,068.21.025
code352.8353.0357.31.0131,173.91,159.61,221.11.047
shell_ops374.0385.5378.10.9961,194.21,245.81,320.31.082
tool_json333.4326.4347.61.0541,184.11,156.01,202.51.028

The two controls agree at C1 within 0.1–3.0% per class, which is the width of the noise floor we are working above. Read the aggregate before the per-class cells: 3.204 / 3.262 / 3.189 tokens per verify step for A / B / A2, a ratio of 1.020 against the control mean. That is the cleanest signal in the card, because everything else is downstream of it.

THE VERDICT WE KEPT

The pre-registered verdict for BR1 was FAIL. Two instrumentation defects, both ours: the env check compared the container environment as an ordered list (Docker returns the same 44 variables in varying order, and the sets were identical on every arm), and one gate used a ±2% bar that was tighter than the class's own r1/r2 spread. Re-scored with the order-insensitive compare and a bar that honours the controls' spread: PASS. Both verdicts are on the record. The change did not fail; the instrument did — and the fix is a re-score with the reason written down, not a quiet delete.

Block verification was promoted to the production lane as v27 at 04:08 CDT on October 26, 2026. One key, a hash-hit boot, a rollback pair stopped-and-kept.

Where block verification raises acceptance: positions two through five Two curves of accepted-at-position over drafts at temperature 1.0 across all five traffic classes. The control and the block arm are identical at position one, 0.793 and 0.795. The block arm is higher at positions two to five, and the five differences sum to the measured 0.058 extra tokens per verify step. 0 0.20 0.40 0.60 0.80 accepted ÷ drafts p1 p2 p3 p4 p5 position in block .793 .604 .463 .191 .153 .617 .484 .202 .163 token-by-token (A, 3.204 tok/step) block verification (B, 3.262 tok/step) position 1 is unchanged: the first accept decision is the same either way. the five gaps sum to +0.058 tokens per step, which is the whole measured gain. dsv41-flash lane, temp-1.0 window: all five classes, served k-schedule (mean 4.1 drafts/step), so p4–p5 come from single-stream steps.

Figure 4. What block verification changed on our lane, per position. It does not make the first draft more likely to be right. It makes the later drafts of an otherwise-good block more likely to survive. That is a 2% effect, and it looks exactly like a 2% effect.

6. Why five beat nine

Card BR2 ran A (production) → K9 (block verification plus num_speculative_tokens 9) → A2 (production again). Note what the arms are: the controls have block verification off, so the K9 ratios measure the pair — nine drafts and joint verification — against production. The k effect is not isolated. That is a flaw in the experiment and it is written down as one.

K9 booted as a hash hit too (529 s): num_speculative_tokens is not in the autotune hash on this image either. Nothing about the engine stops you from drafting past the trained block. The engine has no opinion; the model does.

K9 ÷ mean(A, A2), temp 1.0prosestructuredcodeshell_opstool_jsonmixed
C10.8130.8600.8680.9180.8530.793
C80.8960.8910.9250.9300.882—
tokens/step at C1, k=52.853.033.833.983.632.08
tokens/step at C1, k=92.602.943.664.043.451.82

Nine drafts lost throughput in every class we serve. Look at the prose row: 2.85 tokens per step at k=5, 2.60 at k=9. Drafting four more tokens per step produced fewer tokens per step. That is the whole story, and the per-position histogram says why.

The arithmetic you need before tuning k

Let ap be the chance that a step accepts a draft at position p, counted over all drafts of that position. Then:

marginal_p   = a_p / a_(p-1)          # given you reached p, chance p is right
tokens/step  = 1 + Σ_p a_p            # the +1 is the token the target always samples

Both lines are checkable against the engine's own counters, and both are the reason k is not a free parameter. With prose at one user, k=5 gives

1 + (0.717 + 0.476 + 0.310 + 0.205 + 0.145) = 2.85

and k=9 gives

1 + (0.612 + 0.379 + 0.235 + 0.147 + 0.097 + 0.063 + 0.036 + 0.020 + 0.011) = 2.60

More terms, and a smaller sum. Positions 6 through 9 contribute 0.130 tokens between them.

Marginal acceptance per position for k=5 and k=9 on prose at one user Two curves of the chance a position is accepted given the step reached it. With k=5 the curve stays between 0.65 and 0.72 across all five positions. With k=9 it starts lower at 0.61 and falls to about 0.55 after position six. A dashed line marks the trained block size of five. 0.50 0.55 0.60 0.65 0.70 0.75 marginal acceptance trained block = 5 1 2 3 4 5 6 7 8 9 position in block .717 → .612 even position 1 gets worse 0.55–0.57 past the block k=5 → 2.85 tokens/step prose C1 · 271 tok/s k=9 → 2.60 tokens/step prose C1 · 222 tok/s (−19%) prose, C1, temp 1.0; marginals from the per-position counter step time +11% on the peer 2K/C1 cell (11.0 → 12.2 ms)

Figure 5. Marginal acceptance per position, prose at one user. With k=5 the curve is flat through the block — position 5 is still accepted about 70% of the time when the step gets there. With k=9 the curve starts lower and breaks after position 6, exactly where the trained block size ends.

Two cliffs, not one

The first cliff is at position 7: marginal acceptance drops from about 0.65 to about 0.55 and stays there. The drafter is extrapolating: slot 7 and beyond were never in its training distribution, and the Markov bias is now conditioning on a longer and longer fabricated prefix.

The second cliff is less obvious and more interesting: position 1 gets worse too — 0.717 at k=5, 0.612 at k=9. The anchor's own prediction is degraded by drafting a wider block, because the query shape changed. The drafter was trained to read a five-slot window; you handed it nine. Nothing about the first slot's inputs changed except its company. That is why the k=9 curve never recovers: it is not a decay, it is a different (worse) operating point plus a decay.

But the throughput fell 19%, not 8.9%

Because throughput is not tokens per step. It is

tok/s = tokens/step ÷ seconds/step

and the step got slower as well as less productive. On the peer harness's single-stream 2K codegen cell, step time went from 11.0 ms to 12.2 ms (+11%): nine slots to draft, nine logit positions to verify, and — we think — a wider expert union per step. Two terms moving in opposite directions, one product that lands at 0.813.

Where nine drafts did pay

Same config, different workload. On the peer harness's padded 64K single-stream codegen cell, K9 read 489 tok/s against our 453 (+8%), and tokens per step went from 5.05 to 6.05. At 2K it was flat: 468 vs 466, with tokens/step up 5.13 → 5.70 and step time up 11%.

Our reading, and it is a reading rather than a measured decomposition: on a long single-stream prompt the step is dominated by weight and KV traffic that the wider batch was paying for anyway, so extra draft slots are close to free; and code is the class where marginal acceptance is highest in every arm we ran. Both conditions are needed. Real agent traffic at one user has neither.

One result we could not explain away: at C8 both arms draft three tokens for most steps (the schedule drops k above batch 4), yet K9 still read 0.88–0.93×. That should have been close to parity. One candidate is the larger captured CUDA-graph footprint (32 × (k+1) = 320 tokens of capture on K9 against 192 on A, both above the 128-entry list), and we have not measured it. Flagged, not explained.

What we changed: nothing. k stays at 5 with an effective 3 above batch 4. Block verification — the change that cost nothing and gave 2% — is what got promoted.

7. Five rules I would carry to the next lane

  1. Match k to the trained block, not to your ambition. The checkpoint told us five (dspark_block_size). The engine will let you run nine, the model will let you down, and the cost is a 19% throughput loss on the traffic we actually serve. Read the trained width before you tune the serving width.
  2. Benchmark at the temperature you serve. Block verification is a no-op at temperature 0 by construction, not by accident. A greedy benchmark cannot see it, and a temperature-0 result of "no change" is not evidence against the feature — it is evidence that you measured the wrong branch.
  3. Watch tokens per step, then step time. k is an input, not a KPI. The two multipliers move independently, and a config can win one and lose the other. Print both.
  4. Bracket every change A → B → A2. Two controls turn "it feels faster" into "it is faster than the same lane an hour ago, and the two controls agree within 3%". Ours also caught the defect in our own gate.
  5. Keep the failed verdict, then re-score it in public. Our pre-registered verdict was FAIL, on instrumentation, not on the change. Overwriting it would have been easier and would have taught us nothing. The re-score is the interesting artifact.

8. What this does not prove

Glossary

Drafter / speculator
The cheap model that proposes tokens. Here, the MTP stages in the checkpoint, run through the DSpark speculator.
Target
The full model. It always has the last word, and its sampler is what makes the output exact.
Anchor
The position of the last token the target accepted. It is slot 0 of the next draft block and the only slot with real context behind it.
Noise slot
A query position in the draft block that has no real token yet. It is a prediction slot, which is why the drafter has to guess them all at once.
Verify step
One target forward pass over the draft block plus the accept decision. A step always emits at least one token — the one the target sampled itself (the "bonus" token).
ap
Fraction of drafts that get accepted at position p, over all drafts of that position. Summing ap over p and adding 1 gives tokens per verify step.
Marginal acceptance
ap ÷ ap−1: given the step reached position p, how often is p right. This is the curve that tells you where to stop drafting.
Block size
The draft width the drafter was trained at. Five here. Not a cap the engine enforces, and not the number of MTP stages.
Hash hit
The kernel autotune cache matched the config, so the boot skipped re-tuning. 528 s instead of a much longer tuning boot. Useful as a side check that a flag did not change the compute path.

Credits

Block verification is Sun, Mendlovic, Leviathan, Aharoni, Ro, Beirami and Suresh, "Block Verification Accelerates Speculative Decoding" (2024). Token-level speculative sampling is Leviathan, Kalman and Matias (2023). DSpark and the MTP stages ship in the DeepSeek checkpoint; the serving implementation is vLLM's dspark speculator. The peer harness we compare against is Fraser Price's DeepSeek-4.1-Flash-1017-BRRR recipe and bench, Apache-2.0, ported to stdlib; the expert-sidecar hook is original-el8's. DeepSeek for the checkpoint. James for the box, the lane and the promotion call.

Receipts: results/2026-10-26-br1-block-verification/ and results/2026-10-26-br2-k-ladder-peerbench/ in the GB300 sidecar recipe. Related: One GB300 + one RTX PRO 6000 against four RTX PRO 6000s (the peer-harness post these two cards feed) · The sidecar lane itself · Adaptive Residency, Explained · GB300 topic page.