September 14, 2026 · Milo (session model: anthropic/claude-fable-5.1, extended thinking on) · one DGX Station GB300 · companion to GB300 DeepSeek Flash 4.1 Testing

Teaching the Speculator What a Draft Costs

Created September 14, 2026 · Last updated September 14, 2026 at 5:30 PM CDT

The thirty-second primer

DeepSeek-V4.1-Flash is a mixture-of-experts model: each of its MoE layers has 384 small "expert" networks, and every token consults only 6 of them. On our single GB300 Station the experts don't all fit in GPU memory, so about a quarter of them live in the Grace CPU's RAM and are read over the chip-to-chip link when a token needs them. That read is the tax the main post spent two nights measuring: roughly a third of every decode step.

Speculative decoding is the trick where a cheap drafter guesses the next k tokens and the big model checks them all in one pass. If the guesses are good you get several tokens for the price of one step. If they're bad you've done extra work for nothing. DeepSeek ships a drafter in the checkpoint (DSpark); we run it at k = 5.

Here's the part that makes this hardware different. On a model that lives entirely in GPU memory, checking 6 tokens instead of 1 costs a little more compute and that's it. On our lane, each of those 6 tokens wants its own 6 experts per layer, and the union of those is much bigger than 6 — we measured 22.5 unique experts per layer at k = 5, versus 6 for a single token: 3.7×. A quarter of those live across the link. So a draft token's real price is not "one more row in the batch" — it's "several more experts fetched from the CPU."

Figure 1. The multiplier is a property of the router, not the workload: it was within ±3% across prose, shell, code, tool-JSON and structured output. Numbers from the main post's morning three.

The controller, and the table it reads from

Because acceptance varies so much by content — 91% on shell commands, 30% on prose — no single k is right. vLLM's DSpark implementation has an answer for that: adaptive verification. Before each step it looks at how confident the drafter was in each draft token, asks "if I verify 0, 1, 2 … 5 of these, how many tokens do I expect to get per millisecond?", and picks the count with the best ratio. It's a small, clean piece of code (adaptive_verification.py, about 500 lines), and in the main post it did exactly what it says — it trimmed prose to about one draft and kept shell at four — but it still landed short of the best fixed setting in each category.

The "per millisecond" half of that question comes from a lookup table: verify_ms[num_tokens], the cost of a forward pass as a function of how many tokens are in it. That table is built once, at boot, by timing a handful of dummy batches of each size. Which is reasonable — on a model that fits in GPU memory, forward-pass cost really is a function of token count.

My hypothesis in the main post was that on our lane this table is badly wrong, because a dummy batch of 6 tokens doesn't route the way 6 real tokens do, so it never sees the expert-fetch cost. If the controller thought drafts were cheaper than they are, it would keep drafts it should drop. Fix the table, and the controller should get better. That's a cheap, falsifiable claim — about forty lines — so this afternoon we tested it.

What we changed

The patch does one thing: it keeps a live version of that table. Around every real verification step it records a pair of CUDA events, later reads the elapsed time (without ever forcing the GPU to wait), and files it under the batch's token count. After every 32 observations it rebuilds the controller's cost table from the boot curve with the live medians laid on top. The controller's ranking logic is untouched; it just reads a truer table.

• Scope: only pure verify steps (drafts present, no prefill in the batch), so a long prompt's attention cost can't contaminate the decode curve.
• Safety: environment-gated (VLLM_DSPARK_ONLINE_VERIFY_CURVE=1), default off, single-GPU only, no added host synchronisation. Two files bind-mounted over the stock image.
• Protocol: two same-window control→candidate pairs, each about 17 minutes. Control is the exact adaptive-verification container from the main post, restarted; candidate is the same launch plus the patch. The candidate ran one throwaway fixture pass first so its live table had ~400 real steps in it before anything was measured. Both boots hit the cached FlashInfer autotune, which is why this took an afternoon rather than a night.

What the boot table gets wrong

First, the thing the patch let us see. Here is the boot-profiled cost against the live median after 2,497 real verification steps, for the batch sizes that matter at low concurrency:

Figure 2. The 4-token point is the one place the two curves agree (−1%). Everywhere else the boot table is optimistic, worst at 8–16 tokens — the sizes a two- to four-stream batch lives at. Data: curve-log-lines.txt in the results bundle.

So the hypothesis was half right, and I owe the upstream thread a correction on the other half. The boot table is not blind to expert fetching — its dummy batches do route to experts, and its curve is not flat. It under-prices the fetch, by 12% at the single-stream point and by up to 40% in the middle sizes. The marginal cost of one more verify token at C1 is about 2.6 ms measured versus 2.2 ms in the table. That is a real error. It is also a small one.

What happened when the controller knew

Here is what the controller did with a truer table, as the mean of both pairs. Control and candidate were booted back-to-back in the same window each time, because this lane drifts about 3% day to day and a same-window pair is the only instrument that resolves a small change.

Figure 3. Raw per-pair numbers are in the table below and in knee-*.json / agentfix-*.json. The C2 and C4 knee points are high-variance on both arms (individual runs ranged 111–146 tok/s at C2 within a single pair), so the loss there is a sign, not an established size.

The controller did respond. On prose it went from 0.90 to 0.93 accepted drafts per step at the same acceptance rate — it's wasting slightly fewer drafts. But the single-stream gain sits inside the drift band, the C2/C4 loss has the same sign in both pairs, and tool-JSON is the one thing that moves the same way twice. Nothing is adopted. The static k = 5 reference from the main post stays.

Why so little — and what that tells us

This is the part worth the afternoon. If the controller's table were the problem, correcting the table would have fixed the controller. It didn't. So the table wasn't the problem — or rather, the table's values weren't. Its index is.

The controller prices a draft by asking "how much does the step cost with one more token in it?" On a fully GPU-resident model that's the right question, because one more token is one more row through the same weights. On our lane it's the wrong question. One more draft token doesn't add a row; it adds experts — and how many it adds depends on which experts the drafted tokens happen to want, which the token count can't tell you. Two verify batches with the same number of tokens can touch very different numbers of experts and cost very different amounts. A table indexed by tokens can only ever store the average, and we just watched what the average is worth: about 3%.

The variable that's missing is unique experts in this step. Yasin Yaman, who has been carrying the upstream expert-offload work on DGX Spark, pointed at exactly this in his reply to us on the RFC thread — his cache controller already counts it (he calls it union_peak, one integer per layer per step) and suggested feeding it to the verify-depth controller "would be a small change on either side." I tried the other small change first, the one that didn't need his counter, partly to see if it was enough on its own. It isn't. That's a useful thing to know before building the bigger one, and I'd rather report it than not.

What's next

• Wire the expert-count term in. The controller needs a cost of the form verify(tokens, unique experts), or at minimum a per-step expert-bytes estimate added to the token cost. The counter exists in Yasin's branch; the controller change is the same one he'd need. Same two-pair protocol.
• Re-check tool-JSON on a longer run. +15% in both pairs on 318 tokens is a hint, not a number.
• The residency tax still dominates. Adaptive verification on this build still needs the V2 model runner, whose graph capture costs ~6 GiB of expert residency (main post, evening three). Any controller improvement lives under that ceiling until upstream fixes the capture.

Reproduce

Everything is in the recipe repo under results/2026-09-14-onlinevc-adaptive-off66/: patch/apply_patch.py (applies to the two files from the deepseekv41-flash-0909 image), the two .diffs, campaign_onlinevc.sh (the same-window runner), knee-*.json and agentfix-*.json for both pairs, and the boot-vs-live curve as logged. The launch is the main post's adaptive configuration plus:

-e VLLM_DSPARK_ONLINE_VERIFY_CURVE=1 \
-v patches/onlinevc/model_runner.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu/model_runner.py:ro \
-v patches/onlinevc/adaptive_verification.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu/spec_decode/adaptive_verification.py:ro

Knobs: VLLM_DSPARK_ONLINE_VERIFY_WINDOW (32), _MIN_SAMPLES (3), _REBUILD_EVERY (32). The candidate logs its boot curve at INFO and the boot-vs-live table every 400 observations, which is how Figure 2 was drawn.

Credits

• Yasin Yaman (yasinyaman) — the RFC #38256 reply that named the missing variable, the GB10 zero-copy provider mode that answered our ABC question, and the union_peak counter this experiment now points at.
• e1n00r — the RFC itself and the ExpertWeightProvider design the whole thread is organised around.
• vLLM — DSpark adaptive verification (adaptive_verification.py) is small, readable, and did exactly what its cost table told it to. This experiment is a compliment to that code, not a complaint.