One Line, Thirty-Five Percent: a Batch-Size Schedule for Speculative Decoding
Why one number couldn't win
Two days ago the main post ran a speculation-depth sweep and found something inconvenient. With the drafter guessing five tokens at a time (k=5), agent text — shell, code, tool JSON — decodes a third faster than with k=1, because those tokens are predictable and the drafter is usually right. But at eight or sixteen concurrent requests, k=1 beat k=5 by 23–33% on every point. No single k won both. We shipped k=5 because our lanes are agent lanes, and wrote the multi-stream loss down as the price.
The reason is the one this hardware keeps teaching us. Each verify step on our lane costs experts: a five-token draft window touches 3.7× the unique experts of a single token, and a quarter of those experts live in Grace memory across NVLink-C2C. At one stream, that cost is paid for by the tokens the draft wins. At sixteen streams, the batch already fills the step with real tokens — the GPU is busy, the expert traffic is already high — and speculative drafts are mostly just more experts to fetch for guesses the batch didn't need. The draft that is a bargain when the machine is idle is a tax when it's full.
So the fix isn't a better k. It's a k that knows how busy the machine is.
The schedule
vLLM's speculative config has a field for exactly this, num_speculative_tokens_per_batch_size: a list of (range_start, range_end, k) triples, inclusive, keyed on the number of sequences running in the step. We set:
--speculative-config '{"method":"dspark","num_speculative_tokens":5,
"num_speculative_tokens_per_batch_size":[[1,2,5],[3,16,1]]}'Five tokens at one or two streams, one token at three through sixteen. That is the entire diff from v12 to v13. The drafter, verifier, weights and kernels are untouched; the schedule only changes how many draft tokens are proposed each step, and verification is still lossless, so output quality is by construction the same as v12.
Figure 1. Left of the dashed line the schedule is k=5 and the bars are identical. Right of it the schedule is k=1 and every point gains 26–35%. The C16 candidate measured 430.8 and 427.1 in the two pairs; controls 317.6 and 320.0.
How it was measured
Same protocol as everything else in this series, because the lane's baseline drifts a few percent day to day and a single before/after is worthless here. Each pair is: boot the v12 reference container, run the six-point concurrency sweep (knee.sh, two runs per point, ~1% spread) and the single-stream category fixture; stop it; boot the candidate with only the schedule added; run the same two instruments; stop it. Then do the whole thing again. The bar, set before the run: single-stream within ±1.5% of control and C8/C16 at least +15%, on both pairs.
Both pairs cleared it with room. The candidate's own numbers agree across pairs to within 1% at all six points, which is as tight as this instrument gets. One flag for honesty's sake: the tool-JSON fixture class came in 7% lower than control in both pairs (146.8 and 146.5 vs 157.6 and 158.7) with acceptance identical. That fixture runs single-stream, where the schedule is k=5, so the schedule shouldn't touch it; and across today's four control boots that class ranged 145–159, so it is inside its own spread. Consistent sign though. It's noted in the results README rather than explained away.
A small pleasant surprise: the schedule does not change vLLM's FlashInfer autotune hash. We had budgeted 80 minutes for a fresh MoE kernel tune; the candidate booted in 151 seconds with the cached configs. The schedule changes what the drafter proposes, not which kernels run.
The other result: decode is flat with depth
This lane exists for 1M-token context, and until tonight every decode number we had published was measured on a six-thousand-token prompt. That's a gap. If attention cost climbs with context the way it does on a GPU-resident model, then at a few hundred thousand tokens the story stops being "expert fetch is the bottleneck" and starts being "attention is," and the levers change completely — vLLM's in-flight attention megakernels and sparse-indexer work for this model would jump to the top of the list.
So we measured it. One prompt at each of five depths, streamed, timing from the first output token to the last so prefill and any cache effects are excluded. (The first version of the instrument didn't do that and produced garbage at two depths — it assumed a prefix-cache hit and silently timed a re-prefill. The bad JSON is in the repo next to the good one.)
Figure 2. Decode speed at one stream, k=5, as prompt depth grows 65×. The line stays inside a ±7% band. The 212K point read 183.6 tok/s because the random-word prompt at that seed produced a repetitive continuation the drafter guessed in long accepted streaks (51–60 stream chunks vs ~80 elsewhere); it's a real measurement of an unrepresentative prompt and is excluded from the line.
Flat. Decode at 425K tokens is 121.8 tok/s against 123.8 at 6.5K — a 7% drop over a 65× increase in context. Cold prefill is still 25.8 s at 425K, consistent with the 972K-in-85 s number from the main post; that's the price of a deep context here, not decode.
What this settles: on this lane, at these depths, attention is not the bottleneck, and expert fetch over C2C is — all the way out. The attention-megakernel and sparse-indexer work landing upstream for this model is real engineering, and we'll take it when it ships in an image; but it will not move throughput on this box, and we're no longer going to spend a day building from a moving branch to find that out.
What's next
- Run on v13. The lane's reference container is now
dsv41-vllm-v13-1M-ksched-BOUND-REF; v12 is stopped and kept for rollback. Recipe, launch script (KSCHED=) and results are in J-M-Recipes #16. - Tune the boundary.
[[1,2,5],[3,16,1]]was the first guess and it worked; nobody has tried k=3 in the middle, or switching at C3 vs C4. One boot each, hash hits, cheap. - Watch tool-JSON. If the −7% persists across a third and fourth boot it is real and wants an explanation; if it wanders back, it was the fixture.
--async-schedulingwas also measured tonight: a wash (±3%, hash hit). Not adopted; harmless if a tail-latency reason appears later.- The research-sized idea from the verify-curve post — expert-aware draft selection — is still the only thing on the list that could move single-stream prose. It's a project, and it's now behind a lane that's 35% faster at load than it was this morning.
Reproduce
The recipe's launch script takes the schedule as an environment variable:
MODEL=/models/DeepSeek-V4.1-Flash-df42c109f1defefcbfcedbe7d905718a12266e40 \
TAG=v13-1M-ksched OFFGB=60 UTIL=0.97 SEQS=16 SPEC=dspark:5 KSCHED='[[1,2,5],[3,16,1]]' CTX=1048576 \
EXTRA='--long-prefill-token-threshold 6144' \
bash scripts/launch-dsv41-vllm.shRaw per-boot knee-*.json and agentfix-*.json for all six boots, boot receipts (facts-*.txt), the campaign runner, the depth instrument (v2, and the broken v1), and throughput.csv are under results/2026-09-14-round3-ksched-depth/.
Credits
- vLLM —
num_speculative_tokens_per_batch_sizeexisted before we needed it. This post is a measurement of someone else's good decision. - Fable 5.1 via Nous — the Sept 11 plan review that put the batch-size schedule on the list in the first place, where it sat for three days behind more exotic ideas.