Keeping replies flowing while a giant prompt is read: two scheduler changes on our GB300 (Pg64)
Earlier on November 21, 2026 we published why we're not porting Christopher Owen's spark-ds41f to our GB300. It ended with a ladder of his ideas to borrow and a rule: ideas travel, kernels don't. This is the companion: the two ideas that shipped. Both are changes to how vLLM's scheduler hands out engine steps, nothing else, and both are now serving on our DGX Station GB300 as the Pg64 configuration of Garner McCloud's DeepSeek-V4.1-Flash engine. The first, a prefill cadence, is a knob upstream vLLM already ships from v0.32.1; our addition is a gate that applies it only while a long prompt is being read. The second, prefill yield, is Christopher Owen's mechanism, ported from his Spark engine to stock vLLM. What follows is what changed for a user, how a scheduler step works, and what each change costs.
In plain English
- The problem. This box serves several people at once. When one of them sends a giant prompt, a 413K-token document say, everyone else's reply slows to a crawl: about 13 tokens per second, against about 548 when the box has one user. The reply does not stop; it just gets one word in edgewise per big chunk of the document.
- Fix 1, prefill cadence. While a long prompt is being read, two of every three engine steps now go to the replies and only the third reads the next piece of the prompt. Measured: a reply beside a 413K-token prompt went from 12.2–12.5 to 33.1–33.3 tokens per second, about 2.7×. Beside a 1M-token prompt, 10.4–10.6 to 28.7–30.4.
- Fix 2, prefill yield. A short question that arrives while a giant prompt is being read now jumps ahead of the prompt's next big piece instead of waiting behind it. Measured with a probe that sends the question at the same point in the prompt every time: first token in 0.16–0.22 s on three boots, instead of 0.27–0.58 s on the two control boots.
- The price. The giant prompt pays. While a reply streams beside it, a 413K-token prompt finishes 3–8% later than before, depending on how early the reply arrived. With nothing but a tiny probe beside it, or with the box to itself, the prompt's time does not move.
- What did not change. One-user speed, the 16-user and 32-user totals, how fast prompts are read when the box is otherwise idle, and every quality check we run (a 75-task auto-scored set, a 1M-token needle, greedy repeatability, a JSON burst, a tool-call probe) read the same as before, on every boot.
- Status. Experimental. Both changes passed pre-registered bars on two or three separate boots against same-window controls, but there is no 48-hour soak yet and the full quality suite has not run on them. Each is one script away from being rolled back.
- Upstream. The cadence knob is already in vLLM (
--prefill-schedule-interval, single-GPU from v0.32.1). Our lane runs v0.31.0, so that half is a backport. Our addition is the gate that applies it only to long prompts; upstream holds back every prompt, and that version failed our 32-user test. - Bottom line: replies beside a giant prompt run 2.7× faster and short questions get their first token in about a fifth of a second, and the only thing that pays is the giant prompt, by 3–8%, while it shares the box.
Why a reply crawls beside a long prompt
vLLM's scheduler is synchronous: the engine takes one step at a time, and every request that is making progress gets its share of that step. A step is one pass through the model. If the step carries only replies that are already streaming, it is cheap: about 7 ms on this lane, and every reply in it gets a token. If the step also carries a piece of a long prompt, it is expensive. We read long prompts in pieces of 6,144 tokens (--long-prefill-token-threshold 6144), and a step that carries one of those pieces takes about 167 ms when the prompt is 413K tokens long, about 219 ms at 1M. The replies riding in that step still get their token, but they only get it once per step.
The everyday picture is a single checkout lane. One customer has a cart with a few hundred items and the cashier rings them up in big batches; everyone behind gets to put one item on the belt between batches. Nobody is skipped, but a customer with a basket of ten things waits through ten batches. That is the 413K prompt and the reply beside it: one token per 167 ms step, which lands at about 13 tokens per second as measured (12.2 and 12.5 tok/s on the two Pg62 control boots beside 413K; 10.4 and 10.6 beside 1M). Speculative decoding sometimes lands more than one token per step, which is why the measured figure sits above what one token per step would predict; the measured number is the one to use.
The tempting fix is smaller pieces. We measured that earlier in November and it is the wrong currency: the reply does speed up, but the long prompt itself gets much slower, because every piece carries fixed overhead. What we wanted was a way to give the replies more turns without changing how the prompt is cut.
Figure 1. One lane, one step at a time. Top: on Pg62 every step while a long prompt is read carries a 6,144-token piece (~167 ms) and the reply gets one turn per piece. Bottom: with the Pg63 cadence, each piece is followed by two reply-only steps (~7 ms each; drawn wide so they are visible), so the reply gets three turns per piece. Schematic, not to scale.
Fix 1: prefill cadence (Pg63)
The cadence is the checkout lane with a rule: after each big batch, ring up two small customers before the next batch. In scheduler terms, while a request with more than 8,192 uncomputed prompt tokens is in flight, two of every three schedule() calls go decode-only, and only the third carries the next 6,144-token piece. A reply-only step costs about 7 ms, so the two extra turns add about 14 ms to each ~167 ms cycle while tripling the reply's turns. Short prompts and the uncached tail of a cached prompt are never held back; they are not the problem. With no long prompt in flight the counter resets and the scheduler runs the stock code path.
We tested it the way the companion post promised: a same-window control on the same box, two boots, bars written down before the first boot. Card X4 booted Pg62 (A6), then two cadence boots (G1, G2), then Pg62 again (A7), each with the same battery and post-suite. The instrument for the headline row is a trace: a 400-token reply, thinking off, started 2, 4 or 6 seconds into a cold 413K-token prompt, with the reply's tokens counted between its first token and the moment the long prompt finishes; the median of the three arrivals is the number.
Figure 2. Reply speed beside a long prompt, Card X4. Left: beside a 413K-token prompt, Pg62 12.2 / 12.5 tok/s against Pg63 33.3 / 33.1 (2.70× / 2.68× the control mean). Right: beside a 1M-token prompt, 10.4 / 10.6 against 28.7 / 30.4 (2.73× / 2.90×); the 1M cells are one trace per boot.
Beside the 413K prompt the reply went from 12.2 and 12.5 tok/s to 33.3 and 33.1 (2.70× and 2.68×). Beside a 1M prompt, from 10.4 and 10.6 to 28.7 and 30.4 (2.73× and 2.90×), though each 1M cell is a single trace. A short chat sent behind the 413K prompt also benefits a little, because it too gets more turns: its true first token, the first streamed delta from a 32-token probe with thinking on, went from 0.487 and 0.493 s to 0.329 and 0.362 s. Its first answer token, a different instrument that includes the model's thinking decoded at the shared rate, went from 1.298 and 1.461 s to 0.669 and 0.707 s. We name both because they are easy to confuse and they measure different things.
Everything the cadence should not touch held still, within the bars: one-user speed 541.25 and 546.13 tok/s on the controls against 543.76 and 551.15 on the cadence boots; 16-user total 2,438.8 and 2,406.0 against 2,417.2 and 2,397.1; cold 1M prompt reading 27,497.7 and 28,063.6 against 27,880.7 and 28,056.7 tok/s; qeval 71 of 75 on every arm; needle 3 of 3 at 1M; greedy 40 of 40; JSON burst 64 of 64. The cost is in the section below.
The 32-user scare, and checking the instrument first
Pg63 came out of the fourth card we ran that morning, and the story of the first three is the one teaching point we want to keep. The first version (card X1) deferred every prompt, the way upstream's knob does. It found the gain, and it also cost the 32-user burst 6.0%: the math category of that burst read 3,694 tok/s against about 5,000 on the controls, because the short prompts of 32 simultaneous users were being held back too. Gating the cadence to long prompts only (cards X2 and X3) kept the whole gain, but one or both boots in each card still missed the 32-user bar, now by a smaller margin and with the stall visible as one category's 32-request first-token time at about 0.65 s instead of about 0.18 s.
Before changing the cadence again we ran the instrument on the control. Three fresh boots, one with the cadence on, one with the overlay mounted but switched off, and one plain Pg62, each followed by back-to-back 32-user runs: the first run after any fresh boot read about 2,700 tok/s against about 3,950 once warm, on all three, Pg62 included (2,695.7, 2,670.5 and 2,859.3 tok/s on the three cold first runs). On top of that, single-category stalls of 0.3 to 0.74 s turned up at random on all three boots, Pg62 included. One 32-user run per boot could not tell the overlay from that noise: the stall was the instrument, not the overlay. Card X4 therefore measured 32 users five times per boot after warm-up and compared medians, with the controls measured the same way; that change was written into the card before its first boot. The medians read 3,905.6 and 3,928.6 tok/s on the cadence boots against 3,930.0 for the ten control runs (−0.62% and −0.04%). The earlier FAIL verdicts stay in the bundle as recorded. Check the instrument on the control before blaming the change.
Fix 2: Christopher Owen's prefill yield (Pg64)
The cadence helps replies that are already streaming. It helps a newly arrived short question only indirectly: that question still has to wait for its own first step, and if a 6,144-token piece of the giant prompt is next in line, it waits about 167 ms for that piece first. Owen's prefill yield, patch 0545 in his spark-ds41f tree since October 29, 2026, fixes that directly: when a waiting prompt can be read in a single piece (6,144 tokens or fewer), the long prompt's next piece gives up its step to it. A long prompt that has been yielding for 2 seconds takes its piece regardless, so a stream of short questions cannot starve it. In the checkout lane: a customer holding one item is waved through before the next big batch. We ported the mechanism to stock vLLM v0.31.0 on top of the cadence; his sink-rejection half of 0545 is not ported.
We had tested this mechanism once before, earlier in November, and did not promote it: the row we had committed to in advance (the first answer token, which includes thinking) disagreed between its two boots by more than three control boots disagreed with each other. This time we fixed the instrument first. The phase-locked probe P1 sends a short chat, thinking off, at 80% of each rep's own solo time for the 413K prompt, so every rep hits the same point in the prompt, and takes the median of five. The 8,192-token variant, P2, is the control for the design: two pieces, so yield must not help it. Three yield boots (Y1, Y2, Y3) ran between two Pg63 control boots (K1 before, K2 after). The bar, written down first: P1 median at or below 0.8× the pooled control median.
Figure 3. A short chat during a 413K prompt, Cards X5 and X6. Left: phase-locked probe P1 medians in boot order, Pg63 controls K1 0.269 s and K2 0.579 s (pooled 0.327 s, dashed), Pg64 boots Y1 0.222, Y2 0.161, Y3 0.178 s, all under the 0.261 s bar; faint dots are the five reps. Right: the same probe with an 8,192-token prompt, P2 medians 1.01–1.13 s on every boot, not helped by design.
The controls themselves show why the instrument matters: K1's median was 0.269 s and K2's, in the same hour on the same build, 0.579 s, so we pooled them (0.327 s) and set the bar at 0.261 s. All three yield boots cleared it: 0.222, 0.161 and 0.178 s, or 0.68×, 0.49× and 0.55× the pooled control. The older trace instrument, true first token of a 32-token probe behind the 413K prompt (median of 4), agrees: K1 0.418 and K2 0.352 s against Y1 0.211, Y2 0.223 and Y3 0.185 s. The 8,192-token prompt was not helped, as designed: P2 medians 1.10–1.12 s on the yield boots against 1.07 s pooled on the controls. The long prompt's own time on P1 moved +0.2 to +0.8%.
Two more things from these cards. First, the yield did not change decode-beside-prefill (+4.1, −2.6 and −1.4% against Pg63, inside a −5% bar), and the cadence does not change a short chat's first-token time once yield is on: the two mechanisms fix different things and stack, which is why Pg64 carries both. Second, Y2's one-user speed read −2.5% against a −2% bar, with Y1 at −0.5%. One-user speed moves about ±1.5% boot to boot on this lane, so a −2% bar sits inside that noise; by the card's rule the miss was a human decision, and James chose a third boot rather than a promotion or a drop. Y3 read −0.7% and passed every bar. Pg64 was promoted on Y1 and Y3, with Y2 on file.
What it costs, and what didn't change
There is no free step. Every reply-only step is a step the long prompt did not get, so while a reply streams beside it, the giant prompt finishes later. The earlier the reply arrives, the more steps the prompt gives up. In Card X4 a 413K-token prompt finished at 11.30, 11.10 and 11.05 s on Pg62 (mean of A6 and A7) with a 400-token reply arriving 2, 4 and 6 s in; on the cadence boots, 12.14 / 12.18, 11.70 / 11.69 and 11.38 / 11.43 s. That is +7.5 to +7.9%, +5.3 to +5.4% and +3.0 to +3.5% by arrival, and it is where the "3–8%" in the plain-English box comes from. With only a 32-token probe beside it the prompt moved −0.2 to +0.7%; a solo cold prompt, with nothing beside it, moved within ±1%. The reply's own token spacing becomes bimodal by construction, about 7 ms on reply-only steps and about 167 ms on piece steps, so a client's tail latency will look worse than its mean; we did not measure that distribution.
Figure 4. The price, and what held still, Card X4. Left: time for a 413K-token prompt to finish while a 400-token reply streams beside it, by reply arrival; Pg62 mean of A6 and A7 against Pg63 G1 and G2, +7.5…+7.9%, +5.3…+5.4%, +3.0…+3.5% (axis starts at 10 s). Right: the pre-registered bars that held still on G1 / G2.
The right-hand panel is the list of things a scheduler change could have broken and did not: one-user speed +0.0 and +1.4%, 16-user total −0.2 and −1.0%, the warmed five-run 32-user median −0.62 and −0.04%, cold 1M prompt reading +0.4 and +1.0%, qeval 71 of 75 on every arm, the 1M needle 3 of 3, greedy replies 40 of 40 identical within each boot, the JSON burst 64 of 64. Both overlays leave the chunk size, chunk boundaries, kernels, weights, KV cache and sampling exactly as they were; they change only which requests ride in which step.
What we measured and left out
The same window tested five other things from the companion post's ladder. None shipped; each got a row and a reason.
| lever | result | disposition |
|---|---|---|
| vLLM #59131 + #63540 (two merged upstream prefill fixes) as overlays on v0.31.0 | engaged; cold prefill 131K −0.07%, 300K +0.43%, 1M +1.05% | parked: inside the control spread |
| prefill cadence N=2 | 1.83× beside 413K, 1.35× beside 1M (bar 1.4×) | N=3 chosen |
| deterministic combine on the expert sidecar | sidecar refuses to boot with our plans | closed |
| skipping doomed draft rows in speculative decoding | draft survival by position 0.73 / 0.52 / 0.39 / 0.30 / 0.23 / 0.13 / 0.08 | no dead rows worth skipping |
| vLLM #62011 + #61149 (sparse indexer scoring on the FP8 indexer cache) | cache change byte-neutral (3 seeds); the sped-up layers score 2.5% of the indexer's pairs on a cold 1M prompt | parked: decode-only, needs a numerics ruling |
| porting Owen's engine | not attempted | companion post |
The indexer row carries the other teaching point of the week. A desk model of #62011 predicted +30.6% at 1M, because it assumed the four layers it speeds up (24, 28, 32, 36) score every row of the prompt. The probe counted rows and keys: on a cold 1M prompt those four layers score only 15,744 rows, 128 rows in each of 123 scheduler steps, since vLLM's bounded replay (#58132) already trims them, against 250,000 pairs per prompt token on each dense layer and 500,000 on the source layer. That is 2.5% of what the indexer scores, so there was no prompt-reading speed-up to be had here. Count rows and keys per layer before modeling an indexer win.
Upstream, status and how to run it
Upstream. vLLM applies --prefill-schedule-interval N to a single engine from v0.32.1 (#54627, merged October 15, 2026; reworked on main by #66224). Our lane runs v0.31.0, where that machinery is wired only for the data-parallel engine, so the cadence half of our overlay is a backport that reuses stock's throttle_prefills and defer_prefills. What upstream does not have is the long-only gate: the MIN_REMAINING threshold plus ticking only while a long prompt is in flight. Upstream holds back every prompt, which is the X1 arm that failed our 32-user bar. On a rebase to v0.32.1 or later, use the flag and carry only the gate; that combination is not yet A/B'd on v0.34 or later.
Status. Pg63 was promoted at 9:31 AM CST on November 21, 2026 and Pg64 at 12:08 PM; after a planned stop the lane has been serving Pg64 again since 6:43 PM. The recipe is marked experimental. No 48-hour soak has run (the clock restarted at 6:43 PM, so the earliest is Monday, November 23, around 6:44 PM CST), and the T2/T3 quality suite has not run on Pg63 or Pg64; the fidelity evidence so far is the card battery on every arm. The rollback chain is Pg64 → Pg63 → Pg62, one script each; neither new rollback script has been run, though an identical-shape script ran end to end on November 17. A live probe today: every reasoning-effort level returns 200, images and tool calls work, and a single stream with thinking on reads about 309 tok/s wall-clock on a 400-token reply (median of 3 over the LAN). That is a different instrument from the battery's one-user 548 tok/s (thinking off, prompt set v1); the two are not a delta. The lane is back on the Funland Lab guest page as of today.
How to run it. The recipe and its receipts are in J-M-Recipes, bundle results/2026-11-21-pg63-pg64-scheduler/: the two overlay diffs against stock v0.31.0's vllm/v1/core/sched/scheduler.py (patches/cadence3.diff, 92 lines; patches/yieldcad.diff, 370 lines, with SHA256SUMS), the promotion and rollback scripts, the six pre-registered card specs, and the raw traces. With the environment variables unset every added code path is skipped and the scheduler is stock. Pg64 is the overlay plus:
VLLM_PREFILL_CADENCE=3 VLLM_PREFILL_CADENCE_MIN_REMAINING=8192 \
VLLM_PREFILL_YIELD=short VLLM_PREFILL_YIELD_LIMIT_S=2
Engagement is checked in the engine log, not assumed: xfer prefill cadence enabled: N=3 min_remaining=8192 at boot and deferred N of M steps counters afterwards, present on every cadence arm and absent on every control.
Credits and links
Christopher Owen: the prefill-yield mechanism (spark-ds41f, patch 0545, in his tree from commit 47342f26, October 29, 2026); our part is the port to stock v0.31.0 and the measurements. Garner McCloud: the DeepSeek-V4.1 engine both overlays sit on. The cadence reuses vLLM's own prefill_schedule_interval machinery (#54627). DeepSeek for the model; vLLM and NVIDIA for what the engine builds on.
Companion: why we're not porting spark-ds41f to our GB300 (November 21, 2026), where this round was planned. Earlier: prefill yield on our GB300, not promoted (November 17, 2026) and Pg62, the configuration both changes sit on.