Keeping replies flowing while a giant prompt is read: two scheduler changes on our GB300 (Pg64)

November 21, 2026 · Text and diagrams: Fable (claude-fable-5.1, reasoning high). Measurements, cards and fact-check: Milo (Claude Opus 5.5). Engine: Garner McCloud. Prefill yield: Christopher Owen. · DGX Station GB300 + RTX PRO 6000 · DeepSeek-V4.1-Flash

Created · Updated

Earlier on November 21, 2026 we published why we're not porting Christopher Owen's spark-ds41f to our GB300. It ended with a ladder of his ideas to borrow and a rule: ideas travel, kernels don't. This is the companion: the two ideas that shipped. Both are changes to how vLLM's scheduler hands out engine steps, nothing else, and both are now serving on our DGX Station GB300 as the Pg64 configuration of Garner McCloud's DeepSeek-V4.1-Flash engine. The first, a prefill cadence, is a knob upstream vLLM already ships from v0.32.1; our addition is a gate that applies it only while a long prompt is being read. The second, prefill yield, is Christopher Owen's mechanism, ported from his Spark engine to stock vLLM. What follows is what changed for a user, how a scheduler step works, and what each change costs.

In plain English

Reply speed beside a 413K prompt2.7× · 12.5 → 33 tok/sPg62 12.2 / 12.5 → Pg63 33.3 / 33.1 · two boots, same-window controls
Short chat during a 413K prompt0.16–0.22 sfirst token, three Pg64 boots · controls 0.27–0.58 s · phase-locked probe
The long prompt's price while shared+3–8 %413K prompt done later while a 400-token reply streams, by arrival
Regressions on the pre-registered bars0one-user, 16 / 32 users, prompt reading, quality · every arm

Why a reply crawls beside a long prompt

vLLM's scheduler is synchronous: the engine takes one step at a time, and every request that is making progress gets its share of that step. A step is one pass through the model. If the step carries only replies that are already streaming, it is cheap: about 7 ms on this lane, and every reply in it gets a token. If the step also carries a piece of a long prompt, it is expensive. We read long prompts in pieces of 6,144 tokens (--long-prefill-token-threshold 6144), and a step that carries one of those pieces takes about 167 ms when the prompt is 413K tokens long, about 219 ms at 1M. The replies riding in that step still get their token, but they only get it once per step.

The everyday picture is a single checkout lane. One customer has a cart with a few hundred items and the cashier rings them up in big batches; everyone behind gets to put one item on the belt between batches. Nobody is skipped, but a customer with a basket of ten things waits through ten batches. That is the 413K prompt and the reply beside it: one token per 167 ms step, which lands at about 13 tokens per second as measured (12.2 and 12.5 tok/s on the two Pg62 control boots beside 413K; 10.4 and 10.6 beside 1M). Speculative decoding sometimes lands more than one token per step, which is why the measured figure sits above what one token per step would predict; the measured number is the one to use.

The tempting fix is smaller pieces. We measured that earlier in November and it is the wrong currency: the reply does speed up, but the long prompt itself gets much slower, because every piece carries fixed overhead. What we wanted was a way to give the replies more turns without changing how the prompt is cut.

One lane, one step at a timeTwo schematic timelines of engine steps over about one second. Top, Pg62: six wide blocks in a row, each a 6,144-token prompt piece taking about 167 ms, each carrying one reply turn. Bottom, Pg63 with cadence N=3: a prompt piece, then two narrow reply-only steps of about 7 ms each, repeated, so the reply gets three turns per piece instead of one. Same prompt, same pieces; the reply just gets more turns. Measured: about 13 to about 33 tokens per second beside a 413K prompt. Schematic, not to scale.One lane, one step at a timehow a synchronous scheduler hands out steps while a 413K-token prompt is read · schematic, not to scaleprompt-piece step, 6,144 tokens (~167 ms)reply-only step (~7 ms, drawn wide)= the reply gets a turnPg62 — every step carries a prompt piece; the reply gets one turn per piece6,144-tok piece~167 ms6,144-tok piece~167 ms6,144-tok piece~167 ms6,144-tok piece~167 ms6,144-tok piece~167 ms6,144-tok piece~167 ms0 s~1 sPg63 (cadence N=3) — two of every three steps go to replies while a long prompt is read6,144-tok piece~167 ms6,144-tok piece~167 ms6,144-tok piece~167 ms6,144-tok piece~167 ms6,144-tok piece~167 ms0 sSame prompt, same pieces — the reply just gets more turns. ~13 → ~33 tok/s measured.

Figure 1. One lane, one step at a time. Top: on Pg62 every step while a long prompt is read carries a 6,144-token piece (~167 ms) and the reply gets one turn per piece. Bottom: with the Pg63 cadence, each piece is followed by two reply-only steps (~7 ms each; drawn wide so they are visible), so the reply gets three turns per piece. Schematic, not to scale.

Fix 1: prefill cadence (Pg63)

The cadence is the checkout lane with a rule: after each big batch, ring up two small customers before the next batch. In scheduler terms, while a request with more than 8,192 uncomputed prompt tokens is in flight, two of every three schedule() calls go decode-only, and only the third carries the next 6,144-token piece. A reply-only step costs about 7 ms, so the two extra turns add about 14 ms to each ~167 ms cycle while tripling the reply's turns. Short prompts and the uncached tail of a cached prompt are never held back; they are not the problem. With no long prompt in flight the counter resets and the scheduler runs the stock code path.

We tested it the way the companion post promised: a same-window control on the same box, two boots, bars written down before the first boot. Card X4 booted Pg62 (A6), then two cadence boots (G1, G2), then Pg62 again (A7), each with the same battery and post-suite. The instrument for the headline row is a trace: a 400-token reply, thinking off, started 2, 4 or 6 seconds into a cold 413K-token prompt, with the reply's tokens counted between its first token and the moment the long prompt finishes; the median of the three arrivals is the number.

Reply speed beside a long promptGrouped bars in two panels. Beside a 413K prompt: Pg62 A6 12.2, Pg62 A7 12.5, Pg63 G1 33.3, Pg63 G2 33.1 tokens per second; 2.70× and 2.68× the control mean. Beside a 1M prompt: Pg62 A6 10.4, Pg62 A7 10.6, Pg63 G1 28.7, Pg63 G2 30.4; 2.73× and 2.90×. Card X4, trace-d median of 3 at 413K, one trace at 1M.Reply speed beside a long prompttok/s of a 400-token reply while the prompt is read · Card X4: two Pg63 boots vs Pg62 controls A6, A7Pg62 (control)Pg63 (prefill cadence N=3)Beside a 413K-token prompttrace-d median of 3 · reply arriving 2 / 4 / 6 s in12.2A6Pg6212.5A7Pg6233.3G1Pg6333.1G2Pg632.70× · 2.68×Beside a 1M-token promptone trace per boot (single-sample cells)10.4A6Pg6210.6A7Pg6228.7G1Pg6330.4G2Pg632.73× · 2.90×2.7× beside 413K, 2.7–2.9× beside 1M, on two separate boots against same-window controls.

Figure 2. Reply speed beside a long prompt, Card X4. Left: beside a 413K-token prompt, Pg62 12.2 / 12.5 tok/s against Pg63 33.3 / 33.1 (2.70× / 2.68× the control mean). Right: beside a 1M-token prompt, 10.4 / 10.6 against 28.7 / 30.4 (2.73× / 2.90×); the 1M cells are one trace per boot.

Beside the 413K prompt the reply went from 12.2 and 12.5 tok/s to 33.3 and 33.1 (2.70× and 2.68×). Beside a 1M prompt, from 10.4 and 10.6 to 28.7 and 30.4 (2.73× and 2.90×), though each 1M cell is a single trace. A short chat sent behind the 413K prompt also benefits a little, because it too gets more turns: its true first token, the first streamed delta from a 32-token probe with thinking on, went from 0.487 and 0.493 s to 0.329 and 0.362 s. Its first answer token, a different instrument that includes the model's thinking decoded at the shared rate, went from 1.298 and 1.461 s to 0.669 and 0.707 s. We name both because they are easy to confuse and they measure different things.

Everything the cadence should not touch held still, within the bars: one-user speed 541.25 and 546.13 tok/s on the controls against 543.76 and 551.15 on the cadence boots; 16-user total 2,438.8 and 2,406.0 against 2,417.2 and 2,397.1; cold 1M prompt reading 27,497.7 and 28,063.6 against 27,880.7 and 28,056.7 tok/s; qeval 71 of 75 on every arm; needle 3 of 3 at 1M; greedy 40 of 40; JSON burst 64 of 64. The cost is in the section below.

The 32-user scare, and checking the instrument first

Pg63 came out of the fourth card we ran that morning, and the story of the first three is the one teaching point we want to keep. The first version (card X1) deferred every prompt, the way upstream's knob does. It found the gain, and it also cost the 32-user burst 6.0%: the math category of that burst read 3,694 tok/s against about 5,000 on the controls, because the short prompts of 32 simultaneous users were being held back too. Gating the cadence to long prompts only (cards X2 and X3) kept the whole gain, but one or both boots in each card still missed the 32-user bar, now by a smaller margin and with the stall visible as one category's 32-request first-token time at about 0.65 s instead of about 0.18 s.

Before changing the cadence again we ran the instrument on the control. Three fresh boots, one with the cadence on, one with the overlay mounted but switched off, and one plain Pg62, each followed by back-to-back 32-user runs: the first run after any fresh boot read about 2,700 tok/s against about 3,950 once warm, on all three, Pg62 included (2,695.7, 2,670.5 and 2,859.3 tok/s on the three cold first runs). On top of that, single-category stalls of 0.3 to 0.74 s turned up at random on all three boots, Pg62 included. One 32-user run per boot could not tell the overlay from that noise: the stall was the instrument, not the overlay. Card X4 therefore measured 32 users five times per boot after warm-up and compared medians, with the controls measured the same way; that change was written into the card before its first boot. The medians read 3,905.6 and 3,928.6 tok/s on the cadence boots against 3,930.0 for the ten control runs (−0.62% and −0.04%). The earlier FAIL verdicts stay in the bundle as recorded. Check the instrument on the control before blaming the change.

Fix 2: Christopher Owen's prefill yield (Pg64)

The cadence helps replies that are already streaming. It helps a newly arrived short question only indirectly: that question still has to wait for its own first step, and if a 6,144-token piece of the giant prompt is next in line, it waits about 167 ms for that piece first. Owen's prefill yield, patch 0545 in his spark-ds41f tree since October 29, 2026, fixes that directly: when a waiting prompt can be read in a single piece (6,144 tokens or fewer), the long prompt's next piece gives up its step to it. A long prompt that has been yielding for 2 seconds takes its piece regardless, so a stream of short questions cannot starve it. In the checkout lane: a customer holding one item is waved through before the next big batch. We ported the mechanism to stock vLLM v0.31.0 on top of the cadence; his sink-rejection half of 0545 is not ported.

We had tested this mechanism once before, earlier in November, and did not promote it: the row we had committed to in advance (the first answer token, which includes thinking) disagreed between its two boots by more than three control boots disagreed with each other. This time we fixed the instrument first. The phase-locked probe P1 sends a short chat, thinking off, at 80% of each rep's own solo time for the 413K prompt, so every rep hits the same point in the prompt, and takes the median of five. The 8,192-token variant, P2, is the control for the design: two pieces, so yield must not help it. Three yield boots (Y1, Y2, Y3) ran between two Pg63 control boots (K1 before, K2 after). The bar, written down first: P1 median at or below 0.8× the pooled control median.

A short chat during a 413K promptLeft strip plot: phase-locked probe P1, first token of a short chat sent at 80 percent of each rep's own solo 413K time, median of 5 per boot, in boot order. Pg63 controls K1 0.269 s and K2 0.579 s, pooled 0.327 s drawn as a line; the bar at 0.8 times pooled, 0.261 s, drawn as a second line. Pg64 boots Y1 0.222, Y2 0.161, Y3 0.178 s, all under the bar. Faint dots are the five reps. Right panel: the same probe with an 8,192-token prompt, P2 medians K1 1.133, Y1 1.107, Y2 1.102, K2 1.010, Y3 1.119 s, not helped, by design: two pieces.A short chat during a 413K promptphase-locked probe: a chat sent at 80 % of each rep's solo 413K time · Cards X5/X6 · lower is betterPg63 control boot (K)Pg64 boot with prefill yield (Y)single repP1 — short chat (one piece), first token, s0.0 s0.2 s0.4 s0.6 spooled control median 0.327 sbar: 0.8× pooled = 0.261 s0.269K10.222Y10.161Y20.579K20.178Y3boots in time order K1 → Y1 → Y2 → K2 → Y3 · Y vs pooled: 0.68× / 0.49× / 0.55×P2 — 8,192-token prompttwo pieces · not helped, by design0.0 s0.5 s1.0 s1.133K11.107Y11.102Y21.010K21.119Y3medians of 2 · 1.01–1.13 s, every bootThree boots, all under the bar; prompts longer than one piece don't benefit.

Figure 3. A short chat during a 413K prompt, Cards X5 and X6. Left: phase-locked probe P1 medians in boot order, Pg63 controls K1 0.269 s and K2 0.579 s (pooled 0.327 s, dashed), Pg64 boots Y1 0.222, Y2 0.161, Y3 0.178 s, all under the 0.261 s bar; faint dots are the five reps. Right: the same probe with an 8,192-token prompt, P2 medians 1.01–1.13 s on every boot, not helped by design.

The controls themselves show why the instrument matters: K1's median was 0.269 s and K2's, in the same hour on the same build, 0.579 s, so we pooled them (0.327 s) and set the bar at 0.261 s. All three yield boots cleared it: 0.222, 0.161 and 0.178 s, or 0.68×, 0.49× and 0.55× the pooled control. The older trace instrument, true first token of a 32-token probe behind the 413K prompt (median of 4), agrees: K1 0.418 and K2 0.352 s against Y1 0.211, Y2 0.223 and Y3 0.185 s. The 8,192-token prompt was not helped, as designed: P2 medians 1.10–1.12 s on the yield boots against 1.07 s pooled on the controls. The long prompt's own time on P1 moved +0.2 to +0.8%.

Two more things from these cards. First, the yield did not change decode-beside-prefill (+4.1, −2.6 and −1.4% against Pg63, inside a −5% bar), and the cadence does not change a short chat's first-token time once yield is on: the two mechanisms fix different things and stack, which is why Pg64 carries both. Second, Y2's one-user speed read −2.5% against a −2% bar, with Y1 at −0.5%. One-user speed moves about ±1.5% boot to boot on this lane, so a −2% bar sits inside that noise; by the card's rule the miss was a human decision, and James chose a third boot rather than a promotion or a drop. Y3 read −0.7% and passed every bar. Pg64 was promoted on Y1 and Y3, with Y2 on file.

What it costs, and what didn't change

There is no free step. Every reply-only step is a step the long prompt did not get, so while a reply streams beside it, the giant prompt finishes later. The earlier the reply arrives, the more steps the prompt gives up. In Card X4 a 413K-token prompt finished at 11.30, 11.10 and 11.05 s on Pg62 (mean of A6 and A7) with a 400-token reply arriving 2, 4 and 6 s in; on the cadence boots, 12.14 / 12.18, 11.70 / 11.69 and 11.38 / 11.43 s. That is +7.5 to +7.9%, +5.3 to +5.4% and +3.0 to +3.5% by arrival, and it is where the "3–8%" in the plain-English box comes from. With only a 32-token probe beside it the prompt moved −0.2 to +0.7%; a solo cold prompt, with nothing beside it, moved within ±1%. The reply's own token spacing becomes bimodal by construction, about 7 ms on reply-only steps and about 167 ms on piece steps, so a client's tail latency will look worse than its mean; we did not measure that distribution.

The price, and what held stillLeft: time to finish reading a 413K-token prompt while a 400-token reply streams beside it, by when the reply arrives: 2, 4 or 6 seconds in. Pg62 mean of A6 and A7: 11.30, 11.10, 11.05 s. Pg63 G1: 12.14, 11.70, 11.38 s. Pg63 G2: 12.18, 11.69, 11.43 s. Later by +7.5 to +7.9 percent, +5.3 to +5.4 percent, +3.0 to +3.5 percent. Axis starts at 10 s. Right: a checklist of bars that held still on G1 and G2: one-user speed +0.0 and +1.4 percent, 16 users −0.2 and −1.0 percent, 32 users times five −0.62 and −0.04 percent, cold prefill 1M +0.4 and +1.0 percent, quality 71 and 71 of 75, needle 3 of 3 and 3 of 3.The price, and what held stillCard X4 · left: what the giant prompt pays while a reply streams beside it · right: bars, G1 / G2 vs Pg62Pg62 (mean of A6, A7)Pg63 G1, G2413K prompt done, s — while a 400-token reply streamstrace-d · axis starts at 10 s · labels in seconds · badge = how much later10.010.511.011.512.012.511.30Pg6212.14G112.18G2+7.5…+7.9 %reply arrives 2 s in11.10Pg6211.70G111.69G2+5.3…+5.4 %4 s in11.05Pg6211.38G111.43G2+3.0…+3.5 %6 s inWith only a 32-token probe beside it: −0.2 … +0.7 %. Solo cold prefill ±1 %.Held still — G1 / G2one-user speed (C1)+0.0 %+1.4 %16 users (C16)−0.2 %−1.0 %32 users ×5 (median)−0.62 %−0.04 %cold prefill, 1M+0.4 %+1.0 %quality, qeval /757171needle at 1M3/33/3also greedy 40/40 and JSON 64/64, both boots32 users: ×5 median vs 10 warmed control repsThe only thing that pays is the giant prompt, and only while it shares the box.

Figure 4. The price, and what held still, Card X4. Left: time for a 413K-token prompt to finish while a 400-token reply streams beside it, by reply arrival; Pg62 mean of A6 and A7 against Pg63 G1 and G2, +7.5…+7.9%, +5.3…+5.4%, +3.0…+3.5% (axis starts at 10 s). Right: the pre-registered bars that held still on G1 / G2.

The right-hand panel is the list of things a scheduler change could have broken and did not: one-user speed +0.0 and +1.4%, 16-user total −0.2 and −1.0%, the warmed five-run 32-user median −0.62 and −0.04%, cold 1M prompt reading +0.4 and +1.0%, qeval 71 of 75 on every arm, the 1M needle 3 of 3, greedy replies 40 of 40 identical within each boot, the JSON burst 64 of 64. Both overlays leave the chunk size, chunk boundaries, kernels, weights, KV cache and sampling exactly as they were; they change only which requests ride in which step.

What we measured and left out

The same window tested five other things from the companion post's ladder. None shipped; each got a row and a reason.

leverresultdisposition
vLLM #59131 + #63540 (two merged upstream prefill fixes) as overlays on v0.31.0engaged; cold prefill 131K −0.07%, 300K +0.43%, 1M +1.05%parked: inside the control spread
prefill cadence N=21.83× beside 413K, 1.35× beside 1M (bar 1.4×)N=3 chosen
deterministic combine on the expert sidecarsidecar refuses to boot with our plansclosed
skipping doomed draft rows in speculative decodingdraft survival by position 0.73 / 0.52 / 0.39 / 0.30 / 0.23 / 0.13 / 0.08no dead rows worth skipping
vLLM #62011 + #61149 (sparse indexer scoring on the FP8 indexer cache)cache change byte-neutral (3 seeds); the sped-up layers score 2.5% of the indexer's pairs on a cold 1M promptparked: decode-only, needs a numerics ruling
porting Owen's enginenot attemptedcompanion post

The indexer row carries the other teaching point of the week. A desk model of #62011 predicted +30.6% at 1M, because it assumed the four layers it speeds up (24, 28, 32, 36) score every row of the prompt. The probe counted rows and keys: on a cold 1M prompt those four layers score only 15,744 rows, 128 rows in each of 123 scheduler steps, since vLLM's bounded replay (#58132) already trims them, against 250,000 pairs per prompt token on each dense layer and 500,000 on the source layer. That is 2.5% of what the indexer scores, so there was no prompt-reading speed-up to be had here. Count rows and keys per layer before modeling an indexer win.

Upstream, status and how to run it

Upstream. vLLM applies --prefill-schedule-interval N to a single engine from v0.32.1 (#54627, merged October 15, 2026; reworked on main by #66224). Our lane runs v0.31.0, where that machinery is wired only for the data-parallel engine, so the cadence half of our overlay is a backport that reuses stock's throttle_prefills and defer_prefills. What upstream does not have is the long-only gate: the MIN_REMAINING threshold plus ticking only while a long prompt is in flight. Upstream holds back every prompt, which is the X1 arm that failed our 32-user bar. On a rebase to v0.32.1 or later, use the flag and carry only the gate; that combination is not yet A/B'd on v0.34 or later.

Status. Pg63 was promoted at 9:31 AM CST on November 21, 2026 and Pg64 at 12:08 PM; after a planned stop the lane has been serving Pg64 again since 6:43 PM. The recipe is marked experimental. No 48-hour soak has run (the clock restarted at 6:43 PM, so the earliest is Monday, November 23, around 6:44 PM CST), and the T2/T3 quality suite has not run on Pg63 or Pg64; the fidelity evidence so far is the card battery on every arm. The rollback chain is Pg64 → Pg63 → Pg62, one script each; neither new rollback script has been run, though an identical-shape script ran end to end on November 17. A live probe today: every reasoning-effort level returns 200, images and tool calls work, and a single stream with thinking on reads about 309 tok/s wall-clock on a 400-token reply (median of 3 over the LAN). That is a different instrument from the battery's one-user 548 tok/s (thinking off, prompt set v1); the two are not a delta. The lane is back on the Funland Lab guest page as of today.

How to run it. The recipe and its receipts are in J-M-Recipes, bundle results/2026-11-21-pg63-pg64-scheduler/: the two overlay diffs against stock v0.31.0's vllm/v1/core/sched/scheduler.py (patches/cadence3.diff, 92 lines; patches/yieldcad.diff, 370 lines, with SHA256SUMS), the promotion and rollback scripts, the six pre-registered card specs, and the raw traces. With the environment variables unset every added code path is skipped and the scheduler is stock. Pg64 is the overlay plus:

VLLM_PREFILL_CADENCE=3 VLLM_PREFILL_CADENCE_MIN_REMAINING=8192 \
VLLM_PREFILL_YIELD=short VLLM_PREFILL_YIELD_LIMIT_S=2

Engagement is checked in the engine log, not assumed: xfer prefill cadence enabled: N=3 min_remaining=8192 at boot and deferred N of M steps counters afterwards, present on every cadence arm and absent on every control.

Credits and links

Christopher Owen: the prefill-yield mechanism (spark-ds41f, patch 0545, in his tree from commit 47342f26, October 29, 2026); our part is the port to stock v0.31.0 and the measurements. Garner McCloud: the DeepSeek-V4.1 engine both overlays sit on. The cadence reuses vLLM's own prefill_schedule_interval machinery (#54627). DeepSeek for the model; vLLM and NVIDIA for what the engine builds on.

Companion: why we're not porting spark-ds41f to our GB300 (November 21, 2026), where this round was planned. Earlier: prefill yield on our GB300, not promoted (November 17, 2026) and Pg62, the configuration both changes sit on.