We tried Christopher Owen's prefill yield on our GB300: real, small, and not promoted
In plain English
- What we tried. Last night Christopher Owen's candidate 60 made our four Sparks faster, and the setting that kept guests fast there was his "prefill yield": when a huge prompt is being read in chunks and a short request is waiting, the huge prompt gives up one turn so the short one can be read whole. Our GB300 runs a different engine, so none of his files install. The yield is the one idea from his list that ports as code, so we ported it into stock vLLM 0.31.0 and tested it on the same box, the same night, against three boots of what we serve today.
- What it did. A short chat sent while a 413,000-token prompt is being read got its first token about 40% sooner on the first boot and 46% sooner on the second (0.28 s and 0.25 s instead of 0.43–0.53 s). Nothing else moved: one-user speed, 16-user total, prompt reading, the 1M needle and the 75-task quality check all read the same, and greedy replies were identical to the control's, 40 of 40.
- Why it is not serving. We had promised, in writing, before the first boot, to judge it on a different row: the first answer token of that short chat, which also counts the model's thinking. That row read −26% on the first boot and −9% on the second. The two boots disagreed with each other by more than three control boots did, so the rule said roll back, and we did.
- What we learned. The real guest pain on this lane is not the first token. It is that a reply is written at about 13 tokens per second for as long as a giant prompt is being read beside it. Yield does not touch that.
- Smaller chunks trade badly. Reading the big prompt in 2,048-token chunks instead of 6,144 lifted the reply to about 18 tokens per second, but made the big prompt itself take up to twice as long. Declined.
- What else transfers from his list. As code, almost nothing: his late admission, his 8-bit word-list screen, his draft-layer skipping and his Spark expert kernel each run into something structural about our engine or our box. His testing method, yes: two boots per candidate, identical greedy replies across boots, a JSON burst, a tool probe and a 32-user stress run, and every boot passed them.
- Where it stands. Pg62 is serving. The yield build is kept and one command from serving. James decides.
What we tried, and why
Two words first. Prefill is the engine reading a prompt; decode is the engine writing a reply, one token per step. A 413K-token prompt is read in chunks, one chunk per step, and every other request on the box shares those steps with it.
Last night, on our four Sparks, Christopher Owen's candidate 60 was faster on every row, and his prefill yield was the setting that kept guests fast: a short chat behind a 413K prompt waited 1.51 s with yield off and 0.70 s with it on. The idea is simple. When a long prompt is being read in chunks and a waiting prompt is small enough to be read whole in one chunk, the long prompt's chunk gives its step to the short one. A long prompt that has yielded for 10 s stops yielding until its next chunk runs, so it cannot be starved.
Our GB300 lane does not run his engine. It is Pg62: Garner McCloud's dgx-station-recipes engine plus our patches, overlaid on stock vLLM 0.31.0, on one DGX Station with a GB300 and an RTX PRO 6000 in the same chassis. The 255 hot experts per layer live on the GB300; cold experts are computed on the RTX, which we call the sidecar. None of Owen's Spark files install on that. So a desk worker on our Spark lane ported the yield into vLLM 0.31.0's scheduler: one file, vllm/v1/core/sched/scheduler.py, +190 lines and no deletions, switched on by an environment variable (VLLM_PREFILL_YIELD=short, limit VLLM_PREFILL_YIELD_LIMIT_S, default 10 s). Unset, the scheduler is stock. Ten of ten unit tests pass inside the serving image.
One thing to know before the numbers: stock vLLM 0.31.0 already does part of this. When more than one request is eligible, a long prompt's chunk is capped at 6,144 of the 8,192-token step, so a short request already gets about 2K tokens of room in the very next step. Only when the long prompt is alone does it get the full 8,192. That is why the gain here is small, where on the Sparks it was large.
What happens when a short chat arrives while a 413K prompt is being read
Figure 1. Top: stock Pg62. The short chat shares its first step with a 6,144-token chunk of the big prompt and gets its first token in about 0.45 s. Bottom: with Owen's yield, the chat gets a step to itself and the first token arrives in about 0.25 s. After that the two lanes are the same: every step carries a big-prompt chunk and the reply moves on at about 13 tokens a second until the big prompt is read.
Results: same box, same night, seven boots
Every arm is a full boot of the lane, on the same box, between 22:46 and 01:20 CST. A1, A1b and A2 are Pg62 as it serves (three separate boots; A1 before the window, A1b after an accidental extra restart, A2 after the candidates). Y is Pg62 plus prefill yield; Y2 is the same build booted again, which was the promotion boot. T2 is Pg62 with the chunk cap at 2,048 instead of 6,144, the stock knob closest to Owen's 1107, which shrinks chunks on deep prompts; YT2 is yield plus that cap. Medians unless noted.
| Row | A1 Pg62 | A1b Pg62 | A2 Pg62 | Y + yield | Y2 + yield, boot 2 | T2 chunk 2,048 | YT2 yield + 2,048 |
|---|---|---|---|---|---|---|---|
| True first token, short chat behind a cold 413K prompt (s, median of 4) | 0.428 | 0.525 | 0.454 | 0.280 | 0.252 | 0.317 | 0.252 |
| First answer token, same situation (s, median of 3; arrives after the model's thinking) | 1.472 | 1.521 | 1.397 | 1.089 | 1.328 | 1.116 | 0.865 |
| Reply written while the 413K prompt is read (tok/s, median of 3) | not traced | not traced | 13.0 | 12.8 | 13.3 | not traced | 17.8 |
| Time to read the 413K prompt while a 400-token reply runs beside it (s, 3 runs) | 10.96–11.22 | 10.90–11.18 | 11.01–11.38 | 10.97–11.33 | 11.03–11.37 | 15.96–21.83 | 16.18–21.65 |
| One user, decode (tok/s) | 547.5 | 551.4 | 541.0 | 546.6 | 549.6 | 550.2 | 550.7 |
| 16 users, total (tok/s) | 2,409.7 | 2,437.6 | 2,421.8 | 2,389.7 | 2,407.8 | 2,449.1 | 2,427.5 |
| 32 users, total (tok/s) | 3,898.5 | 3,909.7 | 3,950.2 | 3,913.6 | 3,746.5; reruns 3,934.1, 3,940.3 | 3,884.1 | 3,968.5 |
| Cold prompt read, 131K / 1M (tok/s) | 43,185 / 28,140 | 43,343 / 28,140 | 43,344 / 28,132 | 43,472 / 28,016 | 43,543 / 28,201 | 43,419 / 28,098 | 43,397 / 28,131 |
| Cached 68K document + 1.5K new, behind 413K (s) | 0.562 | 0.623 | 0.554 | 0.534 | 0.475 | 0.404 | 0.475 |
| Next agent turn on a cached 200K session (s) | 0.267 | 0.261 | 0.267 | 0.256 | 0.270 | 0.257 | 0.255 |
| 1M needle | 3/3, 36.0 s | 3/3, 36.0 s | 3/3, 36.0 s | 3/3, 36.0 s | 3/3, 36.0 s | 3/3, 36.0 s | 3/3, 36.0 s |
| qeval (75 tasks) | 71 | 71 | 71 | 71 | 71 | 70 | 71 |
Every arm gave identical greedy replies to 40 fixed prompts twice in the same boot and identical to A1's replies across boots (40/40 for A1b, Y, Y2, T2 and YT2), passed the 64-request JSON burst with the server alive after, and passed the DSML tool-argument probe streaming and not. An idle short chat, with nothing else running, gets its first answer token in about 0.06–0.08 s; everything above it in the first two rows is the big prompt's doing.
The first two rows are two instruments on the same event. The battery's probe counts the first answer token. This lane streams the model's thinking in a field named reasoning, which that probe does not count, so its number is getting in plus the thinking written while the big prompt is still being read. Our trace tool records the first streamed token of any kind, and the decode speed during the big prompt. The trace row is the one that shows what yield does.
Figure 2. True first token of the short chat behind a cold 413K prompt, median of four arrivals per boot. The two yield boots sit at 0.25–0.28 s, the three control boots at 0.43–0.53 s (−40% and −46% against the control mean of 0.469 s). The chunk-cap arms sit between.
Why we did not promote it
The rule was written before the first candidate boot, and it had four lines. One: quality holds (qeval within ±1 of control, 1M needle 3/3, tool probe, JSON burst 64/64, 32-user stress without a crash, greedy agreement no worse than control against control). Two: no regression (one-user decode no worse than −2%, 16 users −3%, cold prompt reading at 32K and 131K −3%). Three: a real gain, at least 20% on the short-chat-behind-413K row as the battery measures it (first answer token), or 5% on another row, with the other mixed rows no worse than +20%; before any candidate boot we added that the trace rows count as rows and that the big prompt's own time while sharing may not grow more than 20%. Four: two boots of the candidate agree within the controls' spread on every row above. A gain that misses one bar narrowly is "James decides", not promoted.
Y cleared lines one to three on its first boot. At 01:10 we promoted it, booting the same build a second time as the production lane, and measured that boot as Y2. The first-answer row came back at −9% against the mean of the three control boots, where the first boot had read −26%. The two boots differed by 0.239 s on that row; the nine control reps across three boots spanned 0.204 s. Line four failed. Rollback started at 01:23 and Pg62 was serving again at 01:30.
Figure 3. Every repetition of the first-answer-token row. Nine control reps across three boots span 1.349–1.553 s. Yield boot 1 (1.076, 1.089, 1.103) sits well below them. Yield boot 2 (1.261, 1.328, 1.412) reaches into the control band.
The honest reading. The first-token gain is real and it repeated: 0.280 s and 0.252 s against 0.428, 0.525 and 0.454. The row we promised to judge on is a different thing. It includes the model's thinking, written at about 13 tokens per second while the big prompt is read, so it is mostly a decode number with a first-token number inside it, and yield can only ever move its front end. We judged on the row we had promised to judge on, and that row did not repeat. By the rule as written this is James's call, and the build waits for it.
Smaller chunks are not the fix here
Owen's 1107 shrinks the chunk size on deep prompts. The stock knob for that is the long-prompt chunk cap, so T2 ran Pg62 with the cap at 2,048 instead of 6,144, and YT2 ran yield plus the same cap. The reply beside the big prompt did speed up: 17.8 tokens per second on YT2 against about 13 on the yield arms and A2. But the big prompt paid for it. Reading the 413K prompt while one reply ran beside it took 16.0–21.8 s instead of about 11 s, up to twice as long, and the rule's added clause caps that growth at 20%. Both arms failed it.
Figure 4. Left: time to read the 413K prompt while a 400-token reply is written beside it, three runs per boot. Pg62 and yield: 10.9–11.4 s. Chunk cap 2,048: 16.0–21.8 s. Right: the reply's speed during that read, about 13 tokens per second on Pg62 and yield, 17.8 with yield plus the cap. T2 alone was not traced.
The rest of Owen's list on a GB300
Five desk workers running DeepSeek-V4.1-Flash on our four-Spark lane read each item of Owen's candidate against Garner's tree and ours. The yield was the only one that came out as portable code. The rest, and why:
| Idea | What we found | Status |
|---|---|---|
| Late admission (his 0222/0226; our Sparks run it at layer 20) | Lives in vLLM's async-scheduling batch queue. Our lane runs the synchronous scheduler, where a request that arrives during step N already joins step N+1. Ported and CPU-tested. | Not booted |
| 8-bit word-list screen on the output head | Our main head feeds a sampled block-verification step that needs the whole vocabulary's mass, so a screen that only returns the top pick cannot be used there. The drafter half is ~0.1 ms per step by model and needs a different drafting mode first. Garner's own 8-bit draft head (−0.07 ms, his measurement) stays off here because it changes numerics; that is a policy question. | Not applicable as is |
| Skip expert layers for draft tokens that will not survive | Our verify row costs ~0.5 ms of a ~7.2 ms step; on his Sparks it is ~4 ms of ~28 ms (both figures from the two trees' docs). Only about one row per step is below break-even here. | Not booted |
| His SM120 expert kernel, for our RTX sidecar | His own receipt puts dispatch + combine at ~8 µs of a ~564 µs layer; the rest needs his forked TileLang. Not where our time goes. | Not pursued |
| spark-ds41f#1194 (JSON-output crash) and the DSML tool-argument fix | The crash cannot occur on stock v0.31.0, which sizes the grammar mask from the draft length; we ran the 64-request JSON burst on every arm anyway. Our GB300 parser already carries the same DSML rule. | Already covered |
| His testing method | Two boots per candidate, identical greedy replies across boots, JSON burst, tool probe, 32-user stress. | Carried over; passed on every boot |
Where this leaves the lane
Pg62 is serving on :30006, as it was yesterday. The yield build is kept, built, and one command from serving if James wants it; nothing in it regressed, and the first-token gain is real. Serving it now would be James's decision, not the rule's: the rule we wrote did not pass, and the gain it did show is about 0.2 s on a row that was already under half a second.
The open problem is the one neither yield nor smaller chunks fixes: a reply written at about 13 tokens per second for as long as a giant prompt is being read beside it, on a box that writes about 550 a second for one user alone. That needs a different kind of fix, either keeping prompt reading from starving replies inside one engine, or putting the two kinds of work on separate hardware. It is not scheduled.
What we did not measure
- Decode speed during the big prompt was not traced on A1, A1b or T2; the instrument was added mid-night, so the ≈13 tok/s reading for stock rests on A2 alone.
- One 32-user run on Y2 read 3,746 tok/s total; two reruns read 3,934 and 3,940. We report all three and do not know what the first one hit.
- One night, one box, synthetic prompts. The 413K prompt is a test fixture, not a guest's.
- No teacher-forced comparison of token probabilities against control; greedy identity only (40/40 on every arm).
- Late admission was ported and CPU-tested but never booted. The 8-bit word-list screen was not booted either.
Sources
- christopherowen/spark-ds41f: prefill yield (patch 0545, from his 1030 release), late admission (0222/0226), the ideas and the test method.
- Our two posts from November 16: candidate 60 promoted on our four Sparks and our desk read of what a GB300 can borrow.
- The Pg62 recipe post, the lane this was tested against.
- GB300 recipe in J-M-Recipes.