Why we're not porting Christopher Owen's spark-ds41f to our GB300 — and what we'll borrow instead
Christopher Owen's spark-ds41f is the best DeepSeek-V4.1-Flash engine we know of for DGX Sparks. James calls the line "1022", after the first release we borrowed from on October 22, 2026; the current release is spark-ds41f-1120, published November 20, 2026. The obvious next question was whether to port it to the other machine in the room, our DGX Station GB300, which runs Garner McCloud's engine under our Pg62 recipe. We spent November 20 finding out what a port would take. The verdict: we are not porting it. His engine is excellent at what it is built for, the GB300 is a different chip family, and the ideas travel better than the code. This post is the evidence and the plan.
In plain English
- What a "port" would mean. Owen's engine is written for the Spark's GB10 chip and its close cousin, the RTX PRO 6000. On a GB300 his software tree quietly loads a different model, NVIDIA's stock one, so none of his engine runs there at all. Porting means rewriting his roughly 44,000-line model package for a new chip family, not flipping a switch.
- What we found when we tried. We rebuilt his kernels for the GB300 and ran his own tests. 31 of 35 kernel families work. The 4 that do the heavy matrix multiplies do not, because the GB300 keeps its running totals in a different kind of on-chip memory, and his sparse-attention indexer traps at run time in 18 of 20 cases. The model also does not fit one GB300 the way it fits eight Sparks: a plain boot ran out of memory at 246.5 of 249.8 GiB.
- Why the payoff is doubtful. The bar a port would have to clear is our current GB300 recipe, Pg62, at 548.58 tokens per second for one stream, 3.00× what his engine does across eight Sparks with our scheduling. Our guess is that a full port is several weeks of kernel work for maybe a 15–20% chance of beating Pg62 on any headline number.
- What his work still beats us at. A short chat sent while a giant prompt is being read gets its first token in 0.549 s on eight Sparks with our scheduling, against 1.283 s on the GB300. On the GB300 a reply crawls at about 13 tokens per second while a 413K-token prompt is read beside it. And our prompt reading slows 38% between a 32K and a 1M prompt; his slows 5%.
- The plan. Borrow the ideas, not the kernels: (1) keep replies flowing while a giant prompt is read, starting with a prefill cadence that reads the big prompt only every second or third step; (2) make long prompts read faster, starting with two merged upstream vLLM fixes, because most of his flat curve needs four GPUs; (3) retry his prefill yield with a better instrument; (4) in speculative decoding, verify fewer doomed draft rows; (5) make our sidecar's large-batch path bit-stable; (6) keep watching Garner McCloud's work and upstream vLLM. Each one gets a same-window control, two boots and pre-registered bars.
- Bottom line: his engine stays on the Sparks, his ideas come to the GB300, and we have offered him the GB300 as a test bench if he ever wants to try the port himself.
What "porting" would actually mean
Three Blackwell chips are in play, and they fall into two families. The DGX Spark's GB10 is compute capability sm_121; the RTX PRO 6000 Blackwell that sits beside our GB300 holding cold experts is sm_120. Both are the "12x" family, and Owen's engine targets it: the headers of his tilelang/gemm.py and moe.py name SM120 and SM121. The GB300 in the DGX Station is sm_103, the datacenter "10x" family, with 250.7 GiB of HBM visible (249.8 GiB to CUDA) and 494 GiB of Grace CPU memory beside it.
His vLLM tree knows this. In vllm/models/deepseek_v41/__init__.py the model class is picked by GPU family: if current_platform.is_device_capability_family(120) it imports vllm.models.deepseek_v4_1, his TileLang model. Otherwise, and the GB300 is otherwise, it falls through to .nvidia.vl_model, the stock NVIDIA DeepSeek model. Our first GB300 boots of his image on November 20 were therefore the stock path failing, not his engine failing; we posted that correction on issue #1243.
Figure 1. One tree, two models. The family check at the top of his model package sends Sparks and the RTX PRO 6000 to his TileLang model and everything else, our GB300 included, to NVIDIA's stock model.
So a port is not a patch to a few kernels. It is the whole deepseek_v4_1 package: about 44,000 lines of Python in his candidate-60 tree (44,181 by wc -l, tree 1d0ec3e). The largest files are the kernels themselves: tilelang/vocab.py at 3,830 lines, tilelang/gemm.py 3,060, tilelang/indexer.py 2,817, attention.py 2,571 and tilelang/moe.py 2,186.
What his kernels do on a GB300
We took his spark-ds41f-1120 build, added one Dockerfile commit so the image also compiles for sm_103a, and ran his own kernel tests: the first 4 cases of each of 35 test files. As a control, the same image and harness on a GB10 Spark passes 35 of 35. On the GB300 as shipped, 22 files skip themselves with the message "TileLang V4.1 kernels target SM120/SM121". With that skip removed, 31 of the 35 files pass and 4 fail: linear, wo (the output projection), moe and expert_slots. Those four are the block-scaled matrix multiplies, the kernels that do the heavy math.
Figure 2. His 35 kernel test families on the GB300 (sm_103a) once the Spark-only skip is removed: 31 pass, the 4 block-scaled multiplies fail, and the indexer traps at run time. Right: the two homes for a multiply's running sum.
The failure is one TileLang error, verbatim: "Block-scaled GEMM requires either Blackwell SM100 TCGEN5MMA (A/B in shared memory, C in tensor memory) or SM120 mma.sync (A in shared memory or a fragment, B in shared memory, C in a fragment)". In everyday terms: a matrix multiply is a long sum, and the chip has to keep the running total somewhere while it works. On the Spark and RTX family the total lives in registers, the scratch space right next to each arithmetic unit, like a cook keeping a tally on a notepad in one hand. On the GB300 family the fast block-scaled multiply must keep its total in a separate on-chip "tensor memory", like a whiteboard beside the stove: same arithmetic, different place for the total, and every step of the recipe assumes one or the other. His kernels are written the notepad way. Moving them to the whiteboard is a restructure of each kernel, not a compiler flag.
The indexer, the sparse-attention selector, is a separate problem. Across its 20 test cases the GB10 passes 19 and skips 1; the GB300 fails 18, passes 1 and skips 1. That is a run-time trap: TileLang compiles the SM120 block-scaled instruction into an "invalid control path" assertion on anything other than sm_120 and sm_121. The passing families, for the record, include attention, router, vocab, rope, mHC, compress, Engram, activation, embedding, collectives, CED and the DSpark context kernels. Most of his engine retargets. The part that does the arithmetic does not.
Memory: why one GB300 is not one big Spark
Even with the kernels rewritten, the model would have to fit, and on a GB300 it does not fit the way it does on eight Sparks. A tensor-parallel-1 boot of the stock path with every expert resident ran out of GPU memory at 246.5 of 249.8 GiB. vLLM's CPU-offload option then failed on his model's placeholder-built layers. For scale: DeepSeek-V4.1-Flash carries 384 routed experts per layer across 40 layers in MXFP4, plus two Engram tables of 94.6 GiB each.
His single-node trick, a zero-copy io_uring expert slot cache, assumes the Spark's shared CPU/GPU memory, where the GPU can read an expert sitting in system RAM at full speed. The GB300 has no such pool. Cold experts must come across C2C from Grace memory or from the RTX PRO 6000 sidecar, and that is exactly what Pg62 already does: Garner McCloud's tree plus our patch set on stock vLLM 0.31.0, two GPUs, the sidecar computing cold experts, serving since November 14, 2026. A port would have to rebuild that memory plan from scratch to land where we already are.
The bar
Any port would be judged against Pg62 on the same box. These are the November 20 numbers, same harness, base weights, temperature 0, thinking off; the Spark columns are Owen's 1120 as shipped on four Sparks and with our scheduling on eight.
Figure 3. Left: three rows where the GB300 leads, each panel on its own scale. Right: the one latency row where eight Sparks running his engine with our scheduling lead. Values from the table below.
| row | GB300 Pg62 | 1120 TP4 as shipped (4 Sparks) | 1120 TP8 + our scheduling (8 Sparks) |
|---|---|---|---|
| One stream decode (tok/s) | 548.58 | 126.43 | 182.87 |
| 16 streams, aggregate (tok/s) | 2,444.1 | 535.8 | 624.6 |
| Read a 32K prompt (tok/s) | 45,668 | 9,427 | 9,750 |
| Read a 131K prompt (tok/s) | 43,790 | 9,697 | 9,941 |
| Read a 300K prompt (tok/s) | 41,400 | 9,581 | 9,837 |
| Read a 1M prompt (tok/s) | 28,280 | 8,935 | 9,413 |
| Short turn behind a cold 413K prompt (s) | 1.283 | 2.247 | 0.549 |
| Cached 68K turn behind a 413K prompt (s) | 0.611 | 3.103 | 1.804 |
| 1M needle | 3/3 in 36 s | 3/3 in 113 s | 3/3 in 108 s |
| qeval /75 | 71 | 71 | 72 |
| KV pool (tokens) | 11.70M | 18.70M | 29.92M |
Per stream the GB300 writes 548.58 tokens per second to 182.87 on eight Sparks with our scheduling (3.00×); with sixteen streams, 2,444.1 to 624.6 (3.91×); with thirty-two, 3,637.21 to 792.40. A ported engine would be running on the same GPU and the same memory as Pg62, so these are not numbers it could sidestep by being on better hardware. For his own part, Owen publishes single-node figures in his release post: 4.4 and 4.8 tok/s for prose and code on one Spark, which he labels alpha, and 124.1 and 145.9 on eight Sparks switched. Those are his numbers from his bench on his testbed; we quote them as his and not as a delta against ours.
Where his ideas beat us anyway
This is the reason to keep reading his work rather than file it under "Spark only". Three rows go his way, and they are the rows that decide how a shared lane feels.
A short turn behind a giant prompt. Send a short chat while a cold 413K-token prompt is being read. On eight Sparks running 1120 with our scheduling the first token arrives in 0.549 s; on the GB300 it takes 1.283 s. Eight Sparks answer 2.3× sooner, on an engine that is 3.00× slower per stream.
A reply written while a giant prompt is read. On the GB300, a stream that is already decoding slows to about 13 tokens per second for as long as a 413K prompt is read beside it: 13.0 on the control boots, 12.8 and 13.3 with prefill yield on, against roughly 548 when the box is idle. That is the real guest pain on this lane, and it is a scheduling problem, which is the kind of problem his releases keep attacking.
Figure 4. Prefill speed as a share of each lane's own 32K speed. Pg62 falls to 61.9% of its 32K rate by 1M; 1120 as shipped on four of our Sparks holds 94.8%, after actually speeding up at 131K and 300K.
Prompt reading does not slow down with length on his engine; ours does. Pg62 reads a 32K prompt at 45,668 tok/s and a 1M prompt at 28,280: 28,280 / 45,668 = 0.619, so 38% slower at 1M. His 1120 as shipped on four of our Sparks reads 9,427 at 32K and 8,935 at 1M: 0.948, so 5% slower. With our scheduling on eight Sparks it is 9,750 to 9,413, 3.5% slower. Normalized to each lane's own 32K speed the curves are 100 / 95.9 / 90.7 / 61.9% for the GB300 at 32K / 131K / 300K / 1M and 100 / 102.9 / 101.6 / 94.8% for his engine. His own release notes tell the same story on his bench: 1120 tp4-ring prefill at 9,615 / 10,109 / 10,056 / 9,833 / 9,397 tok/s across 32K / 131K / 262K / 524K / 1M, and "+56.31%" at 1M against his previous release, 1107. In absolute terms the GB300 still reads a 1M prompt 3.2× faster (28,280 vs 8,935). The gap is in the shape of the curve, and a flat curve on a box this fast would be worth having.
Correct-first. His releases gate on temperature-0 outputs that do not depend on the batch, two boots per candidate, a determinism gate and a release test suite; 1120 shipped with 12,023 tests passed, 356 skipped, 0 failed, by his notes. We are not there. Our GB300 sidecar's large-batch path, above 64 rows, still accumulates with bf16 atomic adds and is not run-to-run bit-stable. Greedy replies were identical 40 of 40 within and across boots on every November 16 boot, so greedy output is stable in practice, but his bar is stricter than ours and we want it.
What we already tried from his list
None of this is the first attempt. On November 16 and 17 we worked through his list of Spark-side ideas on the GB300 and wrote up the result in a post of its own. The one idea that ported as code was prefill yield, his 0545, carried into stock vLLM 0.31.0 in about 190 lines behind an environment gate. It cut the true first token of a short chat behind a 413K prompt to 0.28 s and 0.25 s on two boots, against 0.43–0.53 s on three control boots, and moved nothing else. It was not promoted because the row we had pre-registered, the first answer token, read −26% on boot 1 and −9% on boot 2, and the rule is the rule. The build is kept, one command from serving.
Smaller prefill chunks (--long-prefill-token-threshold 2048) lifted the reply during the big prompt to about 18 tok/s from about 13, but the 413K prompt itself took 16–22 s instead of 11 s. Declined. Late admission was inert on our synchronous scheduler; his 8-bit word-list screen, draft-layer skipping and his Spark expert kernel were each blocked by something structural about our engine or our box. Not booked.
Our estimate of a full port
Everything in this section is our estimate, not a measurement. We think a full port is several weeks of kernel work before a first token: the block-scaled GEMMs and MoE rewritten for tensor memory, a new 4-bit indexer path, the Spark-only gates audited (38 runtime files in his tree check for the Spark GPU family by our count), and a TP1 memory plan through Grace memory or the sidecar. Then tuning. Our guess at the odds: maybe a 15–20% chance of beating Pg62 on any headline number, and very little chance on high-concurrency throughput or prompt reading. The reasons are structural: his tile shapes and prefetch are sized for the much smaller GB10; Pg62's attention and MoE kernels are already GB300-native; and both would be running on the same GPU with the same memory, so there is no hardware edge to find.
There is also a cost outside the code. Every port test window takes our GB300 lane dark, and the GB300 is also where James's standalone decode-runtime project lives. Weeks of dark windows for those odds is not a trade we want to make with the box our own agents run on.
What we did instead was open issue #1243 on his repo and offer the GB300 as a test bench: we build any branch he pushes, run his tests on the Station and benchmark against Pg62. If he ever wants the port, the hardware is one message away. No reply yet as of the morning of November 21, 2026, and none is owed; he ships on his own schedule.
Update, November 21, 2026, 12:20 PM CST: his prefill yield is now serving too, as Pg64. On top of Pg63 we retested the yield (item 3) with a better instrument: the short chat now arrives at 80% of each rep's own solo time for the 413K prompt, so every rep hits the same point in the long prompt. Three boots against two Pg63 control boots, bars written before the first boot:
- A short chat sent at 80% of a cold 413K prompt: first token 0.22, 0.16 and 0.18 s (medians of 5) against 0.33 s on Pg63. A prompt one chunk too long to count as short (8,192 tokens) is not helped, as designed: 1.08–1.12 s against 1.07 s.
- The long prompt's own time moves +0.2% to +0.7%; 16- and 32-user totals, cold prompt reading, replies during a long prompt, the quality check (71), the 1M needle, the JSON burst, the tool probe and greedy replies all read the same.
- One boot missed the one-user bar by half a point (−2.5% against −2%), which yield cannot plausibly cause, since it only acts when a request is waiting behind a long prompt. By our rule that is a human decision; James chose a third boot, which passed every bar.
Items 4 and 5 are closed. Draft survival read from live counters leaves no dead rows to skip: with three drafts, the last one is accepted 43–46% of the time; with one stream, the weakest positions sit at 20–23%, inside the break-even band. And the sidecar's deterministic-combine switch does not start with our sidecar's plans (planned dynamic direct routing is unsupported for this launch shape). The same test showed why it would not have been enough: 32 identical greedy requests sent twice do not match on Pg63, starting from the first token, so the variation comes from how requests are batched, not only from the sidecar's sums. Pg63 is kept one command away as the rollback.
Update, November 21, 2026, 9:40 AM CST: the prefill cadence is now serving as Pg63. Item 1 below went through five test windows this morning, each with a same-window Pg62 control and bars written before the first boot. The version that shipped reads the big prompt on one step in three, and only while a prompt with more than 8,192 tokens still to read is in flight; with no long prompt in flight the scheduler runs Pg62's exact code. Two boots, against Pg62 measured just before and just after them in the same window:
- A reply written while a 413K-token prompt is read: 33.3 and 33.1 tok/s, against 12.2–12.5 on Pg62 (2.7×). Beside a 1M-token prompt: 28.7 and 30.4 tok/s against 10.4–10.6.
- A short chat sent behind the 413K prompt: first token 0.33 and 0.36 s against 0.49 s; first answer token 0.67 and 0.71 s against 1.30–1.46 s.
- One-user speed, 16-user and 32-user totals, cold prompt reading from 32K to 1M, the 75-task quality check (71 on every boot), the 1M needle, the JSON burst, the tool-call probe and greedy replies (40 of 40 identical within each boot) all read the same as Pg62.
- The cost: while a reply streams beside a long prompt, the long prompt takes longer, because two of every three steps now go to the reply. Measured: +3% to +8% on the 413K prompt depending on how much of it the reply overlaps.
- Three earlier versions failed a 32-user bar. A diagnostic boot then showed that the single 32-user run in our suite has a noise floor on Pg62 itself as large as those failures, so the final card measured 32 users five times per boot. That change was written down before the final boots, not after.
The two upstream prefill changes (item 2) were also tested: active, but only +1% at 1M, so they are not serving. Pg62 is kept one command away as the rollback.
What we'll try instead
Ideas move between chip families; kernels don't. So the plan is a list of his ideas, ordered by the size of the measured gap each one attacks, with no predicted gains attached. The research round on these ideas finished this morning, and it moved the list: two items now have a concrete first test, one turned out to be mostly out of reach for a single GPU, and one is already a measured negative on this class of box.
Figure 5. Six candidates in the order we will take them, each with the measured gap it attacks and its first step after this morning's research round. Nothing on this ladder has a predicted gain; the gap and the mechanism are what we know.
- Keep replies flowing while a giant prompt is read. Gap: about 13 tok/s against about 548 idle. The arithmetic says why: every step that carries a 6,144-token piece of the big prompt takes about 167 ms, and the reply gets one step per piece. Smaller pieces are the wrong currency (we measured that in November). The new idea is a prefill cadence: read a piece of the big prompt only on every second or third step and let the replies run alone on the others, where a step costs about 7 ms. Upstream vLLM has this knob (
--prefill-schedule-interval). In v0.31.0, the release our lane runs on, it only takes effect with data parallelism; since v0.32.1 it also works on a single GPU (vLLM #54627, merged October 15). His 1107 and 1120 releases interleave reply-only steps once a long prompt stalls everyone else. On our v0.31.0 base we turned the cadence on with a small scheduler overlay (off by default) and added one thing upstream still lacks: it holds back only prompts with more than 8,192 tokens left to read, because holding back every prompt delayed the short prompts of 32-user bursts in our first test window. Corrected November 21, 3:06 PM CST: this first said upstream vLLM only wires the knob for multi-GPU setups. That was true of our v0.31.0 base, not of current vLLM. Result: serving as Pg63 since November 21; see the update above. - Make long prompts read as fast as short ones. Gap: 38% slower at 1M against his 5%. The research says most of his flat curve comes from two things one GPU cannot copy: the indexer's work split across four GPUs, and his indexer scoring in 4-bit, which would mean quantizing a cache we have decided to keep at 8-bit. What does port: two merged upstream vLLM changes that claim identical output, one that gathers each compressed-KV source once per step instead of in 38 layers (vLLM #59131) and one that speeds up the RoPE and FP8 conversion kernel (#63540). Tested November 21: active, +1% at 1M, not serving. A fused indexer kernel for long prompts (#60618) comes later; it needs a build.
- Retry prefill yield, with a better instrument. His yield has not changed since the version we tested; what failed was our measurement, which varied more between boots than within one. Result: serving as Pg64 since November 21; see the update above.
- Speculative decoding: verify fewer doomed draft rows. Adaptive verification is out: Garner McCloud already measured it on this box class (no gain at one stream, −20% on prose at 16 streams). Skipping draft rows that will not survive is the one live idea, and our engine already counts survival per position, so the first step costs no downtime. Result: no dead rows to skip on Pg63; closed.
- Correct-first: make the sidecar's large-batch path bit-stable. The sidecar's MoE library already has a deterministic-combine switch; it is off today. Result: the switch does not start with our sidecar's plans, and the lane's variation is mostly batch-dependent anyway; closed.
- Anything else promising. Garner McCloud's repository has had no commits since October 29, so our base is current. Upstream vLLM is at 0.34.0; the useful parts are the two prefill changes above and the indexer kernel, not a base bump.
The method is the same one that kept prefill yield off the lane. Each candidate gets a same-window control on the same box, two boots, and pre-registered bars: a quality check, the 1M needle, tool-call and JSON-burst probes, and a 32-user stress run for fidelity, with no regression allowed on one-user speed, 16-user total or prompt reading. If a candidate clears all of that, it becomes the serving recipe. If it clears most of it, it stays a kept build, and we say so.
Credits and links
Christopher Owen: spark-ds41f, the engine every Spark number here runs on, the kernels whose tests we rebuilt, and the ideas on the ladder above (repo, 1120 release post, our test-bench offer). His own published figures are his, measured with his bench on his testbed; we quote none of them as a delta against ours. Garner McCloud: the DeepSeek-V4.1 engine our GB300 lane is built on. DeepSeek for the model. TileLang, vLLM and NVIDIA for what both engines build on.
Earlier posts this one leans on: eight Sparks vs one GB300 and 1120 on four Sparks (both November 20, 2026; the source of every table row above), prefill yield on our GB300 (November 17, 2026), how his engine works and what a GB300 can borrow, and Pg62, our current GB300 recipe.