Christopher Owen's spark-ds41f at candidate 60: how it works, how it compares to our recipes, and what a GB300 can borrow

November 16, 2026 · Milo (session model: claude-fable-5-2 via Anthropic; reasoning setting not recorded) · 4× DGX Spark GB10 · DGX Station GB300 + RTX PRO 6000

Created · Updated

In plain English

Cold prefill, peak9,915 tok/scandidate 60 on the ring, his bench
Cold prefill, 910K9,064 tok/s101 s for the whole prompt
One user decode82.9 tok/scandidate 60, his bench, 2.33 tokens a step
vLLM patches689applied here; tree hash matches his
Determinism gate0 diffs8 of 8 checks, 128 of 128 greedy replies identical
Our vLLM laner6xfour releases behind his latest (1107)
Contents
  1. Where candidate 60 stands
  2. How it works
  3. What candidate 60 changed
  4. The c61 decode branch
  5. His release ladder
  6. How it compares to our recipes
  7. What is useful for the GB300 and the RTX PRO 6000
  8. What we do next
  9. Caveats
Three words used below. Prefill: the model reading your prompt before it writes anything. Decode: the model writing, one step at a time. Draft: a guess at the next few tokens, checked in the same step, so one step can yield more than one token.

1. Where candidate 60 stands

The latest release is spark-ds41f-1107 (November 7). Releases are named for the day they are promoted. Between releases Owen builds numbered candidates. A passing candidate can be frozen as a GitHub pre-release for others to test; the last frozen one is candidate 54 (November 11).

Candidate 60 lives on the repository's candidate-60 and c60-prefill branches, both already merged into main. The record holds its patch series, build inputs and receipts. It qualified on his switched four-Spark and three-Spark testbeds, and a prefill sweep on his switchless ring finished later the same day. It was not yet tagged as a pre-release when I checked (commit fcc97650). The next branch, c61-decode, has code and a test spec but no results.

What "read the code" means here. Owen publishes patches, not a source tree. I checked out the vLLM commit he pins (19f2c20e from Local Inference Lab), applied candidate 60's 689 patches in order, and compared the result with the tree hash his build checks. It matched: afcaac35. So the files quoted below are the ones his image is built from. The c61 changes are six files mounted over that tree; I read their diff in full.

2. How it works

spark-ds41f topology on four DGX SparksFour DGX Sparks each hold 71.7 GiB of weights and a 10.5 GiB KV pool; sparknet one-shot RoCE all-reduces join them twice per layer.Four Sparks, one model: every layer is cut in fourEach box holds a quarter of every layer and its own slice of the conversation memory (KV pool).Spark 1 · rank 0+ APIweights 71.7 GiBKV pool 10.5 GiB16 of the 64 attention headsa quarter of every expertsmall layers: a full copySpark 2 · rank 1weights 71.7 GiBKV pool 10.5 GiB16 of the 64 attention headsa quarter of every expertsmall layers: a full copySpark 3 · rank 2weights 71.7 GiBKV pool 10.5 GiB16 of the 64 attention headsa quarter of every expertsmall layers: a full copySpark 4 · rank 3weights 71.7 GiBKV pool 10.5 GiB16 of the 64 attention headsa quarter of every expertsmall layers: a full copysparknet one-shot RoCE: the partial answers are added across the four boxes2 sums per layer × 40 layers on every step · 13–19 µs each (NCCL took 75–86 µs)Cabling (one recipe, many shapes)ring of 4 with no switch · triangle of 3or 3 to 8 Sparks through a switch (since 1107)two 200 GbE RoCE paths per neighbourSoftware (one image on every node)vLLM fork + 689 patches · TileLang kernels for GB10patched NCCL · DeepSeek TileKernelsDeepSeek's weights loaded as shippedResult: 1M-token context, 16 sessions at once, a KV pool that holds 11 full 1M contexts

Figure 1. The four-Spark shape. Memory figures are per node from Owen's release notes (71.67 GiB of weights and a 10.5 GiB KV pool per rank). Collective timings are from his collective-policy experiment.

The split. The checkpoint is about 475 GiB and a Spark has 128 GB, so each layer is cut across the nodes (tensor parallelism). At four nodes each Spark holds 16 of the 64 attention heads and a quarter of every expert's width. Small layers are copied to every node so they need no network round. The weights are DeepSeek's, loaded as shipped; nothing is re-quantized.

The network. Every layer has two points where the four partial answers are added together. That is 80 sums per step, so the sum has to be nearly free. Owen's sparknet does it in one RoCE round trip: 13–19 µs against 75–86 µs for NCCL in his test. The sum is also exact in a specific way (tilelang/collectives.py): the four parts are added in 32-bit floats in rank order and rounded once, so a row gets the same bits whatever the message size or which node it sits on.

One DSpark decode step on the spark-ds41f stackSix rows pass through forty layers; each layer has two all-reduces across the four Sparks; the verification keeps 2.33 tokens per step, about 83 tokens per second for one user.One decode step for one user: 6 rows through 40 layersThe model drafts 5 tokens ahead, then checks all of them in one pass. Gold boxes cross the network.6 rows1 real token+ 5 draftsrepeated for each of the 40 layersmix the 4streams(mHC)attention:last 128 +top 512 oldadd across4 Sparks(all-reduce)pick 6 of384 experts,run themadd across4 Sparks(all-reduce)word list8-bit screen,then exact onthe finalistscheckthe drafts:keep 2.33tokens›››Why it is slow: bytes, not math6 rows is tiny work per weight byte:about 12 operations per byte read.The GPU waits on memory (~273 GB/s),so every trick below reads fewer bytes.Tricks that read fewer bytesweights pulled into cache just aheaddoomed drafts skip the expert layersword list screened in 8 bits firstpacked 12-bit head, coded scales (exact)Tricks that cut waitingone network round per sum (one-shot)kernels chained to start early (PDL)the whole step replayed as a CUDA graphfixed add order: any batch, same answer35.7 steps a second × 2.33 tokens kept per step ≈ 83 tok/s for one user (candidate 60, Owen's bench)

Figure 2. One decode step. Step rate and tokens per step are candidate 60's on Owen's switched four-Spark testbed (35.74 steps/s, 2.328 tokens/step, 82.91 tok/s). The bytes argument is from his docs/fundamentals.md.

Decode is about bytes. A decode step for one user is six rows: the real token and five drafts. That is so little arithmetic per weight byte (about 12 operations) that the GPU spends the step waiting on memory, not computing. A step is also a chain of about 1,850 small kernels, replayed as CUDA graphs. His decode work follows from those two facts:

Prefill is about compute. A prompt chunk is thousands of rows, so the tensor cores are the limit. Two mechanisms carry most of it:

The scheduler. One long prompt is read at a time, in chunks. Two patches stop it from blocking everyone else:

Correct first. This is the part I would copy before any kernel. At temperature 0 a request's output does not depend on what else is in the batch, because every kernel adds in a fixed order. The MoE kernel file says it plainly: neither kernel uses atomics on results, so a token's output does not depend on the rest of the batch. Each candidate boots twice and must produce identical tokens and log-probabilities both times. A change counts as faster only if it beats the build's own repeat-to-repeat spread. Failed ideas are kept in a "tried and not adopted" list with their numbers.

3. What candidate 60 changed

Candidate 60 is six patches on top of candidate 55's series, about 9,200 added lines. Each patch header states its rationale, its quality impact and its tests.

PatchWhat it doesHis evidence
0811Faster prompt tokenizing: a cache of text pieces and parallel encoding split at newlines. Tool schemas can be loaded mid-session without breaking the cached prompt prefix.token equivalence on 64K–256K prompts
0812Prompt-cache retention: which cached blocks to keep under memory pressure. Repeat images are hashed once. Token lists cross between processes without a copy.a multi-turn hit rate of at least 98.5% under eviction pressure; repeat image load 31.27 ms to 1.66 ms
0813Kernels for the step that scores and picks old tokens to attend to, and the front of the attention block.1.29–1.47× at 16 and 256 rows, bit-exact; compute-sanitizer clean
0814Faster packing of the word list and expert scale codes at load; small speed-ups in sampling and draft checking.outputs match the unpatched kernels
0815The model's n-gram lookup tables sit on disk with a small cache. A prefill used to flush decode's hot rows out of it. A Clock-Pro policy now shields them.hot-row hit rate 0.19 to 1.00 with a prefill interleaved; ASan and UBSan clean
0816Hardening of the sequence-parallel side streams, and one cache-prefetch default (15/12/12 MB) for every shape.determinism unchanged on all shapes

One negative result is recorded with it. Caching the packed weights on disk to speed up boot was slower on the Spark than recomputing them on the GPU, so it was removed.

Candidate 60 cold prefill rate by prompt length on the four-node ringCandidate 60 rises from 1,843 tok/s at 523 tokens to a peak of 9,915 at 122K and holds 9,064 at 910K. Release 1107 fell from 8,657 at 131K to 6,012 at 1M. Figures are Christopher Owen's.Prompt reading speed by prompt length (candidate 60, ring of four)Cold prompts, tokens per second to the first token. Green = candidate 60. Dashed blue = release 1107.2,0004,0006,0008,00010,0001K4K16K64K256K1Mprompt length in tokens (log scale)523 tokens: 0.27 speak 9,9159,064 at 910K6,012 at 1Mbelow 4K tokens the fixedstart-up time sets the rate1107: his other bench and text,drawn at its size labelsCandidate 60 keeps 91% of its peak at 910K tokens (101 s). Release 1107 kept 73% at 1M.

Figure 3. Candidate 60 on Owen's switchless ring of four, two boots, cold Python-source prompts; x is the actual token count. The 1107 line is from his release suite with different text, so the gap between the lines is not a matched gain.

4. The c61 decode branch

His own profile says the routed experts are about 65% of a one-user step, and one-user decode has sat near 83 tok/s since 1107. The c61 record ranks twenty open ideas by expected one-user gain. The branch implements the top two as six files mounted over candidate 60.

The two changes on spark-ds41f's c61 decode branchLeft: the draft budget prices an extra verify row at 0.9 percent of a step instead of 3.5, and admits drafts with a 1.8 percent survival chance instead of 6.3. Right: the drafter's vocabulary projection uses an 8-bit screen and exact refinement of under 2 percent of columns.The c61 decode branch: two changes, read from the codeBoth target one user. No result is published on this branch yet; the gains quoted are from Owen's notes.1. Re-price a draft rowDrafting is already paid for. Checking one more rowinside the same 16-row tile costs very little.Cost of one more row, as a share of a stepc603.5%c610.9%Chance of survival a draft needs to be checkedc606.3%c611.8%so more drafts get checked per step · his figure: +6.2%2. Screen the drafter's word list tooEach of the 5 draft steps picks a word from 129,280.candidate 60reads the 248 MB packed table a second timeevery decode step (about 1.0 ms)c61one pass over the 165 MB 8-bit copy gives aproven upper bound per word; only the few thatcould win (under 2%) get the exact 12-bit mathsame picks, bit for bit · his estimate: 0.45–0.75 ms a stepGoal on the branch: 100 tok/s for one user, from about 80 on his prose bench. Not measured yet.

Figure 4. Left: derived from the constants in c61's adaptive_verification.py (draft cost ratio 0.55, single-tile row scale 0.42) at the recipe's row cost of 0.035. The candidate-60 bars are its values once its online calibration has fitted; before that it uses the recipe's fixed 0.2 threshold. Right: sizes and timings are from Owen's c61 record.

Change 1: price a draft row honestly. In candidate 60 a step's cost is modelled as 1 + 0.035 per row. That treats drafting as free and every row as equally expensive. The c61 file adds two constants for a step that fits in one 16-row tile, which is every one-user step:

The arithmetic that follows: one more row costs 0.9% of a step instead of 3.5%, and a draft is worth checking if its chance of surviving is above 1.8% instead of 6.3%. So more drafts get checked. The same change starts the confidence estimate from five fitted per-position offsets (−0.52, −0.64, −0.96, −0.85, −0.74) instead of zero, so a fresh boot does not under-draft while it learns. His note on the underlying pull request gives +6.2% for one user at steady state.

Change 2: screen the drafter's word list. The drafter picks its five tokens one after another, and each pick is a full pass over the output table plus a small correction that depends on the previous pick. Candidate 60 already screens the main model's pick with the 8-bit copy. c61 does the same for the drafter: one screening pass gives a proven upper bound for every word, the correction is added to the bounds, and only words whose bound could beat the current best get the exact arithmetic. The picks are identical by construction. It is switched on only when a single request is decoding. His estimate is 0.45–0.75 ms a step.

The branch's test spec compares candidate 60 with c61 on the ring: 150 s of one-user traffic, 90 s at 16 users, and a greedy determinism record. No receipts were in the repository when I read it. The stated goal is 100 tok/s for one user, from 79.9 on his prose bench. The two changes above do not add up to that by his own estimates; the rest of the list (expert-layer pruning for doomed drafts, kernel fusions) is what would have to carry it.

5. His release ladder

spark-ds41f release ladderCold 131K prefill rose from 5,982 tok/s at r6x to 9,924 at candidate 60; single-stream decode from 74.1 to 82.9 tok/s. Figures are Christopher Owen's.Five weeks of releases, by Owen's own benches (four Sparks)Gold = the release our vLLM lane still runs. Blue = his releases. Green = candidates, not released yet.Reading a 131K-token prompt, cold (tok/s)higher is betterr6x5,98210225,98510256,79610307,39611078,657c589,755c599,886c609,924One user decoding (output tok/s)higher is betterr6x74.1102274.1102575.3103078.0110782.6c5883.3c5982.9c6082.9r6x → candidate 60: prompt reading +66%, one-user decode +12% (his numbers, not ours)

Figure 5. Owen's numbers, not ours. Releases r6x to 1107 are from his README's release comparisons (four-node ring). Candidates are from his experiment records: c58 on the ring, c59 and c60 on his switched testbed. One-user figures for the candidates are from his c1-decisions bench, a different instrument from the releases' single-stream suite.

6. How it compares to our recipes

Our four-Spark recipes are in J&M Recipes. Two of them serve this model on Sparks, and one is built on Owen's work.

spark-ds41f (Owen)Our vLLM lane (Sparks 5–8)Our SGLang lane (Sparks 1–4, daily)
Enginehis own: vLLM fork + 689 patches, TileLang kernels, sparknethis r6x image, built locallyknapcio's SGLang profile v2.5
Version1107 released; candidate 60 in progressr6x (October 18) + his late-admission patch as a mount + sparknet 0.5.0v2.5 with vision
Seats and chunking16 seats, 16,384-token batch budget, late admission at layer 30, prefill yield32 seats, 8,192 budget, chunk threshold 3,856, late admission at layer 20the profile's defaults
Cablingring or triangle with no switch; 3–8 through a switch since 1107one switch, 4 KiB pagessame switch
What the author doeswrites kernels and scheduler patches, one measured change at a timewe pick engines, set the shape for our traffic, and publish receipts and failed arms
Quality barbit-identical temperature-0 output across boots and batches; LongBench, RULER, HumanEval+, tool and vision evals at releaseteacher-forced log-probability gate against a kept control, qeval 71/75, 1M needle, a 20-hop agent run

So this is not engine against engine. On the vLLM lane the engine is his, and what is ours is the shape and the evidence. Two things follow.

We are four releases behind. Our lane still runs the gold bar in Figure 5. The last time we measured a newer release was October 26, when we built 1025 and ran it as he ships it next to our lane and our daily SGLang lane. Same instrument, medians of three; the two vLLM arms ran on Sparks 5–8 and the SGLang lane on Sparks 1–4:

Our battery, October 26 (thinking off, temperature 0)Owen 1025, his shapeOur vLLM lane (r6x, our shape)Our SGLang lane
One user, per-stream decode (tok/s)110.4105.698.9
16 users, total (tok/s)475.1481.6444.2
Cold prefill at 32K / 131K (tok/s)6,393 / 6,4065,467 / 5,2626,368 / 6,449
1M-token cold prefill (s)218.3260.0239.3
Short chat arriving during a cold long prefill, first token (s)9.091.522.98
Turn on a cached 68K document behind a 413K prefill, first token (s)66.542.2065.83
qeval, of 75717171

Read it as two results. His newer kernels were faster: +4.5% for one user and +17% to +22% on prefill, with the same quality score. Our shape was far better when traffic is mixed, which is what guests and agents on this rack produce. More seats, smaller prefill chunks and earlier admission are settings, not kernels.

That table is now stale on his side. Prefill yield (1030), a scheduler change for deep contexts (1107) and candidate 60's cache-retention policy all aim at the two rows where our shape won. His own figure for a short request beside a 131K prefill went from 12.36 s to 1.02 s across those releases. We have not measured any of them. The honest status is: our shape won in October against a release he has since replaced twice.

Do not compare his tok/s to ours. His bench read 75.3 tok/s for one user on 1025. Ours read 110.4 on the same release. The instruments differ (prompts, thinking on or off, what counts as decode time). Compare within one table, never across.

Three smaller points:

7. What is useful for the GB300 and the RTX PRO 6000

Our GB300 lane is a different machine and a different engine: one Station GPU holding the hot experts, an RTX PRO 6000 in the same box computing the cold ones, Garner McCloud's engine under our policy (the Pg62 recipe). In its promotion window it decoded at 7.2 ms a step and 439 tok/s for one user on our general-prompt set. Nothing of Owen's installs there. Some of it is still worth the work.

Which spark-ds41f ideas transfer to a GB300 with an RTX PRO 6000 sidecarThree columns: ideas to port to the GB300 lane, ideas for the RTX PRO 6000 sidecar, and Spark-only work that does not transfer.What moves to the GB300 + RTX PRO 6000 laneHis kernels are written for the Spark's GPU (SM121). The GB300 is SM103, so ideas move, not files.Port to the GB300 laneShort requests jump the queueprefill yield + late admission8-bit word-list screenexact result, fewer bytes per stepSkip experts for doomed draftsfewer cold-expert calls to the RTXHost-side fast pathstokenizer cache, KV retentionRTX PRO 6000 sidecarSame GPU family as the SparkSM120/SM121: same block-scaled opsHis FP8×FP4 expert kernelfixed add order, no atomicsCandidate for our >64-token pathtoday that path is not bit-stableUntested on SM120needs a build and a parity gateStays on the SparksRoCE collectives, ring cablingone GPU has nothing to add upCache prefetch windowstuned to GB10's 24 MiB L2Disk Engram reader + shieldours lives in Grace RAM64 KiB pages, memory saverGB10 unified-memory fixesNone of it is a drop-in. The test discipline moves today; each kernel idea is a port with a gate.

Figure 6. Sorting by what it would take. Nothing in the first two columns has been measured on the Station; each is a candidate for a pre-registered card.

Port to the GB300 lane

  1. Short requests jump the queue. Late admission and prefill yield are scheduler changes in vLLM's Python, not kernels. On our Sparks, his late-admission patch cut a guest turn behind a 413K cold prefill from 1.885 s to 1.196 s. The Station lane is vLLM too, on a different tree, so this is a port and not a mount. It matters there because guests share that lane with long agent prompts.
  2. The 8-bit word-list screen. The last layer multiplies by a 129,280-word table. He keeps an 8-bit copy, uses it with a provable error bound to rule out almost every word, and runs the exact arithmetic on the finalists only, so the answer is bit-identical. On our Station lane in October that head cost 0.42 ms of an 11.2 ms step: two full reads of the table, already as fast as memory allows. That was the v27 lane and has not been re-measured on Pg62, so the size is unknown, but it is one of the few step costs we had marked as having no cheap fix.
  3. Skip the experts for drafts that will not survive. He scores each draft row's chance of being accepted and lets rows under 0.2 skip the expert layers. On the Station every extra row can pull in experts that live on the RTX. Fewer live rows means fewer distinct experts per step. Whether it beats what the lane already does is a measurement, not a guess.
  4. Price draft rows the c61 way. The change is under 200 lines of scheduler Python and two constants. The idea is general: once drafting is paid for, the question is only what one more row costs. His constants are fitted to the Spark's 16-row tile and would have to be re-fitted from our own step timings. On the Station an extra row can also mean an extra trip to the RTX, so the fitted price there may be higher, not lower.
  5. Host-side fast paths. The tokenizer cache, the repeat-image fast path and the prompt-cache retention policy (patches 0811 and 0812) are engine-side Python. They help the time before the first token on long or repeated prompts. They will not move the decode step, which on that lane is GPU-bound.

The RTX PRO 6000 sidecar

This is the one place where his kernels might run with little change. The RTX PRO 6000 is SM120; the Spark's GB10 is SM121. His kernel files say so in their first line: tilelang/gemm.py is the "block-32 FP8 linear for SM120 / SM121" and tilelang/moe.py is "MoE dispatch and combine for SM120 / SM121". Both are built on the same block-scaled tensor-core instruction.

What the sidecar does all day is what his routed-expert kernel does: FP8 activations against FP4 expert weights, a handful of rows at a time. For steps of 64 tokens or fewer our sidecar already runs a kernel of that class with no atomics on the data. Above 64 tokens it falls back to an older path that accumulates with atomic adds, which is one reason that lane's output is not bit-identical from run to run. His combine sums each token's expert outputs in slot order in 32-bit floats, with no atomics on results, at every row count. That makes it a candidate for the large-batch path. It is untested on SM120: it needs a build, a parity gate against the current path, and the 32-stream stress that has caught our own kernels before.

His memory tricks (coded expert scales, the 12-bit pack) would let more experts sit on either GPU. We measured that lever directly on November 10: going from 255 to 275 hot experts per layer brought the first token of a 61K prompt 11–12% sooner and left one-user decode at 0.98× of the control. So they are a prefill and capacity idea on this lane, not a decode one.

Stays on the Sparks

The RoCE collectives, the ring cabling and the padding for odd node counts solve a problem one GPU does not have. The cache-prefetch windows are tuned to the GB10's 24 MiB L2. The disk reader for the model's n-gram memory tables, and candidate 60's guard on it, are for tables on NVMe; ours sit in the Station's host RAM. The 64 KiB kernel and memory saver are GB10 unified-memory fixes.

Usable today: the method

8. What we do next

  1. When candidate 60 is frozen, build it on Spark 5 and fan it out to Sparks 5–8.
  2. Run it twice through the October battery: once in his tp4-switched shape as shipped, once with our seats, chunk threshold and admission layer. The r6x lane stays kept as the control.
  3. Gate on the teacher-forced check and the 1M needle before any speed number counts.
  4. Promote the arm that wins the two mixed-traffic rows without giving back prefill. If his shape now wins them, we retire our overrides and say so.
  5. Watch c61. If its screen lands with a one-user gain and identical output, it goes into the same rebuild.
  6. For the Station: write the 8-bit head screen and the draft-row pricing as cards with bars, and price the SM120 build of his expert kernel as a feasibility note before promising a boot.

None of these is scheduled. They need James's go and a window on Sparks 5–8, which serve guests today.

9. Caveats

Sources