Kimi-K3 on the same pod, two engines: why TensorFold decodes 2.4–3.8× faster

Scoreboard

Source labels. [R] reproduced by us today, running Ash's own runner on the pod. [A] Ash's handover or logs, not reproduced. [O] our pod's published receipts from October 30–31. [M] matched run today: our vLLM pod booted fresh and fed Ash's exact prompt token ids. [D] arithmetic on the numbers above, shown in the text. Single-stream figures on both engines time decode only; prompt reading is excluded on both sides.

Same hardware, same weights

Everything in this post ran on one pod. The Station is a DGX Station GB300: a Blackwell Ultra GPU with HBM, a Grace CPU with about 494 GiB of LPDDR joined to the GPU by NVLink-C2C, and an RTX PRO 6000 Blackwell (Max-Q) in the same chassis. Eight DGX Sparks (GB10, about 117 GiB usable each) hang off two MikroTik CRS804 switches over RoCE; the Station's mlx5_0 port serves Sparks 1–4 and mlx5_1 serves Sparks 5–8. The checkpoint is nvidia/Kimi-K3-NVFP4 at revision b2428a0b, read-only and unmodified: 93 layers, 896 routed experts per MoE layer, 16 of them picked per token, 82,432 routed experts of 18,579,480 bytes each. Both engines run attention, routing and the final add on the Station GPU, and both send expert work to the Sparks and the RTX. The difference is entirely in how the experts are placed and computed, and in what the program is.

Reason 1 — slice every expert across all eight Sparks

This is the biggest structural idea, and it is simple once you see it. In our pod each Spark owns whole experts: 62 of each layer's 896, so 496 per layer across the eight. When a token's router picks its 16 experts, those picks land on the Sparks unevenly. One Spark might hold three of them, another none. A Spark computing k whole experts takes 73 + 77·k µs, measured on the real bank with a graphed worker, and the 77 µs is just reading one 18.6 MB expert at the GB10's roughly 248 GB/s. The layer cannot finish until the busiest Spark replies. So every layer, 92 times a token, our pod waits for whichever Spark drew the short straw.

TensorFold does not give any Spark a whole expert. Each Spark holds one-eighth of the intermediate columns of all 511 Spark-tier experts. Whatever the router picks, every Spark has exactly one-eighth of the work and all eight finish together; each returns a partial sum and the Station adds the eight partial sums in a fixed order. The arithmetic: 18,579,480 B / 8 = 2.32 MB per expert per Spark, and 2.32 MB at 246 GB/s (the rate Ash reports the Sparks sustaining at five rows) is about 9.4 µs. Nine Spark-tier experts in a layer come to 9 × 2.32 MB ≈ 21 MB, about 85 µs of reading, on every Spark alike. The whole-expert way, the same nine experts could put 3 × 77 µs plus the fixed cost on one unlucky Spark while two others sit idle. The slices live only in GPU memory; the Station sends them at load, about 109 GB per Spark, and the workers keep no weights on disk.

Reason 2 — put the hottest experts in the fastest memory

Not every expert is used equally. Our placement, plan-4tier-v1, ignores that: it is uniform and random, so a heavily used expert is as likely to sit in Grace memory as in HBM. On the Station, reading one expert costs about 3 µs from HBM and about 48 µs from Grace over NVLink-C2C at roughly 345 GB/s, a 48 / 3 = 16-fold difference the uniform plan never exploits.

TensorFold's placement file, place_cal2.bin, ranks each layer's 896 experts by how often drafted decoding of eight calibration prompts picked them. The hottest 104 go to GB300 HBM, the next 58 to the RTX PRO 6000, the next 511 to the Sparks (sliced, as above), and the coldest 223 to Grace. 104 + 58 + 511 + 223 = 896. Grace still holds a quarter of the experts, but they are the ones a token rarely asks for, and the ranking comes from calibration text, so it is only as good as the match between that text and yours.

Reason 3 — one native program

TensorFold is Zig and CUDA. One 308 MB runner binary, built for sm_103a (the GB300), sm_120 (the RTX PRO 6000) and sm_121 (the GB10), runs on the Station and on all nine workers: the same file everywhere. Its transport is Ash's MCDMA fabric library: a UDP rendezvous on the fabric interfaces, then RDMA. There is no Python anywhere on the per-token path.

Our pod is stock vLLM 0.31.0 plus an external Python hook (k3pod.py, no fork) plus a C proxy thread that does the network send and collect. We spent the night of October 30 taking Python out of that path piece by piece: the graph-resident expert step that Javier suggested took a token from 79.9 ms to 63.7, and CUDA-graphing every worker's expert call took it to 54.0. Per layer at batch 1 we are at 0.59 ms, of which 0.305 ms is proxy send to last worker reply and 0.154–0.171 ms is the Station's own attention, router and shared experts. That is a respectable place for a hook on a general-purpose server to be. It is still a hook on a general-purpose server.

The combined effect of reasons 1–3, with no drafting on either side, is plain greedy decoding at about 40.6 ms per token on TensorFold (1000 / 24.6) against 54.0 ms on ours: 1.33×. On the matched prompts the range is 24.1–25.2 tok/s against 18.4, or 1.31–1.37×. We did not measure how the 13.4 ms (54.0 − 40.6) splits among slicing, placement and the native path, so we will not pretend to.

Reason 4 — draft, then check several tokens in one pass (DSpark)

This is the biggest multiplier, and it is the part worth understanding even if you never touch a Spark. A big model writes one token per pass, and a pass is mostly spent reading weights, not doing arithmetic. So checking a second token in the same pass is nearly free if you already know what to check. Speculative decoding supplies the guesses. A small drafter, here RadixArk's Kimi-K3-DSpark at 4.5 GB, proposes up to seven tokens ahead. A confidence head decides how many of them, r, are worth checking. Kimi-K3 then runs one pass over all r rows at once and compares its own choice at each position with the guess. It keeps the guesses up to the first disagreement, adds the token it would have written there itself, and the next round starts from the result. Because the check is greedy, the output is token for token what plain greedy decoding would have produced: faster, not different. TensorFold asserts this on every run (K3_CHECK=1 reruns plain greedy and compares), and every run we saw printed PASS.

Why it pays so well here: Ash's cost model, measured on this pod, is that a pass of r rows costs about 38 + 4.4·r ms. The 38 is the pass; each extra row adds only 4.4 ms, because draft rows of one stream mostly touch experts the pass is reading anyway. So checking three to six guesses costs little more than checking one, and every guess that lands is a token for nearly nothing. Measured today on the Python prompt the drafter was right often: 5.86 drafts checked a round, 5.56 tokens kept a round, 69.7 tok/s. On the chat prompt it was harder to predict: 2.95 checked, 2.44 kept, 44.4 tok/s. Plain greedy on the same engine runs 24.1–25.2 tok/s on these prompts, so drafting alone is 1.79–2.89× depending on how predictable the text is.

Why more rows are cheap on TensorFold

The same chart explains both drafting and concurrency. On our pod, adding a concurrent stream costs (172.1 − 54.3) / 7 = 16.8 ms per pass on our own benchmark prompts, and (189.3 − 54.4) / 7 ≈ 19.3 ms on Ash's eight prompts today. On TensorFold, adding a lane costs (107.6 − 40.6) / 7 ≈ 9.6 ms, about half, and a draft row of the same stream costs about 4.4 ms in Ash's model, because its experts overlap with the rows already in the pass. A new stream brings its own 16 experts a layer; a draft row mostly reuses them. On our pod every row's experts also have to clear the busiest Spark, which is where the whole-expert layout bites again.

Where our build is still ahead

Prompt reading. Ours reads a 3,364-token prompt at 562 tok/s, first token in 5.99 s (remeasured today: 570.6 tok/s, 5.90 s). TensorFold reads prompts at 103 tok/s: 6,144 tokens took 59.9 s. The reason is the mirror image of reason 4. Our prompt path sends each worker 1,024-token chunks and runs them as one batched GEMM, so each expert is read from memory once per chunk and amortised over every prompt token that needs it. TensorFold's Kimi-K3 path has no prompt kernel yet; prompts go through the decode kernels 16 rows a pass, so experts are re-read every 16 tokens. That makes ours about 5.5× faster at the prompt (562 / 103). Ash's own next step is a real prompt path, which he estimates at about 1,000 tok/s; that is his estimate, not a measurement.

It is a server. Our pod is an OpenAI-compatible endpoint with streaming, sampling and eight concurrent sequences at up to 4,096 tokens of context. TensorFold's Kimi-K3 path is a test runner: prompt files of token ids in, greedy token ids out, no HTTP, no streaming, no sampling. It handles prompts up to 8,192 tokens (6,144 tested), and a run takes the whole pod for 10–15 minutes including a 6.5-minute warm load (about 25 minutes cold). Neither of those is a criticism; it is what a research runner is for. But if you want to talk to Kimi-K3 on this pod today, ours is the one that answers.

Is it right?

Speed without correctness is a random number generator, so here is what each engine can show, side by side, with the limits stated.

The two engines against each other. On the same prompts, ours and TensorFold produce identical tokens for the first 12–23 tokens on three prompts and then diverge; on the code prompt the very first token differs (a # heading versus "Here's"), a near-tie. Both outputs are coherent and on topic: both engines open the chat answer with "# Photosynthesis: How Plants Turn Sunlight into Food". Divergence between the engines is not evidence that either is wrong. Ours is not even consistent with itself from run to run, so it cannot be the yardstick; TensorFold, on the evidence above, is reproducible. What neither has is a teacher-forced comparison against a stock copy of the full model, because no single machine here can run one. That gate is still open for both.

What we checked ourselves

We did not take the handover's numbers on trust. Between 09:46 and 10:03 CST today we ran Ash's own k3_run.sh drafted and then lanes on the pod, as his account, with his placement and prompt files. Model load took 368.1 s. The runs produced the same rounds and tokens as Ash's logs and PASS on all four prompts.

Then the other direction. From 10:06 to 10:34 CST we booted our vLLM pod fresh in its exact parked configuration (vLLM 0.31.0 with the k3pod hook, --max-num-seqs 8, all nine workers graphed), warmed it, and fed Ash's exact prompt token-id files through /v1/completions: greedy, streamed, 200 tokens, two repetitions each, decode rate taken from first token to last so prompt reading is excluded on both sides. Steady state was 18.33–18.38 tok/s (54.4–54.6 ms a token; the published 18.4–18.5 stands). The first request after warmup ran slower, 15.41 tok/s, and is reported but not used. Eight of Ash's lane prompts at once, 128 tokens each, two repetitions: 42.31 and 42.21 tok/s over the decode window, so 42.2 against TensorFold's 74.36, or 1.76×. Prompt reading remeasured at 32 / 284 / 1,124 / 3,364 tokens: 45.8 / 260.8 / 418.2 / 570.6 tok/s, consistent with the published 562.

What this means for our build

Two of Ash's ideas transfer to the vLLM pod without changing what it is. Slicing experts across the Sparks replaces "wait for the busiest Spark" with "all eight finish together", and the Spark tier is where our per-token time goes. Frequency-ranked placement puts the experts a token actually asks for in HBM instead of Grace, at a 3 µs versus 48 µs difference per read. Both are placement and worker changes, not engine changes. Drafting is the third thing, and the largest, but it is a different kind of work: a drafter, a verification step and a server that still streams and samples correctly around it. We are not going to promise numbers for any of these before we have measured them. We will say that today's comparison turned "our Sparks set the floor" from a suspicion into a mechanism with a fix attached, and that is worth more than the 13.4 ms.

Credits

Ash Hart built TensorFold and did every TensorFold measurement in this post: github.com/ashhart/TensorFold (main is public, Apache-2.0) and x.com/ashxhart; the transport is his MCDMA fabric library, github.com/ashhart/MCDMA. The Kimi-K3 work is on his lead/k3-cuda branch, which is not public yet; the source and the runner are on the pod under his account. The handover, the placement, the cost model and the drafting results are his; our part was to rerun them and to run our own pod on his prompts.

RadixArk for the Kimi-K3-DSpark drafter. NVIDIA for the NVFP4 Kimi-K3 checkpoint. Moonshot AI for Kimi-K3. Javier (@javierbits) for the graph-resident expert step in our build. The vLLM project for the engine our pod is built on. Our side of the story is in Kimi K3 NVFP4 on Station, and 8 Sparks, the four-tier page and the Station + Sparks fabric how-to.

Provenance

All numbers come from one fact sheet assembled on November 1, 2026 from commands run that day and named receipt files. [R] TensorFold reproduced by Milo, 09:46–10:03 CST, running Ash's k3_run.sh drafted and lanes under his account on the pod: four drafted prompts (200–204 tokens each), their plain-greedy checks, and the 8-lane run. [A] Ash's handover (HANDOVER-kimi-k3-pod.md, November 1) and his logs: the placement counts, the cost model 38 + 4.4·r ms, the 103 tok/s prompt figure, the float64 parity figure and the ~1,000 tok/s prompt-path estimate; not reproduced. [O] our pod's published receipts of October 30–31 (the earlier post, STATUS.md, PARKED-2026-10-31.md). [M] our vLLM pod rebooted fresh 10:06–10:34 CST in the parked configuration and fed Ash's exact prompt token ids, two repetitions each; the 8-stream and prefill remeasurements the same way. [D] arithmetic shown in the text. Modelled numbers (Ash's 38 + 4.4·r line, the ~85 µs slice read, the ~1,000 tok/s estimate) are labelled as such wherever they appear and are never called measurements. The hardware is described in the earlier posts; nothing about it changed between the two engines' runs.