November 6, 2026 at 11:27 AM CST · charter and measurements: James Meadlock with Milo (one session on anthropic/claude-fable-5-1) · writing and diagrams: one headless session on anthropic/claude-fable-5.1 · one DGX Station GB300 (+ RTX PRO 6000 for recipe B)

Can One GB300 Station Run DeepSeek-V4.1-Flash in 5 ms a Step?

A charter, plus the first two days. Our standalone runtime decodes the real model on the Station with its arithmetic kept; the project's own instrument — the 6-row DSpark step, drafts verified by exact greedy match — now exists and reads 20.1 ms a step, 6.9 ms a token (production: 7.36 ms a step, about 2.6 ms a token). The 5.0 ms bar turned out to be a bytes bar, and the bar is now: beat 7.36 and publish ms per token (section 8).

Created November 6, 2026 at 11:27 AM CST · Updated November 7, 2026 at 2:43 PM CST

1. The machine and the model

One GPU with 250.7 GiB of visible HBM, a Grace CPU with 494 GiB of RAM, NVLink-C2C between them; pinned host reads measure around 340 GB/s. DeepSeek-V4.1-Flash-1017 is a 552B-parameter MoE plus 196B of Engram tables, about 475 GiB on disk: 40 MoE layers × 384 MXFP4 experts × 17.93 MiB = 269.0 GiB of expert weights, plus two Engram tables of 94.6 GiB. That cannot all fit HBM (fact, verified 2026-09-10): Engram lives in pinned Grace RAM, and so do some experts. An RTX PRO 6000 (96 GB) sits on PCIe with no P2P path to the big GPU; its results come back through host memory.

Figure 1 — Where the bytes can go. HBM and Grace are facts of the hardware; the ~340 GB/s is a measured pinned-read figure from our notes. The HBM box lists what lives there without a byte total, because we have not published one.

2. Where 11 ms goes

The only full per-role anatomy we have is from the earlier v27 build (2026-10-27), traced with nsys at roughly 0% overhead: 11.216 ms per step, one user, 6 rows per step (DSpark speculative decoding, k=5). GPU 98.6% busy, 0.15 ms idle. 1,305 kernel launches per step; 1,035 ran under 10 µs yet together cost 4.54 ms, 40% of the step. Counters: SMs active 49%, SM issue 6%, tensor 2.3%, DRAM read 32%. Our reading: latency- and occupancy-bound on many tiny kernels; not compute-bound, not launch-bound (CUDA graphs already hide the launches).

Today's lane, Pg61 (vLLM v0.31.0, Garner McCloud's public dgx-station-recipes, our patches), runs the same decode at 7.36 ms (measured, release round 2026-10-31). It has not been profiled per role; that profile is the project's first job.

Figure 2 — The v27 step by role (measured, nsys, 2026-10-27). "Other" is 11.216 ms minus the listed roles and idle: 1.34 ms. "Sidecar wait" is the _finish2 span where the GPU has finished its hot-expert math and is waiting for the RTX card's rows to come back over PCIe. Pg61's 7.36 ms step has no equivalent breakdown yet.

3. Why this is not a megakernel project

Start with what a perfect megakernel cannot remove. Bojie Li's sm_120 microbenchmarks (author's claim, 2026-09-26) put a PDL one-to-one handoff at about 0.37 µs and an all-SM gather at about 1.0 µs; at roughly 1,370 dependency boundaries per step, that is 0.5–1.4 ms of irreducible cost even inside one kernel. Our outside-view model (W1, 2026-10-28) prices everything fused-layer or PDL work can address at 50–70 µs per layer: 2.0–2.8 ms theoretical, −0.7 to −1.4 ms realistic, 6–13% of the 11.0 ms v27 step. Pg61 has already spent some of that pool (it inherits the peer's PDL and fused-sampler tree), so it cannot be subtracted again from 7.36 ms.

Peer evidence agrees (author's claims, 2026-10-28): a PDL reconstruction of the Hazy megakernel reaches 97.3% of a single persistent kernel with 81 ordinary kernels; SGLang #46088 found a fused Mega-mHC slower at every batch size; vLLM-Moet measured a +2.7% step regression from fusing routing into the MoE kernel. Nobody has published one kernel per forward with a PCIe sidecar, host Engram, sparse indexer and speculative decoding in the loop.

The peers who are fast use a chain. NVIDIA's sgl-nvidia-mega SGLang plugin (PR #1, merged 2026-10-19) decodes DeepSeek-V4.1-Flash at a p50 of 3.386 ms per step, batch 1, 6 rows, on four GB300s in TP4: 10–15 right-sized kernels per layer handing outputs through mailboxes, PDL and side streams. Per-GPU weight bytes there are a quarter of ours and no experts live off-GPU; that number is evidence for the shape, not a target for one Station. vLLM's specialized decode for Kimi K3 (2026-10-16) is the same shape: 23 kernels → 13 per layer, 130 µs → 69 µs, on GB300 at TP8/TP16.

So the project (first chartered under the banner "megakernel", renamed the same day) is defined as whatever makes the decode step shortest: first a fused multi-kernel chain with mailboxes and PDL, one persistent kernel per decoder layer as the stretch, one kernel per forward not a goal. The vehicle is a standalone decode runtime: one Python process, no vLLM, no SGLang; Python host, CuTe DSL kernels, Triton glue, CUDA C++ for the scheduler and mailboxes. Only decode gets optimized; prefill speed is a non-goal until the headline is met. Milestone 0 borrows every existing kernel as a library (DeepGEMM MegaMoE, FlashMLA sparse attention, FlashInfer MXFP4 MoE and MXFP8 GEMMs, vLLM's Engram gather, the peer's draft samplers) and replaces a piece only when the step trace names it and the replacement is measured against it in the same window.

Figure 3 — The fusion pool is modeled (W1, 2026-10-28) against the measured v27 step; the hatched slice is the realistic −0.7 to −1.4 ms. Pg61 is measured. The target is a decision. The gap between Pg61 and the line is larger than anything fusion is modeled to buy.

4. The two recipes

Why the round trip costs what it does: per layer at 6 rows, the hot-expert kernel touches about 25 unique experts, roughly 448 MiB, about 65 µs at ~7 TB/s of HBM (modeled, W1; we measured ~81 µs). Then the GPU waits on the RTX card, a ~112 µs MoE section on about 80% of layers: the 1.33 ms _finish2 wait on v27. The hot math is not the problem; the waiting is.

Recipe A comes first and is Grace-only. The thesis: a fused layer gathers the cold expert rows it needs directly from pinned Grace RAM over C2C, inside the kernel. One memory system, no sidecar, no GPU spinning on a PCIe mailbox. Recipe B is the same runtime plus a persistent cooperative kernel on the RTX PRO 6000: the peer's "GP1" shape, the configuration we know is fast. If A's C2C self-fetch loses to B on measurement, that is a published result.

Figure 4 — One MoE layer under each recipe. In both, the hot expert rows are in HBM and Engram stays in Grace. The difference is where the cold experts are computed: inside the GPU's own kernel from C2C-reachable Grace memory (A), or on a second card that answers over PCIe (B).

5. The number and the ladder

The headline metric is single-user decode step time at 6 rows, measured with nsys on the cudaProfilerApi range over a fixed prompt corpus. Baseline: Pg61, 7.36 ms. Bar, restated November 7, 2026: beat Pg61 — under 7.36 ms per 6-row step — and publish ms per token alongside it. The charter's original bar was ≤ 5.0 ms per step, chosen to exceed the modeled fusion pool on purpose; the first two days showed it is a bytes bar the box cannot meet with stock weights (section 8), and James restated it. 5.0 stays on record as the original.

Month labels are targets, not promises. Context ladder is 32K → 128K → 1M; 1M gates production candidacy only. Rows C1–C8 are designed in; C32 is a declared non-goal of the runtime.

Kill rule: milestone 0 by December 18, 2026 (six weeks), or Pg61 is restored and the project parks as desk research.

6. What stays fixed

Stock weights: no NVFP4, no requantization. No KV-cache quantization. No expert pruning, no router masking; all 384 experts stay. Quality gates as calibrated. Determinism is a design requirement: bit-identical output across runs and relaunches, fixed-order reductions, no atomics on the data. Today's sidecar lane fails that (measured: 79,376 of 100,000 sidecar calls differ between two runs of the same binary; reduction order depends on arrival when two or more cold experts are summed). The one thing reopened is layout: weight tiling on disk and in HBM may change, as long as every value stays bit-identical to the pinned checkpoint (HF revision 1bae0e10).

7. Status today

Day one and the night after (November 6–7, 2026). The standalone runtime exists and decodes the real checkpoint on one GB300: DeepSeek's own reference model.py, unchanged, on our tiered loader (hot experts and everything dense in HBM, 218.6 GiB; cold experts and the Engram tables in pinned Grace RAM, read over NVLink-C2C from inside the kernels). Boot is about 60 seconds. The 1-row decode step went 404.9 → 13.9 ms in a day and a night; at 13.9 ms the GPU is busy for 13.95 ms of it, so there is no launch overhead left, only kernel time (2,827 kernels per step, down from 20,550). Every rung was measured with the same instrument and graded against the rung before it: five of the nine are bit-identical (every fp32 logit hashed at every position, 32 steps × 8 prompts, replay and relaunch, memory fills on and off), the other four are kernels we borrowed or wrote that reproduce the reference within one bf16 ulp on a small fraction of outputs. The exact path stays selectable for parity runs.

Figure 5 — The first 36 hours of the runtime. Measured on the Station (cuda events over steps 32–512 of a 512-token greedy generation, 1 row). Receipts: docs/first-light-*.json and docs/NIGHT-LADDER-20261107.md in the repo.

All figures measured on the Station on November 6–7, 2026 (cuda events, steps 32–512, 1 row, greedy). Not comparable to the 7.36 ms production number, which is a 6-row speculative step on a different instrument; the project's headline instrument arrives at milestone 1.

Where the remaining 13.9 ms is, and what it means

The profile at 13.9 ms is 2,827 kernels averaging 5 µs: the 290 projections' GEMV 3.6 ms, the fused norm 1.7, the routed-expert GEMMs 1.6, residual glue 1.4, the fp32 head 0.4, attention 0.4. The bytes the step actually touches (about 15.9 GB: 5.9 GB of fp8 projection weights, 4.5 GB of routed experts — 240 expert calls × 18.8 MB — 2.7 GB of bf16 attention output weights, 2.65 GB of head) would take ~2.5 ms at HBM rate. Corrected November 7: this first said about 9 GB and ~1.2 ms; the routed experts and the bf16 wo_a weights had been left out of the count. The step is latency-bound inside the graph: most kernels move under a megabyte and take 3–20 µs regardless. There are 3–4 ms of straightforward wins left on the 1-row path (in progress), and then the structure has to change: fewer, longer kernels (milestone 2), or more rows per step (milestone 1), which amortizes every latency-bound kernel except the MoE and the head. The design memo for the 6-row speculative step says plainly that no 6-row configuration of today's kernels beats 7.36 ms — the 5.0 ms bar needs the fused chain — and the 6-row measurement followed the same morning (section 8). The 1-row step itself reached 10.5 ms after this profile (T9b-4: the fused norm's reduction split, the split-K reduce folded into the GEMV, three pairs of same-input projections merged).

Parity against production, and what we got wrong

Against production's own greedy outputs over eight fixture prompts and 256 tokens, the runtime is token-identical on two, and on the other six it diverges at a near-tie (positions 1–152) with zero top-1 disagreements before the split and mean |Δlogprob| ≤ 0.05 nats on the shared prefix — unchanged at every rung above. We first suspected the KV-cache format (the reference stores the compressed latent through an FP4 quantize-dequantize; vLLM stores MXFP8); swapping ours to vLLM's moved the divergence points and did not shrink the drift, so that hypothesis is dead. The decisive check was production against itself: vLLM with one KV-format knob changed scores the same 2 of 8 against its own baseline, at the same token positions on three prompts. The charter's parity bar ("no worse than production's run-to-run spread") therefore measures vLLM against itself and nothing else can meet it; a replacement bar (bit-identical self-replay; ≤ 0.05 nats and zero weighted top-1 flips against production; tiered placement bit-exact against all-HBM) is proposed in PROJECT.md and awaits James's decision. Two other misses: the per-role nsys breakdown of production never came (both profiler-wrapped boots stalled once the capture range opened; parked), and the first GEMV round was numerically perfect and slow — no split-K, serial rows — and needed a second round. The glue census also undercounted: it counted launches correctly (~5,300) but at 1 µs each; measured they were 2.3 µs, so the glue was 12 ms, not 5.

The Station lane is dark between measurement windows. Production serves from the four-DGX-Spark SGLang lane. Everything above is in the public repo with its receipts: github.com/jmeadlock/gb300-dsv41-decode (Apache-2.0; docs/LEDGER.md is the running log, docs/NIGHT-LADDER-20261107.md the night's table). Runtime code was written by headless coding workers on claude-opus-5-5 from written handoffs, and measured, merged and fact-checked in one Milo session on claude-fable-5-1.

8. The headline instrument, and what the bar turned out to be

The number the charter asks for exists (November 7, morning). The runtime now runs DeepSeek's speculative decode the way production does: a 6-row verify step (the last accepted token plus five drafts), acceptance by exact greedy match with the confidence head off — what vLLM 0.31 does for this model — then a three-stage DSpark draft, all on the device inside two CUDA graphs. Both halves were proven exact first: the 6-row verify step is bit-equal to the reference model at 6 rows (684 fp32 logit rows, every cache tensor, both compressor parities), and the draft stage is bit-equal to the reference forward_spec (43 draft sets, all three rings). The loop's own accept/commit/emit work costs nothing measurable: one graph and two graphs give the same step time to 0.02 ms.

Figure 6 — The project's own metric, day one. Receipts: docs/headline-20261107{,b,c,d,i-plan-gemv}.json, docs/HEADLINE-20261107.md, docs/LEDGER.md.

Eight fixture prompts, 256 new tokens each, greedy; cuda events over live steps ≥ 32 of each sequence, pooled. Tokens per step by prompt class: prose 2.2, code 3.9, math 4.6, json 4.5, shell 3.7, tool 5.0, long 2.4, story 2.0 — production reports 2.6–2.9 on the same fixture. Every run replays byte-identically; the emitted token stream equals the 1-row greedy stream on 2 of 8 prompts and diverges at a near-tie token on the others (after run 4 the only place the 6-row step still differs from the 1-row step by an ulp is the routed-expert GEMM, which runs a different group shape per row count — the same class of difference production shows against itself; matching the two shapes is the next rung).

What the bar turned out to be

A 6-row step touches about 22 unique routed experts per layer (6 rows × 6 experts, deduplicated), 40 layers, 18.8 MB each: about 17 GB of expert weights, and about 29 GB in all once the dense projections, the head, and the draft stage are counted (the fusion design memo's shape arithmetic, modeled, not measured). At the copy rate we have measured on this box's HBM (6.3 TB/s) that is 4.6 ms of pure data movement, and at the best rate any kernel class has reached here, 6.3 ms. So 5.0 ms a step needs 92 percent of HBM bandwidth for the entire step, with under half a millisecond for everything that is latency-bound. Production's 7.36 needs 3.9 TB/s. The bar is a bytes bar, not a launch-count bar, and no fusion design meets it with stock weights at 6 rows. The two things that are reachable are beating 7.36 (the fusion ladder models about 9 ms, a persistent per-layer kernel about 7.3), and ms per token — the number a user feels, which the drafts turn step time into. James restated the bar the same morning: beat 7.36 and publish ms per token; 5.0 stays on record as the original, with this arithmetic as the reason. The three charter amendments the first day raised (the KV wording, adaptive residency, and the parity bar as a class — bit-identical self-replay always, and against production zero top-1 disagreements before the first near-tie split) are approved.

8.1 The afternoon: five rounds in parallel, two that paid, two that taught

With the instrument in place, the afternoon ran five coding rounds at once — each in its own git worktree with its own files, each box-verified in turn — against the question the morning left open: why does the 6-row step's emitted stream match the 1-row step on only 2 of 8 prompts, when every kernel in the 1-row step is deterministic? The 6-row step is the same arithmetic; a difference can only come from a kernel that reduces in a different order when it sees six rows instead of one.

What did not pay, and what it proved. The first suspect was the expert GEMM: at one row the runtime calls DeepGEMM one way, at six rows another. Making both calls the same shape made every row's expert output provably bit-equal across row counts — and changed nothing: still 2 of 8, and the one-row step got 1.7 ms slower (36 padded blocks for one row). Kept as an option, off by default. The second was the GEMV: a round-2 research memo modeled that the 208 small projections paying ~8 µs each were latency-bound for want of bytes in flight, and that a rewrite would save 2 ms. The rewrite (wide tiles, several K-tiles in flight, the split-K reduced inside the kernel in a fixed order) is row-invariant and nearer the fp64 oracle, and saved 0.26 ms (3.41 → 3.15 ms of GEMV per 1-row step; receipt docs/gemv3-bench-20261107.json). The model was wrong: the fixed cost of a small kernel is ramp and tail, not memory latency, and a better GEMV cannot remove it. Fewer launches can. That is milestone 2's thesis, now with a measurement behind it. Branch parked, finding recorded.

What paid. Four rounds landed on main, each gated on the box:

• Residency. The adaptive expert manager from the charter's second amendment (approved this morning): between graph replays it promotes experts that the last steps kept fetching from Grace into eight spare HBM slots per layer and rewrites a pointer table — no graph is recaptured, no value changes. On the fixture: 0.7 swaps a step, cold share 1.0 → 0.65 percent, manager cost 0.06 ms, tokens and acceptances bit-equal to the run without it, −2.2 percent (22.3 → 21.8 ms).
• The head as bf16. head.weight is bf16 in the checkpoint; the loader had been widening it to fp32 in HBM (2.65 GB read every step). Reading bf16 and widening in registers gives the same fp32 products, and with the fixed-order kernel the same sums, bit for bit — verified on the mini and real checkpoints, the draft stage's Markov heads included. 1.3 GB less per step; the 1-row step 10.52 → 10.37 ms.
• One reduction order per cluster. The fused glue clusters had chosen their reduction order by calibrating against torch at each row count — so the 1-row build and the 6-row build legitimately ran different orders at the Engram sums, the mHC mixes, the norms, and the q projection. Now every cluster's order is a function of the shape alone and each row reduces independently in that order; the census reads 18 of 18 clusters row-invariant. Correct, necessary — and still 2 of 8, because the last site was elsewhere.
• The router. The one site the census still flagged was a cuBLAS fp32 F.linear: the router's gate projection, which picks a different algorithm for 6 rows than for 1. The round that put the router and its 6-row plan into one launch per layer (gate → softplus → top-6 with torch's tie order reproduced on 100,000 recorded and constructed inputs → weights → the dedup plan) first came back slower at six rows (40 → 43 µs a layer: one CTA doing six rows' work serially). The fix gave each row its own program and moved the gate onto the tuned MXFP8 GEMV path — which is row-invariant by construction. Six-row router: 40.5 → 19.6 µs a layer; and the stream matches the 1-row step on 7 of 8 prompts.

Day's end: 20.1 ms a step, 2.91 tokens a step, 6.9 ms a token, from 22.3 / 7.5 at the morning's close; production 7.36 / ~2.6. The remaining prompt (shell) diverges at step 171 with a 0.098 logit margin at the 1-row step — not a near-tie — so one site still reduces differently at six rows; the attention kernel's split schedule, which depends on batch size, is the next suspect, and a per-layer, per-row probe is being built. What the afternoon settled: the 6-row step is now within a hair of being the 1-row step's arithmetic exactly, and the cost that is left is launches, not bytes.

Where the 20.1 ms goes, and the next rungs

The 6-row profile is the 1-row profile plus the MoE: at 6 rows the expert GEMMs, staging, dedup and scale packing grew from about 4 ms to 12.5 before run 3's levers; everything on the attention, projection and norm side is row-invariant in time, as the design memo predicted. Next: the last non-invariant site at six rows (section 8.1), then the fusion ladder from milestone 2 — one kernel per chain segment, the 2,000 graph nodes a step toward about 400. The afternoon's GEMV result says that ladder is the lever: the step's cost is now launches, not bytes.

Corrections

Section 7's 1-row bytes figure first read about 9 GB (~1.2 ms at HBM rate); the routed experts (4.5 GB) and the bf16 attention output weights (2.7 GB) had been left out. It now reads 15.9 GB (~2.5 ms) and is corrected in place. The 6-row design memo's first estimate for a fully ported 6-row verify step was 13–15 ms; measured, it is 19.9 inside the loop, and the gap is the MoE at 6 rows, which the memo had costed at the 1-row figure.