November 6, 2026 at 11:27 AM CST · charter and measurements: James Meadlock with Milo (one session on anthropic/claude-fable-5-1) · writing and diagrams: one headless session on anthropic/claude-fable-5.1 · one DGX Station GB300 (+ RTX PRO 6000 for recipe B)

GB300 DSV4.1 Megakernel: Can One Station Run DeepSeek-V4.1-Flash in 5 ms a Step?

A project charter, not a result. The Station decodes in 7.36 ms a step today; we want 5.0 ms, and fusion alone cannot get there.

Created · Updated

IN PLAIN ENGLISH We serve DeepSeek-V4.1-Flash on one DGX Station GB300, and today each decode step takes 7.36 ms. We want it at or under 5.0 ms on the same box, with stock weights, all 384 experts, and no KV-cache quantization. The obvious move is to fuse the ~1,300 small kernels per step into fewer, bigger ones, but our arithmetic says that pool is worth about 0.7–1.4 ms on the old 11 ms step; the larger cost is the GPU waiting while a second card computes the cold experts and ships the answer back over PCIe. So the bet is a standalone decode runtime, no vLLM and no SGLang, whose fused MoE layer reads the cold expert rows it needs straight out of Grace RAM over NVLink-C2C, inside the kernel, no sidecar round trip. "Megakernel" is the banner because the kernel work is where the engineering is; the number is honest about what has to go. We will also build the sidecar version on the same runtime and publish whichever loses. The kill rule is fixed: a working runtime by December 18, 2026, or we restore today's lane and park the project as desk research.

1. The machine and the model

One GPU with 250.7 GiB of visible HBM, a Grace CPU with 494 GiB of RAM, NVLink-C2C between them; pinned host reads measure around 340 GB/s. DeepSeek-V4.1-Flash-1017 is a 552B-parameter MoE plus 196B of Engram tables, about 475 GiB on disk: 40 MoE layers × 384 MXFP4 experts × 17.93 MiB = 269.0 GiB of expert weights, plus two Engram tables of 94.6 GiB. That cannot all fit HBM (fact, verified 2026-09-10): Engram lives in pinned Grace RAM, and so do some experts. An RTX PRO 6000 (96 GB) sits on PCIe with no P2P path to the big GPU; its results come back through host memory.

Two memories, one model: GB300 HBM 250.7 GiB and Grace RAM 494 GiB joined by NVLink-C2C, with an RTX PRO 6000 on PCIe for recipe B onlyLeft box, GB300 HBM 250.7 GiB holding hot experts, attention and dense weights, and the KV cache. Right box, Grace RAM 494 GiB holding Engram 2 × 94.6 GiB and the cold experts. Two arrows between them labeled NVLink-C2C about 340 GB/s pinned. A dashed RTX PRO 6000 96 GB box hangs off the GPU over PCIe, recipe B only. Conclusion: 269 GiB of experts plus 189 GiB of Engram cannot fit 250.7 GiB of HBM, so some of the model lives in Grace.TWO MEMORIES, ONE MODELGB300 HBM · 250.7 GiBhot expertsattention + denseKV cacheGPU sideGrace RAM · 494 GiBEngram 2 × 94.6 GiBcold expertsCPU side, pinned, no swapNVLink-C2C~340 GB/s pinnedPCIe, no P2PRTX PRO 6000 · 96 GBrecipe B only269 GiB of experts + 189 GiB of Engram cannot fit 250.7 GiB of HBM:some of the model lives in Grace.

Figure 1 — Where the bytes can go. HBM and Grace are facts of the hardware; the ~340 GB/s is a measured pinned-read figure from our notes. The HBM box lists what lives there without a byte total, because we have not published one.

2. Where 11 ms goes

The only full per-role anatomy we have is from the earlier v27 build (2026-10-27), traced with nsys at roughly 0% overhead: 11.216 ms per step, one user, 6 rows per step (DSpark speculative decoding, k=5). GPU 98.6% busy, 0.15 ms idle. 1,305 kernel launches per step; 1,035 ran under 10 µs yet together cost 4.54 ms, 40% of the step. Counters: SMs active 49%, SM issue 6%, tensor 2.3%, DRAM read 32%. Our reading: latency- and occupancy-bound on many tiny kernels; not compute-bound, not launch-bound (CUDA graphs already hide the launches).

Today's lane, Pg61 (vLLM v0.31.0, Garner McCloud's public dgx-station-recipes, our patches), runs the same decode at 7.36 ms (measured, release round 2026-10-31). It has not been profiled per role; that profile is the project's first job.

Where the v27 step went: 11.216 ms measured with nsys, split by kernel roleOne stacked horizontal bar: MegaMoE 3.38 ms, mHC 1.80 ms, attention 1.46 ms, ll_gemm 1.34 ms, sidecar wait 1.33 ms, lm_head 0.42 ms, other 1.34 ms, idle 0.15 ms. A thin bar below shows 1,305 launches per step, 1,035 of them under 10 µs summing to 4.54 ms, 40 percent. Conclusion: GPU 98.6 percent busy, not launch-bound, not compute-bound, many tiny kernels plus a wait on the sidecar.WHERE THE v27 STEP WENT — 11.216 ms, MEASURED WITH nsys0 ms24681011.216 msMegaMoE3.38 msmHC1.80 msattention1.46 msll_gemm1.34 mssidecar wait1.33 msother1.34 mslm_head 0.42 msidle 0.15 msKernel launches per step: 1,3051,035 of them run under 10 µs and together take 4.54 ms (40% of the step)GPU 98.6% busy. Not launch-bound, not compute-bound:many tiny kernels plus a wait on the sidecar.

Figure 2 — The v27 step by role (measured, nsys, 2026-10-27). "Other" is 11.216 ms minus the listed roles and idle: 1.34 ms. "Sidecar wait" is the _finish2 span where the GPU has finished its hot-expert math and is waiting for the RTX card's rows to come back over PCIe. Pg61's 7.36 ms step has no equivalent breakdown yet.

Start with what a perfect megakernel cannot remove. Bojie Li's sm_120 microbenchmarks (author's claim, 2026-09-26) put a PDL one-to-one handoff at about 0.37 µs and an all-SM gather at about 1.0 µs; at roughly 1,370 dependency boundaries per step, that is 0.5–1.4 ms of irreducible cost even inside one kernel. Our outside-view model (W1, 2026-10-28) prices everything fused-layer or PDL work can address at 50–70 µs per layer: 2.0–2.8 ms theoretical, −0.7 to −1.4 ms realistic, 6–13% of the 11.0 ms v27 step. Pg61 has already spent some of that pool (it inherits the peer's PDL and fused-sampler tree), so it cannot be subtracted again from 7.36 ms.

Peer evidence agrees (author's claims, 2026-10-28): a PDL reconstruction of the Hazy megakernel reaches 97.3% of a single persistent kernel with 81 ordinary kernels; SGLang #46088 found a fused Mega-mHC slower at every batch size; vLLM-Moet measured a +2.7% step regression from fusing routing into the MoE kernel. Nobody has published one kernel per forward with a PCIe sidecar, host Engram, sparse indexer and speculative decoding in the loop.

The peers who are fast use a chain. NVIDIA's sgl-nvidia-mega SGLang plugin (PR #1, merged 2026-10-19) decodes DeepSeek-V4.1-Flash at a p50 of 3.386 ms per step, batch 1, 6 rows, on four GB300s in TP4: 10–15 right-sized kernels per layer handing outputs through mailboxes, PDL and side streams. Per-GPU weight bytes there are a quarter of ours and no experts live off-GPU; that number is evidence for the shape, not a target for one Station. vLLM's specialized decode for Kimi K3 (2026-10-16) is the same shape: 23 kernels → 13 per layer, 130 µs → 69 µs, on GB300 at TP8/TP16.

So "megakernel" here means whatever makes the decode step shortest: first a fused multi-kernel chain with mailboxes and PDL, one persistent kernel per decoder layer as the stretch, one kernel per forward not a goal. The vehicle is a standalone decode runtime: one Python process, no vLLM, no SGLang; Python host, CuTe DSL kernels, Triton glue, CUDA C++ for the scheduler and mailboxes. Only decode gets optimized; prefill speed is a non-goal until the headline is met. Milestone 0 borrows every existing kernel as a library (DeepGEMM MegaMoE, FlashMLA sparse attention, FlashInfer MXFP4 MoE and MXFP8 GEMMs, vLLM's Engram gather, the peer's draft samplers) and replaces a piece only when the step trace names it and the replacement is measured against it in the same window.

What fusion can and cannot buy: v27 at 11.2 ms with a modeled fusion pool of 0.7 to 1.4 ms, Pg61 at 7.36 ms, and the 5.0 ms target lineTwo horizontal bars on a 0 to 12 ms axis. The v27 bar reaches 11.2 ms and its last 0.7 to 1.4 ms is hatched as the modeled fusion pool. The Pg61 bar reaches 7.36 ms. A dashed vertical line marks the 5.0 ms target. Conclusion: fusion alone does not reach 5.0 ms; the sidecar round trip has to go.WHAT FUSION CAN AND CANNOT BUYfusion pool, modeled: −0.7…−1.4 msv27 step 11.2 msv27 (measured)Pg61 today 7.36 msPg61 (measured)024681012 mstarget ≤ 5.0 msFusion alone does not reach 5.0 ms. The sidecar round trip has to go.

Figure 3 — The fusion pool is modeled (W1, 2026-10-28) against the measured v27 step; the hatched slice is the realistic −0.7 to −1.4 ms. Pg61 is measured. The target is a decision. The gap between Pg61 and the line is larger than anything fusion is modeled to buy.

4. The two recipes

Why the round trip costs what it does: per layer at 6 rows, the hot-expert kernel touches about 25 unique experts, roughly 448 MiB, about 65 µs at ~7 TB/s of HBM (modeled, W1; we measured ~81 µs). Then the GPU waits on the RTX card, a ~112 µs MoE section on about 80% of layers: the 1.33 ms _finish2 wait on v27. The hot math is not the problem; the waiting is.

Recipe A comes first and is Grace-only. The thesis: a fused layer gathers the cold expert rows it needs directly from pinned Grace RAM over C2C, inside the kernel. One memory system, no sidecar, no GPU spinning on a PCIe mailbox. Recipe B is the same runtime plus a persistent cooperative kernel on the RTX PRO 6000: the peer's "GP1" shape, the configuration we know is fast. If A's C2C self-fetch loses to B on measurement, that is a published result.

Recipe A versus recipe B for one MoE layer: Grace-only C2C self-fetch versus an RTX PRO 6000 sidecar over PCIeTwo panels. Recipe A, first, Grace-only: a GPU box running the fused layer kernel exchanges rows with HBM hot expert rows below it and with pinned Grace RAM cold expert rows over C2C to its right; caption one memory system, no spin. Recipe B, sidecar: the same GPU box exchanges rows with HBM and sends work over PCIe to an RTX PRO 6000 whose persistent cooperative kernel computes cold experts; caption the peer's GP1 shape. Conclusion: A tests the thesis, B is the known-fast configuration, whichever loses is a published result.RECIPE A vs RECIPE B — ONE MoE LAYERRecipe A (first): Grace-onlyGPUfused layer kernelGrace RAM (pinned)cold expert rowsC2CHBMhot expert rowsone memory system, no spinRecipe B: sidecarGPUfused layer kernelRTX PRO 6000persistent cooperativekernel computescold expertsPCIeHBMhot expert rowsthe peer's GP1 shapeA tests the thesis; B is the known-fast configuration.Whichever loses is a published result.

Figure 4 — One MoE layer under each recipe. In both, the hot expert rows are in HBM and Engram stays in Grace. The difference is where the cold experts are computed: inside the GPU's own kernel from C2C-reachable Grace memory (A), or on a second card that answers over PCIe (B).

5. The number and the ladder

The headline metric is single-user decode step time at 6 rows, measured with nsys on the cudaProfilerApi range over a fixed prompt corpus. Baseline: Pg61, 7.36 ms. Bar: ≤ 5.0 ms per step, about 1.5× Pg61 (decision). It exceeds the modeled fusion pool on purpose: reaching it means removing the MoE-section round trip, not just fusing kernels.

MTargetDone when
W0first dark window, ~1 dayPg61 per-role nsys step breakdown at 7.36 ms; per-boundary PDL gap histogram; 1-row greedy logit capture as the parity reference. Taken while Pg61 still exists; then Pg61 goes down. 2026-11-06: parity capture and step cell done; the nsys breakdown is blocked (see Status).
M0Dec 18, 2026Standalone runtime, borrowed kernels end to end, greedy 1-row parity vs the W0 capture, 32K context, step trace with named boundaries, step ≤ Pg61 + 10% at 1-row-equivalent.
M1Jan 2027DSpark k=5 in the runtime, 6-row step, headline instrument live → first honest ms/step vs 7.36.
M2Feb 2027Recipe A fused chain + C2C self-fetch of cold rows — the thesis test; plus an "everything in Grace" bounding measurement.
M3Mar 2027Recipe B: persistent cooperative RTX kernel on the same runtime.
M4—Minimal OpenAI-compatible surface; 128K → 1M context; production-candidacy review.

Month labels are targets, not promises. Context ladder is 32K → 128K → 1M; 1M gates production candidacy only. Rows C1–C8 are designed in; C32 is a declared non-goal of the runtime.

Kill rule: milestone 0 by December 18, 2026 (six weeks), or Pg61 is restored and the project parks as desk research.

6. What stays fixed

Stock weights: no NVFP4, no requantization. No KV-cache quantization. No expert pruning, no router masking; all 384 experts stay. Quality gates as calibrated. Determinism is a design requirement: bit-identical output across runs and relaunches, fixed-order reductions, no atomics on the data. Today's sidecar lane fails that (measured: 79,376 of 100,000 sidecar calls differ between two runs of the same binary; reduction order depends on arrival when two or more cold experts are summed). The one thing reopened is layout: weight tiling on disk and in HBM may change, as long as every value stays bit-identical to the pinned checkpoint (HF revision 1bae0e10).

7. Status today

Charter signed. Production DeepSeek-V4.1-Flash moved to the four-DGX-Spark SGLang lane on 2026-11-06 (consumers repointed, gateway alias shelved); the Station is the project's dev box for the duration. The first measurement window (W0) ran the same day. Measured: on our own driver, Pg61 decodes at 7.72–7.75 ms per step across three 512-token greedy generations (2.6–2.9 tokens per step); that is a different prompt and instrument from the 7.36 ms release number, not a regression claim. Two identical greedy runs over eight fixture prompts agreed byte-for-byte on seven and diverged at token 121 on the code prompt: that is Pg61's own run-to-run spread, and the bar milestone 0 is graded against. The nsys-wrapped boot did not deliver the per-role breakdown: the server booted and answered warm requests, but once the CUDA profiler range opened the engine stopped making progress (GPU idle, the engine process spinning on the host) and had to be stopped after six minutes. CUPTI kernel tracing and the sidecar's spin-wait protocol do not coexist on this lane; that is the first finding, and the first thing the project has to instrument around. Pg61 is now stopped-and-kept as a one-command rollback. No runtime code exists yet. The repo is public: github.com/jmeadlock/gb300-dsv41-megakernel, Apache-2.0, from the first commit (W0 receipts and scripts are in it). Peers invited after milestone 0.