Why one GB300 doesn't beat 4× RTX PRO 6000 at single-stream decode — and what it would take
In plain English
What's being raced. One person, one chat, how fast the model types its answer (tokens per second, "C1"). On one specific test, a long coding prompt with thinking turned off, four RTX PRO 6000 cards type 703 tokens/s (their published number). Our one GB300 Station plus a helper card types 466 on the same test (measured). On ordinary prose, both setups land around 250–300 tokens/s, so there we're level.
Why we're behind on that test: two reasons.
- Each step takes longer: 11.2 ms for us vs 8.45 ms for them. In every step our GPU runs about 1,305 separate small programs ("kernels"). Over 1,000 of them finish in under 10 microseconds. Every tiny program pays a fixed start-up and wind-down cost, so together they eat 4.5 ms, 40% of the step, mostly on overhead instead of real work (measured). Their software merges each model layer into about 4–5 programs. On top of that, the model doesn't fully fit on our GB300, so 89 of the 384 "experts" in every layer run on a second card over PCIe, and waiting for it costs about 1.3 ms per step (measured).
- Each step produces fewer tokens: about 5.2 for us vs 5.9 for them. Both setups guess several tokens ahead and keep the ones the model agrees with. We guess 5 ahead; they guess 7. Guessing further only pays off when steps are cheap. Theirs are; ours aren't yet.
What won't help: more memory bandwidth (we already have plenty), CPU or launch tuning (the GPU is already busy 98.6% of the time, measured), or a newer vLLM (its new fast path for this model needs four GB300s).
Ideas to catch up, in order:
- Stop repeating work in attention. 38 layers each rebuild a lookup table, but only 8 of those tables are actually different. Sharing them is a small win, about 0.1 ms per step, with identical output (modeled). Being tested today.
- Make the helper card answer faster. Replace its per-call Python loop with a persistent one that never stops to check in. It targets the 1.3 ms wait; the saving is about 0.8–0.9 ms per step (modeled). Its offline safety harness is running today.
- Tune the small matrix multiplies, and use an 8-bit draft output head. Each is worth roughly 0.1–0.3 ms (modeled).
- Then guess 7 ahead like they do, once steps are cheap enough for it to pay.
- The big one: write fused kernels that merge dozens of tiny programs per layer into a few. That's 2–4 weeks of kernel work, and it's the only path all the way to ~700.
Bottom line: ideas 1–4 should get us to roughly 510–600 tokens/s on this test (modeled, not measured). Matching 703 needs idea 5. The rest of this post is the evidence.
The short answer
Fraser Price's 703 tok/s (padded codegen, thinking off, k=7, 4× RTX PRO 6000 Max-Q) is real. Other people have reproduced it on the same prompt shape. On that prompt our lane does 466 on the same harness, so we're 1.5× behind. The gap splits into 1.32× step time × 1.15× tokens per step. Their step is faster because their fused decoder runs about 4–5 kernel launches per layer; ours runs 1,305 kernels per step (measured, nsys). They also hold every expert on-device, while we compute 89 of 384 experts per layer on a PCIe sidecar.
Today's nsys trace changed what we think the fix is. Launch overhead is not the problem: the GPU is 98.6% busy and idle only 0.154 ms per step, because CUDA graphs already hide the launches. The cost is that 1,035 of those kernels run under 10 µs each and add up to 4.54 ms per step (40% of it). Kernels that short are latency-bound: ramp-up, prologue, tail and grid sync, not bandwidth. Only fusion removes that. Host-side and launch-overhead work is closed on evidence.
On prose the picture is different. The published 4× RTX PRO 6000 numbers sit between 249 and about 300 (our span over the four cited peer numbers below, not a quoted range), level with our 275. Portable, bit-identical work could get us roughly +15–20% on the codegen class (modeled). Reaching their codegen number takes an op-fusion kernel program for sm_103 that doesn't exist yet. No config flag gets us there.
I'm a wannabe inference engineer; this is why a box with ~8 TB/s of HBM (spec) loses a decode race to four workstation cards, and what we got wrong getting there.
The decomposition
Tokens per second = tokens per verify step × steps per second. On the 2K / C1 cell of their benchmark (bench.py; we run a stdlib port of it):
| theirs: 4× RTX PRO 6000 Max-Q, k=7 | ours: GB300 + sidecar, k=5 (v27) | |
|---|---|---|
| decode tok/s, p50 | 703 (padded codegen, thinking off, k=7, 4× Max-Q); published, copied, not re-run by us | 466 (measured, same harness) |
| tokens per verify step | 5.94 (published; 4.94 of 7 drafts accepted, 71%) | 5.17 (measured; 83% per draft token) |
| step time | 8.45 ms (derived: 5.94 ÷ 703) | 11.2 ms (measured, p50 inter-arrival, 14 reps) |
703 / 466 = 1.51× = 1.15× tokens/step × 1.32× step time. (5.17 / 11.2 ms is 462 tok/s, and multiplying the two rounded factors gives 1.52 — our arithmetic, 1.15 × 1.32 — so the factors are good to about a percent. Last week's post used BR2's 11.0 ms and 5.13 and got 1.30 × 1.16; today's figures are a 14-rep decomposition on the current production config, v27.) Card I1's clean re-run this morning, no profiler attached, read the same thing: 11.23 ms per step, 5.23 tokens per step, 465 tok/s.
Measured today, and ruling out the easy explanations: the step is flat with context (11.13–11.21 ms at 2K, 11.22–11.27 ms at 64K) and the same at temperature 0; the GB300 sits at its 2,070 MHz max at 540–620 W, not power-capped. The sidecar costs 1.2 ms per step on codegen (about 39.6 round trips at ~30 µs) and 0.32 ms on prose, so GB300-local work is about 10.0 ms. To hit 703 at k=5 our step would need about 7.35 ms, a 3.85 ms cut; the sidecar can give back at most 1.2 of it.
The token factor is mostly their k=7. Our own k=9 arm on the same harness reaches 5.70 tokens/step at 2K and 6.05 at 64K, so the checkpoint and drafter can yield their tokens. But each extra draft position costs us about 0.3 ms per step (11.0 → 12.2 ms from k=5 to k=9), so at 11.2 ms k=7 is a wash; it pays once the step is under about 9.5 ms (modeled). Their kernels are tiled for ≤8 rows, so at C1 the k=7 rows ride nearly free — our reading of their source; their README doesn't say why they chose 7.
Where the 11.2 ms goes (measured, Card I1b)
Card I1b wrapped the production server in nsys (--cuda-graph-trace=node, capture driven by vLLM's CUDA profiler hook) on the exact v27 config: arguments and environment equal to production except the profiler flag, KV pool 4,400,013 tokens, autotune hash hit, no foreign requests, production restored afterwards. Step boundaries are _rejection_kernel, which runs exactly once per speculative step. Over 83 captured steps the median wall step was 11.216 ms against 11.23 ms with no profiler, so nsys costs about 0% here. The torch profiler we used this morning stretched the step by 23%. This is the first per-kernel timeline of this lane we trust.
Figure 1. Measured per-step budget, Card I1b (nsys, 83 steps). The first seven bars are kernel rows from the table below. “Other” is a residual we computed: 11.216 − (3.38 + 1.80 + 1.46 + 1.34 + 1.33 + 0.42 + 0.154) = 1.33 ms. The remaining rows in the table add up to more kernel time than that, because some kernels run concurrently on side streams, so read the bars as a budget, not an exact partition.
| kernel (per step) | n | avg µs | ms/step | note |
|---|---|---|---|---|
| MegaMoE (hot experts) | 43 | 78.6 | 3.38 | at most ~1 ms above a ~2.3–2.7 ms bandwidth floor (modeled) |
attention (SparseAttnFwdMode=1, decode) | 43 | 34.0 | 1.46 | |
ll_gemm low-latency block-scaled | 132 | 10.1 | 1.34 | |
sidecar _finish2 | 40 | 33.3 | 1.33 | the lane's own counter says 1.18–1.22; nsys includes the kernel's own work |
| mHC (pre 161 × 5.1, prenorm GEMM 81 × 6.1, post 81 × 5.3, mega 5 × 11.9 µs) | 328 | ~5.5 | 1.80 | |
nvjet_tst_128x8 | 7 | 65.5 | 0.46 | bimodal; the two big ones (2 × 208 µs, 0.42 ms) are the BF16 lm_head reads, target + DSpark draft |
| fp8×fp4 1d1d GEMM | 43 | 10.0 | 0.43 | |
_q_kv_norm_quant | 43 | 8.7 | 0.38 | |
_route_send2 | 40 | 7.1 | 0.29 | |
| q-norm-rope-insert, LLBf16SplitK, dsv4 top-k, mxfp8 quantize ×2, prepare-megamoe, etc. | ~400 | 2–5 | ~1.2 | |
| idle | 0.15 | almost all in gaps under 5 µs |
And the same 1,305 kernels per step, bucketed by how long each one runs:
| kernel duration | kernels/step | ms/step |
|---|---|---|
| < 3 µs | 363 | 0.69 |
| 3–6 µs | 477 | 2.32 |
| 6–10 µs | 195 | 1.53 |
| 10–20 µs | 154 | 1.70 |
| 20–50 µs | 65 | 2.21 |
| ≥ 50 µs | 52 | 4.30 |
Three readings, all measured on Card I1b:
- The GPU is not waiting on launches. Busy 98.6% of the step, idle 0.154 ms. CUDA graphs already hide launch overhead, so host-side, launch-gap and scheduling PRs (async scheduling, graph-capture tricks, CPU-bubble removal) are worth about nothing at C1. We've closed them.
- The excess is about 1,000 small, latency-bound kernels. 1,035 kernels under 10 µs take 4.54 ms per step, 40% of it. Each one pays ramp-up, prologue, tail and grid sync, and the dense, mHC and norm/quant/route work gets chopped into ~5 µs pieces far from roofline. That pool is what BRRR's 4–5-kernels-per-layer decoder removes.
- The big rows are close to done. MegaMoE has at most ~1 ms of headroom (3.38 ms against a ~2.3–2.7 ms modeled bandwidth floor), and a faster small-T MegaMoE would be a kernel project, not a flag. The lm_head is 0.42 ms (two 208 µs BF16 reads), so an FP8 draft head is a real, small item.
Before the trace, we'd written a kill condition for the fusion program: if launch gaps were under 0.5 ms per step and dense GEMMs ran above ~60% of HBM bandwidth, drop it. I1b meets the first half (gaps are tiny) and fails the second (the dense and mHC work is chopped into kernels too small to reach bandwidth). So fusion stays, and now it has a measured target. What I1b did not measure is DRAM bytes per step: the floor model's ~28.3 GB per step at k=5 (modeled) is still unconfirmed by a counter, and we keep its ±40% caveat. This morning's torch-profiler pass (1,370 launches, 99.1% busy, under the 23% inflation) agrees on the shape; the nsys numbers replace it.
What we tested today and why it failed: Card MHC0
The cheapest idea was one switch: MEGA_MHC_OVERLAP=0, which swaps mHC's two-stream TileLang pair for the single-kernel Mega-mHC. The flag came in with RonanLabsAI's launcher; original-el8's hook comment says the two-stream path "exists to hide the TP all-reduce; at TP1 there is none to hide"; no receipt showed it was ever A/B'd here. We bracketed it A → C → A2 on the production pair.
Null (measured): step 11.195 vs 11.236 ms control (0.9964×, bar ≤0.98×); 451.4 vs 456.4 tok/s; two per-class cells a hair low (tool_json C1 0.971×, shell_ops C8 0.975×); mixed-traffic C1 212.5 vs 212.2; flips 2.167% vs a 2.166% floor; gauntlet 12/12. The engine-source read explains it: each mHC boundary already runs its "stats" pre-pass on a side stream that hides ~0.65 ms, so collapsing to one kernel loses the overlap it saves in launches. Closed; reopening needs a different mHC kernel. The pre-registered verdict read INVALID, not FAIL — see the corrections box.
What the field says
We went through every published single-stream number we could find for this checkpoint on 4× RTX PRO 6000 or on one Station. Only numbers with a stated method count.
The 703 class (padded codegen, thinking off, k=7, 4× Max-Q) reproduces. Dataphil3 gets 669 on the BRRR image; Moet r21 (kacper-daftcode/vLLM-Moet) gets 680 median / 741 best at 400 W with a memory overclock; 0xSero ran BRRR's unmodified bench.py on Moet r21 at 300 W and got 634. So 4× RTX PRO 6000 does roughly 550–740 at C1 on padded codegen — a genuine 1.2–1.5× lead over our 466. (BRRR's commit 2aff972, October 27, fixed a misaligned indexed-key load in decoder.cu / sparse_decode.cu after the benchmark run; the README numbers weren't re-run.)
On prose they're level with us:
| prose / real-text C1 | tok/s | source |
|---|---|---|
| Moet r21, 4× RTX PRO 6000, short prose | ~277 | Moet README (their measurement) |
| Karmic Kraken, 4× RTX PRO 6000 | 250 | commdata2338 benchmark paper (their measurement) |
| LIL stock Karmic Kraken, 4× RTX PRO 6000 | 249–258 | LIL digests (reported) |
| Fraser Price, BRRR, reasoning | "~300ish" | author's estimate |
| ours, v27, prose class | 275.1 | measured (MHC0 control arm, temp 1.0, 256 forced tokens) |
| RonanLabsAI, one Station + RTX sidecar, real 8K news | 273.6 | his receipts |
Prompts differ, so this is parity within noise, not a win. On every stack that publishes both, step time doesn't move with prompt class; tokens per step does, by 1.7–2.9× codegen-vs-prose on the same 4× RTX PRO 6000 box. BRRR's 8.45 ms step at our prose acceptance would land near 354 (modeled; nobody has measured it) — about 1.3×, the most a faster step buys on prose.
Nobody with a published method beats our 466 on one Station (GarnerFox's 556–695 has no harness, acceptance or repo). No other engine runs this model on one GB300: TensorRT-LLM V4.1 needs 4 GPUs without expert offload, Flare is TP3/TP4, SGLang's low-latency mode is TP4 only.
What it would take
vLLM main has nothing for this lane. The one GB300 decode PR, #61868 (merged October 21), needs TP4 plus an opt-in backend, skips MegaMoE layers and hard-codes TP4 shapes; its tray number (4× GB300, all experts resident: 5,355 → 4,375 µs per step at batch 1) is different hardware, not a bar we're failing. The mHC files are byte-identical between our pin and main, so a rebase buys no decode speed.
Tier 1: portable, bit-identical, roughly +15–20% (modeled)
| scenario | step (ms) | tok/step | tok/s | basis |
|---|---|---|---|---|
| today | 11.2 | 5.17 | 466 | measured |
| + persistent sidecar | 10.35 | 5.17 | 500 | modeled; measured stall 1.2 ms (1.33 ms in nsys, which includes the kernel's own work) → ~0.35 ms, inside the design doc's 0.3–0.45 ms estimate |
| + attention/indexer key-sharing | 10.2 | 5.17 | 507 | modeled |
| + dense low-latency GEMM tactics/shapes | 9.9–10.1 | 5.17 | 512–522 | modeled |
| + k=7 once the step is fast enough | 10.2–10.4 | 5.52–6.02 | 531–590 | modeled |
The dense item has a measured cause: in the real chain three low-latency GEMMs run 1.2–2.3× slower than in isolation (wqa 7.88 vs 3.39 µs, wq_b 9.92 vs 7.53, wo_b 11.43 vs 9.43), and production leaves MEGA_FI_TUNE_FILE empty. A tuning defect, not physics. I1b adds a small item the table doesn't carry: the two BF16 lm_head reads are 0.42 ms, and an FP8 draft head is worth about 0.1 ms (modeled).
Honest ceiling without new kernels: about 590–600 tok/s on the codegen class, ~85% of 703 (padded codegen, thinking off, k=7, 4× Max-Q), closing 55–60% of the gap. The floor model (8.81 ms → 587 for the portable tier) and the engine-source pass (non-fused TP1 floor ~9.5 ms) agree on the shape.
Tier 2: the fusion program
Matching 703 (padded codegen, thinking off, k=7, 4× Max-Q) means removing most of that 4.54 ms small-kernel pool — 1,305 kernels per step down to roughly 500 (modeled): a BRRR-shaped split fusion — one persistent per-layer kernel for wo_a → wo_b → mHC → norm → gate → top-k → shared expert, with our MegaMoE and sidecar protocol unchanged. I1b names the densest targets: mHC (328 kernels, 1.80 ms per step) and the per-layer norm/quant/rope/route chain around attention and MoE. It's new sm_103 work (BRRR's decoder.cu uses sm_120a-only block-scaled MMA and TP4 constants — a rewrite, not a port): 2–4 weeks, modeled at 7.2–9.0 ms per step (610–830 tok/s at 5.5–6.0 tokens/step; the floor model's ~500-launch graph lands at 7.23 ms → 716). Numerics change — float atomics aren't bit-identical — so the gate becomes KLD against self-noise, as BRRR gates itself. Beating that number needs fusion and k=7 paying off.
Card order after I1b
| order | work | expected | status |
|---|---|---|---|
| done | Card I1 / I1b: nsys timeline of the v27 step | decided the plan on evidence | measured, this post |
| 1 | H1: attention/indexer key-sharing + #61161 backport, bit-identical | −0.12 ms (modeled) | running next |
| 2 | SC6: persistent sidecar, option A (3–4 dev days; offline harness first) | −0.8 … −0.9 ms of the measured 1.33 (modeled) | offline harness running next |
| 3 | L1 microbench: the offline microbench that gates the dense low-latency GEMM bundle | −0.08 … −0.30 ms (modeled) | queued |
| 4 | FP8 draft head | −0.1 ms (modeled) | queued |
| 5 | k=7 at C1–C2 on the improved step | +0.35 tok/step (modeled) | queued |
| 6 | fusion spike targeted at mHC + the per-layer norm/quant/rope/route chain (~3 ms of small kernels) | −1 … −2 ms (modeled), 2–4 weeks | queued |
The two lane cards at the top, H1 and the persistent-sidecar offline harness, are being run next. Their results are not in this post. Key-sharing, the persistent sidecar and the dense GEMM bundle together are the "+15–20%" plan: about 5–6 dev days and about 8 box hours.
Open correctness check (not speed). vLLM #62417: DSpark corrupts thinking/chat output on DeepSeek-V4.1-Flash-1017 — degenerate reasoning loops, no structured tool calls, at every k, clean with speculation off. It closed October 27, 08:07Z, with #59986; our pin predates both. Our probe of v27 (5 requests: math, word problem, tool call, under reasoning_effort: high, thinking: true, enable_thinking: true) saw no corruption — clean answers, correct math, a structured get_gpu_alert call — and 0 characters of reasoning_content in every case, so it never exercised thinking-mode separation. Not cleared. The reporter's stack (8× RTX PRO 6000 SE, TP8+EP, FULL-cudagraph sparse prefill) differs from ours. A thinking-mode separation gate now precedes any spec-decode promotion.
Corrections to things we said
- "Attention runs a prefill kernel at C1." This was in our lab notes and in the brief for this round, and it's wrong. FlashMLA instantiates one fused template in two modes. Our decode step calls the decode instantiation (
SparseAttnFwdMode=1) 43.5× per step. The prefill mode runs exactly once per layer, during prompt prefill. "prefill" is only the C++ namespace. There's no wrong-kernel bug to fix. - "The 6000 number is harness-dependent by 3×." Wrong, and we said it in messages too. The 3× came from comparing a slow SGLang adapter on the older 0909 checkpoint against a tuned vLLM run, so most of it was engine, not harness. The real same-stack codegen-vs-prose factor is 1.7–2.9× (cross-read of peers' published numbers).
- "#62417 closed on October 19." It didn't. It closed October 27, 08:07Z, with #59986. Our pin (
af7f9488) predates both. - "Their decoder runs everything after attention as one persistent kernel per step" (previous post). Not quite. It's one persistent work-queue kernel per decoder layer, plus a fused prologue kernel for ≤8 rows. Attention and the indexer stay separate. That works out to about 4–5 launches per layer.
- "Launch overhead their persistent fused kernel removes" (previous post). The nsys trace says the GPU isn't waiting on launches (98.6% busy, CUDA graphs hide them). What fusion removes is the latency of about a thousand sub-10 µs kernels.
- MHC0 instrument bug. The pre-registered verdict read
RESULT INVALIDbecause the validity gate marked all three arms BAD. The cause was a parse bug: the foreign-request counter file ended in a newline, and the parser didn't strip it. The re-score fixes only that parse, leaves every bar unchanged, and readsRESULT FAIL: validity PASS, controls PASS, win FAIL, no-regression FAIL, fidelity PASS, gauntlet PASS. Both files are kept in the bundle. The lever is closed on the re-scored FAIL, not on the INVALID.
Credits
- Fraser Price for BRRR: the 703 table (padded codegen, thinking off, k=7, 4× Max-Q), the fused
decoder.cu/sparse_decode.cukernels, andbench.py, which we ported with its Apache-2.0 credit header. - Local Inference Lab for b12x and the vLLM fork BRRR builds on, and for the stock Karmic Kraken numbers.
- original-el8 for the MegaMoE peer hook our lane runs.
- RonanLabsAI (@SJRonanMD) for the sidecar protocol and launcher, and the 273.6 real-text receipt.
- Dataphil3 (669 reproduction); kacper-daftcode / vLLM-Moet (r21: 680 / 741, prose ~277); 0xSero (BRRR's unmodified bench on Moet r21, 634); commdata2338 (Karmic Kraken paper); knapcio (decode adapters
draft_tau,verify_cap,draft_head_fp8). - Authors of vLLM #61868, #62417, #59986, #61161, #60118, #61371, #61039, #63063; FlashMLA and DeepGEMM maintainers; Hazy Research (1.3 µs graph-launch figure in the floor model); DeepSeek (checkpoint, DSpark).
Every peer number is theirs, copied from their posts, not re-run by us unless stated.
Provenance
Measured: c1-gap decomposition (14 reps, v27, October 27, 07:35 CDT); PROF27 torch profile (diagnostic boot 07:52–08:16 CDT, production restored); MHC0 bracket (October 27); Card I1 clean re-run (v27 restored 11:42 CDT) and Card I1b nsys capture (83 steps, v27 restored 12:08 CDT, watchdog re-armed, lane check passed); peer-harness cells from Card BR2 (October 25–26). Modeled: four desk workers this round (engine source, peer code, floor/tokens, community), key claims verified by the orchestrator against source, gh and our trace. Peer numbers are copied from public cards, READMEs, PRs and posts. Labels: measured = our run; modeled = arithmetic or a floor model, not a run; published = a peer's number.