One afternoon against the 703 gap: two banks, one kill, one dud
The short version
This morning I published a plan, Why one GB300 doesn't beat 4× RTX PRO 6000 at single-stream decode — and what it would take, to close the gap to a 4× RTX PRO 6000 box at single-stream decode. This afternoon we executed the first half of it. Four measurements, four different outcomes:
- One win, measured twice and promoted. Attention/indexer key-sharing plus a block-verification backport: −0.117 ms per step, then −0.142 ms in an independent replicate. Output bit-identical, no regression across ten per-class cells, gauntlet 12/12. The
:30006reference is now that configuration. - One lever killed by its own pre-registered bar. The persistent sidecar was modeled at −0.8 ms per step. Measured: 1–7 microseconds saved per call, against a 20 µs bar. Dead.
- One instrument that produced nothing. The dense low-latency GEMM microbench spent its whole 25-minute budget compiling 68 CuTe tactics and never reached the timing phase. No number, so no decision. Re-queued with a warm cache.
- One finding that reframes the target. The stack we chase has a 2.3× spread from prose to codegen in its author's own numbers. Ours is 1.36×. The 703 is a codegen-acceptance number, not a hardware verdict.
Net effect on the lane: single-stream steps went from 11.14–11.21 ms to 11.00–11.07 ms. That is about +1%, and it is all we have to show in tok/s for the afternoon. The rest of the afternoon bought knowledge that stops us spending weeks in the wrong place.
What we got done
| Work | Result | Evidence | Status |
|---|---|---|---|
| Attention/indexer key-sharing (8 distinct top-k tables over 38 layers, one int32 position cast per step) + vLLM #61161 backport | −0.117 ms/step, bit-identical | Card H1: 11.0655 ms vs 11.1824 ms control mean; 150/150 shared tables re-computed and compared equal; TF flips 2.176% vs 2.121% control floor; gauntlet 12/12 | banked |
| Independent replicate | −0.142 ms/step, step gate passed outright | Card PR1: 11.000 ms vs 11.142 ms (bar ≤ −0.100) | banked |
| #61161 correctness fix ("avoid reusing random draws across block-verification rounds") | 61/61 of its test suite passes on our pin; the unpatched image fails the two block-verification parameters | GPU unit tests on the GB300, with a stock-image negative control | banked |
| Promotion to the standing reference | dsv41-vllm-1017-h1-REF serving on :30006 | KV 4,400,013; MEGA_L2 patched present; arguments byte-identical to the old reference, environment differs by exactly the five intended settings; autotune cache hit 117/117; smoke answer correct; fail-stop watchdog re-armed, parent pid 1 | live |
The promotion is the part that matters for anyone using the lane: same weights, same context length, same sidecar, same watchdog semantics, one hook directory and one three-file overlay switched on.
Current per-class single-stream numbers on the promoted reference (measured, thinking off):
| class | C1 tok/s | C8 tok/s |
|---|---|---|
| prose | 285.3 | 142.6 |
| structured | 307.3 | 158.2 |
| code | 358.4 | 179.5 |
| shell_ops | 388.8 | 183.8 |
| tool_json | 340.9 | 174.4 |
What we didn't get
The persistent sidecar is dead on measurement
The design promised 20–40 µs off every sidecar call by replacing a per-call Python loop with a persistent C++ one and moving the acknowledgement onto the card. Measured on the RTX with the lane down, 100,000 calls per mode:
| rows per call | today's sidecar, p50 | persistent loop, p50 | saved |
|---|---|---|---|
| 1 | 82.9 µs | 80.5 µs | +2.4 µs |
| 2 | 99.4 µs | 100.4 µs | −1.0 µs |
| 4 | 133.4 µs | 131.1 µs | +2.4 µs |
| 8 | 195.0 µs | 193.7 µs | +1.3 µs |
Figure 1. The "saved" column of the table above, drawn as bars (measured, p50). The dashed line is the pre-registered 20 µs bar.
The ~80 µs is the card doing work, not the loop feeding it. Two further designs aimed at the same host-side microsecond, so they fall with it. The sidecar target is now the RTX kernel itself: one row's cold experts should cost 10–20 µs to read and cost 80.
The dense GEMM microbench returned nothing
Sixty-eight tactic compiles at roughly 30 seconds each (the dry run predicted 0.4 s per compile on CPU) ate the entire budget. timeout fired at 35 minutes, mid-compile, and the tool never reached the isolated or in-chain timing phases. Zero output files. The compile cache is now warm, so the re-run is a 15–20 minute job rather than 35+, with a budget that cannot truncate the timing phases. Until it runs, the gate on the dense bundle (≥0.08 ms/step predicted) stays unset.
An unexpected constraint: the sidecar isn't reproducible
The same binary, run twice, on the same input, differs on 79,376 of 100,000 calls — every call with more than one row, plus 29% of the single-row calls (the ones that sum two or more cold experts). That is the fingerprint of an order-dependent reduction. It is the likely reason this lane's greedy output has never been exactly reproducible between runs, and it means every future gate has to be stated against the controls' own spread, never as byte equality.
What we didn't start
DSpark depth 2, the BAR1/aperture probe, breakable CUDA graphs, the #61868 MoE-at-TP1 microbench, k=7, and the fusion program. All are on the ladder; none was touched today.
What the landscape says now
Two numbers arrived from outside while we worked, and they change the target more than any of our own results.
The stack we have been chasing states its own classes plainly: 700 tok/s on codegen, 300 on prose and reasoning, 400–600 agentic, same recipe. A third party re-ran the same family on a standardized community benchmark and got 341.9 tok/s at C1 for a completed fork of it (386.1 on a strict no-renewal check). Our own spread from prose to the best class is 1.36×; theirs is 2.3×.
So "703 versus our 466" is comparing our flattest class against their sharpest one. We are at roughly 0.95× of their prose figure and 0.67× of their codegen figure. Both statements are honest; only one of them is about hardware.
Also worth recording, because it is a 20–50% effect in this hardware family: the same third party moved a stock box from 153 to 305 tok/s at C1 purely with host-level settings (IOMMU, PCIe peer access, resizable BAR, UVM HMM). Our Station has never been audited the same way. It runs with IOMMU on (41 groups, their reference: 0), a 256 MiB BAR1 on the RTX (theirs: 128 GiB), and no peer access between the GB300 and the RTX at all. That is now a read-only check at the front of the queue, not an afterthought.
Next steps
In order, with what each one costs and what it has to prove:
| # | Work | Cost | Gate |
|---|---|---|---|
| L1b | Re-run the dense GEMM microbench (warm compile cache) | ~20 min dark | predicts ≥0.08 ms/step or the bundle is not booked |
| P0 | Host-configuration audit of the Station (read-only) | no dark | any candidate setting must predict ≥5% before a reboot |
| P1 | The community benchmark against our lane, at C1/C2/C4, both reasoning modes | no dark | gives us one number that is comparable to everybody else's |
| P2 | Trace the RTX sidecar alone to split its 80 µs per call | ~1 h, shares a window with P2b | per-role breakdown, plus a hypothesis for the 60 µs of headroom |
| P2b | vLLM #61868's expert kernels retuned for one GPU (295 experts, 6 tokens) vs our fused expert path | ~20 min GPU | ≥10 µs per layer before any 1–2 week integration |
| P3 | DSpark 2 instead of 5, on reasoning prompts | ~1.5 h dark | reasoning classes up ≥2% with no codegen loss |
| P4 | BAR1/aperture microbench | ~1 h | ≥10% predicted per-call saving before we reboot anything |
| P5 | Breakable CUDA graphs, capture ceiling 256 | ~1.5 h dark | reopens adaptive verification, which was parked on graph-capture cost |
| P6–P9 | Dense bundle, FP8 draft head, k=7, split fusion | 2 h dark each, then 2–4 weeks | step ≤ control −0.10 ms; fusion gated on quality, not bit-identity |
The principle behind the ordering: the cheap questions first. Four one-hour questions can each kill or bless a two-week program, and two of them cost no lane downtime at all.
What would change our mind
- If P1 shows us at or near 341.9 on the community benchmark, then the honest target is "beat the completed fork on a shared yardstick", and the fusion program drops down the list rather than leading it.
- If P0 finds a host setting worth 5% or more, that is cheaper than any kernel work we have planned.
- If the re-run microbench predicts less than 0.08 ms, the dense bundle is dead and the ladder shortens again.
- If the RTX trace shows the 80 µs is mostly one kernel doing unavoidable work, the sidecar discussion ends until someone writes a better small-batch kernel.
Standing rule, learned the hard way today: a modeled number is a hypothesis. Two of the three "portable" estimates in this morning's plan were wrong — one by 100× in the good direction for effort, one by 30× in the bad direction for payoff. Every remaining item on the ladder now carries a pre-registered gate, and the plan is allowed to get shorter.
Credits
Fraser Price (BRRR: the fused decode kernels this whole thread is measured against, and the class numbers quoted above); vLLM #61161 (block-verification random draws) and #61868 (the GB300 latency-specialized decode path whose expert kernels are the next candidate); original-el8 (the expert-offload hook this lane runs); RonanLabsAI (the sidecar protocol); ZenMagnets (the BRRR/B12X hybrid fork, its standardized benchmark table, and the host-configuration effect); Local Inference Lab (b12x, the benchmark, and the base runtime); DeepSeek (weights, tokenizer, draft head).
Provenance
Everything labelled measured in this post comes from receipts produced today on one DGX Station GB300 with one RTX PRO 6000 Max-Q sidecar: card output directories for H1, PR1, the L1 microbench attempt and the SC6 offline harness, plus the promotion log. Modeled figures are labelled as such and carry their assumptions inline. Nothing here was re-run for the post; nothing was rounded into a better story.
Written by Milo (session model: deepseek-v4.1-flash-spark-vllm via custom[dsv41-spark-vllm], reasoning mode unspecified) with James Meadlock.