Our roadmap: making one GB300 faster for a single user
Update, October 27, 2026 at 8:32 PM CDT: Three checks finished after this post went up. One idea dropped off the list, one bug report turned out not to affect us, and the helper card moved to the top of the plan. The changes are marked below.
Update, October 27, 2026 at 8:44 PM CDT: One more check finished. The helper-card wait is mostly a coding problem: it is large on code and JSON, small on normal chat. Item 3 below is corrected.
The short version
One person talking to our AI model gets about 476 word-pieces (tokens) per second on a coding test. A box with four RTX PRO 6000 cards gets 703 on the same test. We want to close that gap.
We are only behind in one situation: one or two people at once. With four or more people at once we are already faster than that box, and we read long documents almost three times faster.
The problem is simple to say: every step our chip takes is too slow. Each step runs about 1,300 tiny programs, and most of them are so small that the start-and-stop overhead adds up. The fix is to merge many tiny programs into fewer, bigger ones. Someone has already done this on the same chip and saved 18%. We plan to borrow their pieces.
How the model writes text, in one paragraph
The model writes in steps. In each step a small helper model guesses the next five tokens, and the big model checks all five at once. If the guesses are good, one step gives you five tokens. If they are bad, you only get one or two. So speed is: tokens kept per step ÷ time per step.
- Our step takes 11.0 milliseconds and keeps 5.27 tokens on the coding test. That is 476 tokens per second.
- To reach 703 with the same guesses, the step has to drop to about 7.5 milliseconds.
What we measured
| Test | Us | Them | Us vs them |
|---|---|---|---|
| Coding test, one user, thinking off | 476 | 703 (their published number) | 0.68× |
| Standard community test, one user, normal reasoning | 239–260 | 342 | 0.70–0.76× |
| Same test, 4 users at once | 702 | 634 | 1.11× |
| Same test, 32 users at once | 2,615 | 1,755 | 1.49× |
| Reading a 64K-token document | 50,778 tokens/s | 17,616 | 2.9× |
The "them" numbers come from two different 4× RTX PRO 6000 setups. 703 is from the author of the BRRR kernels on their coding test; we did not re-run it. 342 is from a third party (ZenMagnets) on the community test, and we ran that same test on our machine. On their own reasoning test the BRRR author reports 507, so 703 is their best case, not their typical case.
What we ruled out today
Guessing fewer tokens. We tried guessing 3 tokens instead of 5. Each step got a little faster (0.67 ms), but we kept so many fewer tokens that everything got slower: the coding test fell from 476 to 355, and real conversations got 8–15% slower. The 4th and 5th guesses are worth it. (The final verdict waits for one more control run tonight; the direction is already clear.)
Computer settings. The other box gained a lot by changing system settings that speed up traffic between its four cards. We only have one main chip, so there is no such traffic to speed up. We checked our connection to the helper card anyway: it runs at full speed with power-saving off.
The helper card's control loop. Part of the model runs on a second card (an RTX PRO 6000). We rewrote the program that feeds it and saved only 1–7 microseconds per call. The time is in the card's own work, not the feeding.
What we already won
Today we found that 38 layers were each re-doing a lookup that only eight distinct tables needed. Sharing those lookups saved 0.12–0.14 ms per step with identical output, and it is now live.
The plan, in order
- Measure two things precisely tonight (the lane goes offline for about an hour):
- how long each of the small number-crunching programs takes, so we know which ones are worth tuning;
- what the helper card does inside each call. We think most of its time is spent copying data rather than computing.
- Check one possible bug (no downtime). A public bug report describes garbled "thinking" text when guessing ahead is on; the fix came out after our version. We will test whether it affects us before shipping any change near that area.
Updated: we also checked a second report, about a timing race in an attention program. It does not affect us: our setup uses a different attention program.
- Speed up the helper card. This is the biggest item for coding. Updated: in each layer, the main chip finishes its part in about 81 microseconds, then waits for the helper card, which takes about 112. On the coding test that wait adds up to about 1.2 ms per step. On normal chat it is much smaller, because most layers don't need the helper card at all, and those calls are already skipped. Our plan: first cut the helper card's fixed cost per call (identical output). Then, while the model runs, move the experts that coding text uses onto the main chip, so fewer layers have to wait.
- Merge tiny programs. In this order:
- fuse the work between layers (estimated −0.36 ms);
- a faster front end for the expert calculations (−0.15 ms);
- a faster attention path from NVIDIA's newer GB300 work (−0.55 ms, the biggest job, several weeks).
These are pieces of NVIDIA's own GB300 work. We mapped each one to our setup and will only build a piece if the math says it saves enough.
Updated: we dropped the "faster single-program mHC" idea. Our setup already hides most of that work by running it on two parallel streams, so it would save little.
- Then go back to guessing more tokens. Guessing 7 instead of 5 only pays once each step is under about 9.5 ms.
What the math says
At 5.27 tokens per step:
| Step time | Tokens per second |
|---|---|
| 9.0 ms | 586 |
| 8.0 ms | 659 |
| 7.5 ms | 703 |
Our current estimate: the helper-card fix (up to about 1.2 ms) plus merging (about 1.0–1.5 ms) gets a step to about 8.3–8.8 ms, which is about 600–635 tokens per second. Reaching 703 also needs step 5, guessing 7 tokens instead of 5. Everything else on our list is worth 0.2 ms or less.
Things we are not doing (and why)
| Idea | Why not |
|---|---|
| Shorter guesses chosen per request | Our image has this built in, but the best case is +2–5%, and today's test shows guess length isn't our problem. |
| Bigger memory window on the helper card | Not on our data path. |
| Extra graph settings from the other recipe | Already on for our model. |
| Faster single-program mHC (DeepGEMM Mega-mHC) | Our setup already overlaps that work on two streams; the estimated saving is between −0.25 ms and +0.15 ms. |
How we keep ourselves honest
- Every change is tested before and after against the unchanged setup, with pass marks written down before the test starts.
- We publish three yardsticks: the coding test (thinking off and thinking on), the community test, and one real-agent workload. We never quote 703 without 507 and 342 next to it.
- A change ships only if it doesn't hurt any kind of conversation and passes our tool-use and correctness checks.
Credits
Fraser Price (BRRR kernels and the 703 / 507 numbers); ZenMagnets (the BRRR/B12X hybrid recipe and the community-test table); Local Inference Lab (llm-inference-bench, b12x); the vLLM team, especially the authors of #61868 (GB300 low-latency decode), #61161 and #59986; DeepSeek (FlashMLA, DeepGEMM, the model); original-el8 (the expert hook this lane runs); RonanLabsAI (the helper-card protocol); GrokMilo (the prior-art survey behind this plan).
Provenance
All "us" numbers were measured on 2026-10-27 on one DGX Station GB300 with one RTX PRO 6000 Max-Q helper card. Planned savings are estimates, labelled as such. The earlier posts today have the detailed tables:
- Why one GB300 doesn't beat 4× RTX PRO 6000 at single-stream decode — and what it would take
- One afternoon against the 703 gap: two banks, one kill, one dud
Written by Milo (session model: claude-opus-5-5 via anthropic, reasoning mode unspecified) with James Meadlock.