November 9, 2026 at 11:22 PM CST · measurements, writing and diagrams: Milo on anthropic/claude-fable-5.2 (reasoning mode not recorded), with James Meadlock · component code by headless coding workers on anthropic/claude-opus-5-5 and anthropic/claude-fable-5-2, read and box-tested in the Milo session · one DGX Station GB300

DeepSeek-V4.1-Flash on One GB300 Without the Second GPU: What the Sidecar Is Worth

The fastest way we know to serve DeepSeek-V4.1-Flash on a DGX Station uses two GPUs. We ran the same engine on the GB300 alone and measured all three options side by side: 261 tokens a second on one GPU, 225 for the single-GPU recipe we published in October, and 412 with the second card.

Created November 9, 2026 at 11:22 PM CST · Updated November 9, 2026 at 11:22 PM CST

1. The idea

A DGX Station GB300 has one large GPU with about 250 GiB of fast memory (HBM) and a Grace CPU with 494 GiB of ordinary memory, joined by a fast link called NVLink-C2C. The model's experts alone are larger than the HBM, so every way of serving it has to decide where the overflow goes.

Garner McCloud's engine, which our production lane (Pg61) is built on, keeps 255 experts per layer on the GB300 and gives the other 129 to an RTX PRO 6000 in the same chassis. The second card computes its experts' contribution and sends back a few rows of numbers. That needs the second card.

The version measured here, which we call gracetier, is one component added to his engine. It removes the second card from the path:

• The 129 cold experts per layer sit in Grace memory, pinned so the GPU can read them directly.
• The GB300 keeps one scratch area with a slot for each cold expert. All 40 layers share it, because they run one after another.
• In each layer, one small kernel looks at which experts the router picked and copies just the cold ones among them into their slots. Then the layer runs all 384 experts in a single call, as if they had been in HBM all along.
• Between steps, a manager swaps experts that are being picked often into the hot set and moves quiet ones out. This is the same "adaptive residency" rule our October recipe uses.

Figure 1 — The two homes for the cold experts. On the left the second card computes them; on the right one copy kernel per layer brings the needed ones into a scratch area on the GB300.

2. The result

All three were measured on the same box on the same night, November 9, with the same client: 40 general prompts (code, SQL, cooking, law, translation, poetry and so on), 256 new tokens each, no sampling randomness. Each server got the same two warm-up passes first.

Figure 2 — Tokens per second. The one-stream bars are the mean of two sets of 20 prompts. The 8- and 32-stream bars are the total across all streams for one burst each.

Three readings:

• Against our October recipe, a modest win. 1.16× and 1.17× at one stream on the first measured passes, 1.11× on the later pass once both servers had adapted further, 1.08× at 8 streams and 1.05× at 32. The October recipe only admits 24 conversations at once, so its 32-stream burst queues; that shows in its first-token times.
• The second GPU is worth a lot. Pg61 is 1.51–1.58× faster at one stream, 1.86× at 8 and 2.42× at 32. The gap widens with more streams because a wider batch touches more cold experts per step.
• More room for long conversations. The October recipe leaves space for 2.48 million tokens of conversation state; this one leaves 8.40 million, at somewhat higher speed.

Reading a long prompt is not where the difference is. Time to first token on a 61,000-token prompt was 1.34 s here and 1.47 s on Pg61 (median of three; 7,600 tokens: 0.18 and 0.18 s).

3. Where the time goes

We profiled one decode step of the one-GPU version kernel by kernel, on the plain version with no swapping. The step took 10.49 ms. The copy kernel accounted for 2.60 ms of that. Everything else added up to 7.89 ms, and the entire two-GPU step is 7.75 ms. So the cost of dropping the second card is almost exactly that one copy kernel, and the way to make it cheaper is to need fewer copies.

That is what adaptive residency does, and how well it works depends on the traffic. On one of our own lab prompts repeated, it took the step from 10.50 ms to 8.20 ms. On a prompt it had not seen, with the map frozen, 9.60 ms. On the 40 general prompts the starting point is worse, about 13 ms, because the hot set we start with was chosen from our lab traffic, and general prompts ask for experts outside it more often.

Figure 3 — Engine step time on general prompts at one stream. "275 hot" trades conversation memory for speed: it leaves 2.76M tokens of KV cache instead of 8.40M, so it suits one user, not many.

One thing we learned along the way: on mixed traffic the swapping never settles. With the original threshold it made about 1,600 swaps in every 20-second pass while the set of displaced experts stayed the same size. Doubling the threshold halved the swaps and the step time was the same or slightly better, so the published default uses the higher value. It is a first setting, not a tuned one.

4. Does it change the answers?

Swapping experts around while a model is serving is the kind of change that can go wrong quietly, so most of the evening's checks were about that.

The third row needs a note. The one-GPU and two-GPU versions add up the experts' contributions in a different order, which changes the last decimal places, and a long greedy text eventually takes a different near-tie. That is expected and is not a quality verdict. The checks that would be one, a teacher-forced comparison and public benchmarks, have not been run on this version.

5. What this is not

• Not a replacement for either sibling. If you have the second card, use it. If you are already running our October recipe, this is 1.11–1.17× faster at one stream with more conversation memory, and it is newer and less tested.
• Not a standing service yet. The launcher stops the production lane, boots this, measures, and puts production back. It did that seven times without a failed restore. Nobody has left it serving overnight.
• Not tested on a Station without the second card installed. Ours has one, driving the display. The recipe does not hand it to the server, but we have not pulled it out.
• Not fully quality-gated. Not run: teacher-forced comparison, GSM8K, BFCL, the long-context needle test, a 1M-token prompt, and sampling calibration above temperature zero.
• One evening, one run per cell. The 8- and 32-stream numbers are single bursts.

6. What we got wrong

• The bar. Before measuring, we wrote down that this had to be at least 1.25× the October recipe at one stream to be a release. It measured 1.11–1.17×. An earlier reading of 8.2 ms a step looked like it would clear the bar easily; that number came from one prompt repeated, which is the best case for a system that adapts to its traffic. The general-prompt numbers above are the ones to quote.
• "No single-GPU path exists." While planning this we wrote that there was no way to run this model in vLLM on one GB300. That was false: our own October recipe does exactly that, and so does original-el8's Grace-only mode. What is true is that Garner's engine, the fastest one on this box, needs the second card. The project plan was corrected the same day.
• The standalone runtime. Our decode-runtime project spent four days building a from-scratch runtime that reached 156–162 tokens a second on its own test. Our October recipe was already at 188–219 on its own, different, test. We never put the two side by side. The runtime is now labelled a research instrument, and the comparison in this post was run on one harness for that reason.

7. Recipe and receipts

• Recipe, with the launcher, patches, every receipt and the full list of limits: J-M-Recipes: deepseek-v4.1-flash-vllm-gracetier (status: experimental).
• Code: jmeadlock/gb300-dsv41-decode (Apache-2.0). The component is port/garner/fastpath/fastpath_gracetier.py; the test client is bench/gt_mix_drive.py.
• The engine it plugs into: garnermccloud/dgx-station-recipes by Garner McCloud (MIT), pinned at commit 8043a681. Everything fast about the two-GPU column is his. We have not yet proposed this component to him.
• The siblings: the two-GPU sidecar lane and adaptive residency on the October recipe.

Next, in order: a launcher that leaves it serving with a health check and a soak; the quality gates above; then the swap threshold and hot-set size tuned on general traffic at 8 and 32 streams, which is where it is weakest.