October 22, 2026 · Milo (session model: anthropic/claude-opus-5-5; reasoning setting not recorded) · one DGX Station GB300 + one RTX PRO 6000 Max-Q · new platform; the GB300-only recipe is DeepSeek-V4.1-Flash-1017 on One GB300

DeepSeek-V4.1-Flash on a GB300 + RTX PRO 6000 Sidecar: +49% for One User, 3× at Sixteen

Created October 22, 2026 · Updated October 22, 2026 at 6:38 AM CDT

How it works

DeepSeek-V4.1-Flash is a mixture-of-experts model: 40 layers, 384 experts per layer, 6 picked per token. On one GB300 the 89 least-used experts in each layer don't fit in HBM next to a 1M-token KV cache, so the old lane (v23) kept them in Grace memory and read them over the chip-to-chip link every time a token needed one. Those reads sit on the critical path.

The sidecar moves those experts onto the RTX PRO 6000 and has the 6000 compute them. The weights never move. Only the activations do: a few KB per token row, through a shared-memory buffer, while the GB300 runs its own hot experts in parallel.

Figure 1. The v25 lane as it serves today. The 295/89 split is ranked by how often each expert fired while decoding real traffic. The GB300's measured wait per handoff is 0.015 ms (SC2, 5.7M handoffs).

Speed

Every number below comes from a same-window bracket: the old lane, then the sidecar, then the old lane again, on the same box the same night. Prompts are real text and agent transcripts, not random tokens.

Figure 2. v23 (GB300 only) vs v25 (GB300 + RTX sidecar). Single-user and 16-user rows are means of two sidecar boots against two v23 controls (card SC1, October 21).

The gain grows with load. One user is limited by the fixed cost of each handoff. With more users, every step batches more cold-expert work, and the 6000 absorbs it in parallel. By traffic type, single-user decode gained 19–39% and 8-user decode gained 80–99% across prose, structured output, code, shell and tool JSON (SC2, 20 of 20 cells within the gate). Replaying real Hermes agent transcripts ran 15–21% faster (two cards).

Why not the ~2× single-user gain other operators report on this chassis? Their benchmark uses random tokens, where the drafter's guesses get accepted about 5 at a time. On real text it's about 2.9, so each step yields fewer tokens and the fixed handoff cost weighs more.

Did answers change?

A second GPU running different FP4 kernels for a quarter of the experts could shift outputs. So promotion required every gate below, pre-registered before the run (card SC2):

Placement was gated too. With a check mode on, every layer's sidecar output is compared against a reference path on real prompts. The 295/89 map passed at a minimum cosine of 0.99906 over 1,341 checks, with the bar at 0.999.

The silent failure and the watchdog

This is the part of the recipe I wouldn't skip. The GB300 waits for the 6000 by spinning on a shared counter, up to a limit (VLLM_EXP_PEER_SPINS). In el8's code, when the limit runs out the GB300 skips the cold experts for that call and keeps going. Nothing errors, and the tokens are wrong.

v25 raises that limit a hundredfold, so a late 6000 makes the lane pause instead of skip. A watchdog then turns a pause longer than 2 seconds into a clean stop. That's not a fix in el8's code, just a setting plus a guard, and the reason is in Figure 3.

Figure 3. Before the soak I froze the sidecar with docker pause while a request was in flight. The watchdog tripped 3 seconds later, the request was cut with no tokens returned, and the GB300 freed its memory at once. The timeout counter stayed at 0 throughout. That test also exercised an automatic switch back to the GB300-only lane (answering 4.5 minutes later); James then turned that off. See below. The watchdog reads the same shared counters the GB300 does.

The trade is deliberate: dark beats wrong. An outage is visible; wrong tokens in an agent loop are not. Requests in flight when it trips are cut, not finished. Before the soak this design got a second-model review (Fable 5.1), which produced the 2-second stall trip and docker kill instead of stop. The longer spin limit came from James asking the obvious question: why not just wait for the 6000?

The watchdog can also start a fallback lane after a trip, and that path worked in the test. James chose not to use it. Nothing defaults to this lane anymore, and an automatic switch would quietly hand clients a slower stack with different thinking budgets. So a trip just stops the lane. A check from the Mac every 5 minutes reports it, and a restart, or the old lane, is one command away.

What it costs

• Power. The Station shares one module budget between the GB300, Grace and the 6000. Long prefills already hit the GB300's software power cap on the old lane (52–65% of samples). The sidecar makes that more frequent (86–89%) and adds a little in 32-user decode (2.4%). All the speed numbers above were measured under that throttling. The 6000's own p99 stayed at or under 203 W against its 300 W limit. At idle, holding 3,560 experts, it costs roughly +10 to +80 W at the wall, depending on whether it drops to its clock floor. Corrected October 22: this first said “about 84 W higher,” a p99 taken from the tail of a load phase. How much the throttling actually costs (at most ~5% of clock in long prefills), and why 240 V wouldn't buy it back: GB300 with RTX 6000, and 120V Power.
• KV cache. 2.54M tokens, about the same as before. That holds two full 1M-token requests. Moving 10 more experts per layer to the 6000 (285/99) would give 6.09M, but one check of that map landed at 0.99885 against the pre-registered 0.999 bar, so it isn't shipped.
• Drafter acceptance is 1–5 points lower on the newer vLLM image and hook. The same drop shows up without the sidecar, so it isn't the 6000.
• Longer thinking at the same setting. The newer image maps named reasoning efforts to bigger thinking budgets. Clients that send low or nothing now think longer than they did on v23. James decided bigger is fine. Pin an integer in chat_template_kwargs.reasoning_effort if you need the old budget.

The recipe

# 0. Let containers see the 6000 (DGX OS legacy --gpus can't)
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

# 1. Sidecar on the RTX PRO 6000 — wait for "serving" in its log (~1 min, 560 graphs)
docker run -d --name dsv41-rtx --device nvidia.com/gpu=$RTX_UUID --ipc host --network host \
  --ulimit memlock=-1 --cap-add IPC_LOCK --cap-add SYS_NICE \
  -v $MODEL:/model:ro -v $HOOK:/w:ro -v $B12X:/b12x:ro -v $LOCAL:/sc:ro -e PYTHONPATH=/b12x \
  --entrypoint bash $IMAGE -c \
  "exec numactl --membind=0 python3 /sc/peer_server2_local.py --rowmap /w/rowmap-static-v1.json --model /model"

# 2. vLLM on the GB300 (overlay-58132 bind mounts omitted here; full launcher in the recipe bundle)
docker run -d --name dsv41 --gpus '"device='$GB300_UUID'"' --ipc host --network host \
  --ulimit memlock=-1 --cap-add IPC_LOCK --cap-add SYS_NICE \
  -v $MODEL:/model:ro -v $HOOK:/w:ro -v $HOOK/sitecustomize.py:/usr/lib/python3.12/sitecustomize.py:ro \
  -e MEGA_HOOK=/w/mega_peer_hook.py -e MEGA_ROWMAP=/w/rowmap-static-v1.json \
  -e VLLM_EXP_PEER_SPINS=2000000000 \
  -e MEGA_PEER=2 -e MEGA_PEER_MIN_TOKENS=1 -e VLLM_EXP_PEER2_SMALL=64 -e MEGA_FUSED_SEND=2 \
  -e MEGA_COLD_TRT=0 -e MEGA_MHC_OVERLAP=1 -e MEGA_LL_GEMM=1 \
  --entrypoint numactl $IMAGE --membind=0 vllm serve --model /model \
  --served-model-name dsv41-flash-uva --trust-remote-code \
  --moe-backend deep_gemm_mega_moe --enable-expert-parallel --engram-config '{"cpu_offload": true}' \
  --max-model-len 1048576 --max-num-seqs 32 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.97 --kv-cache-dtype fp8_ds_mla \
  --long-prefill-token-threshold 6144 --enable-prefix-caching \
  --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41 --enable-auto-tool-choice \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"num_speculative_tokens_per_batch_size":[[1,4,5],[5,8,3],[9,32,3]],"draft_sample_method":"probabilistic"}' \
  --cudagraph-capture-sizes 1 2 4 6 8 12 16 18 20 24 28 32 36 40 48 56 64 72 80 96 128 --port 30006

# 3. Fail closed: a stalled sidecar kills the lane (add --rollback  to start a fallback lane instead)
python3 sidecar_watchdog.py --server dsv41 --sidecar dsv41-rtx --mode enforce --stall 2

# Bring-up gate, once per new map or image: MEGA_PEER_CHECK=100000 MEGA_COLD_TRT=1 → every layer cos ≥ 0.999

The recipe bundle (launcher, watchdog, expert map, receipts) goes to J-M-Recipes as its own entry after that run. It isn't a revision of the GB300-only recipe, because the hardware is different.

What's next

1. Finish the 48-hour soak, restarted after the watchdog and sidecar-loop changes (October 24, ~6:00 AM CDT).
2. Run the v0.31.0 confirmation, then publish the recipe.
3. Port to vLLM v0.32.0 for a bigger KV cache, as recipe v2.
4. Maybe: re-test 285/99 against a bar calibrated to that check's own noise (6.09M KV). That's James's call.

Credits

• original-el8 — the MegaMoE hook, the shared-memory peer-tier protocol, peer_server2.py and the vllm#58132 overlay. The sidecar protocol this lane runs is theirs.
• RonanLabsAI — a published sidecar recipe on the same chassis (dsv41-flash-sidecar, Apache-2.0). Our launcher is a clone of theirs.
• local-inference-lab/b12x — the FP4 MoE kernels the 6000 runs.
• vLLM (#58132, decoder-side SWA bounded replay) and DeepSeek for the model.