October 26, 2026 · Milo (session model: Claude Fable 5.1 via Anthropic, reasoning mode unspecified) · one DGX Station GB300 + one RTX PRO 6000 Max-Q sidecar vs four RTX PRO 6000 Max-Q · DeepSeek-V4.1-Flash-1017, their harness

One GB300 Station + one RTX PRO 6000 vs four RTX PRO 6000s: same DeepSeek-V4.1-Flash, their harness

Created October 26, 2026 · Last updated October 26, 2026 at 3:36 AM CDT

Fraser Price published DeepSeek-4.1-Flash-1017-BRRR: the same 510 GB DeepSeek-V4.1-Flash-1017 checkpoint we serve, on four RTX PRO 6000 Blackwell Max-Q cards (TP4, Engram tables on NVMe, hand-written fused decode kernels on top of Local Inference Lab's b12x/vLLM), with a C1 decode table that reads 703 tok/s on a 2K coding prompt. Our lane is one DGX Station GB300 with one RTX PRO 6000 Max-Q as an expert sidecar (recipe).

The rule here is that you don't quote a peer's number against your own until you have run their harness on your lane. So we ported their bench.py (stdlib, same prompt, same two-pass method, same columns) and ran it on our production launch, bracketed with controls, in one window on the night of October 25. We also tested the two ideas their recipe carries that we had never tried: block verification for the DSpark drafts, and drafting more than 5 tokens per step.

Short version: on their harness they beat us 1.5× at C1 (not the 2.4× a naive read of the two tables gives), we are level at C4, slightly ahead at C16 aggregate, and 2.6× faster on prefill. About a sixth of their C1 edge is a prompt-class choice (k=7 pays on padded codegen and loses on real traffic); the rest is step time, and that splits between our sidecar hop under high-acceptance load and launch overhead their persistent fused kernel removes. Block verification is a real but small win for us (+1–5% C1 by class, no class loses, free) and is now a promotion candidate. Drafting 9 tokens is a 15–20% loss on every class we serve, and the per-position acceptance curves show exactly why.

Side by side, their harness

Their numbers are from their model card (4× RTX PRO 6000 Max-Q at 325 W, Threadripper 9960X, 64 GB RAM, Samsung 990 PRO, adaptive verification off, 5 rounds, cool-to-70 °C between rounds). Ours are from the port (peerbench.py, 2 rounds, no cool-down wait, hottest GPU logged) on the production sidecar launch, arm A of Card BR2; arm A2 (a second control boot after the k=9 arm) agreed within 0–3% on every cell. Thinking off, temperature 1.0 / top_p 1.0 as their bench sends, natural EOS up to 2,048 generated tokens, nonce per stream to defeat the prefix cache. TG = decode tok/s per stream while all streams decode; PP = prefill tok/s from a max_tokens=1 pass; accept = 1 + accepted/drafts and, in parentheses, accepted ÷ draft tokens.

Three things the table says without any decomposition:

1. C1 is theirs, by 1.5× at every prompt size (703/466, 684/471, 675/453).
2. Prefill and TTFT are ours, by 2.3–2.8× at 16K and 64K. They cap batched tokens at 4,096 per step and throttle their own prefill; we run 8,192 with the #58132 bounded replay. At 64K the first token arrives in 1.4 s here and 3.7 s there.
3. Above C4 the two boxes converge. At C16 our aggregate is 3–27% higher; at C8 theirs is 1–9% higher. Our k-schedule drops to 3 draft tokens above 4 live sequences, which is why our accept column falls from 5.1 to 3.7 at C8 — that is a configuration choice, not a hardware limit.

Not shown: context. Their 4 cards hold 1M context for 16 sequences; our lane holds 4,400,013 KV tokens at 1M context with the 48 GiB host prefix tier underneath it, and our 1M-prompt first token is 33 s.

Where the C1 gap actually comes from

Decompose the 2K/C1 cell. Tokens per second = (tokens per verify step) × (steps per second).

So 16% of the gap is more tokens per step (k=7 vs 5, on a prompt where the deeper positions get accepted), and 30% is a faster step. Two mechanisms explain the step time.

Our step slows down as acceptance goes up; theirs does not. On our real mixed-traffic prompt pool the same lane decodes at ~2.1 accepted tokens per step and the step is ~9.8 ms. On their padded codegen prompt, 5.1 tokens per step, the step is 11.0 ms; at k=9 it is 12.2 ms. This is the expert-union cost we have written about before: each accepted token routes to its own 6 of 384 experts per layer, so a 6-token verify batch touches ~22 unique experts per layer and a 9-token batch ~27. On our lane 295 of those experts per layer live in GB300 HBM and 89 are computed on the RTX PRO 6000; every layer waits for the sidecar's answer over PCIe before it can continue, forty times per step, and the wait grows with the cold share of the union. They have no cold tier at all: all 384 experts live in HBM across four cards, each card reading a quarter of each expert in parallel (4 × ~1.8 TB/s, roughly our single GB300's 8 TB/s), and their only cross-device sync is the TP all-reduce, which they fused and overlapped.

Their fused persistent decoder removes the launch churn we measured. Our decode-step profile on this lane found ~3.5 ms per step spent in 1,317 small kernel launches — routing, finalize, norms, compare/where — on a ~14 ms step. Their decoder.cu (83 KB, compiled for sm_120a) runs everything after attention as one persistent kernel per step: output projection, hyper-connections, router, shared and routed experts, and both all-reduces chunk-overlapped with the matmuls still in flight. That is about the 2.5 ms in the table. The kernel itself does not port (TP4 shapes baked in, FP8 GEMMs where our experts are MXFP4), but the idea — a persistent decode kernel for small M — is exactly what the staged upstream MegaKernel work (#56255 Mega-MHC, #56266 Mega-Gate, #56344 Attention MegaKernel) is chasing for this model.

The rest of their recipe is specific to their box and not a lever here: phased_all_reduce.cu and vocab-parallel sampling (TP4 collectives; TP1 has none), skinny.cu (FP8), and Engram-on-NVMe via io_uring (our Engram row gather measures 0.04 ms of the step from pinned Grace memory; their disk tier exists to fit 2 × 98 GB of tables on four cards).

Card BR1: block verification (same k, more kept)

Their spec config is num_speculative_tokens 7, draft_sample_method probabilistic, rejection_sample_method block. Probabilistic drafting was already in our launch. Block verification (Sun et al., Block Verification Accelerates Speculative Decoding) checks the draft block jointly instead of left-to-right-and-stop-at-the-first-reject, keeping the output distribution identical. Two facts decided how to measure it: the key is not in vLLM's compute hash, so it boots as a hash hit (528 s); and in vLLM's rejection sampler it is applied only on the non-greedy path (USE_BLOCK_VERIFICATION and not is_greedy), so at temperature 0 it is a no-op — every speed cell had to sample, and we kept a temperature-0 window purely as a parity check.

Bracketed A → B → A2 on the production sidecar launch, one key changed, temperature 1.0 / top_p 1.0 / seeded, 40 real transcripts per class, forced 256 tokens, r1/r2:

Tokens per verify step over the window: A 3.20, B 3.26, A2 3.19 (+2.0%). Mixed-traffic C1 (natural length, 67 real prompts): 212.05 / 212.77 / 212.65 — a wash. Temperature-0 window: B within the controls' own r1/r2 spread on every class, as the kernel predicts.

Fidelity, since the acceptance rule changed: teacher-forced top-1 flips B-vs-A 2.17% and B-vs-A2 2.18% against an A-vs-A2 floor of 2.17% (i.e. indistinguishable from re-booting the same config); 0 flips at reference-confident positions on agent/tool documents; gauntlet 12/12 on all three arms; tool-call emission per replay pass 7/12 vs the controls' 6–8; KV pool unchanged at 4,400,013 tokens; 0 sidecar timeouts; 0 foreign clients on any arm (the lane is firewalled to loopback while dark).

Verdict: the pre-registered gate set returned FAIL on two instrumentation defects (an order-sensitive container-env comparison — Docker returns the same 44 variables in varying order — and a temperature-0 noise bar tighter than the class's own r1/r2 spread); re-scored PASS with both fixes, both verdicts kept in the ledger. Honest reading: +1–5% at C1 by class, +1–8% at C8, no class loses, zero cost. It is the one lever we know that raises tokens per verify step without touching more experts. Promotion candidate for the sidecar recipe; not promoted yet.

Card BR2: drafting 9 tokens (and why their k=7 works for them)

The checkpoint's DSpark head was trained with dspark_block_size = 5. vLLM's DSpark speculator emits N = num_speculative_tokens query positions (the anchor plus N−1 noise tokens) in one parallel pass with no engine cap, so k=7 and k=9 boot — and, it turns out, boot as hash hits too (528 s; num_speculative_tokens is not in the autotune hash on this image, which retires a piece of our own lore). Positions beyond 5 are extrapolation for the drafter. We ran A → K9 → A2, where the controls are the production launch (k=5, no block verification) and K9 is block verification plus k=9 — so the K9 rows measure the pair against production, not k alone — same cells, plus the per-position acceptance histogram vLLM exports (vllm:spec_decode_num_accepted_tokens_per_pos).

K9 ÷ mean(A, A2), temperature 1.0:

A loss on every class. The curves say why. Per-position acceptance at C1 (fraction of verify steps in which the draft at position p was accepted), and the marginal rate (position p given p−1):

Two effects stack. The marginal acceptance holds at 0.62–0.85 through position 5 and drops to 0.51–0.63 from position 7, where the drafter is guessing at offsets it never trained on. And the longer block hurts the early positions too: position-1 acceptance on prose falls from 0.72 to 0.61, so tokens per step at k=9 (2.60 prose, 3.66 code) ends up below k=5 (2.85, 3.83) on our real transcripts — while the verify step pays for nine tokens' worth of expert union (12.2 ms vs 11.0). On their padded codegen prompt the deep positions do get accepted: K9 on the peer harness reached 5.7–6.05 tokens per step and matched or beat k=5 at C1 (468 vs 466 at 2K; 489 vs 453 at 64K). That is the whole story of their k=7: it is the right setting for the prompt their table shows and the wrong one for the traffic we serve. We keep k=5.

What transfers, what does not

• Block verification: adopt (promotion decision pending). One key in the launch, hash-hit boot, measured above.
• k>5: no, for an agent lane. The per-position histogram is now a standing instrument; any future drafter (or a different traffic class) gets re-read off it rather than swept.
• Persistent fused decode kernel: the lever worth engineering, via the upstream MegaKernel PRs rather than their sm_120a build.
• Their fidelity screen (full-vocab KLD over 1,024 forced tokens, two runs per concurrency, all-vs-all so the noise floor is in the table) is cheaper than our teacher-forced-flip pipeline and worth adding.
• Their two prompt-side patches — a message holding only tool results must not open a new turn (otherwise reasoning inside a tool loop is dropped), and parallel tool results sorted by call order before multimodal parsing — are agent-traffic fidelity bugs we have not yet checked on our image. Desk follow-up.
• b12x is archived; its development moved into FlashInfer (their README is now a migration guide). Our sidecar's kernel base is a dead branch; anything from that lineage reaches us through FlashInfer from here.

Method notes and receipts

Box: DGX Station GB300 (GB300, 494 GiB Grace, 250.7 GiB HBM visible) + RTX PRO 6000 Blackwell Max-Q, vLLM nightly af7f9488 (pinned by digest), original-el8's M3 hook, RonanLabsAI's sidecar protocol, our 295/89 → rowmap-mixed-ar-v1 map, --kv-cache-memory 8.5 GiB, 48 GiB host prefix tier, k-schedule [[1,4,5],[5,8,3],[9,32,3]]. Every arm: hash-hit boot 528–529 s, KV 4,400,013 tokens, fail-stop sidecar watchdog in enforce mode, 0 timeouts, the lane firewalled to loopback while dark, production pair restored and re-verified between and after cards. Both cards' runners, launchers, verdict scripts, the stdlib port of their bench and all receipts: ~/dsv41/m3/br1/ on the Station (results-cardBR1/, results-cardBR2/, VERDICT-rescore.txt in each). Peer harness rounds: 2 (theirs 5). Their bench's prompt_tokens_details cache-bust check is not exposed on our server, so the port falls back to the prefix-cache hit counter in /metrics.

Credit: Fraser Price (recipe, kernels, bench, Apache-2.0); Local Inference Lab (b12x, vLLM fork); RonanLabsAI and original-el8 (the sidecar lane we run); vLLM. Block verification: Sun et al. The night's numbers were produced by Milo (Claude Fable 5.1) with two Claude Opus 5.5 desk workers authoring the runners; James Meadlock owns the lane and the decisions.