Shipped: 484.5 tok/s on one GB300 — split attention at 4 splits, plus the crash fix that unblocked it
Release note for today's two posts: Splitting attention on one GB300: a 3.8% single-stream win, and the crash that blocks it and One stray 4-byte read: how the split-KV crash on one GB300 was actually a buffer that ended on a page boundary. Both are now resolved by this release.
The short version
The single-GPU lane that serves DeepSeek-V4.1-Flash is now on a new reference configuration, h2-REF. Two things changed: decode attention is split across the GPU, and a latent upstream bug that had been silently killing test runs is fixed.
Net effect on the coding workload: 484.5 tok/s at 10.67 ms per step, up from 467.5 — 3.6% faster per step, with every correctness gate still green.
What shipped
| Change | Detail |
|---|---|
| Split-KV decode attention | FlashMLA PR #25 built as a side extension (_flashmla_splitkv_C) next to the untouched shipped library, driven by a vLLM #57711-derived overlay. Splitting applies only to 1 request at exactly 6 tokens, at 4 splits |
| Rejection-sampler guard | vLLM #60945 backported onto our pinned overlay copy. Three lines, bit-identical, fixes an out-of-bounds read |
| Unchanged | Same checkpoint (stock weights), same sidecar expert offload, same KV host tier, same DSpark k=5 schedule, KV 4,400,013 |
Measured, this window MEASURED
Controls ran the previous production config twice (start and end of the bracket); the split arm ran in between.
| Measure | Controls (h1-REF) | h2-REF (4 splits) |
|---|---|---|
| Coding workload, 2K prompt, T=0 | 467.5 tok/s · 11.10 ms/step | 484.5 tok/s · 10.67 ms/step (1.036×) |
| Per-class, 1 user | 1.00× | prose 1.047 · structured 1.044 · code 1.043 · shell 1.021 · tool_json 0.992 |
| Per-class, 8 users | 1.00× | 0.989–1.011× |
| Community bench, 1 user | 1.00× | 0K 1.036× · 64K 1.069× |
| Community bench, 8/32 users | 1.00× | ~1.00× |
| Prompt processing | 1.00× | 1.001–1.021× · first token 0.980–0.999× |
Fidelity and safety: teacher-forced divergence within the controls' own run-to-run spread (weighted 2.784 against a bar of 3.192, 4 confident flips against a bar of 4, zero on agent/tool prompts), gauntlet 12/12, thinking-mode probe 16/16, long soak 4/4 with no loops, and a 900k-token prompt returning its needle in 31.2 s. Zero new Xid lines and zero engine errors in all four arms — the first split arms that have ever been clean.
Why 8 splits lost
Eight splits was faster in isolation (0.657× the unsplit kernel time) but slower in the server: 477.8 tok/s is only 1.022× on the coding workload against a 1.025× bar, and it drifted more (6 confident flips against a bar of 4). The extra combine work spends the kernel's win. Four splits keeps it.
The bug that had been faking a split-KV problem
Five earlier card runs died with a CUDA illegal-memory-access at 32 concurrent users, always on a step that never split, and always with our build loaded. The address was always the same offset — and the last block before it turned out to be an innocent bystander: MoE weight-scale buffers that merely pushed a 512-byte tensor to the end of a 2 MiB allocation.
The actual fault: draft_sampled[logit_idx + 1] in the speculative rejection sampler. The guard was sized for the configured number of draft steps (5), but our schedule drops to 3 steps at 9–32 concurrent requests, so the last row's read stepped one 4-byte slot past a 512-byte buffer. Normally that reads junk harmlessly; when the buffer sits at a segment edge, the next page is unmapped and the engine dies.
Upstream fixed it on 2026-10-12 in #60945; our pin predates it. With the backport, the identical failing workload runs clean while the unpatched code shows 20 invalid 4-byte reads under compute-sanitizer. Production had the same latent exposure all along — it was luck, not correctness, that kept it alive.
This corrects the earlier root-cause post: the MoE weight-scale buffers it identified were the bystander next to the fault, not the source of the read.
Reproduce
- Recipe: J-M-Recipes →
dgx-station-gb300/deepseek-v4.1-flash-vllm-uva-dspark-sidecar. The recipe update for this release is landing; it is not merged yet. - Split kernel: FlashMLA fork PR #25, mounted as a side extension; shipped
_flashmla_Cuntouched. - Guard: vLLM #60945 on
vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py. - Rollback: stop
dsv41-vllm-1017-h2-REF(+-rtx), startdsv41-vllm-1017-h1-REF(+-rtx), re-arm the watchdog.
Credits
Split-KV attention: FlashMLA PR #25 and vLLM #57711 (both closed unmerged upstream — we built and gated them ourselves). Crash fix: vLLM #60945. Watchdog and lane plumbing: existing.