Splitting attention on one GB300: a 3.8% single-stream win, and the crash that blocks it
The short version
The model we serve — DeepSeek-V4.1-Flash-1017 — runs on one GB300 GPU with a second, smaller GPU (RTX PRO 6000) holding the "cold" experts. For a single user the speed is set by how long one step takes: about 11.0 ms today, five tokens per step, roughly 470 tokens per second.
Most of that time is not maths. Measured on the card during real single-user decode: the GPU's compute units are busy only 49% of the time, they issue an instruction 6% of the time, the tensor cores are 2.3% busy, and memory reads run at 32% of peak. The chip is mostly waiting.
So the useful work is removing waits. This post is about the one big wait we found and removed — and the crash that stops us shipping it.
Result 1: attention can be split across the GPU MEASURED
Decode attention for one user is tiny: 6 query tokens, 64 heads, 512-wide values. On this engine it runs one block per token — six blocks on a 152-SM GPU — so it uses about 4% of the machine.
We built FlashMLA's split-KV patch (PR #25) plus vLLM's #57711 overlay as a separate extension, and timed the kernel first, on its own:
| Kernel, 6 tokens, cold L2 | Time | vs unsplit |
|---|---|---|
| Unsplit | 29.07 µs | — |
| 4 splits | 20.22 µs | — |
| 8 splits | 19.09 µs | 0.657× |
Output check: the unsplit path stays byte-identical to the shipped library; the split path matches 99.9% of FP8 codes and ~1e-6 on LSE.
In the real server, with one user:
| Measure | Split off | Split on |
|---|---|---|
| Step time | 11.0 ms/step | 10.675 ms/step |
| Tokens per second | 468.7–470.2 (two controls) | 487.9–488.3 (two split-on runs): 1.038× |
| Workload classes | every class improved, 0.3% to 4.9% | |
That is the first sizeable step-time win since the L2 key-sharing change.
Result 2: it crashes at 32 users, and only when the split path ran OPEN BUG
Three separate builds — two that replaced the whole FlashMLA library, one that loaded the split kernel as a side extension — all die the same way at the 32-request decode cell of the llm-inference-bench sweep: CUDA_ERROR_ILLEGAL_ADDRESS, engine dead, at a step of 32 requests × 4 tokens = 128 tokens.
What makes it strange:
- The failing step never splits. Splitting is gated to ≤8 tokens, so 128 tokens runs the ordinary path.
- With the extension loaded but splitting switched off, the same 32-user cells pass clean: 2584.7 / 2577.4 tok/s at 0K and 2610.9 / 2545.5 tok/s at 64K, zero errors.
- A standalone harness is clean. It replays the serving step sequence on CUDA graphs — 738 steps, 600 of them splitting, one shared buffer pool, 4 splits — and is clean under NVIDIA's memory checker.
- Narrowing doesn't help. Restricting splitting to the single-user shape (1 request, exactly 6 tokens) still crashes.
So the fault is not the kernel's maths. Something the split path leaves behind in the serving process breaks a later 32-request step. That is the open bug.
What else changed this week MEASURED
- Thinking mode is clean on this lane: 20/20 probe cells including mixed prefill+decode rounds, and 12/12 chat checks with thinking on and off. The upstream corruption bug (#62417) does not reach us.
- Single-user community benchmark: 259 tok/s at 0K averaged over five runs (239–299 — this benchmark is noisy), 1130–1180 at 8 users, ~2600 at 32 users.
- Prompt processing: 0.21 s to first token at 8K, 0.40 s at 16K, 1.58 s at 64K, 3.22 s at 128K, 40–42k tok/s throughout. A 900k-token prompt returns its needle in 31.3 s.
Where we sit: 476 tok/s is 0.68× of a 4-GPU RTX PRO 6000 recipe (703, author-reported) and 1.4× of a hybrid 4-GPU lane's single-user number (341.9). Different hardware, so read it as placement, not a scoreboard.
Closed, with the measurement that closed it
| Idea | Result |
|---|---|
| Shallower drafts (k=3, k=2) | 0.67 ms faster per step but loses 7–17% tokens/step; 354 / 281 tok/s vs 476 |
| Mega-mHC overlap off | tied; decode takes the other path anyway |
| Host / PCIe settings | ASPM off, Gen5 x16 in every in-decode sample; idle "Gen 1" was a power state |
| Dense GEMM tactic tuning | 0.0026 ms/step in the real chain, against a 0.08 ms bar |
| Zero-copy sidecar I/O | 25–30 µs per call slower than the five copies it replaced |
| Breakable CUDA graphs | already on |
| PDL for the main kernels | already on (656 early-start links per step) |
One operational lesson
A crash core dump on this box was 602 GB and filled the 1.9 TB root filesystem. We deleted it to protect the serving lane, which cost us the post-mortem. Cores are now capped and the card refuses to start with less than 400 GiB free.
Next
- Find what the split path leaves behind. Ranked: graph memory-pool interaction, the side extension's per-call buffers, an async fault that only surfaces two steps later.
- If it cannot be fixed quickly: ship split-KV for single-user traffic only, behind a restart-on-fault watchdog, and keep the ordinary path for anything concurrent.
- Then L2 weight prefetch (modeled 0.25–0.4 ms) and the sidecar's fixed cost.
Provenance
Numbers marked measured come from this lane's receipts on one DGX Station GB300 with one RTX PRO 6000 Max-Q helper card. The 703 figure is author-reported, not re-run by us; the L2 prefetch saving is modeled, not measured.
Reproduce: ~/dsv41/m3/att2c/ (side extension, overlay, cards W6–W8), receipts per card under results-card*/. Production reference is dsv41-vllm-1017-h1-REF on :30006.