One stray 4-byte read: how the split-KV crash on one GB300 was actually a buffer that ended on a page boundary
Follow-up to Splitting attention on one GB300: a 3.8% single-stream win, and the crash that blocks it, which left this crash as the open bug.
The short version
Yesterday's post left an open bug: splitting decode attention across the GPU made one user 3.8% faster, but the engine died with a CUDA illegal-memory-access whenever the server also handled 8 and 32 users. Splitting was off on the step that died, so it looked like state left behind by the split kernel.
It wasn't the split kernel at all. It was a 4-byte read that starts exactly at the end of a 2 MiB block of PyTorch memory holding the MoE weight-scale transforms, at an address where the next page happens to be unmapped. Our build only changed the layout of GPU memory, which moved that buffer to a segment edge — so a read that had always been harmless started landing in a hole.
How it was narrowed MEASURED
Each step is a card, and each has receipts.
| Card | Question | Result |
|---|---|---|
| W6 | Is the crash from our rebuilt FlashMLA library? | Split-off with the library loaded: 32-user cells passed, 2,584–2,611 tok/s |
| W8 | Is single-user splitting stable? | Yes: three single-user passes plus a standalone 32-user run, 2,600.9 tok/s, zero errors |
| W9 | Is it set at boot? | No: 32 users first, then 16, then 32 at 64K, then a single-user soak, then 16, then 32 — all clean |
| W10 | Is it the cell order (C8 then C32)? | No: eight cells in that order, each a separate benchmark process, all clean |
| W11 | Is it one process running every cell? | Yes — reproduced on the first arm, and again with splitting OFF |
What the fault actually is MEASURED
Both W11 arms died with the same GPU error (Xid 31, FAULT_PDE, ACCESS_TYPE_VIRT_READ) at an address whose low 29 bits are identical to every earlier crash (0x1F200000, 2 MiB-aligned):
- The address is exactly 0 bytes past the end of a 2 MiB PyTorch segment.
- That segment holds the MoE weight-scale-transform buffers allocated by
_transpose_sf_for_utccpandtransform_weights_for_mega_moeduring weight finalization. - The last block in it is 253,952 bytes (163,072 requested) and ends precisely at the faulting address.
- The following page is an unmapped CUDA reservation, so a read there is a page fault, not a wrong value.
Splitting off, same mount, same fault. So: not the split kernel, not the second library — an out-of-bounds read in the MoE weight-transform path, exposed by where our build places memory.
Why it only shows with our build loaded
Our library adds a few megabytes of allocations at start-up. That is enough to move a 163 KB transform buffer to the tail of a 2 MiB segment. Before, the buffer had another allocation behind it, so a 4-byte over-read hit valid memory and nobody noticed. With our build, it hits the boundary. The bug is older than the speed work; the speed work only made it visible.
What happens next
- DeepGEMM upstream fixes are being checked against our vendored pin
e1f418c2(#480, #451, #462). - Fallback mitigation, independent of upstream: allocate the transform buffers with at least one page of slack so a segment boundary can never follow them.
- Then re-run the same failing cell shape on the fixed build, with the address tracer on, and expect zero new Xid lines.
If that holds, split-KV ships for single-user traffic and the 3.8% becomes real.
Reproduce
~/dsv41/m3/att2e/ (Card W11), receipts under results-cardW11/S6/owner.txt, S7/owner.txt and maptrace/*.snap.pickle. Production reference remains dsv41-vllm-1017-h1-REF on :30006.