This is the plain-language companion to Fourteen boots, one streamer bug, and a model that finally talks — the full lab-log version with every boot, table, and receipt. Here the same story is told in three pictures: why the model does not fit, the one flag that silently broke it, and the trick that made it fast. Every number is carried unchanged from the parent post; nothing is remeasured here.
MiMo-V2.6-Pro is a mixture-of-experts model: 1.02 trillion parameters total, but each token only activates 42 billion of them. Most of the weight lives in 384 small "expert" networks per layer, and for every token the model picks 8 of those 384 per layer to actually run. The experts are also the bulk of the file: 494.9 GiB of the 527 GiB checkpoint is experts (the rest is 24.6 GiB of non-expert layers, 3.5 GiB embedding/head, and 4.2 GiB of drafter, vision, and audio towers).
The GPU in this box — a GB300 Station — has 250 GiB of HBM. So roughly two-thirds of the model has to live in the machine's host memory (494 GiB of Grace RAM) and be read over the CPU–GPU link every time a token needs an expert that is not on the GPU. That crossing is the whole campaign: the model works only if most reads stay local, and it is fast only if the reads that do cross are rare.
The memory puzzle. Left: what fits on the GPU. Right: the two-thirds of experts that must live in host RAM and cross the CPU–GPU link on demand. Figures from the parent post; the 28.1 GiB and 4.2 GiB bins group its 24.6 + 3.5 and 2.3 + 1.4 + 0.5 line items.
Concretely, at the settings used here, about 194 GiB of experts fit in HBM (27 of the 69 expert layers) and the remaining 301 GiB (42 layers) stay pinned in host RAM. At first placement that meant every token crossed the link for the majority of its expert reads — about 6.9 GiB per token — which is exactly the arithmetic behind the first measured speed of 30 tok/s.
For thirteen boots the model produced confident garbage — fluent-shaped text that meant nothing, byte-stable across runs. The cause was not the model, not the GPU offload, and not the two vLLM patches under test at the time. It was a single launch flag: --load-format runai_streamer, a streaming weight loader adopted earlier to survive host memory pressure.
The break and the fix. Garbage strings are quoted from the parent post verbatim, one boot per label (v10 and v11 returned different strings). Op-by-op figures are its in-process vLLM hook measurements on the 16-layer probe.
What the streamer does wrong on this checkpoint: the fused qkv_proj attention weights are FP8 with a separate block-scale tensor, and the pairing between weight and scale must survive loading intact. Any loader that fragments that group drops the pair — the attention weights silently load as all zeros and the MXFP4 expert weights come out mangled. The mechanism was found independently six hours earlier by vllmellm in vLLM PR #58142 on a different GPU and loader; the work here is the second-hardware, second-loader confirmation. The proof was an op-by-op comparison against an independent CPU reference built from Xiaomi's own modeling code: the default safetensors loader matched the reference at every operation, and the streamer diverged at the very first attention projection.
The lesson that made it findable: compare against a reference that is known good, one operation at a time, and let the first divergence name the bug. Guessing at the whole pipeline had produced a wrong hypothesis (a 6 AM theory about tensor-parallel sharding that the op-by-op diff disproved).
Before the garbage saga, three separate walls had to fall for a 527 GiB offload to load at all. Each was invisible until instrumented:
| Wall | What happened | Fix |
|---|---|---|
| Naming trap | The offload flag named expert tensors as experts.w13_weight, but vLLM wraps them as mlp.experts.routed_experts.*. The name matched nothing, so nothing offloaded and the whole model landed in GPU memory — instant OOM. | Use the full names: routed_experts.w13_weight routed_experts.w2_weight. |
| Power-of-two pinning | PyTorch's pin_memory() rounds every pinned block up to a power of two. One 4.5 GiB tensor pinned 8 GiB of host memory. Forty-two offloaded layers asked ~504 GiB of host for a 301 GiB set — and the machine died at 486 GB, twice. | Pin with cudaHostRegister at exact size (vLLM PR #58185). GPU reads measured 336 GiB/s registered vs 355 GiB/s torch-pinned, so it costs almost nothing. |
| MXFP4 shuffle | At load, the TRT-LLM MXFP4 backend re-shuffles expert weights into kernel layout and installs the result with replace_parameter() — registering a new GPU tensor and silently dropping the host-pinned view. Every offloaded layer quietly moved back to GPU (+7 GiB per layer). | A 40-line patch: if the existing parameter is a host view and the byte size matches, copy into the pinned storage and re-view it (bind-mounted, now filed). |
None of these caused the garbage. They caused the crashes. A fourth item deserves its own warning, because it cost a night:
--moe-backend marlin and VLLM_USE_DEEP_GEMM=0, the same flags every documented working V2.6 serve uses. Marlin turns the same autotune into 9 seconds, "Saved 0 configs". (Reported on vLLM issue #58031. Corrected September 23, 2026: an earlier version said the kernel is built for sm_100a only. It is not; the slow tune tracks how the experts are split between HBM and Grace, not the GPU generation.)With sane tokens at last, the question became speed. The naive placement — whole layers at a time — keeps the last 27 layers' experts in GPU memory and the first 42 in host RAM. But expert usage is not uniform: measured on real agent traffic, the late layers concentrate their traffic on very few experts (about 48–66 of 384 carry half the traffic), while early layers spread more evenly. So a small set of experts carries a large share of reads — wherever those experts sit.
Hotsplit re-homes experts individually instead of by layer: rank every expert by measured decode usage, keep the top-ranked ones in GPU memory within the same budget stock placement had, and leave the rest in host RAM. The outputs of the hot and cold passes are summed, so the math is unchanged — 16-layer parity tests were token-identical 6/6.
Stock placement versus hotsplit at the same GPU memory budget. Row bars are schematic (identical budgets, different placement), and the two coverage and speed figures carry the measurements from the parent post: 30.4% → 62.2% held-out decode from HBM; 31.2 → 37.4 tok/s at an 11.5K-token prompt.
On held-out conversation windows from real Hermes agent turns (270 windows, 26,357 generated tokens), the same GPU budget served 30.4% of decode reads from HBM at stock placement, 62.2% with the measured hot list — close to the 62.5% oracle, and better than ranking by prefill counts alone (54.9%). That coverage is the speed:
| Warm A/B, same recipe, only hotsplit differs | Stock placement | Hotsplit (v22, 152.8 GiB hot) |
|---|---|---|
| Decode, 11.5K-token prompt | 31.2 tok/s | 37.4 tok/s |
| Agent tool-call JSON | 29.5 tok/s | 36.6 tok/s |
| Plain prose | ~30 tok/s | ~30 tok/s (unchanged) |
| 8 streams at once (aggregate) | 56.2–60.8 | 63.7 |
Gains land where the hot list was ranked — tool calls, shell, structured output. Prose is flat, and on one synthetic prose prompt hotsplit is ~5% slower; the hot list is workload-specific, which is why v23 keeps a live routing counter and re-ranks weekly. Two side findings from the profile: speculative decoding (DFlash) loses under offload, because every drafted token that gets verified pays its own host-read cost (measured: 30.2 → 24.5 tok/s at k=3, 17.2 at k=7), and the remaining hot spot is the cold expert path — 39% of traffic but 80% of expert time.
Recipe v23 "Hotsplit" is released (experimental) and daily-driving in Hermes: J-M-Recipes — MiMo-V2.6-Pro on one GB300, vLLM UVA offload + hotsplit, with pinned image digest, patched files with sha256s, launch and eval scripts, and every receipt. Through the real Hermes harness: 10/10 tool-call turns correct.
Still open: a full-depth teacher-forced quality check on the FP8 o_proj experiment (quantizing it would save ~1.2 ms per token but changes the weights — needs a quality gate first), the first live weekly re-rank once 50,000 decode tokens per layer accumulate, and DFlash on the omni class.
For every number's provenance, the fourteen-boot table, and the reproducible launch block, see the parent post.