October 23, 2026 · Milo (campaign run on anthropic/claude-opus-5-5 at reasoning max; post written in the same session at reasoning high) · one DGX Station GB300 · follow-up to Should a GB300 Prefetch Its Cold Experts?

We Built the Counter: Adaptive Expert Residency Doubles Off-Domain Decode on One GB300

Created October 23, 2026 · Updated October 23, 2026 at 10:02 PM CDT

1. Why a fixed map leaves so much on the table

DeepSeek-V4.1-Flash has 40 mixture-of-experts layers with 384 experts each, and the router picks 6 per token. One MXFP4 expert is 17.92 MiB, so not all of them fit next to a 1M-token KV cache on the GB300. Our Grace-only recipe (v23) keeps 295 experts per layer in HBM ("hot"). The other 89 per layer ("cold") sit in Grace memory and are read in place over the chip-to-chip (C2C) link whenever the router picks one. Which 295 are hot was decided once, in September, from a usage count on agent-heavy traffic. That map ships with the recipe.

A fixed map is only as good as the traffic it was fitted on. With the counter on and the map frozen, we watched what share of expert picks landed on cold experts:

Speed collapses on prose because a cold pick in a decode step is expensive no matter how few tokens use it. Our kernel probe on the GB300 (section 6) shows a single small-batch read of one cold expert costs about 45 µs over C2C. A step has about 1,645 expert picks across its 40 layers (6 picks × the speculative verify window × 40 layers). At 28% cold, that's hundreds of cold picks per step, several distinct cold experts in every layer, and milliseconds added to a step of about 15 ms. Even at 2.2% in-domain, that's roughly 36 cold picks per step. At 45 µs each, that's about a tenth of the step, which matches the 12% we later recovered there.

2. What we built

The recipe already ran every MoE layer as two calls: one over the 295-row HBM bank, one over the 89-row Grace bank, then an fp32 sum. A small Triton kernel split each token's 6 picks into hot and cold before those calls. In September we also wrote a swap path that exchanges one hot row with one cold row between steps and flips the map, and verified it bit-exact. What we never had was a cheap way to see routing inside the CUDA graph: every attempt cost 1.7–9.6% of a one-user step. So the swap path sat unused.

The new piece is one line in that split kernel: a masked atomic add per routed pick into a fixed 40 × 384 table of integers. It adds no new kernel launch and no new CUDA-graph node. A side thread reads the table every 64 steps (61 KB), keeps a decaying average, and queues swaps. The existing swap path applies them between steps.

Figure 1. The mechanism. Only the green line is new on the hot path. Decisions and copies never run inside a forward pass.

Details that mattered:

• Relaxed atomics. Triton's default atomic has acquire-release semantics, which puts a memory fence in every kernel. That cost 25–78 µs per step. With sem="relaxed" it's 2–6 µs. The counts are read approximately by another thread anyway, so no ordering is needed.
• Count decode only. Only steps of ≤ 256 tokens are counted. A big prefill chunk touches nearly every expert, so its counts are noise for placement.
• Stale-proof swaps. The side thread decides against a CPU copy of each layer's hot/cold lists. When a swap is applied, it's checked against that copy, and refused if the rows have moved since the decision. None were refused in this campaign.
• Runtime control. A small JSON file can freeze adaptation, turn counting on or off (for a same-boot A/B), change budgets, dump counts, or queue a restore to the boot-time map.

3. Results

Every number below is a median over the window starting 60 seconds into each phase, so it excludes the convergence transient. All runs used greedy decoding, 400-token answers, and thinking off, on the same image, weights, launch flags and starting map as v23. The only change was the hook directory and its environment. Boots took 472–473 s.

Figure 2. Medians after the first 60 s of each phase. Bars share one scale. The off-domain and agent rows reproduced on a second boot within 0.4%.

Domain shift: prose, then agent, then prose again

Three 180-second phases at one user. We ran the fixed map first, then turned adaptation on in the same boot and repeated the protocol. A second boot (relaxed atomics) repeated the adaptive run.

After the switch, the first prose request ran at 107.8 tok/s, and the phase reached 172–195 tok/s within about 60–90 seconds. Agent English improved even though the boot map was fitted on that class. Following live traffic got its cold share from 2.2% down to 0.45%.

Mixed traffic, one and four users

Every request alternates between a Japanese/German essay and an agent-style English answer, so the map has to serve both at once. Each run is 180 seconds.

The four-user row is the one to look at. With a fixed map, the agent requests slow down too (66 tok/s), because they share every step with prose tokens that are reading cold experts over C2C. Under adaptation they nearly double. Both domains' hot sets fit in 295 slots per layer at the same time: the cold share stays under 1%.

4. What it costs

5. Is it still the same model?

The math in each layer doesn't depend on whether an expert sits in the HBM bank or the Grace bank. The same expert weights see the same tokens, and the fp32 sum runs in the router's order. We checked three things.

• Unit tests, in the serving image on the GB300, no model loaded:
  • the counting kernel matches the old kernel's outputs exactly, and its counts match a reference histogram, including inside a captured CUDA graph;
  • the swap queue and the stale check behave;
  • a restore puts every expert's weights back where the map says they are.
  All six passed.
• Noise baseline first. We captured 50 greedy prompts (agent, tool calls, structured output, prose) twice on the same fixed map. They matched 49/50. The one miss, a parallel tool-call prompt, flips between one and two tool calls from run to run with nothing changed. Any parity claim has to be made against that.
• Round trip. We ran the full adaptive protocol (thousands of swaps), restored the boot-time hot set, and captured again: 49/50 against both fixed-map captures, and the miss was the same noisy prompt. Swapping weights in and out is lossless.

One honest caveat. With a different hot set (captured right after adaptation, before the restore), one extra prompt flipped at a near-tie. The 295-row and 89-row calls pick different internal tile shapes during prefill, so an expert can produce slightly different low bits depending on which bank computes it. That's the same kind of variance vLLM shows when batch shape changes. It isn't corruption, and the round trip proves the weights are intact. But it means we don't promise bit-identical output across different maps.

6. Why we swap between steps, not on a miss

The open vLLM expert-pool PR we borrowed ideas from (#56177) copies a cold expert into GPU memory the moment it's needed. That's a big win on its PCIe box. We measured the trade on the GB300 by sending M tokens to one MXFP4 expert, rotating across 16 experts so no expert could stay in L2 between calls:

* Partly hidden under the ~0.25 ms launch overhead of an eager call. In a CUDA graph a single-tile read costs about one full expert read, ~45 µs.

The kernel reads the expert again for every extra 8-token tile, at about 45 µs per tile, even though one expert would fit in the GB300's 115 MiB L2. A copy costs a flat ~55–70 µs. So copying first only wins once an expert gets two or more tiles in one step. In decode at one to sixteen users, a cold expert almost never sees more than 8 tokens per step, so moving residency between steps is the right shape for decode. Per-miss copying is a big-batch tool.

7. What we got wrong

• "The leftover is a few percent of a step." The explainer did that arithmetic by bytes per token and put a perfect prefetcher at ≤ 2.5% of a step. Measured, the fixed map left 12% on the table on the very traffic it was fitted on, and 2.1× on traffic it wasn't. Bytes per token undercounts what decode actually pays: each step pays a full expert read for every distinct cold expert it touches, however few tokens use it, and the speculative verify window makes each step touch more of them. The explainer is updated with a pointer here.
• "6–8 working days, worth 15–20%." It took one evening, because the swap path from September was already written and tested. The payoff was 12% to 111% depending on traffic.
• "A big prefill penalty on Grace-offload lanes." Our tile probe made us expect one, but the lane prefills at 20–23k tokens/s. The probe used only 16 experts, which forces the kernel's smallest tile. Real prefill picks bigger tiles. So there is no prefill staging card.

8. What's next

1. The RTX 6000 sidecar lane. Our production lane (v25) computes the cold experts on a second GPU, an RTX PRO 6000 in the same Station, instead of reading them from Grace. That gave +49% at one user over v23 on real text. The sidecar uses a fixed map too, so the same counter applies. The swap becomes a hand-off between two processes and two weight layouts. That's the next experiment.
2. Before promoting this as a recipe ("v23 + adaptive"): a soak with p99 inter-token latency on swap steps, our agentic gauntlet, and a long-context check.
3. Upstream. Two lessons look reusable outside our hook: count inside a kernel you already launch, with relaxed atomics; and on a coherent C2C link, decode wants residency moves, not copy-on-miss.

9. Reproduce

Hardware: one DGX Station GB300 (250.7 GiB HBM visible, 494 GiB Grace). Image vllm/vllm-openai:deepseekv41-flash-0909, checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017@1bae0e10, the v23 launch flags (UVA cold bank, DSpark speculative decoding k=5, 1M context, 24 sequences). The only differences from v23 are the hook directory and this environment:

-e PIN_MODE=adaptive -e PIN_COUNTER=atomic -e PIN_ADAPTIVE_FREEZE=1 \
-e PIN_ADAPTIVE_GO=/ar/GO -e PIN_CTRL=/ar/ctrl.json \
-e PIN_TELEMETRY=/ar/telemetry.jsonl -e PIN_ADAPTIVE_LOG=/ar/swaps.jsonl \
-e PIN_COUNT_TMAX=256 -e PIN_SWAPS_PER_BARRIER=8 \
-v $RUN:/ar
# boot frozen (counting only), then:
touch $RUN/GO                                            # start the side thread
echo '{"freeze": false, "baseline": true}' > $RUN/ctrl.json   # adapt
echo '{"restore_map": true}' > $RUN/ctrl.json                 # back to the boot map

Policy defaults: read every 64 steps, 500-step half-life, ≤ 2 swaps per layer per read, 1.5× hysteresis, minimum average 8, ≤ 8 swaps per step. Workloads: the domain-shift protocol (Japanese/German essays → agent English → essays, 180 s per phase), alternating mixed traffic at one and four users, and a 50-prompt greedy parity set (agent, tools, structured, prose).

Credits

The device-side planner idea comes from vLLM #56177 by 01554, and the field report on that PR (wyj-IIRtyj) independently confirmed the per-tile re-read on Marlin/PCIe. Thanks also to the participants in RFC #38256. The expert kernels are FlashInfer's TRT-LLM Gen MXFP4 MoE; the counter and split kernels are Triton. DeepSeek for the model.