Adaptive Residency, Part 3: The Seed Was the Prize
We set out to build a faster expert swapper. The map we boot from was doing all the work — and making the swapper 2–4× more aggressive bought nothing measurable.
1. The vocabulary, briefly
DeepSeek-V4.1-Flash is a mixture-of-experts model. Each of its 40 layers holds 384 experts, and the router sends each token to 6 of them. One MXFP4 expert row is 17.93 MiB, so a full layer does not fit in HBM next to everything else the model needs. Some of it has to live elsewhere and be fetched on demand.
Our lane splits each layer: 295 of its 384 experts stay resident in HBM (the "hot" set), and the other 89 sit in Grace host memory (the "cold" set), read over C2C when a token actually needs one. Which experts hold those 295 slots is the map, or rowmap. That list is the only thing this post is about.
Three terms, and the whole argument becomes readable:
- Cold share
- The share of expert lookups that have to read host memory instead of HBM. This is the scoreboard: 0% means every lookup landed in HBM.
- Swap
- Copying one 17.93 MiB expert row from host memory into an HBM slot, and the row it displaces back out. The policy decides how often a swap may happen.
- Adaptive residency
- Letting the map follow live traffic instead of freezing a September guess about what is hot.
Figure 1 — The two tiers, for one of the 40 layers. The map is a decision about which 295 experts hold HBM slots; a swap is the only way that decision changes at runtime. Measured: one applied swap costs ~1.9× the p95 of a swap-free step, of which the row copy is 0.26 ms.
2. What Part 1 got wrong, and what it pointed at
Part 1 ended with adaptive residency converging "within 60–90 seconds" of a domain shift. That number was wrong, and the correction is already live on that post: re-analysis of the campaign's own telemetry puts the cold share back under 1% in 9–16 s, and each prompt back at ~90% of its converged speed in ~12–20 s. The 60–90 s read came from 30-second medians over five rotating prompts whose converged speeds differ by up to 19%.
The re-analysis also named what sets recovery time: not the decay half-life, not the hysteresis, but the swap queue budget — 2 swaps per layer per 64-step read. After a hard shift, 31–40 of the 40 layers sit pinned at that budget for about ten seconds while the map catches up.
So the next experiment looked obvious. Raise the queue rate, drain it more often, and watch the transient shrink.
3. Simulate first, boot second
Boots cost about eight minutes and a domain shift is hard to reproduce by hand, so we simulated before spending any. The simulator replays the hook's exact policy code — same EWMA decay, same thresholds, same per-layer and per-step budgets — over public routing traces of this exact checkpoint that 05yuki published (ten DeepSeek-V4.1-Flash-1017 decode traces: Japanese prose, Python, math, Chinese, JSON). We checked it against the production functions in lockstep: 28,329 decisions, zero divergence.
It pruned a five-cell ladder down to three cells and made two falsifiable predictions for the box: doubling the queue rate (drain every 32 steps) would not halve recovery time, and quadrupling it (drain every 16) would.
A literature round killed the spicier ideas before they cost a boot. Promoting an expert on a single hit lost in both systems that tested it directly: a two-touch gate cut page promotions 11× in Intel's tiered-memory work, and a dwell-0 cache in llama.cpp ran 0.84× against 1.21× with dwell. Every published "trial tier" is free entry plus earned promotion — the reverse of making experts earn their way into HBM. Our misses are cheap in-place C2C reads, so promotion evidence is worth less here than anywhere.
4. What the box said: nobody halves anything
Three cells, same seed-map boot, each compared against a same-boot control. Higher recovery time is worse.
Figure 2 — Recovery time (T90: the time to clear 90% of the peak-to-converged cold-share excess) for the shipped policy and for 2× and 4× the queue rate. Three repeats per cell. All speeds are within noise of each other.
| cell (knobs) | agent→prose T90 | prose→agent T90 | vs control | steady churn | converged C1 (prose / agent) |
|---|---|---|---|---|---|
| B0 — shipped (drain 64, 2/layer) | 8.7 s | 14.0 s | 1.00× | 95/min | 189.6 / 218.9 |
| B1 — 2× rate (drain 32, 2/layer) | 9.3 s | 14.0 s | 1.08× / 1.00× | 103/min | 189.6 / 218.2 |
| B2 — 4× rate (drain 16, 2/layer) | 10.0 s | 14.0 s | 1.15× / 1.00× | 112/min | 189.5 / 217.9 |
Cold-share T90 in 2 s windows, 3 repeats per cell, all cells on the seed-map boot over the same window. "Steady churn" is swaps per minute; converged C1 is decode speed at one concurrent request.
Both simulator predictions missed — in the most informative way possible. The simulator ran its ladder from the September static map, where a shift sends cold share to 25–30% and the queue saturates. The box ran the cells the way the card specified: on the seed-map boot. And on a seed boot there is almost no transient left to fix.
Figure 3 — The same domain shift, two boots. On the September map the shift peaks at 25.4% cold share and the queue sits saturated for about ten seconds; on the seed map it peaks at 5.7% and never saturates. With nothing backing up, the queue rate stops mattering.
The reason is geometry, not tuning: 295 hot slots per layer hold both domains' working sets at once. That is the same fact the four-user mixed result in Part 1 showed from the other direction — one map, both workloads resident, no thrash.
One secondary metric did move: time to reach 2× the converged cold share improved from 14 s to 6 s on the prose→agent shift. At a 3–6% peak there is very little excess left for that metric to describe, so we are not counting it as a win.
So the aggressive-policy axis is closed with receipts, and the right policy is the boring one we shipped in Part 1.
5. The seed map: +44.5% for zero new mechanism
The seed map is nothing clever, which is exactly the point:
Figure 4 — Building the seed map. No new kernel, no new geometry: the same 295/89 split, with a better-informed choice of which 295.
Take one cumulative dump of the live counter's decayed counts, keep the top 295 experts per layer, write the rowmap, boot from it. Same 295/89 geometry, same kernels, autotune cache hit, ~8-minute boot. It was scored on held-out prompts — the map was fitted on the campaign pools, so in-sample numbers do not count.
Figure 5 — Frozen-map comparison, held-out prompts, same window. Adaptation is frozen on both arms, so the only difference between the bars is which experts booted into HBM.
| frozen-map comparison, held-out, same window | September static map | seed map | Δ |
|---|---|---|---|
| mixed traffic C1, aggregate | 127.1 | 183.7 | +44.5% |
| agent English C1 | 202.7 | 201.4 | −0.7% |
| first prose request after boot | 85.0 | 177.9 | 2.1× |
Adaptation frozen on both arms. With adaptation on, both boots converge to the same steady state (189.6 / 218.9 tok/s) — the seed changes where you start, not where you end.
Quality went through the same gates as every candidate: agentic gauntlet 12/12 on the seeded lane, greedy parity 49/50 against the static map, with the one miss being agent-tool-1 — the prompt that flips run to run even on a fixed map.
Prior art note: fitting a static map from route counts was original-el8's move first (his rowmap-mix-v1, September 26, ships as his hook's default); Part 2 now credits that properly. The part that is ours is seeding from the live counter's own dump and scoring held-out.
6. What a swap actually costs, measured for the first time
This campaign added a per-step telemetry ring to the hook — one 4-int64 write per host step, about 0.5 µs — so for the first time we can price swaps instead of estimating them. A step that applies swaps runs at ~1.9× the p95 of a swap-free step: roughly 1.9–2.4 ms attributed per applied swap at the host-step level, of which the GPU row copy is only ~0.26 ms (CUDA events around the copies). That ratio held in every cell, including the shipped-policy control, so it is a property of the mechanism, not of aggressiveness.
It sounds bad and mostly isn't. At steady state the lane swaps ~100 times a minute across ~5,000+ steps, so swap-bearing steps are rare, and what decides promotion is the overall inter-token p99 — exactly what the running soak measures. If it does bind, the follow-up is already designed: keep one spare row per bank and rotate asynchronously, which removes the in-step stall for the price of one resident expert per layer.
7. Dead, with receipts
- Aggressive queue rates — section 4. No gain at 2× or 4×, and the one shift that moved got slower.
- Promote-on-a-single-hit — loses where it has been tested (two-touch gate: 11× fewer page promotions; llama.cpp dwell vs dwell-0), and collides with this lane's prefill-count exclusion and apply budget.
- A dedicated trial tier in HBM — published tiers are free-entry plus earned promotion, the reverse design; and a third expert bank means a third kernel call per layer every step, or a new autotune geometry.
- Empty-HBM cold start ("earn everything") — strictly dominated by seeding. Purity would cost roughly 2× on every first request.
- Per-session maps — one contiguous weight tensor per bank, and the four-user result shows both domains coexist in 295 slots anyway.
8. What's next
The v26 candidate — the Part 1 counter, this seed map, the shipped policy, and the new telemetry ring — is serving its 48-hour soak now (started October 24, ~8 PM CDT). Owed before any promotion: the soak itself, a 128K-context check, and the overall-p99 inter-token number from the ring. The two experimental recipes get updated after that, receipts included. The sidecar lane (Part 2) still holds the absolute speed crown on this box; the v26 candidate is the single-GPU answer.
Credits
05yuki published routing traces of this exact checkpoint plus hot-lend policy data in sglang#42557 — the traces made the desk simulation possible. N3mezis's trace replays in the same thread supplied the half-life evidence. original-el8's rowmap-mix-v1 is prior art for traffic-fitted maps on this lane. The device-side planner idea behind Part 1 remains vLLM #56177 by 01554. DeepSeek for the model.