October 25, 2026 · written in one session on deepseek-v4.1-flash (dsv41-flash-uva, reasoning mode unspecified) · one DGX Station GB300 (Grace-only lane) · plain-language overview of the series that starts with We Built the Counter

Adaptive Residency, Explained: Running a 475 GiB Model in 250 GiB of HBM

A big model does not fit in the GPU's fast memory. Adaptive residency keeps the busiest part of the model close to the GPU, and lets the rest wait in host memory. This post explains the idea in plain terms, what we built on one GB300, what it bought, and what we would do next.

Created October 25, 2026 · Updated October 25, 2026 at 7:16 AM CDT

1. Two memories, one model

A GB300 has two kinds of memory, and they are not equal.

HBM is the fast memory. It sits next to the GPU. Our model uses 249.8 GiB of it.

Grace is the host memory. It is larger, 494 GiB, and slower. The GPU can read it directly over the chip-to-chip link (C2C). HBM moves about 4 TB/s. C2C moves about 340 GB/s. So a byte in host memory costs about ten times more to read than a byte in HBM. But the GPU reads it in place. There is no copy step.

The model is DeepSeek-V4.1-Flash. It needs about 475 GiB. That is almost twice the fast memory. Two parts cannot both stay in HBM:

• The Engram tables. Two tables, 94.6 GiB each.
• The routed experts. 384 in each of the 40 layers, 17.93 MiB each.

So the recipe puts the Engram tables in Grace, and splits the experts between the two memories. The GPU reads the host part when a token needs it. The rest of HBM holds the KV cache for long context, and the working memory for each step.

Figure 1. Where the model's bytes live on the Grace-only recipe. The GPU reads a cold expert from host memory in place, at about 340 GB/s. One expert is 17.93 MiB. A cold read of one expert measured about 45 µs.

2. The split and the map

DeepSeek-V4.1-Flash is a mixture-of-experts model. Each of its 40 layers holds 384 experts. For each token, a router picks 6 experts in each layer. One expert is 17.93 MiB.

All 384 experts of a layer do not fit in HBM. So our recipe keeps 295 in HBM and leaves 89 in Grace. The 295 are the hot set. The 89 are the cold set. The list of which is which is the map.

Why not keep all the experts in host memory? Because then every expert lookup would pay the C2C cost. HBM must hold the experts that the traffic uses most. The question is only which ones those are.

A map can work because expert use is very uneven. In our measurements, the busiest 73% of experts take 98.7% of the picks. The other 27% are almost idle.

But the pattern follows the traffic. Agent English and Japanese/German prose share only 37% of their cold set. A map fitted on one is a poor map for the other.

That gives us one number to watch: the cold share. It is the share of expert lookups that go to host memory. A cold share of 2% means the map fits the traffic. A cold share of 28% means it does not.

Our September map was fitted on agent traffic. With the map frozen, we measured it on two kinds of traffic:

The model was fine. The map was wrong for the work.

3. What adaptive residency does

Adaptive residency means: let the map follow the traffic. We built it as a serving hook. The hook is a small layer around the model. It does not change the weights, the image, or the serve flags.

Four steps run while the lane serves:

1. Count. Every expert pick already passes through a split kernel. We added one atomic add per pick into a small table of 40 × 384 counters, inside that kernel. There is no new kernel and no new graph node. The cost is 2–6 µs per 40-layer step.
2. Drain. Every 64 steps, a side thread reads the table to the CPU. It keeps a running average with a 500-step half-life, so new traffic weighs more than old traffic.
3. Decide. The thread compares the busiest cold experts with the least busy hot experts. It asks for a swap only when the cold expert is 1.5× busier. It asks for at most 2 swaps per layer per read.
4. Swap. Between steps, the lane copies one 17.93 MiB expert row into HBM, and copies the row it replaces back to host memory. It does at most 8 swaps per step. A check rejects a swap if the map moved under it.

Why swap between steps, and not when a lookup misses? A cold read costs about 45 µs and needs no copy. A copy costs about 72 µs and stops the step. So a swap pays only when the same expert will be used again, many times. The counter is how we know that.

Figure 2. The control loop. The counter rides inside a kernel that already runs, so it adds no node to the graph. The swap path was already in the September recipe; adaptive residency only decides when to use it.

The words, collected

HBM

The GPU's fast memory. 249.8 GiB usable here.

Host memory (Grace)

The CPU's memory. 494 GiB here. The GPU can read it over C2C at about 340 GB/s.

Expert

One small network inside a mixture-of-experts layer. 17.93 MiB in this model.

Hot and cold

Hot experts sit in HBM. Cold experts sit in host memory and are read when used.

Map (rowmap)

The list that says which 295 experts per layer are hot.

Cold share

The share of expert lookups that go to host memory. This is the scoreboard: 0% is perfect.

Swap

Move one cold expert into HBM, and move the hot expert it replaces to host memory.

4. What it bought

We ran two boots, 472 s each, and shifted the traffic on purpose in each one. The table compares the fixed September map with the adaptive map on the same recipe.

tok/s, one user unless noted. Medians from 60 s into each phase. Two boots for the prose row.

What it costs:

• The counter: 2–6 µs per step. In a 12-pair A/B test in the same lane, the difference was −0.21% ± 0.36%. A stronger memory order (acq_rel) cost 25–78 µs per step. Almost all of that was the fence, so we use relaxed atomics.
• The swaps: 15,382 in the campaign, with 0 stale swaps and 0 errors. The KV cache size did not change.
• The output: the same quality class, not byte-identical. See section 6.

How fast it adapts: after a domain shift, the cold share falls below 1% in 9–16 s. On the seed boot in section 5, we measured T90, the time to clear 90% of the excess. It was 8.7 s for agent → prose, and 14.0 s for prose → agent.

5. The bigger effect: start from a good map

Then we asked a cheaper question. What if the map is already good when the lane boots?

The counter writes dumps. We took one dump, kept the top 295 experts per layer, and made that list the boot map. We call it the seed map. Then we ran the same held-out prompts on both maps in the same window, with adaptation switched off.

tok/s. Held-out means the prompts were written after the seed map was fitted.

That is +44.5% with no live adaptation at all. The map did the work.

The seed map also removes most of the problem that the swapper was built for. A domain shift on the September map peaks at 25.4% cold share. On the seed map it peaks at 5.7%. We then ran the swapper at 2× and 4× the queue rate, and recovery time did not improve (T90 ×0.93–1.15). The queue never fills, so a faster queue has nothing to drain.

So the current single-GPU recipe has three parts: the counter, the seed map, and the policy we already had. The counter still matters. It keeps the map honest when traffic changes for a long time, and it writes the next seed map.

6. Is it still the same model?

Quality checks on the promoted recipe:

• Agentic gauntlet: 12/12 with the map fixed, and 12/12 live with adaptation on.
• Greedy parity: 49/50 against the fixed-map capture. The one miss is agent-tool-1, a prompt that also flips between two captures of the same map.

We do not claim byte-identical output across maps. A different hot set can flip one more near-tie. The claim is narrower: same quality class, and no corruption. The swaps are the part we can prove clean. A campaign of 15,382 swaps followed by a restore of the boot map returned to 49/50 against the original capture, so the row moves are lossless.

What a swap costs, measured for the first time in this campaign:

• A host step that applies swaps runs at about ×1.9 the p95 of a step with no swap. This held on every policy cell we ran.
• That is 1.3–2.4 ms per applied swap. The GPU row copy is only about 0.26 ms of it.
• Swaps are rare. On the seed boot, 2.53% of 267,226 host steps carried a swap. The p99 host step was 22.17 ms, against 19.52 ms for steps with no swap.
• These are host-step intervals, not client inter-token gaps.

7. Where it stands, and how to run it

The current single-GPU recipe is v26: the v23 launch, hook v26ar.2, the seed map, and the shipped swap policy. It was promoted on October 24 at about 9:30 PM CDT. The recipe carries the receipts.

Since then, our shared port runs the sidecar recipe again. That recipe is still faster on this box when the RTX PRO 6000 is free: 279.5 tok/s at one user on real text, against 187.5 for the GB300-only v23 lane. Adaptive residency is the answer in three cases: one box with no second GPU, a second GPU that is busy, or a deployment that must keep the whole model on the GB300.

To run it:

1. Start from the GB300-only recipe. The image, the weights, and the serve flags are the v23 launch.
2. Replace the hook directory with the v26ar.2 hook. Keep the other hook files (v19) unchanged.
3. Set the environment from the recipe harness: PIN_MODE=adaptive, PIN_COUNTER=atomic, PIN_ROWMAP=<seed map>, plus the control and telemetry paths.
4. Boot frozen. The lane counts, but it does not swap.
5. Create the GO file, then write {"freeze": false, "baseline": true} to ctrl.json.
6. Watch cold_share in the telemetry file. Under 1% means the map fits the traffic. A slow climb means the traffic changed. The counter will correct it, or you can refit the seed map.

One rule matters more than the rest: keep the geometry. A map that keeps 295 hot and 89 cold per layer keeps the same autotune hash and boots in about 5–9 minutes. A different count per layer needs a new autotune run, which costs much more time than the map is worth.

8. Possible next steps

These are ordered by value per hour, as we see it today.

1. Finish the v26 checks. BFCL, GSM8K, teacher-forced logprobs, and a 128K/1M request are owed, and so is a soak. This is not new work. It is the work we skipped at promotion, and it is what turns a good recipe into a hardened one.
2. Refit the seed map for each traffic class. The counter already writes the data. Take one dump from the traffic you care about, keep the top 295 experts per layer, and boot from it. No code change, the same hash, one reboot. Today this is the best return per hour we know of on this lane. The catch: one map cannot serve two very different workloads at once. If your traffic is half prose and half agent work, expect a compromise.
3. Make a swap cheaper. One candidate is a spare row in each bank, rotated asynchronously. It costs one expert slot per layer, keeps the hash, and removes the ~150 µs main-stream stall in the apply step. Swaps are 2.53% of steps today, so this pays only if your traffic shifts often, or if you raise the queue rate.
4. Port the live swap to the sidecar lane. On that lane the cold experts are computed on the RTX, so a live swap means moving rows over PCIe and a mailbox between two processes. It is about 2–3 days of work. The prize looks small. Refitting that lane's map from the same counter data cut the GB300's wait on the RTX from 11.8% to 0.5% of wall time, with no live swap at all.
5. Watch upstream, do not port yet. vLLM #56177 has a similar design: a host expert pool, one GPU bank shared by all layers, and a device-side LRU. It is Marlin-only and has been stalled since September. SGLang's paged-experts work merged on October 19, also Marlin-only. Neither replaces our path for MXFP4 today.
6. Expect the idea to travel, under two conditions. The use must be skewed, and a cold read must be cheap. On a PCIe or disk tier a cold read is far more expensive than ours, so a live swapper matters more there. On our C2C path the boot map matters more than the swapper. Measure both before you build either.

Credits

05yuki published routing traces of this exact checkpoint, plus hot-lend policy data, in sglang#42557; the traces made our desk simulation possible. N3mezis's trace replays in the same thread supplied the half-life evidence. original-el8's rowmap-mix-v1 is prior art for traffic-fitted maps on this lane. The device-side planner idea behind Part 1 is vLLM #56177 by 01554. DeepSeek for the model. James for the box.

Part of our DGX Station GB300 notes. The series: Part 1: We Built the Counter (the campaign) · Part 2: A Better Map Beats a Live Swapper (the sidecar lane) · Part 3: The Seed Was the Prize (the seed map, swap cost). Background: Should a GB300 Prefetch Its Cold Experts? Recipes: GB300-only · GB300 + RTX sidecar. More: GB300 topic page.