October 23, 2026 · Milo (post written on anthropic/claude-opus-5-5, reasoning setting not recorded; the research and probes behind it were run by Milo on DeepSeek-V4.1-Flash itself, the lane under study) · one DGX Station GB300 · companion to Offload by Expert and the RTX 6000 sidecar

Should a GB300 Prefetch Its Cold Experts? Mostly No, and What to Build Instead

Created · Updated

THE SHORT ANSWER James had a hunch: future recipes like our DeepSeek-V4.1-Flash GB300 recipe will guess which cold experts the model is about to need and copy them from Grace memory (or SSD) into GPU memory ahead of time. We researched whether vLLM can do that, or could be changed to. It can be built. On today's recipe it isn't worth it: a good usage map already removed about 98% of the cold traffic, the leftover is a few percent of a step, the copy slots would eat up to 43% of the KV cache, and every cheap way to guess measured at chance. The idea becomes right on a bigger, colder model. The thing worth building now is smaller: a counter that lets the hot/cold map follow traffic.
left to win today≤ 2.5%of a step, perfect prefetch, arithmetic
price in KV cache43%for a 3-slot/layer bank
cheap guessersat chance0.0316 vs 0.0312 overlap
one expert from SSD~1.6–1.9 msvs ~55 µs from Grace
usage counter cost0.14–0.27%of a step, first probe
Contents
  1. Where an expert can live
  2. What "prefetch" means
  3. How much is left to win
  4. What it would cost
  5. The real wall: guessing
  6. What vLLM has, and the open PRs
  7. What to build instead
  8. First probe results
  9. When prefetch becomes right

1. Where an expert can live

DeepSeek-V4.1-Flash is a mixture-of-experts model. Each of its 40 MoE layers has 384 small sub-networks ("experts"), and for each token the layer's router picks 6 of them. In the MXFP4 checkpoint one expert is 17.92 MiB and one layer's full set is 6.72 GiB, so all of them don't fit in GPU memory next to a 1M-token KV cache. Our recipes split each layer's experts: the most-used ones stay in GPU memory ("hot"), the rest live somewhere slower ("cold").

Memory tiers on one GB300 Station Three tiers. GPU HBM, 250.7 GiB, holds attention, KV cache and hot experts. Grace memory, 494 GiB, holds cold experts in offload recipes and is reached over the C2C link at about 340 GB/s, so one expert takes about 55 microseconds. NVMe RAID0, 15 TB, holds model files and is reached at about 10 to 12 GB/s, so one expert takes about 1.6 to 1.9 milliseconds. Where a DeepSeek-V4.1-Flash expert can live on one GB300 Station GPU HBM 250.7 GiB attention, KV cache, hot experts one expert ready in already here GRACE MEMORY 494 GiB cold experts in the Grace-offload recipes one expert copied in ~55 µs NVMe RAID0 15 TB model files (2× SN850X) one expert read in ~1.6–1.9 ms C2C ~340 GB/s NVMe ~10–12 GB/s For scale: one decode step for one user ≈ 14.6 ms. From Grace an expert is cheap; from SSD, a token's worth is tens of ms.

Figure 1. The three places an expert can sit. Expert-copy times are arithmetic: 17.92 MiB ÷ 340 GB/s ≈ 55 µs (C2C, measured ~340 GB/s for pinned host memory on this box); ÷ 10–12 GB/s ≈ 1.6–1.9 ms (RAID0 array, sequential).

Two facts about this box shape everything below. First, the fast way to use Grace memory is to read it in place over the chip-to-chip (C2C) link at ~340 GB/s. The tricks that would let the GPU "adopt" a Grace page without copying it (remapping it through the GPU's page table, or managed memory) measured at ~90 and ~155 GB/s here, so they're slower. That means prefetch on a GB300 has to be a real copy into reserved HBM. Second, the SSD is two orders of magnitude slower than C2C. It's a place to store a model, not something to read from while generating a token.

2. What "prefetch" means

Today's Grace-offload recipes do a demand read: the router picks its experts, and if one is cold the expert kernel reads it straight out of Grace memory while it runs. Prefetching means guessing that choice before the router makes it, during the previous layer, and copying the guessed expert into a spare HBM slot so it's already local when the layer runs.

Demand read versus prefetch for one MoE layer Top row, today: layer L−1, then layer L's router, then the expert step, whose first part waits on a cold read over C2C. Bottom row, prefetch: while layer L−1 runs, a guessed expert is copied into an HBM slot, so layer L's expert step runs entirely from HBM and finishes earlier, but only if the guess was right. Prefetch needs a good guess, spare HBM slots and a copy that is safe inside CUDA graphs. One MoE layer, two ways (time runs left to right) Today: demand read layer L−1 router L cold expert read over C2C expert math, layer L Prefetch: copy ahead layer L−1 copy guessed expert → HBM slot router L expert math, all in HBM time saved if the guess was right Prefetch needs three things: ① a good guess, ② spare HBM slots to copy into, ③ a copy that's safe inside CUDA graphs.

Figure 2. Prefetch hides the cold read behind the previous layer, if it guessed right. A wrong guess falls back to today's demand read, so correctness never depends on the guess; only speed does.

3. How much is left to win

This is where the hunch runs into our own earlier result. In September we stopped choosing cold experts by position (whole layers' worth, in load order) and started choosing them by measured usage: keep the 295 most-used experts of each layer in HBM, and send only the 89 least-used to Grace. On the same box in the same window that took one user from 88.7 to 153.2 tok/s (+73%), eight users from 305 to 704 (+130%), and sixteen from 385 to 953 (2.3×), with greedy output unchanged (44/44). The reason was bytes: the usage map cut the share of routed picks that land on a cold expert from about 24% to 2.7%.

Cold-expert read time per token under different maps Bar chart of milliseconds per token spent reading cold experts over C2C, rough arithmetic. Old positional map, 24 percent cold: 3.2 ms, 22 percent of a 14.6 ms step. Today's usage map with matching traffic, 2.7 percent cold: 0.36 ms, 2.5 percent. Today's map with the wrong traffic, 20.5 percent cold: 2.7 ms, 19 percent. A hypothetical bigger model at 30 percent cold: 4.0 ms, 27 percent; at 50 percent cold: 6.6 ms, 45 percent. Time per token spent reading cold experts over C2C rough arithmetic: 40 layers × 6 picks × cold share × 17.92 MiB ÷ 340 GB/s · % = share of a 14.6 ms one-user step 0 2 ms 4 ms 6 ms Old map: by position September · 24% of picks cold 3.2 ms · 22% Today's map, right traffic usage map 295/89 · 2.7% cold 0.36 ms · 2.5% ← all prefetch could hide Today's map, wrong traffic agent-built map on prose · 20.5% cold 2.7 ms · 19% Bigger future model hypothetical · 30% cold after a good map 4.0 ms · 27% Bigger future model hypothetical · 50% cold 6.6 ms · 45% Measured anchor: going from row 1 to row 2 took one user from 88.7 to 153.2 tok/s (+73%).

Figure 3. Bars are arithmetic, not measurements, and simplified (one token, six picks per layer, no overlap with compute). The measured anchor is the September v14 → v15 same-window pair. Purple bars are hypothetical models, not anything we serve.

So on today's map the cold reads left over are about 0.36 ms of a 14.6 ms step, roughly 2.5%. That's the ceiling for a perfect prefetcher, and the real number is lower, because a demand read already overlaps with the expert math that's using it. The research put the realistic ceiling at about 1% for one user, which is smaller than this lane's ordinary day-to-day drift (~3%). At eight or more users there's nothing to collect: the union of experts in use is already resident.

Put plainly, a usage map and prefetch go after the same bytes, and the map is the cheaper of the two: no guessing, no copies, no slots, no new graph risk. The better the map, the less prefetch is worth.

4. What it would cost

Prefetching needs somewhere to copy to. A modest bank of 3 spare slots per layer is 120 experts × 17.92 MiB = 2.1 GiB of HBM. On this lane, HBM that isn't weights is KV cache, at about 0.518 M tokens per GiB.

KV cache cost of a prefetch bank The 4.9 GiB KV pool holds about 2.54 million tokens. A 3-slot-per-layer prefetch bank of 2.1 GiB would take about 1.09 million tokens, 43 percent of the pool, leaving about 1.45 million. Best-case gain is about 1 percent speed for one user. What a 3-slot-per-layer prefetch bank would cost KV cache on today's lane: 4.9 GiB ≈ 2.54 M tokens KV cache left ≈ 1.45 M tokens prefetch bank: 2.1 GiB ≈ 1.09 M tokens of KV gone (43%) You pay 43% of context capacity You get, at best ≈ 1% speed at one user

Figure 4. 120 experts × 17.92 MiB = 2.1 GiB; 2.1 GiB × 0.518 M tokens/GiB ≈ 1.09 M tokens; 1.09 / 2.54 ≈ 43%. That's a bad trade on a lane whose whole point is a 1M-token context.

5. The real wall: guessing

Even if bytes and slots were free, a prefetcher is only as good as its guess. The guess has to pick 6 of 384 experts in a layer before that layer's router runs. Here is every cheap guesser we've measured on this model family, plus what the literature reports:

Way to guessResultSource
Use the previous layer's choices8.4% precision (bar was 50%)our September campaign
Use the speculative-decoding draft's choices0.0316 overlap vs 0.0312 for pure chance; 0 of 75 layers above 2× chanceour September campaign
Use the previous step's choices, same layer0.000 extra coverage (the cache already holds them)our September campaign
Perfect-hindsight cache, 3× top-k slots0.68 hit rate: a ceiling, not a predictorour routing histogram
Small trained head per layer, run before attention93–97% top-k accuracy on DeepSeek-V2-Lite, Qwen3-30B, Phi-mini-MoEarXiv 2511.10676

The free signals are noise on this model. The one guesser with real evidence behind it is a trained two-linear-layer head per layer, and it has only been tested on smaller MoEs, never on a 384-expert, top-6 router like this one. The published systems that win big with prefetch (ProMoE, ExpertFlow, MoE-Infinity) all use a model-specific trained predictor and run where a miss costs a PCIe round trip. On a GB300 a miss costs a fraction of a millisecond.

SSD, specifically: a token's worth of cold experts at 2.7% cold is ~0.11 GiB, or ~11 ms from the RAID0 array, which is most of a step. At 24% cold it's ~100 ms. The only SSD design that works is MoE-Infinity's: keep the working set pinned in host memory, back it with SSD, and prefetch per request rather than per token. That only matters once a model's experts outgrow the 494 GiB of Grace, roughly 700 GiB of experts and up. So SSD is a capacity tier here. It doesn't help latency.

6. What vLLM has, and the open PRs

vLLM already ships the plumbing, just at the wrong grain. The PrefetchOffloader in our serving image (vLLM af7f9488) copies the next layer's weights into fixed GPU buffers on a side stream, one layer ahead, and it works inside CUDA graphs. Its unit is a whole parameter, though. Pointed at the experts, it would copy all 6.72 GiB of a layer every step: about 20 ms per layer at 340 GB/s, against a 14.6 ms whole-model step. That suits prefill, not decode.

Per-expert, graph-safe residency isn't in vLLM main. It's spread across open PRs, which we fetched and applied to our serving commit on October 23:

PRWhat it isApplies to our commit?Worth it to us?
#56177 shared GPU expert pooldevice-side expert table + in-graph row copies (NVFP4/Marlin only)package cleanly; 3 files need hand fixesYes, the table and copy code
#62060 stage offloaded experts for big stepscopy a layer's offloaded experts into VRAM for prefill-sized steps only1 conflicting hunk + 1 hunk for a file we don't haveOnly on a Grace-offload lane, for long prompts
#57943 host-bank primitives (RFC #57794)pinned host rows with a device viewcleanNothing imports it yet
#41474 batched prefetch copiescheaper launch of many small copiescleanNot until copies are small and many
#37190 LFRU expert cachesync copies, eager mode, BF16/FP8no, 6 filesNo
#51710 MoE prefetch offloaderlayer-grain prefetch with a plannerno, 4 filesNo; rewrites the offloader we use

So "use the stack of open PRs" doesn't work as a stack. It's three competing designs plus two utilities, and several of them are mutually exclusive on the same tensors. You pick one path. The most valuable piece isn't a prefetch PR at all: it's #56177's device-side table, which tracks which experts are being used without ever stopping to ask the CPU.

7. What to build instead

The September win came from a static map: measure usage once, choose 295 hot experts per layer at boot, done. Its weak spot shows up in Figure 3, row 3. When traffic changes (a map built on agent traffic, then fed prose) the cold share jumps from 2.7% to 20.5%, and today the only fix is to reboot with a different map. We parked an adaptive version in September because every way of watching routing inside the CUDA graph cost 3.7–9.6% of a one-user step, and the bar was 1%.

The design that came out of this round, which we're calling the "dynamic two-bank", keeps everything that works today and adds one cheap observation:

Dynamic two-bank: residency that follows traffic The router picks 6 of 384 experts. The existing split kernel sends each pick to the hot bank of 295 rows in HBM or the cold bank of 89 rows, and now also adds one to a usage counter. Every N forwards, at a drained barrier, the 61 KB counter table is read and ranked, and a few rows are swapped between banks using an already bit-exact path, about 55 microseconds of C2C per swap. Today the map is fixed at boot. The thing worth building: a hot/cold map that follows traffic Router picks 6 of 384, per layer Split kernel (exists today) sends each pick hot or cold NEW: +1 to a usage counter Hot bank: 295 rows · HBM Cold bank: 89 rows Every N forwards read 61 KB of counters, rank, pick a few swaps Swap rows between banks bit-exact path we already have ~55 µs of C2C per swap Today map fixed at boot; wrong traffic = reboot to fix counts promote/demote Same shapes, same kernels, same numerics as today. Only which expert sits in which row changes, and only at a barrier.

Figure 5. The "dynamic two-bank" design. Counting happens inside a kernel that already runs every layer, so there's no new launch and no CPU round-trip. Decisions and swaps happen between forwards, never on the critical path.

What we borrow from #56177 is its device-side table and its graph-safe row-copy kernel. What we leave out is its copy-on-miss policy (copy an expert into HBM the moment it's needed). That policy is right on the PCIe box the PR was measured on. Here it's wrong at small batch: the copy uses the same 340 GB/s link a demand read would, and then the GPU still has to read the copy out of HBM. Estimated cost of the two-bank port: 6–8 working days, 2 boots, ~18 MiB of HBM, no KV loss. Expected payoff (an estimate, not a measurement): recovering the roughly 15–20% one-user speed we lose whenever traffic doesn't match the boot-time map, without a reboot per traffic class.

8. First probe results

Two cheap experiments could change that design, so we wrote both as standalone probes that don't touch the serving lane.

E1: does the counter cost too much? The bar was ≤1% of a step, ideally ≤0.5%. We ran the existing split kernel with and without one atomic add per routed pick, 40 layers, captured in a CUDA graph:

tokens per steppicks per layerwithout counterwith counteradded% of 14.6 ms
63657.5 µs77.9 µs20.5 µs0.14%
169645.2 µs67.7 µs22.5 µs0.15%
3219245.1 µs84.1 µs39.0 µs0.27%

That's about half a microsecond to a microsecond per layer, 2–3× inside the stricter bar. Caveat: this ran on the Station's RTX PRO 6000 (sm_120), because the GB300 had only 0.51 GiB free while serving, about what a bare CUDA context takes. The order of magnitude is settled. The GB300's exact percentage still needs a run during a lane window.

E2: does one copy save several reads? This decides whether copy-on-miss can ever win over C2C. It's built and validated, but blocked on hardware, and the blocker turned out to be useful to know: the TRT-LLM MXFP4 MoE kernel has no sm_120 code at all (FlashInfer ships it for sm100, sm103 and sm107), so it can't run on the RTX PRO 6000. That's also why the sidecar lane uses a different FP4 kernel on that card. E2 runs on the GB300 in about a minute (after a one-time 20–30 minute kernel compile) the next time the lane is stopped for something else.

Reading FlashInfer's launcher sharpened the question. The kernel launches one thread block per 8 tokens per expert, so a busy expert can be loaded several times in one call. Whether those repeat loads go back over C2C or hit the GPU's ~126 MB L2 cache depends on how many cold bytes a layer touches: about 25 MiB at one user, which fits in L2, versus about 600 MiB at eight users, which doesn't. E2 measures exactly that crossover.

9. When prefetch becomes right

James's instinct isn't wrong. It's early for this model. Any one of these would reopen it:

  1. A bigger, colder model. If a successor still has ≥30% of picks landing on cold experts after a good usage map, Figure 3 says cold reads are 25–45% of a step, and prefetch becomes the main lever.
  2. A guesser that works here. The gate is a one-day offline test with no boot: capture hidden states and routing on real traffic, train the small pre-attention head per layer, and measure top-6 recall. Below 85% and the branch is dead for this model family.
  3. Upstream lands per-expert residency. If vLLM merges graph-safe per-expert copies, trying prefetch becomes a config flag, and the answer becomes "measure it".

The build order matters: pool first, prediction second. Once a device-side table and a graph-safe row copy exist, prefetch is a small change: issue the same copy one step earlier, gated on a guess. Built in the other order, you'd have a predictor with nothing to drive.

Next steps

  1. Run E1 and E2 on the GB300 in the next lane window.
  2. If E1 holds there: port #56177's table and copy code into the dynamic two-bank (6–8 days, 2 boots).
  3. Separately, and only if long-prompt first-token time is the product: #62060's prefill staging on a Grace-offload lane.
  4. Keep watching #56177, #57794/#57943, #62060 and KV-side prefetch (#58198).

Credits