Offload by Expert on GB300: The Skew Is Real, the Kernel Is Not
Why this question matters
On a GB300 serving an MoE model with weights that don't fit HBM, the standard move is --cpu-offload-gb with vLLM's UVA backend: pin a chunk of the routed-expert weights in Grace RAM and let the GPU read them in place across C2C. That is the whole trick that makes a 475 GiB DeepSeek-V4.1-Flash fit on one Station — and it works, at a cost.
The cost is measurable. Against the same model family on a two-Station all-HBM reference, our single-Station decode step is ~11.1 ms against their ~7.1 ms. About 4 ms of the step is the expert-fetch tax — roughly 36% of decode, spent pulling routed-expert rows over the Grace link. Moving 12.7 GiB of experts back into HBM bought 9.7 tok/s (0.76 tok/s per GiB, twice what a linear-bytes model predicted). The lane is fetch-sensitive, and where the experts live is the knob.
So the obvious follow-up: don't offload the first N layers positionally — offload the N experts each layer uses least. That is the question this post answers. It is the right question, and it runs into two walls.
The skew is real — the histogram
Before building anything, we collected a per-(layer, expert) selection histogram on the lane's real workload. Using --enable-return-routed-experts, an agent-shaped fixture produced 3,545 decode tokens and 850,000 routed selections across 40 MoE layers.
The skew is extreme and universal.
Measured, decode-only, main MoE layers:
| Metric | Value | Uniform ideal |
|---|---|---|
| Top-73% of experts, share of decode selections | 98.7% | 73% |
| Top-50% share | 91.8% | 50% |
| Top-25% share | 71.4% | 25% |
| Mean Gini across layers | 0.632 | 0 |
So a static usage-aware cold set would work, in principle. Two warnings before the walls:
- The ranking is domain-specific. Agent-class and prose expert ranks correlate at just 0.15; a cold set chosen on agent traffic only gets prose down to 20.5%, not 2.7%. Placement must be profiled on the lane's own traffic, not a borrowed fixture.
- A per-token cache is not the answer. An oracle cache at 3× top_k hits 68% with 384 experts and k=6 — too small to matter, and real caching would be far below oracle.
The kernel interface is the first wall
Here is the fact that reframes everything. I pulled the actual kernel source out of the running image, not from memory: vllm/model_executor/layers/fused_moe/experts/trtllm_mxfp4_moe.py and the FlashInfer entry points it calls.
vLLM's --cpu-offload-gb is per-parameter. The offloader walks modules in order and stops at the byte budget. A layer's w13_weight / w2_weight is one single tensor for all 384 experts. So the offload lever can move a whole layer's weight — it cannot move some rows of that tensor, and it cannot express "these 384 experts are cold." The current offload is layer-granular and positional, and it is that way by construction.
The FlashInfer kernel is the same story. Both trtllm_fp4_block_scale_moe and trtllm_fp4_block_scale_routed_moe take gemm1_weights / gemm2_weights as one contiguous [num_experts, ...] tensor, addressed by a linear expert index. There is no per-expert pointer list, no tuple-of-tensors path, no per-expert placement hook.
And the tempting expert_map is a red herring. It is accepted in the vLLM layer's apply() signatures, but it is never passed through to the FlashInfer call — it is the expert-parallel global-to-local id remap, consumed upstream. It cannot redirect an expert to a different physical row or a different memory backing. Permuting expert rows via expert_map changes order, not where the bytes live.
The bandwidth wall is the second — and it is decisive
Suppose you add the kernel indirection. You still have to put the cold rows somewhere, and here the hardware bites. We measured every GPU→Grace read path on this box (driver 595.84, CUDA 13.0).
The three spikes, precisely:
| Path | GPU read | Placement control | Verdict |
|---|---|---|---|
cudaHostAlloc pinned (UVA today) | 340 GB/s | No | Current baseline; platform ceiling |
VMM host-NUMA cuMemCreate (EGM) | 91 GB/s | Yes, per-row | Elegant but dead — ¼ speed |
| Managed + PreferredLocation=CPU | 90 GB/s | Yes | Death — same slow path |
| Plain managed memory | 155 GB/s (oversubscribed) | No | Dead — worse than pinned |
The VMM mechanism is sound and cheap — all six mechanical steps pass: host-NUMA cuMemCreate on this driver, mixed HBM/Grace backing under one virtual range, a torch view over it, remapping a row in place with the pointer unchanged. But the GPU reads EGM-backed host memory at ~91 GB/s against ~350 GB/s for the pinned memory UVA uses today, over the same C2C link. So a usage-chosen cold set at 2.7% traffic times a 4× per-byte cost is still less than 24% at normal cost — but it is far short of the 2.7%-vs-24% the histogram promised, and it moves the cold rows off the fast path off60 just won.
What it would actually take
Given both walls, there is exactly one design that keeps full speed and gets usage-aware placement:
int32 row-map the kernel consults at runtime so an expert id resolves to a row in either tensor.
| Design | Kernel change | Autotune | Parity gate | Speed + control | C1 ceiling | Effort |
|---|---|---|---|---|---|---|
| (B) two-tensor + row-map | indirection | 74 min | mandatory | Yes | ~+20% in-domain | 2–3 days |
| (A) physical per-expert split | rewrite/gather | 74 min | mandatory | Yes | ~+20% in-domain | 2–3 days |
(C) row permutation via expert_map | none | none | none | No — doesn't change backing | 0 | trivial |
- (B) is the only one that keeps the fast path and adds placement. It costs a kernel change, a new FlashInfer autotune hash (74-minute tune), and a mandatory token-exact parity gate — for a single-stream gain bounded by the remaining ~3 ms fetch tax, ~+20% C1 in-domain only.
- (A) is strictly harder than (B) — it changes the tensor geometry, not just the lookup — for the same ceiling.
- (C) is a false cheap path. Permuting rows changes order, not physical backing. It is a no-op for the memory problem.
The upstream doors (worth tracking, not building): vLLM's ExpertWeightProvider ABC (RFC #38256) would give a clean integration seam, and cuDNN's SM100 discrete-mode grouped GEMM is the only per-expert-pointer MXFP4 path — but neither is wired into this vLLM's FlashInfer path today. If either lands, offload-by-expert becomes a plug-in instead of a fork.
The honest takeaway
The residency axis is closed at off60 — and it is closed for a hardware reason, not a lack of insight.
- The skew is real and we measured it: 98.7%. Nobody had published a routed-expert histogram for this model before.
- The clean design (one pointer, mixed HBM/Grace backing, kernel and autotune hash unchanged) is dead on this driver because the placement-controlling memory path runs at ¼ speed.
- The only surviving design (two-tensor + row-map) costs a kernel change, a 74-minute autotune, and a parity gate, for ~+20% single-stream in-domain only. That was judged not worth it.
- What it bought instead: +14.7% from moving 12.7 GiB of experts back into HBM (the v11-to-v12 off60+util0.97 change), a fetch tax measured at ~27–36% of the decode step and known to be link-bound, and three negative numbers that anyone designing an expert-offload provider for GH200/GB200/GB300 needs before they design one.
If you are running an MoE model on a coherent-memory Grace-class box and thinking about usage-aware expert placement, the answer on this hardware is the two-tensor row-map — and you will pay the autotune and parity tax for it. If you are building the upstream provider, the data point that matters is the one we left on vLLM RFC #38256: on a coherent C2C system, a cache miss does not need a slot and a copy — but the placement-controllable host path is four times slower than the pinned path it replaces.