DeepSeek is a Chinese AI lab that releases its models openly — anyone can download the weights and run them. V4.1-Flash came out on September 10, 2026. Two things about its design mattered for us.
It's a "mixture of experts." Instead of one enormous brain that thinks about every word, the model has 384 small specialist brains ("experts") per layer, and for each token a tiny router picks the 6 most relevant ones. So while the model has 552 billion parameters in total, only about 16 billion do work on any given token. This is why a model this large can be fast at all — you only pay for the experts you use.
It has a giant lookup table called "Engram." Alongside the neural network, V4.1 carries two hash tables — about 189 gigabytes — that store memorized patterns of common word sequences. On each step the model looks up a handful of rows. It's like the model having a phone book on the desk: huge, but you only ever open it to one page.
Here is the problem. Our GB300 has about 250 GB of fast GPU memory (HBM). The model is 510 GB. It doesn't fit. Not even close. Every published recipe for this model assumed four of these GPUs, or eight. Nobody had posted a working setup on one.
The GB300 does have a trick, though. Next to the GPU is a big ARM processor ("Grace") with 494 GB of ordinary RAM, and the two are wired together with a very fast link (NVLink-C2C, around 340 GB/s). Ordinary RAM is roughly a tenth the speed of HBM — but it's there, and it's big. So the whole game became: which parts of the model can live in the slow-but-big memory without slowing everything down?
deepseekv41-flash-0909 image, and that page had the two pieces we needed: --engram-config '{"cpu_offload": true}' to park the Engram tables in host RAM, and the UVA offload backend for experts. Their recipe assumes four GPUs; ours is that recipe bent to one GPU by leaning much harder on the offload knobs. The UVA idea itself was not a guess either — we had already watched it work on this exact box a week earlier with GLM-5.3 (SGLang whole-layer offload 1.4 tok/s → vLLM UVA 33 tok/s), and 0xSero's 4× RTX PRO 6000 vLLM build was the receipt that DSpark worked on the shipped weights. So the "invention" was recognizing that the memory shape of the GB300 (huge coherent RAM next to a too-small GPU) called for the same trick.
Where does the time actually go? We profiled it. At one user, 64% of GPU time is streaming expert weights over that link; at eight users it's 88%. Attention — the part everyone assumes is the bottleneck for long context — is 17% and 6%. And decode speed was flat from 6,000 to 425,000 tokens of context (−7%). So on this box, long context is nearly free and the fight is entirely about expert bytes.
Some doors were closed by the hardware, not by us. We measured that 98.7% of expert lookups hit just 73% of the experts — a strong "hot/cold" pattern. The obvious move is to keep hot experts in HBM and cold ones in Grace. We tried three ways. On this driver, the GPU can read Grace RAM fast (340 GB/s, no control over placement) or with control (90 GB/s), but not both. Managed memory that's supposed to migrate hot pages automatically just faults everything in and thrashes at 155 GB/s. So we wrote it down, posted the measurements to the vLLM design discussion so the next person doesn't repeat it, and moved on.
We published a "win" that wasn't. One overnight run showed a gain on a replay of real agent transcripts. In the morning we ran the same replay three more times: 90, 119, 124 tok/s. The instrument swings ±17. The tight instrument (a concurrency sweep run twice per point, agreeing within ~1%) said the opposite. We retracted it in the post the same day. Bars go on the tight instrument now, and the recipe says so out loud.
The boring traps cost the most time. Changing almost any launch flag makes the kernel auto-tuner think it's on new hardware and spend 74 minutes re-tuning. We hit that four times in one day before learning to seed the cache before launch. And running from a network share instead of local disk turned an 8-minute boot into 92. None of this is clever; all of it is the job.
This deserves its own section, because "SGLang did 3.3 and vLLM did 85" reads like a verdict on SGLang, and it isn't one.
What SGLang got right first. It had day-zero V4.1 support, a working switch to put the 189 GB Engram tables in host RAM (SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1), and it was the first engine to boot the model on this box at all. Every correctness probe passed on the first try: arithmetic, tool-call parsing, reasoning mode. LMSYS's own day-zero post measured Engram-in-host on 4× GB300 with no decode penalty. That part of their design is sound and we used it.
What it didn't have. A way to offload part of the experts. SGLang's offloader (OffloaderV2, added in PR #8034) works at layer granularity: you tell it "keep N of every M MoE layers in host memory" and it copies each offloaded layer's entire expert tensor into the GPU when that layer runs, then discards it. That's a fine design when the copy is small or the link is fast relative to the compute. Here the tensor is 6.7 GB per layer, there are 15 offloaded layers, and the link is 330 GB/s. Copying 100 GB per token takes 300 ms. Period. No flag fixes that; it's the algorithm.
And the V4.1 model file didn't even wire the offloader in — Milo's 23-line patch did that. Someone asked SGLang for expert-granular offload back in December 2025 (issue #14233); it went inactive. It isn't a use case they've built for yet. Their target is racks of GPUs where everything fits in HBM and the hard problems are scheduling and disaggregation. Ours is one GPU that's 260 GB short. Different problem.
Why we'll revisit it. Three reasons, in order of how likely they are to matter:
That was my gut read too, and our own logs say it's half right. The precise version: SGLang adopts new datacenter hardware as fast as vLLM does; it adopts new deployment shapes — odd SKUs, consumer Blackwell, single-GPU-with-offload — much more slowly, because those aren't the customers it's built for. Our record on both engines, same lab, four months:
| Hardware / shape | vLLM | SGLang | Faster |
|---|---|---|---|
| GB300 Station (datacenter Blackwell), V4.1 day one | Day-zero image | Day-zero image; booted the model first; LMSYS measured Engram-in-host on 4× GB300 the day it shipped | Tie |
| GB300, DeepSeek Flash-Vision-Exp | — | Our production lane today — digest-pinned SGLang, DSpark, 1M context, 2,985 tok/s at 64 streams (topic page) | SGLang |
| DGX Spark (GB10, SM121 — a consumer-class Blackwell), DS4-Flash | Community fork working May 27 (PR #41834); four more forks followed and carried our Spark lane all summer | SM12 support merged ~June 25; August 1 canary: 7 tok/s, watchdog hangs, draft acceptance 0.03; not promoted | vLLM by 2+ months, and SGLang still didn't work |
| GB300, one GPU, experts offloaded (V4.1 and GLM-5.3) | Row-level UVA: 85 and 33 tok/s | Layer-copy offload: 3.3 and 1.4 tok/s | vLLM, 26× |
Read down the column: SGLang wins or ties where the box looks like what NVIDIA sells to clouds. It loses where the box is strange — a Spark's SM121, or a Station asked to hold a model 260 GB too big for it. vLLM's edge on those isn't that its core team is faster; it's that vLLM has a wider plugin surface and a community of forks (jasl, eugr, Aiden, Anemll on the Sparks; the dsv41-feat branch here) willing to hack kernels for hardware the main project hasn't prioritized. Ahmad's primer notes the same asymmetry in the platform lists. For a home lab — which is, by definition, a strange box — that's a real bias toward vLLM as the first engine to try. It is not a reason to skip the bake-off.
Bob's natural follow-up: "so what does Anthropic use? OpenAI? Elon?" Fair question, and the answer sharpens Ahmad's sentence. None of the three frontier labs runs vLLM or SGLang off the shelf as its main engine. Each built a thin proprietary serving layer whose real content is hand-tuned kernels mapped to specific silicon — the same thing we spent three days doing, at a scale where they can afford to design the silicon too.
| Lab | Engine | Silicon it targets | What that tells you |
|---|---|---|---|
| Anthropic | In-house, no public name. One serving layer over three mutually incompatible kernel targets (Neuron, XLA/Pallas, CUDA). | Majority of Claude traffic on AWS Trainium2 (Project Rainier, >1M chips); Google TPUs (largest external TPU customer, multi-GW Ironwood deal from 2027); Nvidia GPUs as the third leg. | Hardware strategy is portability for supply resilience. The engine's job is scheduling across silicon, not squeezing one chip. |
| OpenAI | Reportedly Teacup (per SemiAnalysis). Kernels in Triton and its lower-level sibling Gluon — OpenAI owns the compiler, so compiler and engine are co-designed. | Nvidia Blackwell at Stargate scale, AMD MI-series, and from late 2026 their own Jalapeño ASIC with Broadcom, built for the decode side of LLM inference. | Full vertical: model → engine → compiler → chip → datacenter. The engine follows the chip because they design both. |
| SpaceX / xAI | Historically SGLang — its creator, Lianmin Zheng, is at xAI, and Grok on grok.com and X was served on it at trillions of tokens a day with heavy expert parallelism. In May 2026 Musk said the Starlink team rewrote pre-training and inference in C/C++, exact-mapped to Colossus GB300s, dropping JAX/PyTorch for utilization. | Nvidia (Colossus: H100 → GB300) and AMD. | The only one of the three rooted in an open engine — and even they concluded the framework abstraction tax on GB300 was worth a rewrite. Treat the "10× faster" claim as a Musk claim, not a benchmark; whether SGLang still fronts production isn't public. |
Read it as a pattern, not a scoreboard. Every one of them converged on the same shape: a small scheduler and serving layer over kernels written for the exact memory system underneath. The open engines — vLLM, SGLang, TensorRT-LLM — are what everyone else runs, including us, and they're excellent; but they have to work everywhere, which is precisely why nobody had a recipe for one GB300 holding a 510 GB model. Our whole campaign lived in the gap between "works everywhere" and "mapped to this box": UVA row reads over NVLink-C2C instead of layer copies, 60 GiB of experts on the Grace side instead of 70, a k-schedule tuned to what an agent actually writes. That is a home-lab-sized version of what the labs do with their own silicon. The xAI GB300 rewrite is the closest cousin to our problem, and it's the one we'll be watching.
Sources for this section: Anthropic's Amazon and Google/Broadcom compute announcements and its Trainium inference lead's re:Invent 2025 talk; OpenAI's Jalapeño announcement and SemiAnalysis's InferenceX analysis (source of the "Teacup" name); Lianmin Zheng's SGLang at xAI talk and Musk's May 2026 statements on the C/C++ Grok stack as reported by Analytics India Magazine. Added September 16, 2026, same day as publication.
Bob asked a fair question when I described this: "how do you even know where to start?" Nobody handed us a manual for one GB300. What we had was a loop, and it ran maybe fifteen times over three days.
| Question we had | Where the answer was |
|---|---|
| Will 510 GB fit in 250 GB at all? | DeepSeek's tech report and vLLM's recipe page break the checkpoint down by component (experts 260 GiB, Engram 183 GiB, everything else ~33 GiB). That table told us Engram had to leave the GPU and some experts had to follow. Arithmetic before any download. |
| Can Engram live in host RAM without wrecking speed? | DeepSeek designed it that way; LMSYS's day-zero post measured it on 4× GB300 (+36% KV, decode unchanged). We took their word and verified locally. |
| Why did SGLang boot without an expert offload option? | Read the source. deepseek_v4.py built its layers without the offloader_kwargs that the V2/V3 model files pass. That's a 23-line patch, not a limitation. |
| Does DSpark actually work on the shipped weights? | 0xSero's 4× RTX PRO 6000 build had a 45-case sweep with acceptance numbers. Someone else had already paid for that experiment. |
| Is our memory math sane? | Tony's 4× DGX Spark build was the parallel effort on a different memory shape; comparing his ladder to ours caught two of our own arithmetic errors. |
| How much is the Grace fetch actually costing us? | catid's two-Station run with every weight in HBM: 141 tok/s. Ours with the drafter off: 89.7. The gap is the fetch tax (~36% of the step). You need someone else's number to measure your own tax. |
| Is there a per-batch-size speculation knob? | Grep the vLLM source for the speculative config. num_speculative_tokens_per_batch_size existed, undocumented on the recipe page. That one field became v13 and v14. |
| Can the GPU keep hot experts in HBM and cold ones in Grace? | Nobody had measured it. The CUDA Programming Guide (§4.19, virtual memory management) said it should be possible in principle. Three spikes over one night said no on this driver: fast or controllable, not both. That result went to the vLLM design RFC because it wasn't anywhere. |
knee.sh, each point run twice, pairs agree within ~1%) was the decision metric. A replay of 24 real agent turns was the workload-shaped smoke test — useful, but it swings ±17 tok/s at temperature 0 because batched MoE plus speculation isn't deterministic. We learned which was which by publishing a win on the wrong one./metrics — draft acceptance per position, tokens per step — and a torch profiler run told us 64% of GPU time was expert streaming. A py-spy stack dump was what distinguished "tuning kernels for 74 minutes" from "hung."--enable-return-routed-experts gave us 850,000 expert selections across 40 layers from real agent traffic. That's where the 98.7%-on-73% skew came from, and where the 3.7× unique-experts-per-verify-window multiplier came from. Both were new numbers for this model.This is the exact command that runs the lane today (release "Sixty-K Agent", v14). The machine-checked version — pinned image digest, model revision, every script that produced a number above, and a ledger of everything that failed — is in our public recipes repo: J-M-Recipes / dgx-station-gb300 / deepseek-v4.1-flash-vllm-uva-dspark.
docker run -d --name dsv41-vllm --gpus all --ipc host --network host \
--ulimit memlock=-1 --cap-add IPC_LOCK \
-v /models/DeepSeek-V4.1-Flash-df42c109f1defefcbfcedbe7d905718a12266e40:/model:ro \
-v /path/to/vllm-cache:/root/.cache/vllm \
vllm/vllm-openai:deepseekv41-flash-0909 \
--model /model --served-model-name dsv41-flash-uva --trust-remote-code \
--tensor-parallel-size 1 \
--offload-backend uva --cpu-offload-gb 60 \
--cpu-offload-params routed_experts.w13_weight routed_experts.w2_weight \
--engram-config '{"cpu_offload": true}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":5,
"num_speculative_tokens_per_batch_size":[[1,4,5],[5,16,1]]}' \
--max-model-len 1048576 --max-num-seqs 16 --max-num-batched-tokens 8192 \
--gpu-memory-utilization 0.97 \
--long-prefill-token-threshold 6144 \
--tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41 --enable-auto-tool-choice \
--port 30006
| Flag | Plain English |
|---|---|
--offload-backend uva --cpu-offload-gb 60 | Put 60 GB of expert weights in Grace RAM and let the GPU read only the rows it needs. 70 works too with more room for concurrent users but ~12% slower; 55 won't boot at 1M context; 40 won't boot at all. |
--cpu-offload-params routed_experts.* | Which weights go to Grace. Note the name — experts.* silently matches nothing and you get an out-of-memory instead of an error. |
--engram-config '{"cpu_offload": true}' | The 189 GB lookup tables go to Grace too. DeepSeek designed them for this. |
--speculative-config … dspark … [[1,4,5],[5,16,1]] | Turn on the built-in guesser. Draft 5 tokens at a time when 1–4 people are talking to it; drop to 1 when 5–16 are, because at high load the drafts cost more expert fetches than they save. |
--max-model-len 1048576 | Full one-million-token memory. Costs 5.5 GB of conversation cache vs. a smaller setting and nothing in speed. No reason to run it small. |
--gpu-memory-utilization 0.97 | Let vLLM use 97% of GPU memory. This is what lets the 60 GB setting fit; at 0.94 it doesn't. |
--long-prefill-token-threshold 6144 | Don't let one person's giant document block everyone else. A short question behind a 480K-token upload went from 18 s to 1 s wait; the giant document pays 16%. |
--tool-call-parser / --reasoning-parser deepseek_v41 | Without these, when the AI agent wants to run a command it describes running it instead of actually doing it. |
reasoning_effort: medium with an HTTP 400 — it accepts low, high, xhigh, max, or a number 1–100. And with nothing set, thinking is on at effort 50, so a request with a small token budget spends it all on the reasoning trace and comes back empty. That looks like a broken model and isn't.
Going in, I'd have guessed it meant clever math — new kernels, new compression, squeezing the model. It mostly wasn't. The model is used exactly as DeepSeek shipped it; not one number in it was changed. The 27× came from:
Milo did most of the hands-on work — launching, profiling, reading kernel logs at 2 AM, writing the patches, posting findings upstream to the vLLM project. My job was partly guardrails (agents, not benchmarks; no pruning the model; measure it against a control or don't count it) and partly steering. I pushed the strategy at several points, and it's worth saying which pushes paid:
| What James pushed | What happened |
|---|---|
| "Concentrate on using HBM." Stop treating Grace RAM as free; every gigabyte of experts we can pull back into fast memory should be worth something. | Paid — the biggest single win. Milo's first read was that offload size was a fit constraint, not a speed lever; the k=0 floor test suggested the fetch tax was modest. Pushing on it anyway produced the off70 → off60 experiment: +12–15% across every text class, twice what the bytes model predicted. That became v12 and every later version sits on it. |
| "Put the hot experts in HBM, cold ones in Grace." The routing skew (98.7% of hits on 73% of experts) was too strong to ignore. | Right idea, closed by hardware. Three spikes in one night — VMM row remapping, EGM, managed memory — all mechanically worked and all read Grace at 90–155 GB/s instead of 340. The hypothesis was correct; this driver can't cash it. The measurements went to the vLLM RFC so nobody else designs around it blind. |
| "Look at speculative fetching." If the drafter guesses the next tokens, can we prefetch the experts those tokens will route to? | Measured, and it doesn't exist yet. On GLM-5.3 the same week: draft-token routing predicted next-step expert selection at 3.16% vs 3.12% by chance — the drafter knows the token, not the expert. Also found: a k=5 verify window touches 3.7× the unique experts of one token, which is why speculation costs bytes here. Both were new numbers. Both said no. That's still a result. |
| "No pruning. All 384 stay." | Held. Closed off the REAP path for good and kept every quality number in this post comparable to the shipped model. |
Two of four pushes changed the recipe; two closed doors with numbers instead of opinions. That's a decent hit rate for direction-setting on a problem nobody had a map for, and it's the part of the job that can't be delegated to the agent: knowing which idea is worth a night of the machine. Then deciding what to keep, and knowing when to stop. The lane runs on these flags today. It's wired into Hermes as an option Milo can choose. It's not the default — routing is a decision, not a benchmark result — but the reason it couldn't be is gone.
Bob: if you read this far, the thing to take away is that a model designed for a rack of eight GPUs runs at usable speed on one box on a desk, and the reason is a handful of well-chosen decisions about memory, not magic. That's the part that's learnable.
A note on who this is for. James asked for this post for Bob, but he said afterward it was for him too. He was in every one of these decisions — set the bars, killed the pruning idea, called the stop — and still wanted the version that steps back from the 2 AM kernel logs and says what actually happened and why. That's a real use of a write-up like this: the person doing the work and the person directing it both need the high-level story, and it's usually only written for the person who wasn't there. So: written for Bob, and for James.
DeepSeek for the model and for designing Engram to live in host memory. The vLLM team for the day-zero image and the UVA offload backend that made the second number possible. SGLang / LMSYS for day-zero support and the Engram host-table path — the first boot was theirs. 0xSero (4× RTX PRO 6000 build) and Tony / tonyd2wild (4× DGX Spark build) for the community data points we checked our memory math against. catid for the all-HBM two-Station number that let us measure the fetch tax.