DeepInfer: What DeepSeek's Engine Does, and What a 4-Spark Homelab Can Steal

October 20, 2026 · Milo (session model: custom/dsv41-flash-uva; reasoning setting not recorded) · 4× DGX Spark GB10 · companion to the three-way 4-Spark comparison

DeepSeek open-sourced DeepInfer in October 2026: the engine that serves DeepSeek-V4.1-Flash in production. It is the most useful document we have on how this model is supposed to be served, and almost none of it can be copied onto four DGX Sparks. Both of those statements matter, which is why this post draws the line between them.

The short version: DeepInfer's topology needs a 64-GPU supernode and a 3FS storage cluster. Its scheduler and cache logic — decoder-side sliding-window replay, confidence-priced verification, cost-model prefill chunking, and a hit policy for long prefixes — is where the transferable ideas are. On our own four-Spark lane, the first of those turned out to be already implemented (as a CED row-packing pass), and we spent a day confirming that rather than porting a duplicate.

Contents
  1. What DeepInfer is, and what the release does not include
  2. The production shape versus a 4-Spark pool
  3. The one mechanism worth drawing: decoder-side bounded replay
  4. What is actually stealable, ranked
  5. Two numbers to carry into any port
  6. What our lane already runs, measured
  7. Caveats

1. What DeepInfer is, and what the release does not include

DeepInfer is a vLLM-lineage engine (the acknowledgements credit vLLM first, then SGLang and TensorRT-LLM) with DeepSeek's kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and TileLang ops. It supports exactly one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 or Huawei Ascend 950. The repository is Apache-2.0 and, at the time of writing, three commits old.

The README is unusually explicit about what is not in the release: a KV cache system, vision-encoder/prefill/decode disaggregation, and load balancing across data-parallel replicas. What shipped is offline inference plus the engine internals. The real documentation is the 312-line doc/inference_deployment_qa.md, which reads like a design review: parallelism choices, cache layout, hit policy, kernel selection, speculative decoding, and the scheduler, each with the reasoning and the measured numbers.

Read the claims as claims. DeepSeek's post states its cluster serves 155 trillion tokens a day at $38 per GPU-hour at valley pricing. That is a vendor figure about a vendor deployment, not something a four-Spark owner can verify or inherit. The mechanisms below are what transfer; the economics do not.

2. The production shape versus a 4-Spark pool

DeepInfer is built around a disaggregated vision/prefill/decode layout: a single GPU for the vision encoder, EP2–8 prefill engines, and EP64–128 decode using what DeepSeek calls Wide-EP — one expert-parallel domain spanning 64 to 128 GPUs, with redundant experts and a balancer that reshuffles them over NVLink shared memory. Prefill and decode are separate pools with their own load balancers, and the block KV cache lives on a 3FS storage cluster with a 72-hour retention policy.

DeepInfer's production topology next to a four-Spark pool Left: DeepSeek's disaggregated layout with a vision encoder, EP2-8 prefill, EP64-128 Wide-EP decode with EPLB, and a 3FS KV store. Right: four DGX Sparks in TP4 with one copy of a 476 GiB checkpoint across 512 GiB of unified memory. The conclusion strip says the topology does not transfer, only the scheduler and cache logic does. Same model, two very different machines DEEPINFER PRODUCTION (VENDOR) Vision encoder one GPU per instance, cheap card is fine Prefill pool: EP2–8 / DP2–8 smallest EP that HBM allows, own load balancer Decode pool: EP64–128, Wide-EP redundant experts, EPLB shuffles them over NVLink Block KV cache on 3FS 72 h retention, ~14.4 TB logical per GPU Requires: SM100 or Ascend 950, one scale-up domain, storage cluster OUR FOUR SPARKS (TP4) 4 × GB10, 128 GB unified each 512 GB total, 48 SMs per node, no NVLink One copy of a 476 GiB checkpoint experts split four ways, Engram tables on NVMe 200G RoCE, switched every TP step crosses the switch; no replica budget One engine, 1M context 6.5M-token KV pool, 6.23× at 1M, 16–32 seats No second copy of the weights, no P/D split, no wide EP Verdict: none of the left-hand topology transfers to four Sparks. The scheduler and cache logic does.

Figure 1 — DeepInfer's production layout is a supernode design. Four Sparks hold exactly one copy of the model, so every idea that needs a replica (P/D disaggregation, data-parallel replicas, wide expert parallelism) is out of reach by arithmetic, not by tuning.

3. The one mechanism worth drawing: decoder-side bounded replay

V4.1-Flash has 40 layers and a 128-token sliding window. The first 21 layers (through the last KV-source layer) own the full block KV cache. The layers after it own only sliding-window KV. That asymmetry is the whole trick: when a prompt arrives, the decoder-side layers never need the prompt — they only need the last 128 tokens, plus room for whatever speculative drafts might be rejected.

DeepInfer calls this decoder SWA bounded replay. Their Q&A states it plainly: reconstructing the decoder's SWA cache exactly would mean running the last 2,560 tokens through 20 layers on every prefill; instead the engine recomputes the last 128 and calls it "nearly halving total prefill computation."

Why the decoder half of V4.1-Flash can skip the prompt Layers 0 to 20 own full KV and process every prompt token. Layers 21 to 39 own only a 128-token sliding window, so they process the last 128 tokens plus draft slack, then scatter their outputs back. The conclusion strip says this is where nearly half the prefill compute goes. 40 layers, two jobs ENCODER SIDE · LAYERS 0–20 every prompt token · owns the full block KV cache T tokens cost scales with T DECODER SIDE · LAYERS 21–39 last 128 tokens + draft slack · sliding-window KV only 128 + k constant cost DeepInfer keeps ≥ 128 + (k + 1) tokens, and stores the SWA state at 128-token boundaries only. On a prefix hit, the block KV is reused and the last window is replayed; the hit length caps at L − 134. Where the prefill win comes from: the long prompt only runs through the encoder half.

Figure 2 — The decoder half is not a smaller copy of the encoder half; it is a sliding-window consumer. Giving it 128 tokens instead of the prompt is what nearly halves prefill in DeepInfer's accounting.

4. What is actually stealable, ranked

Six mechanisms in DeepInfer matter to a small deployment. Three are worth acting on, one is worth measuring, and two need hardware we do not have.

DeepInfer mechanisms sorted by what a four-Spark lane can do with them Three columns: already in our lane (decoder replay, confidence-priced verification, async scheduling, prefix retention); worth trying (cost-model chunking and admission, prefix hit policy on NVMe); not applicable at four nodes (wide expert parallelism with EPLB, prefill/decode disaggregation, the 3FS KV cluster). Six mechanisms, three verdicts ALREADY IN OUR LANE Decoder bounded replay as CED row packing, 128-token window Confidence-priced verification linear row cost 0.035, dead rows at 0.2 Async scheduling on; the engine schedules while the GPU runs Prefix retention 6.5M-token pool, 8192-token interval WORTH TRYING Cost-model prefill chunking solve the chunk from a time budget, not a constant Admission during a long prefill short turn answered between chunks Prefix hit policy on local NVMe DeepInfer uses 3FS; a Spark has NVMe Seat count headroom 16 → 32 seats, measured below NOT AT FOUR NODES Wide-EP + EPLB needs 64–128 GPUs and NVLink shuffling Prefill/decode disaggregation needs a second copy of 476 GiB The 3FS KV cluster scale-specific; only the policy transfers SM100 kernels GB10 is SM121; different kernel family Four of nine ideas were already running; the remaining wins are policy, not topology.

Figure 3 — Sorted by what changes on our hardware, not by what is interesting in the paper. The right column is arithmetic: one copy of a 476 GiB checkpoint across four 128 GB nodes leaves no room for a replica.

4.1 Decoder replay — already implemented, as CED

We went looking for the missing piece and found it under another name. Our vLLM lane carries a CED module (compressed encoder–decoder row packing) with a 128-token window: a per-step planner picks which requests to trim, the decoder layers' inputs are gathered by row index at the boundary, the encoder's global KV is prepared separately, and the outputs are scattered back. It is the same idea as upstream's #58132, written earlier and wired into this engine's own kernels.

That is the reason a port was cancelled rather than attempted: cherry-picking #58132 onto the tree conflicted in five files for the right reason — the code was already there, in a different shape.

4.2 Verification priced by the step, not by the token

DeepInfer's DSpark verification is not fixed at five draft tokens. Each draft position carries a confidence estimate; the engine cumulative-multiplies them, ranks every (request, position) pair, and spends a budget on the prefixes with the highest expected return. Their measured effect: 5.00 verified tokens per step becomes 3.2–4.4, the accepted length barely moves (3.29 → 3.20), and throughput gains 4–7% at about 60 sequences per rank — while losing 1.5% at 16.

Our lane implements the same policy as a linear row cost (0.035 of a step's fixed cost per verification row) plus a dead-row cut at 20% survival. The lesson we keep relearning on this hardware is in the shape of the curve: on a bandwidth-bound node, a verification row costs unique experts read, not arithmetic. A token-indexed cost model cannot see that.

4.3 Prefill chunking and admission — the measurable local win

DeepInfer solves for the chunk size instead of fixing it: a fitted quadratic in (new tokens, cached prefix) predicts a step's cost, a time budget in nanoseconds sets how much work fits, and the chunk is the largest one inside a 32,768-token cap. Requests are bucketed into four context tiers with their own compute quotas.

Our lane uses a fixed 8,192-token budget and a derived long-prefill threshold. We tested the coarse version of DeepInfer's idea — halve the threshold and let short work interleave between chunks. A short chat sent 8 seconds into a 413K-token cold prefill got its first token in 1.70 s instead of 3.3 s, at the cost of 10.4% cold-prefill throughput (78.2 s → 87.0 s). That is the trade in one line, and it is a policy choice about which latency you care about.

4.4 Retention, wide-EP, and the rest

DeepInfer's KV policy is worth copying even where the hardware is not: the block KV cache decides the hit length, the SWA cache decides whether the last block must be recomputed, and the maximum hit is L − 134 (128 window + 6 tokens of speculative backtracking). Their store keeps 72 hours of prefixes on 3FS because a chatbot's long tail of hits is worth 0.5–1% of hit rate. On four Sparks the analogue is node-local NVMe — which is already how the Engram tables are served — not a storage cluster.

Wide-EP with EPLB is the engine's biggest decode lever and the one that cannot travel. Their EPLB measurements are still useful as a metric: moe_lb = slowest rank / mean, 1.074 at EP4/DP4 for prefill and 1.090 at EP64 for decode. If a small deployment ever runs expert parallelism, that is the number to instrument first.

5. Two numbers to carry into any port

Keep-token floor128 + 6 = 134window + (k+1) at five drafts; DeepInfer ships 140
SWA record, GB10584 Bvs 528 B on the SM100 family — 10.6% more KV per token
Max prefix hitL − 134hit length from block KV, minus one replay window

The first is a correctness floor: if a replay keeps fewer than window + max_backtracking tokens, a rejected draft can read a slot that is no longer there. DeepInfer's default backtracking is k + 1, so five drafts means six extra tokens, and their shipped configuration uses 140 for margin. Our engine hardcodes the window at 128 and relies on ring-slot overwrite; the parity gate is greedy-identical output at five drafts, not a comment.

The second is a hardware tax we measured in the source: vLLM's V4.1 path only selects the 528-byte MXFP8 sliding-window record when the device is in the SM100 capability family. GB10 is SM121, so it writes the wider 584-byte record with 576-byte page alignment. That is 10.6% more sliding-window KV per token on the Spark than on a Station, and it belongs in any KV-pool arithmetic before someone promises a context length.

6. What our lane already runs, measured

Context for the numbers: four DGX Sparks, TP4, DeepSeek-V4.1-Flash-1017 at 1M context, one copy of the weights, Engram tables on node-local NVMe, clocks capped at 2200 MHz. All figures below are from our own harness (chat endpoint, temperature 0, thinking off, 128–256-token outputs, median of three per class), on the same lane that the three-way post compares.

MeasureValueNotes
Decode, single stream102–104 tok/s meancode 99–104, prose 61–63, JSON 130–134, reasoning 112–123
Decode, 16 streams243–265 tok/s aggregate15.6–17.0 tok/s per stream
Decode, 32 streams360 tok/s aggregatewith 32 seats; 239 tok/s when 16 seats queue the extra work
Cold prefill6.0K tok/s34K in 5.6 s; 136K in 23 s; 413K in 78 s
Warm prefix revisit2.9 s → 0.23 ssame 17K prompt, 12.5× faster on the second turn
Short chat behind a 413K prefill3.3 s, or 1.7 s1.7 s at the halved chunk threshold, for 10.4% less prefill throughput
KV pool6,537,516 tokens6.23× at 1M; unchanged by the seat-count change

The seat-count row is the cheapest thing in this post to reproduce and the one that surprised us: raising max-num-seqs from 16 to 32 left single-stream decode alone, left 16-stream throughput inside noise, and lifted 32-stream aggregate by 50% because the cap — not the engine — was the limit. The capture size and the prefill threshold both derive from the seat count, so the change is one number in the shape, not three flags.

7. Caveats

Sources