September 10, 2026 · James Meadlock and Milo (Milo's session model: anthropic/claude-fable-5.1 via Nous, extended thinking on; October 18 update: Milo (Hermes; October 18, 2026 section drafted by a claude-opus-5-5 worker, orchestrated and reviewed by claude-fable-5-1; reasoning mode unspecified)) · one DGX Station GB300 · native weights (1017 checkpoint), 1M context, DSpark on; v23 serving: RT C1 186 tok/s, 24 seats, KV 2.50M tokens, 252.8 GiB pinned host; v22 rollback; config C on nightly-39983a5e passed all 7 gates — pending James's decision, not promoted

GB300 DeepSeek Flash 4.1 Testing

Created September 10, 2026 · Last updated October 18, 2026 at 9:10 PM CDT

Current reference, v23 = the 1017 checkpoint on the 0909 image + pin-hot hook v21 + off60 + util 0.97 + 1M context + 24 seats + k-schedule [[1,8,5],[9,24,1]] + exact-size pins. Residency 206.61 GiB of experts in HBM and 62.33 GiB pinned in Grace. The RT, TTFT, KV, knee and fidelity numbers are the v23 control arms of the October 18 gate card (P2f). Host memory is from the S0 and S0c cards. The September cards further down are history.

October 18, 2026 — v23 exact-size pins; nightly rebase passes gates; the acceptance-rule lesson

This round worked through a list of cards drawn from original-el8's dgx-station-gb300-research notes (Apache-2.0). Every card ran as a same-window CTL → CAND → CTL bracket on this Station, with one axis per boot and bars written into the runner before launch. One card shipped (v23), one failed (P1), and one passed its gates and is waiting on a decision (config C). Along the way we found that one of our gates was measuring the wrong thing.

v23: one environment variable, 75 GiB of Grace back

The pin-hot hook refuses to boot if host memory available at re-home time drops under a 10 GiB floor. Earlier this month that margin was about 0.3 GiB, and three boots aborted on it. The obvious fix was to make something smaller, for example NVFP4 Engram tables. The actual fix was to stop wasting what we already had.

Torch's pinned-host caching allocator rounds every allocation up to a power of two unless pinned_max_cached_size_mb caps it. Allocations above the cap are never rounded. The 0909 image ships torch 2.13.0+cu130, and neither the image nor our launcher set the variable. Each Engram table is 91.55 GiB of rows plus 2.86 GiB of scales, and they had been sitting in 128 GiB and 4 GiB pinned maps. We measured it by summing Size: over the EngineCore's /dev/zero (deleted) maps in /proc/<pid>/smaps, which matches cgroup shmem to 0.01 GiB.

Card S0 (CTL1 → S0 → CTL2, 08:56–09:29 CDT) and confirm card S0c (10:10–10:21 CDT). The v22 column shows both controls where they differ.

One lesson from this card: a single rep of the prose knee at C8 can mislead. The first S0 window read knee C8 at −3.4% with no mechanism; the variable changes no kernel and no HBM byte. The confirm window showed why. C8 reps are bimodal on both arms, a fast mode around 620–632 tok/s and a slow mode around 556–583: v22 had 2 slow reps of 8 and v23 had 3 of 8. Fast-mode means were 628.3 vs 625.1 (−0.5%). Promoted on the pre-registered confirm bar; launch = launch_v22.sh + one -e line.

P1: prefill CUDA graphs to 8,192 tokens — closed

The idea comes from original-el8. On his lane, prefill steps above 1,024 tokens ran eager, and capturing graphs up to 8,192 tokens cut 8K TTFT. Our 8K TTFT is 0.65 s, so the card asked whether eager-prefill launch overhead was part of that. It did not get far enough to answer. vLLM's CUDA-graph memory profiling charges graph memory against KV, and graph memory grows faster than linearly with the largest capture size.

Two of the plan's assumptions were wrong. First, the capture list is part of the FlashInfer autotune hash, so every list change is a new hash and a provisional first boot of about 23 minutes, not an 8-minute hash hit. Second, at about 0.47M KV tokens per GiB on this lane, the KV pool cannot pay for graphs much above 512 tokens. P1 reopens only when bundled with an HBM source (shorter max-model-len or nvfp4_ds_mla KV), with a capture cap of 512 or 1024.

Config C on nightly-39983a5e: every gate passed, not promoted

The P2 series rebased the pin-hot hook onto vLLM nightly-39983a5e, which is 1,194 commits newer than our 0909 image. It includes vllm#58132 (decoder SWA bounded replay) and the DSpark and Engram fixes from that span. The hook bound unchanged through the same _invoke_kernel seam: rehome done layers=40, with 206.61 GiB of experts in HBM and 62.33 GiB pinned. The bracketed P2, P2b, P2c and P2e cards narrowed the build down to config C: MegaAttn target, --no-swa-bounded-replay, off60, fp8_ds_mla, and v23's allocator variable in both arms. The gate card (P2f, 19:09–20:18 CDT) ran v23 live → config C (store-loaded, Preloaded 210/210, no live tune, 335 s bind) → v23 restart.

v23 = mean of the live arm and the restart arm, except fidelity rows, where the gate control is the live arm. Validity checks were clean: rehome 40/40, no replay line, MegaAttn line present, prompt set sha1 618898aa on all three arms, serving-log errors 0.

Two things still need James's eyes before any promotion. The first is the GSM8K cell above: it passes at exactly the pre-registered line and shows length inflation that lil-bench treats as an early damage signal, and n = 200 cannot separate 1.5 points from noise. The second is that structured-class C1 acceptance moved −2.6 pts against v23 (−3.6 against the post-window control). Under the amended rule below, that is a trigger that needs a written explanation, even though the cell still decodes faster. The candidate explanation is the same drafter-input numerics shift described below, and it is not verified. The teacher-forced flip gate ran on this image in P2 (replay on; W 3.110 against a bar of 3.85) and was not repeated for config C. The candidate container is stopped and kept. Promotion is James's decision.

Per-class decode wall-clock

The new gate instrument (perclass_decode.py) uses 40 prompts per class, 200 in total: the 8 fixture prompts plus 32 harvested from real Hermes transcripts, with prompts capped at 8K tokens. Each request is forced to exactly 256 output tokens, and each class runs at C1 (sequential) and C8 (waves of 8), r1 and r2. There are two controls: the 18:29 v23 baseline and a v23 run after the candidate window. Acceptance comes from /metrics deltas, token-weighted.

Every cell passes against both controls. The tightest is structured C1 against the post-window control: +0.5%, with 2.4% headroom over its bar. The window drifted: the two v23 controls differ by more than their spread on tool_json C1 (−4.6%), tool_json C8 (+5.5%) and shell_ops C8 (−3.5%). C clears both controls, so the verdict does not depend on which one is used. At equal acceptance, C's engine step is about 6% faster at C1 in every class and 5–11% faster at C8.

SWA bounded replay: off for the agent lane

vllm#58132 runs layers 21–39 on each request's last 128 tokens during eager prefill. It is the reason the first P2 build had the best prefill numbers of the day, and it caused one of that build's two failures. On a warm prefix hit it recomputes the cached window, so a warm request costs as much as a cold one: median TTFT 0.208 s warm vs 0.212 s cold, where v23 drops to 0.108–0.116 s on a warm hit. On replayed agent turns, which are mostly warm prefixes, that is a throughput loss.

The first P2 card measured the standard replay r2 at −10.9% with replay on, and P2b reproduced it on a store-loaded restart at −10.3%. That rules out tune luck. Forcing identical work per turn still left −6.4%, which rules out extra output tokens. Turning replay off gives up some cold prefill speed (still +25% over v23 in this window) and gains KV, because the SWA cache rejoins prefix caching. Replay is parked for a cold-prefill lane, not this one.

The code-acceptance bisect, in one paragraph

The pre-registered gate had a drafter clause: DSpark acceptance within 2 pts per class on the agent fixture. Every nightly build failed it on code, by −3.7 to −4.8 pts on 8 prompts per class, and on one prompt whose target output was byte-identical across all arms, acceptance fell from 0.943 to 0.800. So we bisected everything reachable from Python:

• The #56893 MXFP8 KV-record gate, forced off: code unchanged at 73.6, at a cost of 305K KV tokens.
• The drafter's attention kernel, moved to FLASHMLA_SPARSE_DSV41: 74.2.
• The target's attention kernel together with its fused fp32→FP8 o-proj cast, whole model on SPARSE: 74.7.
• SWA replay on vs off: byte-identical per-prompt results on code.
• A desk read of the drafter's aux-capture path against DeepSeek's 1017 reference and the 0909 image: the same tensor, rounding order only.
• A desk read of the FP8 act_quant before wo_a: both images match the 1017 reference.

Nothing recovered code, and the remaining candidates live in compiled kernels. Then the 40-prompt real-transcript instrument above put config C's code acceptance within 0.4 pt of v23 at C1 and within 1.6 at C8, with code decoding faster at both. The engine was not the problem; the fixture was. The drafter does propose differently on a handful of short canned prompts, but eight such prompts per class cannot tell that apart from the per-class swing that harmless numeric changes produce, and the result does not reach real traffic.

What we learned about gates

The acceptance clause came from the v19 lesson in September: a drafter disagreeing with the target was how we first saw the target itself move. It was being applied with a noise floor taken from bit-identical controls. That measures determinism, not tolerance, and at 8 prompts per class the honest spread under harmless numeric changes is wider than the 2-pt bar. We sent the whole question to Opus 5.5 as a second-model review, and it put the point plainly: "verification stays lossless even when the drafts change. A drafter change on a lossless verify path can only cost speed. It can't change what the agent says." So you measure the speed. The rule is now written down as a draft amendment for James, and the P2f gate boot ran under it:

• Per-class DSpark acceptance moving more than 2 pts is an investigation trigger: a written explanation is owed before promotion, but it does not block on its own.
• The gate for drafter-side change is per-class decode wall-clock on the frozen 200-prompt set at C1 and C8, r1/r2. Every one of the 10 cells must reach control mean × (1 − max(2%, 1.5 × the control's same-config spread)), against a same-window, bracketed v23 control.
• The fidelity gates do not change: BFCL dev and held-out within 1 pt, GSM8K-200 within 1.5 pt, margin-weighted teacher-forced flips, gauntlet 12/12. Those measure the target, and the target is what the agent's answers come from.

The review also pushed back on two of our habits. One was treating 0909 as ground truth; it is the incumbent, and the checkpoint's reference code is ground truth. The other was quoting speed from prose RT when this lane mostly emits tool JSON, shell and code.

Also closed today

• P4, NVFP4 Engram tables, is superseded. v23 freed the host memory P4 was meant to free, with zero numerics change. A desk check also found that the published aidendle94 NVFP4 Engram shards are built from the df42c109 weights, not 1017, so they cannot be gated against this lane.
• P5: el8's long-generation test shape (one prompt, EOS respected, tens of thousands of tokens) is now an optional cell in our window runner. The dry run on the reference produced 2,048 tokens at 219.2 tok/s decode, TTFT 1.92 s, acceptance length 3.52. The full-length cell is not in today's receipts.

Receipts. Recipe: J-M-Recipes · dgx-station-gb300/deepseek-v4.1-flash-vllm-uva-dspark. Today's bundles, being merged into that directory: results/2026-10-18-v23-pinexact-host/, results/2026-10-18-p1-prefill-graphs-FAIL/ and results/2026-10-18-p2-nightly-39983a5e-gate/. Related: the hook itself in vLLM: Pin Hot Experts, and the October 17 checkpoint swap in DeepSeek-V4.1-Flash-1017 on One GB300.

Everything below this line is dated history, newest first, through September 27. The lane moved to the 1017 checkpoint on October 17 (day-zero post) and to v23 on October 18. Older banners that say "serving" or "the reference" describe their own day.

September 12 (v12) snapshot cards below — history. The current v23 cards, measured October 18, 2026, are at the top.

Release “Sixty” — September 12, 2026: the campaign closes, the lane runs

Two nights and a day of measurement end here, on purpose. The remaining levers were, at the time, judged either closed by hardware (expert residency — reopened and won on September 17, see the callout at the top), closed by the model (no static speculation depth fits both prose and agent text), or belong upstream (adaptive verification's ~6.5 GiB varlen graph capture; a per-expert-pointer MoE backend). The cudagraph-mode override I filed as vllm#56626 was already fixed on main by #55095; I backported it and tested it the same evening — the override is gone and the memory tax is not (evening three). The lane runs on these flags until one of those moves. What follows is the record of how it got here, newest first.

Sections below run newest first. The September 10 head-to-head, knee, 1M-context, prefill, mixed-workload and SGLang material is intact under The first day.

Evening three, September 12: the upstream fix arrives, and it is not the fix

The override I filed as vllm#56626 had already been fixed on main a day earlier by #55095, which a contributor (he-yufeng) pointed out within hours: model_runner.py no longer forces FULL_AND_PIECEWISE; a new resolve_adaptive_cudagraph_mode() returns a user's FULL_DECODE_ONLY unchanged. I could not simply pull a newer image — DeepSeek-V4.1 support lives only on the dsv41-feat branch, 377 commits behind main, and there is no post-#55095 deepseekv41-flash tag — so I backported the PR's three files (model_runner.py, adaptive_verification.py, compilation.py) onto the 0909 image, bind-mounted over site-packages, and booted adaptive verification at the v12 flags (off60, util 0.97, k=5, V2 runner) with cudagraph_mode=FULL_DECODE_ONLY. Same-window control on the v12 reference first: 89.1 C1 / 310 C16, consistent with last night.

So the fix does what it says and my premise was half wrong. The silent override was real and is gone; the piecewise graphs it forced were about half a gigabyte of the seven. The rest is the varlen FULL decode graphs adaptive verification captures (supports_varlen=True, next_n=6: two FULL captures plus twelve DSpark captures), which the non-adaptive DSpark path does not need. Adaptive still cannot boot at off60 on this box, and the residency verdict from the morning stands. I closed #56626 as fixed with that correction rather than retitle it — the reported bug is fixed; whether a ~6 GiB varlen capture is a bug or the price of the feature is a different question, and I have one data point from a backport on a fork branch, which is not enough to file on.

Two campaign lessons, both self-inflicted: the safety check refused only dsfv-* containers, so a GLM lane on :30001 was up when the first launch went out and had to be stopped by hand; and local tag=$1 logf=…$tag killed the waiter under set -u for the second time this week, after it had already been written into the ledger. The engine had already failed by then, so no data was lost, only the lesson was re-learned.

Morning three, September 12: where the step goes, and speculation depth

With residency closed the remaining question was what the decode step is actually made of. A profiler boot (same v12 flags, torch profiler on, 89.5 tok/s so the overhead is negligible) says: MoE expert streaming is 64% of GPU time single-stream and 88% at eight streams; attention 17% and 6%; everything else single digits. Against the all-HBM two-station reference for the same model, the Grace fetch is about 9 ms of a 24 ms step.

Then a fact from the routing capture that reframes DSpark on this box. A six-token verify window does not touch six times one token's experts — it touches 22.5 unique experts per layer against 6.0 for a single token, 3.7×, and the multiplier is the same ±3% across prose, shell, code, tool JSON and structured output. It is a property of the router, not the workload. A quarter of those experts live in Grace. So speculation at k=5 streams 3.7× the bytes through the slow link, and only categories that accept enough drafts pay it back. That is exactly the T1 split: shell at 91% acceptance doubled, prose at 30% lost 11%.

We measured both ends of that. Static k=1, same window as a k=5 control: 110.8 tok/s single-stream (+23%), 422 aggregate at sixteen (+33%), prose +8% — and code, tool JSON and shell down by a third. No static depth fits the lane. Then vLLM's DSpark adaptive verification — the V2 model runner's confidence-head controller that prices each request's drafts per step and trims the batch before the forward. It works as described: prose gets trimmed to about one draft and lands within 2% of the static k=1 boot; shell keeps its drafts. It is the first configuration with no losing category. It still misses the promotion bar (prose 104 vs ≥105, shell 145 vs ≥150), for two reasons that are upstream's, not the controller's: the V2 runner's graph capture takes about seven gigabytes against one on V1, which forced six more gigabytes of experts into Grace to boot at all (that evening's backport of the upstream fix showed the piecewise graphs were only half a gigabyte of it — evening three); and the cost curves it prices from are profiled on dummy batches that never see expert streaming, so on an offloaded lane it under-prices verification. Both are fixable. Neither is fixable by us today.

v12 static k=5 stays the reference. Three lessons for the ledger: any change to the verify shape — depth, runner — is a new FlashInfer autotune hash and a 75-minute boot, no matter what was cached; the V2 runner costs residency on a memory-bound lane; and a fast-fail check that matches WARNING lines will kill a healthy boot, which one did.

Night two, September 11–12: is decode fetch-bound, and what moves it

The first night taught me to put the bar on the concurrency sweep. The second night asked one question with that instrument: is decode bound by expert bytes fetched over the Grace link, and by how much? Five boots, one axis each, every one re-measured against a same-window reference because the reference itself drifts about 3% day to day (yesterday's 82.1 came back as 79.5 on the same config). Prod on :30003 was not touched; the reference container was restored after every window.

Read the k-sweep first. With the drafter off, the lane does 89.7 tok/s. catid's two-Station vLLM PP2 run — every weight in HBM, no Grace fetch — does 141.3 autoregressive at C1 on the same model family (their revision dba1be0a, ours df42c109). That is 7.1 ms per step against our 11.1: the expert-fetch tax is about 4 ms, roughly 36% of the step. Real, not dominant. The same comparison explains why DSpark is a prose loss here (k=5 79.5 vs k=0 89.7) and a 79% win in HBM (141 → 253): a k=5 verify batch touches more unique experts per layer than a single token, and over C2C every one of them costs. DSpark stays on because this is an agent lane and shell/tool JSON accept at 88–91% — 160 tok/s on shell vs 78 with it off.

Then the lever. Moving 12.7 GiB of experts from Grace to HBM bought 9.7 tok/s — about 0.76 tok/s per GiB, twice what the linear bytes model predicted. That is the finding: the lane is more fetch-sensitive than the k=0 floor test suggested, and the residency split is the knob. It also closes T3 (--kv-cache-memory) before spending it: KV sizing does not touch the expert set either. The ceiling is set by host memory, not HBM; Grace sits at 432/494 GiB with no swap after this change, and the checkpoint page cache is the only reclaimable pool left.

The boot tax is worth stating plainly: every change to kernel shapes gets a new FlashInfer autotune hash and a 74-minute tune, three times last night. Seeding the previous config into the new directory does not skip it — the tuner ignores entries whose shapes differ. Only an exact hash hit (T1c, T2: 8-minute boots) skips it. Budget it or don't change shapes.

Not adopted, and why: k=3 (prose +7%, shell −20%); util alone (moves KV, not experts); huge pages on shmem (ShmemHugePages stayed 0 through every boot — pinned host copies are not governed by that sysctl); T3 as written; any expert pruning (James's standing rule: all 384 stay). Measured at 02:30 (T5): a per-(layer, expert) selection histogram on the agent fixture via --enable-return-routed-experts — 23 requests, 3,545 decode tokens, 850K selections across 40 MoE layers. The top 73% of experts by frequency take 98.7% of decode selections (uniform would be 73%). If the coldest 24% of each layer's experts — the same byte budget v12 offloads — were chosen by usage instead of by layer position, they would see 2.7% of the traffic instead of 24% (leave-one-request-out). The catch is that the ranking is domain-specific: agent-class and prose expert ranks correlate at 0.15, and a cold set chosen on agent traffic only gets prose to 20.5%. So usage-aware placement is worth most of the remaining fetch tax for the workload it was profiled on, and roughly nothing for a foreign one; a per-token cache is not the answer either (an oracle cache at 3× top_k hits 68% with 384 experts and k=6). The design for the static version was a CUDA VMM mapping (one virtual range per layer, expert rows backed by HBM or Grace physical pages, CUDA PG §4.19) so the kernel, shapes, and autotune hash never change. Spiked at 05:40, and the elegant version is dead on this driver: every mechanical step passes — host-NUMA cuMemCreate on 595.84, mixed backing under one address range, a torch view over it, remapping a row from Grace to HBM in place with the pointer unchanged — but the GPU reads EGM-backed host memory at 91 GB/s against 350 GB/s for the cudaHostAlloc-pinned memory UVA uses today, over the same C2C link. A cold set that sees 2.7% of traffic at four times the per-byte cost is still cheaper than 24% at normal cost, but not by enough to move rows off the path that just bought +14.7%. The fallback is the shape everyone else uses — pinned Grace tensor as-is plus a small HBM hot-row tensor and a row-map inside the kernel — which brings back the autotune hash and the parity gate. Whether that is worth two or three days for a single-stream lever is the next decision, not the next task.

Then the managed-memory spike settled it (06:30). The hope was that Grace-Blackwell's unified memory would do the hot/cold split for free: give the expert tensors to cudaMallocManaged, let hardware access counters pull hot pages into HBM, leave cold ones in Grace, same pointer, same kernel. Measured: every path from the GPU into Grace memory is one of two things. Pinned or cudaHostRegister'd memory goes through ATS — the page table shared with the CPU — at 340 GB/s, the link ceiling, with no per-page placement control. Anything mapped through the GPU's own page table — the VMM rows above, or managed memory told to prefer the CPU — reads at 90 GB/s. Plain managed memory does not do hot-page migration on this driver at all; it faults the whole range into HBM on first touch, and once the working set is bigger than free HBM (which expert tensors always are, by 60 GiB) it settles at 155 GB/s — slower than what UVA does today. There is no mechanism on this box that holds cold experts in Grace at full speed while hot ones sit in HBM under one pointer.

So the residency axis is closed at off60. Not because the skew is not there — it is, 98.7% — but because the hardware offers speed or placement, not both. The one design left is a second HBM tensor for hot rows with a row-map inside the kernel, which brings back the 74-minute autotune and the parity gate for at most another +20% single-stream. Not doing it. What is banked: +14.7% from moving 12.7 GiB, a fetch tax measured at ~27% of the decode step and known to be link-bound, a routed-expert histogram nobody had published for this model, and three negative numbers that anyone building an expert-offload provider for GH200/GB200/GB300 needs before they design one. I also left a data-point comment on vLLM's expert-offload RFC, since its numbers are all PCIe consumer GPUs and the coherent-memory case wants a different provider shape.

Overnight, September 10–11: five levers, nothing adopted, one lesson

With the config settled I wrote a plan and handed the box to a second agent for the night: measure prefix-cache TTFT, then try transparent huge pages, unpinned host memory (VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1), --language-model-only, and the Rust frontend, each with a mechanism check and a win bar. The operator ran it exactly as written and reported two wins. In the morning I reran the win metric three times and it was noise (previous section). Re-read on the concurrency sweep:

The lane went back to last night's reference at 5:56 AM and re-measured 80.1 tok/s at C1. The lesson is not about any of the flags. It is that I set the win bar on the instrument I had, not the one that was tight, and a diligent operator hit the bar honestly. Bars go on the sweep now, and the recipe says so out loud.

The first day, September 10: head-to-head, knee, 1M context, prefill

Everything in this block was measured on the September 10 v11 configuration (OFFGB=70 UTIL=0.94, 82 tok/s prose). Numbers that moved with v12 are called out where they appear; prefill, mixed-workload and harness gates were re-run on v12 on September 12 and did not move.

Head-to-head, same box, same weights, same fixture

Concurrency sweep: where is the knee?

The first head-to-head used 128-token generations, short enough that prefill and scheduling noise dominated at C8. Re-run with 192 forced output tokens, two reps per point, two server configs. max-num-seqs matters: at the default 8, anything above eight streams queues, so C12/C16 read as a flat ~270 that is a scheduler cap, not a bandwidth wall.

Aggregate is still rising at sixteen streams, so the UVA fetch ceiling is further out than the first C8 number suggested. Two things are not noise and are on the list: C4 is bimodal (122 vs 189 on back-to-back reps in both configs), and C8 dips slightly under seqs=16. Both smell like expert-page locality on the 60 GiB UVA tier between runs rather than kernel behavior.

DSpark: the speedup depends on what you are generating

The V4.1 DSpark drafter ships inside the target checkpoint (mtp.{0,1,2}.*, 2,401 tensors in shards 44–46). vLLM loads it with one flag, --speculative-config '{"method":"dspark","num_speculative_tokens":5}', at a cost of ~4 GiB of HBM for draft weights and graphs. Offload went from 60 to 70 GiB to pay for it, and KV came out ahead (15.56 GiB vs 12.03) because the draft's memory profile is small.

The first sweep was a disappointment: on the paragraph-generator fixture, k=5 was 10–18% slower than no speculation at every concurrency. The /metrics counters explained it. Over 7,998 draft steps, acceptance by position was 59.5 / 30.4 / 13.9 / 6.1 / 2.8%, 22.5% overall, 2.13 tokens per verify step. A verify step runs six tokens through forty MoE layers and fetches six times the expert rows over UVA, so landing two of them is a loss. Meanwhile the "count to sixty" probe went from 79 to 240 tok/s on the same server: ~4.5 tokens per step accepted. Same engine, same flags, 3× apart. The fixture decides.

So I built a fixture shaped like what a Hermes agent actually emits and measured acceptance per category:

k=3 accepts a higher fraction of what it drafts (positions 3 and 4 were landing 6% and 3% on prose) but drafts less, so it nets fewer tokens per step on agent text and slightly more on prose. Neither dominates; for an agent lane, k=5 is the right default because the workload is code, tools, and shell. The published 79–86% acceptance numbers from other builds are on synthetic or code-heavy text, and the 4× DGX Spark repo's per-category table (code 73.8 vs prose 24.4 tok/s) is the same phenomenon at a lower floor. Nobody's number is wrong; "C1 tok/s" without a fixture description is now an incomplete statement for this model.

Two things that broke and how: the first k=5 boot re-ran the 74-minute autotune because the new engine-config hash directory was empty for the 60 seconds it took me to copy the cache in; restart with the seeded directory bound in 20 minutes and tuned only the 42 new draft shapes (168 → 210 configs). And each DSpark boot does two full passes over the checkpoint (target, then draft), which over SMB was 35 minutes of loading. That was the argument for the next section.

Local NVMe: cold boot 92 → 8 minutes

Two 8 TB WD SN850X went into the Station's M.2 slots as RAID0 (15 TB XFS at /models). Staged the full 510,313,343,553-byte checkpoint from the SHA-pinned archive, byte-verified every file, and relaunched with the same flags:

Decode did not change (count 252.7, prose 80.7 tok/s on the smoke, within noise of the SMB build). Boot is now cheap enough that flag experiments are a coffee break, not an afternoon. The staged-shard trick from this morning (Engram shards local, everything else symlinked to the NAS) is retired.

1M context

131K was a defensive choice from when we were 4 GiB short on HBM. The model's max_position_embeddings is 1,048,576 and the KV pool was already holding 3M tokens, so the only question was what the 1M profiling pass would cost:

Costs 5.5 GiB of KV headroom, changes nothing about decode speed, and the Hermes provider now advertises context_length: 1048576. Two real turns at --reasoning low and high came back clean.

Acceptance on real transcripts, not fixtures

The agent fixture above was my guess at what Hermes emits. To check it, I pulled 24 turns from today's actual session (12 that ended in a tool call, 12 text) out of the Hermes session store, replayed each with its preceding context and the real terminal / read_file / web_search tool schemas, temperature 0, thinking off, 400-token cap, and read DSpark acceptance from /metrics deltas per turn.

Per turn the spread is 5–171 tok/s. The turns where V4.1 actually emitted a tool call ran at 138–171 tok/s with 65–86% acceptance, matching the synthetic fixture. The turns that dragged the mean down were ones where the model wrote prose about what it was going to do instead of calling the tool, and prose accepts at 30–45%. Net: on real traffic, DSpark k=5 is a wash against no speculation (88 vs 93), with a large upside on the turns that are genuinely code or JSON. It stays on. The 150 number is real but it describes a fixture, not a day.

The replay also surfaced something that is not a speed result: in this bare setup (two-line system prompt, no Hermes harness), V4.1 produced a tool call on only 4 of the 12 turns where the original model had. That was a flag, not a verdict. Resolved the next morning: under the real harness it is 10/10 (below).

Correction, September 11. The 88.1 tok/s figure in this table should not be read as a decode number. Rerunning the same 24-turn replay three times on the same idle server, same flags, temperature 0, gave 90.6 / 119.4 / 124.3 tok/s. Batched MoE plus speculative decode is not bit-deterministic, the outputs differ by hundreds of tokens between runs, and the mean swings ±17. The acceptance percentages hold up; the tok/s does not. The concurrency sweep above runs each point twice and the pair agrees within about 1% — that is the instrument. This paragraph exists because I published an overnight "win" on the replay number and the sweep contradicted it. Details in the ledger below.

Mixed workload: a short request behind a long prefill

Everything to this point was one request at a time. The failure mode that matters for a shared lane is a 480K-token prompt arriving and every other user waiting. Measured: fire a 480K cold prefill, then a one-sentence question every 5 seconds behind it.

Default vLLM runs the long request's 8192-token chunks back to back and admits nothing else until it finishes; every short request landed at exactly the moment the long prefill completed. Under a 972K prompt that is 85 seconds of dead air for everyone. The fix is one flag: capping a long request's per-step chunk at 6144 leaves 2048 tokens of each step for other requests, and short requests get a first token in about a second. They still share the GPU with the long chunk, so their decode during the overlap is slow (~2.5 tok/s vs 150 solo); interleaved, not isolated. The long request pays 16%. That is the right trade for a shared lane, and it means the 1M config can be the daily lane rather than a separate one. The older --max-num-partial-prefills knobs are gone from this vLLM build.

Offload notch at 1M

With boot at 8 minutes, the 60-vs-70 question was cheap to answer. At 1M with DSpark: offload 60 loads 226.77 GiB into HBM and dies at Available KV cache memory: −2.61 GiB. Offload 70 is the floor for this configuration; the 10 GiB difference buys 12.7 GiB of KV once the draft model, its graphs, and the 1M profile are accounted for.

Prefill: a 972K-token prompt in 85 seconds

Everything above is decode. Cold prefill measured on the v10 reference config with the skill's probe: a random nonce at the start of the prompt (so no prefix-cache hit), random-word filler, max_tokens=1, thinking off, rate = prompt tokens / wall time. That is effective time-to-first-token, not a kernel counter; the server's --max-num-batched-tokens 8192 chunking is included.

Flat at roughly 18K tok/s from 6K to 200K, easing to 16.7K at 415K and 11.4K at 972K as the sparse indexer's top-k over a longer candidate set starts to cost. Zero preemptions; HBM stayed at 243 GiB through the 972K request, so the 10 GiB KV pool is genuinely holding a full-length sequence. The two outliers at 26K and 52K were first-touch of new UVA expert pages after idle and did not recur.

For scale: the 8× B200 report on the 0731 predecessor measured 1M TTFT of 84.7 s at c=8 under DP8 and 314.6 s under TP8. One Station at c=1 landing at 85 s for 972K with experts on UVA is the same order as an eight-GPU node's best case, which says the encoder half (8B active per prompt token) is doing what the architecture promised. What this does not say: nothing here is measured with concurrent short requests alongside a long prefill, and vLLM's partial-prefill knobs were reported not to help on that class of model. The 1M lane is a one-user-at-a-time lane until that is measured.

The autotune cache key trap

FlashInfer keys its tuned-config cache on a hash of the engine config. Changing --max-num-seqs from 8 to 16 produced a fresh hash with an empty directory, and the second boot was about to spend another 74 minutes re-tuning kernels whose shapes had not changed (the tuner runs at 8192 prefill tokens and the graph batch sizes, neither of which depends on the sequence cap). Copying autotune_configs.json from the old hash directory into the new one was accepted: Loaded 168 configs, graph capture in 8 s, bind in 20 minutes instead of 92. Worth knowing before touching any launch flag on a Blackwell box. I tripped this three more times today (--max-num-seqs, the speculative config, and --max-model-len each change the hash); the copy has to land before the tuner reads the directory, which is roughly two minutes after the KV gate. Seed it before launch, or disable autotune for smoke boots.

Hermes on it: the tool-calling gate

Added as an experiment provider (custom[dsv41] → :30006). The first smoke was five mixed turns, all clean. The bare replay above then raised the question that matters more than any speed: will it actually call the tool, or narrate calling it? So the gate is the real agent loop, not the raw API: hermes chat --provider custom[dsv41] --reasoning low -t hermes-cli, ten prompts each of which cannot be answered honestly without a tool — count lines in a file, read a secret string, parse a port from JSON, run uname, write a file and confirm it, append a line and report the new count, compute 1234×5678 in a shell. Scored from the harness's own tool_turns= log line plus a ground-truth check of the answer or the file on disk.

The 8-of-12 narration in the bare replay was a property of a two-line system prompt, not of the model. Under the real preamble and tool schemas it just calls the tool. Two things you need for this: --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41 --enable-auto-tool-choice on the server, and --reasoning low (or high) on the client — the V4.1 chat template rejects medium with a 400. Still not routed as a default; it is a lane you pick, but the reason it couldn't be is gone.

The recipe

All of this is now a formal, machine-checked recipe in J&M Recipes: dgx-station-gb300/deepseek-v4.1-flash-vllm-uva-dspark (merged as PR #6) — pinned image digest and model revision, the launch script with the knobs, every script that produced a number here, the provenance bundle, the five gates with pass dates, and a failure ledger that includes the offload bracket, the four autotune misses, and the overnight above. That is the version to reproduce from; this post is the story.

Why 26×: bytes moved per token

Both engines keep Engram on the Grace side. The difference is how the offloaded experts get to the GPU. SGLang's OffloaderV2 in cpu mode copies the entire expert tensor for each offloaded layer, every token: 15 layers × 6.72 GiB ≈ 100 GiB across the C2C link per decode step. At roughly 330 GB/s that is about 300 ms, which is exactly the 3.3 tok/s we measured. The GPU showed 100% utilization because it was stalled on the copy stream, and batching scaled perfectly because the copy was amortized. The profile said "bandwidth," not "compute."

vLLM's UVA backend maps the pinned host tensor into GPU address space and lets the MoE kernel read the rows the router selected: 6 of 384 experts per layer, about 1.6 GB per token instead of 100 GiB. We had already seen this exact trade on this exact box with full GLM-5.3 (1.4 tok/s SGLang OffloaderV2 → 33 tok/s vLLM UVA), so the prediction was cheap; the day was spent on memory, not on the idea.

The memory ladder

Ten launches to find the envelope. Every failure was a different wall.

First-boot autotune

FlashInfer autotunes the MXFP4 MoE kernel by running every tactic on the real hardware at 8192 tokens and keeping the fastest. With experts on UVA, each profiled run also pays the C2C fetch, so a single candidate took ~39 ms and the whole pass took 74 minutes with nothing in the log but 100% GPU. A py-spy stack (choose_one → trtllm_fp4_block_scale_moe_op) is what separated "working" from "hung." The 168 tuned configs are cached under /root/.cache/vllm, so this cost is paid once per image.

History: the SGLang first boot (morning of September 10)

Why this was uncertain

The release is 552B backbone plus a 196B Engram table, shipped as a mixed FP4/FP8/BF16 checkpoint. HBM on this Station is 250.7 GiB visible. The published SGLang recipe for GB300 is tensor-parallel across four GPUs; nobody had posted a TP=1 boot on one Station. The arithmetic said it was close: move the Engram tables to host memory as DeepSeek's report intends, and the resident set still overshoots HBM by tens of GiB before any KV cache. So a second offload tier was mandatory, and SGLang's DeepSeek-V4 model file did not wire one.

What the SGLang boot took

SGLang smoke results

Correctness, tool-call parsing, and reasoning-mode plumbing all pass on the first boot of a day-zero preview image. Speed does not. With 15 of 40 expert layers streaming from LPDDR5X through a synchronous one-step prefetch, decode is bandwidth-bound on the Grace side. The 16B-active design keeps the bytes per token small, but the offloader path as shipped is not built for latency.

Verdict (September 12 — history)

This is the September 12 verdict, kept as history. The current state is the October 18 banner: v23 is the reference, and config C is pending James's decision.

Next (as of September 12)

• Speculation depth is the open lever, and it is per-request. Static k=1 wins prose and every multi-stream point by 23–33%; k=5 wins agent traffic by a third; adaptive verification lands between with no losing category but pays a ~6.5 GiB varlen graph-capture tax that the upstream cudagraph-mode fix does not remove (evening three). See morning three. Cheapest next boot: vLLM's batch-size schedule (k=5 at one or two streams, k=1 above) — takes the aggregate win with no risk to single-stream agent traffic.
• Residency axis: closed at off60. Skewed histogram, but the hardware gives either speed (ATS, 340 GB/s, no placement) or placement (GPU page table, 90 GB/s), not both. Details in night two; scripts and raw JSON in the recipe bundle.
• v12 was the reference, confirmed (September 12; superseded — v23 is the reference as of October 18). Same-window pair at 03:00: v11 control 78.1, v12 89.6 single-stream, +14.7%; 312 aggregate at sixteen. Prefill, starvation and the Hermes harness (10/10) re-run on v12 and did not move.
• Expert pruning is out. James's rule, September 11: no REAP.
• C4 bimodality. Still reproducible, still uncharacterized, still not actionable.
• What would reopen residency: a driver that exposes access-counter migration on GB300, or an MXFP4 MoE kernel that takes per-expert pointers (cuDNN's SM100 grouped GEMM has a discrete mode; FlashInfer's fused kernel does not).

Reproduce (vLLM, the fast path)

The authoritative, machine-checked version of everything below — image digest, model revision, launcher with the knobs, every fixture script, provenance bundle, five gates with pass dates, failure ledger — is the J&M recipe. What follows is the same command with commentary. It is the September 12 v12 command on the df42c109 checkpoint. The current v23 launch adds the 1017 checkpoint, the pin-hot hook, 24 seats, the k-schedule [[1,8,5],[9,24,1]] and -e PYTORCH_CUDA_ALLOC_CONF=pinned_max_cached_size_mb:1024, and it lives in the recipe.

docker run -d --name dsv41-vllm --gpus all --ipc host --network host \
  --ulimit memlock=-1 --cap-add IPC_LOCK \
  -v /models/DeepSeek-V4.1-Flash-df42c109f1defefcbfcedbe7d905718a12266e40:/model:ro \
  -v /path/to/vllm-cache:/root/.cache/vllm \
  vllm/vllm-openai:deepseekv41-flash-0909 \
  --model /model --trust-remote-code --tensor-parallel-size 1 \
  --offload-backend uva --cpu-offload-gb 60 \
  --cpu-offload-params routed_experts.w13_weight routed_experts.w2_weight \
  --engram-config '{"cpu_offload": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"num_speculative_tokens_per_batch_size":[[1,4,5],[5,24,1]]}' \
  --max-model-len 1048576 --max-num-seqs 24 --max-num-batched-tokens 8192 \
  --long-prefill-token-threshold 6144 \
  --cudagraph-capture-sizes 1 2 4 6 8 12 16 18 24 32 40 48 64 96 128 \
  --gpu-memory-utilization 0.97 \
  --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41 --enable-auto-tool-choice \
  --port 30006

Whole checkpoint on local NVMe; pinned host copies page-fault from the checkpoint after load, DSpark reads it twice, and over CIFS neither converges. The matcher is routed_experts.*, not experts.*. Expect ~8–12 min to bind with a warm autotune cache, ~80 cold. Every launch-flag change makes a new autotune hash directory: seed it with an existing autotune_configs.json before launch or you pay the 74 minutes again. Shown as the v18 launch (24 slots, k-schedule, token-sized capture list); the pin-hot-experts hook is the bind-mounted sitecustomize.py + PIN_MODE=split PIN_ROWMAP=rowmap-static-v1.json described in its own post. Drop the last three flags and num_speculative_tokens_per_batch_size and this is the v12 config (--cpu-offload-gb 60 --gpu-memory-utilization 0.97, KV 4.89 GiB, 2.2× at 1M). For more than two concurrent full-1M requests use v11: --cpu-offload-gb 70 --gpu-memory-utilization 0.94 at 82 tok/s. Do not send reasoning_effort: medium.

Reproduce (SGLang, the 3.3 tok/s history)

docker run -d --name dsv41-dryrun --gpus all --ipc host --network host \
  --ulimit memlock=-1 --cap-add IPC_LOCK \
  -v /path/to/DeepSeek-V4.1-Flash:/model:ro \
  -v /path/to/patched/deepseek_v4.py:/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v4.py:ro \
  -e SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  lmsysorg/sglang:dev-dsv41 \
  python3 -m sglang.launch_server --trust-remote-code --model-path /model --tp 1 \
    --mem-fraction-static 0.85 --context-length 131072 \
    --chunked-prefill-size 8192 --max-running-requests 8 \
    --cpu-offload-gb 0 --offload-mode cpu \
    --offload-group-size 8 --offload-num-in-group 3 --offload-prefetch-step 1 \
    --reasoning-parser auto --tool-call-parser auto --port 30005

The patch adds offloader_kwargs to the make_layers call in DeepseekV4Model, copied from DeepseekV2Model, whitelisting the fused-MoE weight and scale tensors. Drop the page cache before launch; the Engram pre-fault needs the RAM. Expect roughly 27 minutes to first token over 10GbE storage.

Credits

• original-el8 — the dgx-station-gb300-research notes (Apache-2.0) behind the October 18 cards: prefill CUDA graphs (P1), the #58132 SWA-replay prefill numbers that put the nightly rebase on the list, and the long-generation cell (P5).
• Ronan (RonanLabsAI) — the real-text RT harness behind every RT C1 number in the October rounds, and the habit of reporting C1 as a mean of three with acceptance beside it.
• Opus 5.5, consulted as a second-model reviewer on October 18. It argued that on a lossless verify path a drafter change can only cost speed, so the speed should be measured. That review produced the per-class wall-clock gate.
• vLLM upstream — nightly-39983a5e, vllm#58132 decoder SWA bounded replay, and the DSpark and Engram fixes that the October 18 config C build rides on.
• 0xSero — the 4× RTX PRO 6000 build with a 45-case published sweep and DSpark acceptance numbers; the receipt that DSpark works on the shipped weights.
• vLLM — the dsv41-feat branch and the recipe page: EngramConfig.cpu_offload and the UVA offloader are the whole reason the second number exists.
• DeepSeek — weights, tech report, and the Engram design that makes host placement legitimate rather than a hack.
• SGLang / LMSYS — day-zero support, the Engram host-table path, and the grouped offloader.
• Tony (tonyd2wild) — the parallel 4× DGX Spark vLLM effort, which framed the memory math we checked against.