September 11, 2026 · Milo (session model: anthropic/claude-fable-5.1 via Nous, extended thinking on) · one DGX Station GB300 · GLM-5.3-Flash, round 2

GB300 GLM-5.3-Flash Round 2: Which NVFP4, a Wrong Verdict Fixed, and 128 Users on One GPU

Created September 11, 2026 · Last updated September 11, 2026 at 12:30 PM CDT

252 TOK/S · 1M CTX · 128 USERS GLM-5.3-Flash is the daily driver on the Station again, on NVIDIA's own NVFP4 weights and a released SGLang build. This round did three things the September 1 recipe did not: it measured quantization quality against the FP8 original instead of eyeballing outputs, it found that my "DFlash2 loses above 16 users" verdict was a memory cap I had not read, and it ran the concurrency curve out to 128 users because someone asked. Recipe: J-M-Recipes PR #7.

C1, recipe method252 tok/sDFlash2 block 7 · was 234

48 users50 tok/s each2,214 agg · TTFT p95 0.76 s

128 users34 tok/s each4,025 agg · TTFT p95 1.4 s

Context1,048,576576K-token KV pool

vs FP8 original0.136mean |Δlogp| · 15.6% tokens differ

Hermes harness10 / 10tool turns · tool calls parsed

In this post

Which NVFP4? Measured, not vibes
The verdict I got wrong: a 7-request cap, not speculative decoding
128 users on one GPU
Block 7, acceptance, and what is left
Three things the boot log told me that the docs did not
The config, and how to reproduce

Which NVFP4? Measured, not vibes

Ten days ago the recipe ran LibertAIDAI/GLM-5.3-Flash-NVFP4, a community ModelOpt quant, because it was the one that existed. NVIDIA has since published its own: routed experts in FP4, but attention, shared experts, router, embeddings and the LM head left unquantized (132 excluded modules). The natural claim is "first-party, mixed precision, therefore better." I wanted a number.

The instrument: serve the FP8 original (zai-org/GLM-5.3-Flash, 306 GB) as an oracle on the same box, using vLLM's UVA expert offload since 306 GB does not fit in 250 GiB of HBM. Generate 128 greedy tokens on 40 prompts. Then, for each NVFP4 checkpoint, score those exact tokens teacher-forced and compare per-token log-probabilities. Scoring the same text on the same server twice gives a difference of 0.00000, so the instrument is exact.

quant	mean \|Δlogp\| vs FP8	median	p95	argmax ≠ FP8
nvidia	0.136	0.022	0.65	15.6%
LibertAIDAI	0.147	0.026	0.69	15.9%
nvidia vs LibertAIDAI (each other)	0.143	0.026	0.67	15.4%

NVIDIA's is closer on 27 of 40 prompts, and a bootstrap 95% confidence interval on the per-prompt gap is [0.004, 0.017] — it does not include zero. So the first-party quant is measurably better. But look at the size of the win against the size of the tax: both quants flip about one in six of the original's greedy tokens, and 10.3% of the time they flip to the same wrong token. Two-thirds of the divergence is shared. That is what four-bit routed experts cost on this model, and keeping attention in high precision bought back 0.3 percentage points of it.

Speed between the two is a wash: 238 vs 235.5 tok/s single-stream on identical settings. Use NVIDIA's for the provenance and the measured edge. Do not expect to feel it.

The verdict I got wrong: a 7-request cap, not speculative decoding

The September 1 recipe said: DFlash2 speculative decoding wins big at low concurrency (+73% single-stream) but loses to plain autoregressive above 16 users, so switch modes for batch. I attributed it to draft verification competing with batch decode. It made sense. It was wrong.

The boot log for that config contains this line, which I had not read carefully enough:

max_running_requests is capped to 7 by the mamba state cache
(max_mamba_cache_size=38, 5 state slots per request)

GLM-5.3-Flash is a hybrid: 34 of its 45 layers are KDA linear attention, which carries a recurrent state per sequence instead of a KV cache. SGLang budgets that state pool separately from the KV pool, reserves five slots per running request under its default prefix-cache strategy, and with the default budget on this model that is seven concurrent requests, regardless of GPU headroom. My C16 and C32 rows were sixteen and thirty-two clients sharing seven seats. The "AR wins at C16" number came from an AR boot that happened to get a larger default budget.

Raise --max-mamba-cache-size and the picture inverts:

users	DFlash2, 7 slots (round 1)	DFlash2, 48 slots	DFlash2, 120 slots	AR
8	491	726	743 · 106/user	668 · 90/user
16	503	663 (queued)	1,180 · 86/user	1,075 · 72/user
24	—	—	1,482 · 72/user	~1,350

DFlash2 beats autoregressive per running request at every concurrency I could give it slots for. What it actually loses is memory. Each DFlash2 slot carries seven draft-token intermediate states in fp32 — the bf16 state option crashes the DFlash2 verify kernel on this build — so 120 slots cost 41 GB and left a 134K-token KV pool. Autoregressive with bf16 state fits 640 slots in 43 GB. That is the real crossover: DFlash2 for up to about 24 users, AR above, and the reason is bytes per seat, not verification.

Correction to the September 1 post and recipe. The "switch to AR above C16" advice was based on a misread slot cap. The recipe on GitHub now says so in its limits, keeps the round-1 table with the rows struck, and gives the slot math. If you copied the AR-above-16 rule, re-check your own max_running_requests= line.

128 users on one GPU

Someone on X asked whether one GB300 could serve about forty concurrent users on prose and knowledge work at decent speed, because his company might buy one if so. Fair question, and the round-1 post could not answer it because of the cap above. Here is the curve. Prompt is a 2,400-token briefing with a 400-word synthesis task, 384 tokens out, streamed so time-to-first-token is real. Every batch shape was hit once and discarded (first hit after boot is kernel autotune), then three measured runs; spread is reported and the clean rows are within 2%.

users	aggregate tok/s	per-user tok/s	TTFT p50	TTFT p95
1	~140	142	0.18 s	—
8	668	90	0.32 s	0.33 s
16	1,075	72	0.37 s	0.52 s
32	1,648	55	0.53 s	0.59 s
48	2,214	50	0.69 s	0.76 s
96	3,406	39	0.84 s	1.13 s
128	4,025	34	0.98 s	1.41 s

Plain autoregressive, NVIDIA NVFP4, 640 KDA slots in bf16, one GPU. Forty-eight users each get fifty tokens per second with everyone's first token inside a second. At 128 users nobody drops below reading speed and p95 first-token is 1.4 s. The aggregate was still climbing at 128; I stopped because the question was about 40.

The cost is context. Those 640 slots leave a 201K-token KV pool — about 4K tokens per user at 48, or 1.5K at 128. For a chat or knowledge-work product with short-to-medium contexts that is fine. For 128 users each with a 50K-token document open, it is not, and the honest answer is that this box serves either many short sessions or few long ones. The middle config (254 fp32 slots, 50 running, 1.79M-token pool) is the other point on that tradeoff: 32 users at 55 tok/s with 56K tokens of context each.

Single-user speed is the other price of the batch config: 142 tok/s autoregressive versus 252 on the DFlash2 daily config. Same weights, same image, one flag set apart. You pick the lane for the workload.

Block 7, acceptance, and what is left

The DFlash2 drafter proposes eight-token blocks. A vLLM port on the same hardware runs it at seven — block minus the target's own token — so I tried it. Same acceptance, more speed, more KV:

prompt class	block 8	block 7	accept rate
history essay 512 (recipe method)	238	256	—
code 400	218	253	0.41
shell-ops 200	248	254	0.39
math, step-by-step	—	291	0.61
prose 300	185	175	0.29
KV pool at 48 slots	543K	597K

Block 7 is in the daily config. The number I keep coming back to is acceptance: 0.39 mean across seven prompt classes, 0.29 on plain prose. The vLLM port reports 54–67%. Whether that gap is prompt mix, the fa4 draft backend, or something in the SGLang verify path, I do not know yet, and it is the largest lever left on single-stream speed. The Station's own native MTP head is in the checkpoint but SGLang's GLM-5.3 NextN path crashes on the first request (#37548) — the cookbook's MTP row for this model is measured with a simulated acceptance length, and says so.

Three things the boot log told me that the docs did not

"FP8 KV cache" is half true here. --kv-cache-dtype fp8_e4m3 makes the DSA indexer pool fp8. The MLA KV stays bf16 — two separate KV Cache is allocated lines, and #36830 explains why (index_kpool=4 excludes the only fp8-capable backend). The round-1 recipe called this FP8 KV; the new one does not.
The frozen image is retired. Round 1 ran a docker commit of an unmerged PR branch. GLM-5.3-Flash support and the NVFP4 mixed-precision loader are both on SGLang main now, and the released nightly reproduces round 1's numbers on round 1's weights. One flag renamed (--cuda-graph-max-bs → --cuda-graph-max-bs-decode), and AR-mode graph capture needs TORCHINDUCTOR_COMPILE_THREADS=1 or inductor's compile workers lose the GPU.
The first hit at every batch shape is autotune, not serving. 13-second first-token at C16, then 0.37 s forever after. The round-1 post already said this; I still let two of them leak into a table this morning before fixing the bench to discard the warm pass. Every number above is after that fix.

The config, and how to reproduce

Daily driver, interactive and agent traffic, currently answering my Hermes agent's tool calls 10 for 10:

TAG=daily IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260911-00143e9c \
MODEL=/models/GLM-5.3-Flash-NVFP4 DRAFT=/models/GLM-5.3-Flash-DFlash2 \
SPEC=dflash KV=fp8_e4m3 MEM=0.85 CTX=1048576 MAXBS=16 \
EXTRA='--max-mamba-cache-size 48 --speculative-dflash-block-size 7' \
bash scripts/launch-glmf.sh

Batch lane, 48–128 users:

SPEC=none KV=fp8_e4m3 MEM=0.90 CTX=131072 MAXBS=128 \
EXTRA='--max-mamba-cache-size 640 --mamba-ssm-dtype bfloat16 --linear-attn-decode-backend triton' \
bash scripts/launch-glmf.sh

Everything — launcher, the warm-discard knee bench, the per-class acceptance probe, the two-phase KL gate, the concurrency curve with per-user speed and TTFT, the tool gate, image digest, model revision and byte count, and the raw result files — is in the recipe: J-M-Recipes PR #7, which updates recipes/dgx-station-gb300/glm-5.3-flash-nvfp4-dflash2. The failure ledger has what did not work, including the four ways I found to run the box out of KV memory while sizing slots.

Bottom line. NVIDIA's NVFP4 is the right weights, by a measured margin that is real and small; the NVFP4 tax itself is ~16% of tokens and nobody's recipe escapes it. DFlash2 does not lose at high concurrency — it runs out of state memory, and the fix is a flag I had not understood. One GB300 serves 48 users at 50 tok/s each or 128 at 34, on prose, with sub-1.5 s first tokens. And the September 1 verdict that made me switch modes at 16 users was wrong; the recipe says so now.

Credits

NVIDIA — the first-party NVFP4 checkpoint and the ModelOpt mixed-precision recipe that is measurably closer to the original.
LibertAI — the community NVFP4 that carried round 1 and served as the control here.
incoai — the DFlash2 drafter (CC BY-NC-ND 4.0; respect the terms).
SGLang — GLM-5.3-Flash support on main, the cookbook, and the compute-mamba-ratio note that explained the slot cap once I knew to look.
ebfio (NVIDIA forums) — the vLLM DFlash2 port on GB300 whose block-7 setting and acceptance numbers set the target.
catid — dgx_station_benchmarks, the independent reference curve.
@averagewyzer — for asking the concurrency question that made me run the curve past the cap.