GB300 GLM 5.3 Flash Testing
This is the GLM-5.3-Flash chapter on the DGX Station GB300. Day one ended at 141.8 tokens/second single-stream and two open mysteries. By the end of day two the same box was doing 234 tok/s — a 65% gain from a research round, one warmup discipline, and one draft model. Everything below is reproducible from the open-source recipe. The full 744B GLM-5.3 on the same Station is its own story: GB300 GLM 5.3 Testing.
Start with the map, not the flags
Before touching a single launch flag we ran four parallel research agents against the open questions: quantization options, engine support, speculative decoding, and SGLang tuning. Twenty minutes bought a clear picture. Our stack — SGLang PR #36507 plus the NVFP4 checkpoint — was the only viable path: vLLM's GLM-5.3-Flash support was unmerged and TensorRT-LLM handled the big GLM architecture but not Flash's. More usefully, catid's DGX Station benchmarks gave us numbers on identical silicon: 187 tok/s single-stream with DFlash2 speculative decoding, ~965 aggregate at 16 streams. We were at 141.8 and ~220. The gap was configuration, not physics.
Mystery #1: the concurrency cliff that wasn't
Day one measured a hard wall at 8+ concurrent streams: ~220 tok/s aggregate with 15-second time-to-first-token. The research pointed at CUDA-graph fallback. The logs disagreed — cuda graph: True throughout, and the server was decoding at 700 tok/s internally while clients sat waiting. The real cause was first-hit kernel autotune: every new batch shape pays a one-time JIT and tuning cost after server start. A cold benchmark measures that cost, not the server.
The fix is a procedure, not a flag. After every server start, run one warmup wave at each concurrency level you intend to serve — about 90 seconds total. Warm, the same server does 648 tok/s aggregate at 8 streams and 1,051 at 16. That is a 5× difference that was never a hardware limit. If you benchmark inference servers and skip the warmup, your numbers are wrong.
Mystery #2: speculative decoding
GLM-5.3-Flash has a purpose-built drafter, DFlash2: a ~1B block-diffusion model that proposes 8-token blocks which the 355B model verifies in one forward pass. It is verified-lossless — the output distribution is identical to plain decoding. On prose it lands about 3 accepted tokens per verification, so each expensive big-model pass is amortized across ~3 emitted tokens.
Result: 135.5 → 234.2 tok/s single-stream, ahead of the published 187 on the same hardware. The catch: speculative decoding loses above roughly 16 concurrent streams, where verification passes start competing with batch capacity. The recipe ships both launch configs and says which to use when.
The decode table
| Concurrency | Day one | Tuned AR | DFlash2 | Best config |
|---|---|---|---|---|
| 1 stream | 141.8 | 135.5 | 234.2 | DFlash2 |
| 4 streams | 372 | 376 | 511 | DFlash2 |
| 8 streams | ~220 · 15 s TTFT | 648 | 825 | DFlash2 |
| 16 streams | 216 | 1,051 | 796 | AR |
| 32 streams | 221 | 1,163 | 836 | AR |
Aggregate output tok/s, warm steady-state, 256-token generations, benched on-box. Machine-readable results, bench scripts, and the failure ledger are in the repo.
Prefill: a novella in four seconds
Long-context ingestion on the production config. Each prompt carries a leading nonce so the prefix cache cannot cheat, and TTFT is counted from the first streamed delta of any kind:
| Prompt | Time to first token | Prefill rate |
|---|---|---|
| 6.6k tokens | 0.29 s | ~22,500 tok/s |
| 26k tokens | 1.10 s | ~24,100 tok/s |
| 53k tokens | 2.01 s | ~26,200 tok/s |
| 105k tokens | 4.00 s | ~26,300 tok/s |
Same warmup rule: the first request in each prompt-size class after a restart pays ~16 seconds of one-time autotune. Warm, a hundred thousand tokens — a short novel — goes in four seconds.
The bug that passed every benchmark
After all the tuning, the server aced every throughput test and then failed in real agent use: tool calls never fired, and chain-of-thought leaked into replies. SGLang needs --tool-call-parser glm47 --reasoning-parser glm45 to turn GLM's tool-call markup into OpenAI-style tool_calls and to split thinking from content. Without them, plain chat works perfectly and anything agentic breaks silently — invisible to every benchmark. The launch scripts now carry the flags. The lesson generalizes: smoke-test the workload you actually serve, not the one that is easy to measure.
One recipe, many weights
Freezing the runtime had a useful side effect: any same-architecture NVFP4 checkpoint is a drop-in. Same image, same flags, same drafter, same client-facing model name. A swap is a container relaunch plus the warmup sweep, and clients never notice. The repo ships swap-model.sh for this; we use it to rotate between the stock checkpoint and a de-guardrailed research variant on the same box.
Measured, not assumed: swapping to Blackfrost's DERISKED variant (modified weights, same architecture) while keeping the stock-trained drafter cost 6.5% — 218.8 vs 234.2 tok/s single-stream — with accept lengths essentially unchanged (2.69–3.69 vs 2.77–3.95). A surgically-modified variant keeps nearly the whole speculative win without retraining the drafter. Total swap including warmup: about eight minutes.
What is in the recipe
- Launch scripts for both configs (DFlash2 for interactive, AR for batch), pinned model revisions, and an image-build script that freezes the runtime so production boots need no network
- The warmup script — the 90 seconds that separates real performance from autotune artifacts
- The failure ledger — all ten failed launches with exact error strings: transformers version traps, AutoConfig collisions, TP-style failures, and the proof that full GLM-5.3 in FP8 (704 GB) cannot fit one Station
- Diagrams — memory budget, serving topology, and how block-speculative decoding works
Driver memory mode
On day three we enabled NVIDIA's Coherent Driver-based Memory Management (CDMM) on the box so the kernel stops allocating into HBM behind the driver's back. Flash throughput did not move — the model already lives entirely in HBM, so there was nothing for the kernel to steal — and the tool-call smoke, warmup sweep, and a single-stream decode probe all re-verified after the reboot. The change matters for offloaded workloads, which is why the details live in the GLM 5.3 post.
Not yet tried
Two leads from the research pass are parked. SGLang's DSpark speculative decoding is built precisely for the concurrency crossover in the table above (verification length shrinks as load rises), but no GLM-5.3-Flash drafter has been published for it. And the frozen image predates a Blackwell FA4 fix and session-aware prefix caching in SGLang 0.5.18 — both worth an upgrade trial the next time we have a reason to re-bench.
Standing on shoulders
This work exists because catid published theirs: the reference numbers, the runtime pins, the recipe discipline. Publishing ours back, failure ledger included, is the repayment. Credits also to LibertAIDAI for the NVFP4 quant, inco.ai for the DFlash2 drafter, and the SGLang PR #36507 authors for the model support everyone is quietly running.
Provenance: all figures measured September 1–2, 2026 on the hardware in the repo README: DGX Station GB300, driver 595.84, CUDA 13.2, SGLang PR #36507, transformers 5.16.1, frozen image glm53-nvfp4-sglang:gb300-v2. Related: GB300 day one · GB300 GLM 5.3 Testing · the Milo-Ark model archive.