NVIDIA Flare on One GB300: GLM-5.3-Flash TP1 vs the SGLang Daily Lane

Created October 20, 2026 · Updated October 21, 2026 at 8:27 AM CDT

October 20, 2026; tuning section added October 21, 2026 · Milo (session model: anthropic/claude-fable-5-1; reasoning setting not recorded) · one DGX Station GB300 · companion to GB300 GLM Flash recipe and the GB300 topic

What Flare is

NVIDIA/flare-local-inference (Apache-2.0, 698 commits, cloned at 47581cc from October 20) is a two-stage server. Ignition is Python: it loads the checkpoint with PyTorch/Transformers, tunes kernels with FlashInfer and NVIDIA's b12x kernel set, compiles every forward shape, captures CUDA graphs, and writes a signed runtime package. Helios is a Rust runtime that replays those graphs with no Python in the serving path: scheduling, paged cache, prefix snapshots, and the OpenAI API live there. Nothing is compiled or allocated after the server starts. It is aimed at "workstations and compact systems such as DGX Spark" — long conversations, coding agents, repeated tool use — and it trades vLLM's breadth for a fixed set of recipes: Qwen3.8 Flash/27B, Nemotron Bolt, GLM-5.3-Flash, DeepSeek-V4/V4.1-Flash, Kimi-K3, MiMo-V2.6, Step-5.

Why not DeepSeek-V4.1-Flash

We run DeepSeek-V4.1-Flash on this Station every day (vLLM, UVA expert offload, pin-hot-experts), so that was the model we wanted under Flare. It cannot happen as shipped:

• ignition/recipes/deepseek41/bootstrap.py raises "DeepSeek V4.1 Flash preparation requires four distinct GPUs"; the recipe table declares ranks={4: …} only.
• Every weight is device-resident: device_bytes=306,557,453,640 (285 GiB) against 250.7 GiB of HBM. Only the 188.8 GiB Engram tables live off-GPU (disk or host RAM, row-cached). Flare's design forbids runtime allocation outside captured graphs, so there is no equivalent of vLLM's --cpu-offload-gb to lean on.
• DeepSeek-V4-Flash-Vision is 2- or 4-GPU. Kimi-K3 is TP16. Step-5 is TP4/8/16.

So the one-Station recipes are GLM-5.3-Flash (TP1/2/4, 177.8 GB), Qwen3.8-Flash, Qwen3.8-27B and Nemotron Bolt. GLM-5.3-Flash is the useful one: it is the same model family as our :30001 SGLang daily lane, and Flare's own defect notes mention testing it on "a GB300 (SM103, one GPU)". The four-Spark TP4 DeepSeek test belongs on the Sparks and is running there separately.

Getting it to run on SM103

The README says "NVIDIA SM120/SM121 GPUs". The code disagrees: ignition/planning.py has SUPPORTED_ARCHITECTURES = "SM103, SM120, or SM121", and there are SM103-specific components (moe_sm103.py "GB300/B300 through FlashInfer's SM100-family MoE kernels", indexer_sm103.py, mhc_sm103.py, gdn_prefill_sm103.py) because b12x's block-scaled MMA kernels target SM120/121 only. On SM103 the routed experts, sparse-attention indexer and mHC run through FlashInfer's native kernels instead.

Toolchain on the Station: Rust 1.94.1, uv-managed Python 3.12.15, torch 2.14.1+cu132 against the box's CUDA 13.2 toolkit, driver 595.91.07, and the pinned local-inference-lab/flashinfer@622cec10 fork (reports 0.7.1). uv sync --locked plus cargo build --release took about 15 minutes, including one retry after a transient git fetch failure on the FlashInfer fork.

Two packaging bugs in that fork stop every SM103 boot, and nobody on a Spark would ever see them:

1. Catalog JSON not installed. indexer_sm103.py → flashinfer/experimental/deepgemm_dense_mqa/dense_mqa.py reads dense_mqa_catalog.json beside itself. The fork's setup.py ships only **/*.cu, *.cpp, *.cuh, *.h, *.pyx, so the JSON is in the git tree but not in site-packages. Same for mega_moe_v3/catalog.json and two kernel manifests.
2. Hashed kernel sources not installed. After a successful 31-minute autotune, the final bind called cake_dsa_train and ninja failed on a missing cake_dsa_h64_train_257d4674…_kernel.cu — a whole directory of generated *_kernel.cu/*_binding.cu under experimental/*/csrc/ is absent from the installed package (the kimi_k3_vision_tower and SM90 MegaMoE trees too).

The fix was one line: rsync -a --ignore-existing <uv git checkout>/flashinfer/ .venv/…/site-packages/flashinfer/. Python files are identical; only data and generated sources were missing. The cleaner fix is the docs' own development path (editable install of the sibling checkout) or a package_data patch upstream.

Boot timeline, TP1, concurrency 16, MTP five drafts

Helios reported 180.75 GiB loaded · 68.31 GiB cache budget · 10,485,760 KV tokens · 128 shared snapshot slots, 1,048,576 tokens/request, max concurrency 16, text only (the vision tower is not loaded in this recipe). The autotune and the package are now cached under /models/flare-cache/; a second ignition run binds in about six minutes.

Same-window results

Same Station, same hour (22:04–22:28 CDT), one engine at a time, nothing else on the GPU. Flare: :30007, package above. SGLang: the :30001 daily lane exactly as it serves every day — glmf-sglang:0922-582389ce-revert39688-tf5.16.1 (sglang 0.0.0.dev1+g582389cec, torch 2.13.0+cu130), nvidia/GLM-5.3-Flash NVFP4 checkpoint, --speculative-algorithm DFLASH --speculative-dflash-block-size 7 --enable-linear-replayssm-spec, fp8 KV, --max-running-requests 16 (the engine clamps it to 9 at boot, see caveats), 857,576 KV tokens. Both lanes were warmed before measurement.

Probe knee: three prompt classes, 512 output tokens, thinking on

A 40-line stdlib client, box-local. "Per-stream" is each request's own decode rate from first to last token, averaged; "wall" is total completion tokens over the slowest request's wall time, which is the honest aggregate when requests queue. Both C1 cells are the mean of two runs; the rest are single runs, so treat ±5% as noise.

Read it as three regimes. At C1 and C4, Flare is faster on every class (+11…+25% wall). At C8 it is a wash with the sign flipping by class (prose +11%, code −14%, JSON −12%). At C16 wall throughput is within ±6% on code and JSON and Flare is +31% on prose — but SGLang's per-stream numbers at C16 are not what they look like: the daily lane runs nine requests at a time and queues the other seven (TTFT 2.3–2.9 s), while Flare runs all sixteen with 68–91 tok/s each. Which you prefer at C16 depends on whether you are optimizing per-seat latency or queue depth.

llm-inference-bench v0.7.7 (its own prompt, 30-second windows, contexts 0 / 32K / 128K)

Aggregate tok/s as the harness reports it; TTFT is p50. The SGLang 128K × C8 cell is "∅": the harness skips cells that do not fit the engine's KV budget (857K tokens vs 8 × 129K).

Prefill (harness scout, single request)

Flare's prefill is flat at ~30K tok/s from 32K to 128K; SGLang's is flat at ~26.8K. The 8K gap (+55%) is the chunked-prefill floor: SGLang's --chunked-prefill-size 8192 turns an 8,198-token prompt into two steps.

Quality and tools

• GSM8K-200 (pinned test set, exact final-number match, 131,072 max tokens, all 200 in flight): Flare 194/200 = 97.0%, SGLang 194/200 = 97.0%. Five of the six misses are the same items on both (run indices 127, 131, 147, 159, 160); Flare also missed 100, SGLang also missed 14. No truncations, no parse failures on either.
• Tool calls: both return a parsed tool_calls array with get_weather on the standard smoke. That is a smoke, not the four-turn replayed gate we require before anything goes on an agent lane.
• Output read coherent on every probe class; no empty replies, no repetition loops at C16.

Tuning: speculative token capacity

Added October 21, 2026, morning. The first night's C16 numbers had a tell: Flare's per-stream rate fell from 261–315 tok/s at C1 to 68–91 at C16, while SGLang held 110–150 per running request. The cause is in ignition/components/capacity.py. Flare prepares a speculative graph for a (batch, verification width) pair only when batch × width ≤ speculative_token_capacity (width one always exists), and the default capacity is the larger of the padded concurrency and the maximum width — 16 for this export. With five drafts (width 6), the default catalog is:

So at 16 live requests the default package decodes one token per step per request. The graph catalog went from 84 to 89 entries, which is exactly those five added shapes. The rest of Flare's tuning surface is either already spent or does not apply on one GB300: the 31-minute b12x tune is cached, --tuning-gpus needs a second matching GPU, CSF scales are already expanded at load on SM103, stream overlap is already on, and --expert-scale-compression targets DeepSeek and Step. The MTP design note's "small-row MoE band through eight rows" is a statement about b12x on SM12x; on SM103 the routed experts run FlashInfer's trtllm-gen kernel, so the right envelope had to be measured, not inherited.

The capacity-32 export reused the cached tune (188 s instead of 1,854 s), compiled the 59 forwards in 102 s, then spent 1,140 s warming them. During that warm, Inductor logged a Triton signature mismatch on FlashInfer's fused RMSNorm (ROUND_BEFORE_WEIGHT missing from the call), 36 times, and recovered. Package write was 695 s. Total ≈ 37 minutes from launch to bind, of which a cached reload is 13 s of weight load and under a minute to serve.

Same probe, same box, one package after the other, control first (07:31–07:33 CDT) and capacity 32 second (08:14 CDT). Per-stream is the mean of each request's first-to-last-token rate; wall is total completion tokens over the slowest request's wall time. C1 is the mean of two control runs and five capacity-32 runs; every other cell is one run.

• C16 is the win: +26% per-stream on both classes, +13% (prose) and +16% (code) wall-clock. That jump is also the proof the 16×2 graph is actually scheduled — a width-one C16 cannot produce it, and the export log does not name graphs on this path.
• C8: +6% / +11% per-stream, +2% / +8% wall — batch 8 can now verify three drafts instead of one.
• C1 and C4 are unchanged within noise. The first two capacity-32 code C1 runs read 270 tok/s, which looked like an 11% regression; three more runs read 305, 295 and 311. The five-run mean (290) against the two-run control (303) is inside this probe's run-to-run spread. Prose C1 is 262 vs 263.
• Memory: the five extra graphs cost 0.25 GiB of cache budget (68.31 → 68.05 GiB, 10.49M → 10.44M KV tokens).
• Quality: GSM8K-200 on the capacity-32 package is 196/200 = 98.0%, no truncations or parse failures (97.0% on the default package, same harness).

Against last night's SGLang daily-lane C16 (one run, different hour): SGLang still has more per-running-request throughput (110 prose / 131 code) because DFlash keeps drafting under load, but it only runs nine at a time; Flare at capacity 32 now runs all sixteen at 95–97 tok/s. Capacity 48 (which would admit 16×3) is the next point on this curve and has not been measured.

What this comparison is not

1. Different checkpoints. Flare's recipe pins local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD@dec48abd (NVFP4 experts, MXFP8 elsewhere, lossless CSF scale compression, QAD). The daily lane serves nvidia/GLM-5.3-Flash-NVFP4@09b04e5e. Same base model, different quantization recipes. Identical GSM8K and shared misses say they are close; a teacher-forced comparison was not run and cannot be run the usual way because of item 3.
2. Different speculation. Flare uses the checkpoint's MTP layer with five draft tokens, adaptive verification and separate proposal/verification graphs. The daily lane uses the DFlash2 draft model (block 7) plus the replay-SSM trick we documented in September. Acceptance rates were not exported by either harness in this window.
3. Flare clamps sampling to the model card. /v1/models advertises temperature ∈ [0.95, 1.0], top_p ∈ [0.95, 1.0], and requests outside the range are clamped, not rejected. Greedy decoding is impossible, so the GSM8K runs above were effectively sampled at T≈0.95 on Flare and T=0 on SGLang — and still matched. Any logprob/greedy parity gate against another engine needs a different design (Flare returns up to 20 top logprobs per token).
4. Concurrency caps differ. Flare was exported at --max-concurrency 16 and runs sixteen. The daily lane asks for 16 but SGLang boots it at max_running_requests=9 under DFlash; the C16 rows reflect that queueing. Flare's KV plan (10.5M tokens) is also 12× the daily lane's (857K) at the same context length, which is why the harness could run 128K × C8 on Flare and not on SGLang.

Also: Flare's GLM export is text-only; the prefix-cache hit on repeated prompts gives Flare 0.02 s C1 TTFT versus 0.17 s on SGLang, which looks like prefix-snapshot reuse rather than prefill speed; and every number is a single evening's measurement, not a bracketed CTL→CAND→CTL window.

What's next

• Measure capacity 48 (adds 16×3, 8×5–6) and capacity 64 the same way; stop at the first point that does not beat its predecessor at C16 or costs C1.
• File the two packaging gaps against local-inference-lab/flashinfer (package_data for *.json and generated csrc/ sources) and the README arch line against flare-local-inference.
• Run the daily lane's real gates on Flare before anyone calls it a candidate: four-turn replayed Hermes tool conversations, the needle ladder to 1M, a C64 repetition audit, and a logprob-based fidelity check that respects the sampling clamp.
• Export the daily lane's own nvidia/GLM-5.3-Flash-NVFP4 checkpoint through Ignition (local directories resolve by config.json architecture) to remove the checkpoint variable, and export at --max-concurrency 32 to see where Flare's per-stream curve goes.
• Decide what the TP4 DeepSeek-V4.1-Flash result on the four Sparks means for the Station: Flare has no host-expert tier, so a single-Station DeepSeek-V4.1 under Flare would be a new recipe, not a flag.

Reproduce

# Station: toolchain + checkpoint (GPU-free)
sudo apt-get install -y build-essential cmake ninja-build pkg-config protobuf-compiler libssl-dev python3-dev liburing-dev
curl -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.94.1
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/NVIDIA/flare-local-inference && cd flare-local-inference && git checkout 47581ccac82660953a5b59cb04b2750d64edc97d
uv sync --locked --extra prepare --extra test
cargo build --release --manifest-path flare/Cargo.toml
hf download local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD --revision dec48abd33efa73c3bb7c95b74eee10cad34f9be \
  --local-dir /models/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD-dec48abd

# The SM103 packaging fix (until the fork ships its data files)
SRC=$(ls -d ~/.cache/uv/git-v1/checkouts/*/622cec10e)/flashinfer/
rsync -a --ignore-existing --exclude __pycache__ "$SRC" .venv/lib/python3.12/site-packages/flashinfer/

# ~/.config/flare/config.toml: put the 176 GiB package and kernel caches on the big disk
# [paths] package_cache = "/models/flare-cache/packages"  kernel_cache = "/models/flare-cache/kernels"  cuda = "/usr/local/cuda"

# Export + serve in one go (GB300 only; the RTX PRO 4000 display card stays hidden)
CUDA_VISIBLE_DEVICES=GPU-<gb300-uuid> .venv/bin/python -m ignition run /models/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD-dec48abd \
  --gpus 0 --max-concurrency 16 --mtp-draft-tokens 5 --prefill-chunk-size 4096 \
  --speculative-token-capacity 32 --host 0.0.0.0 --port 30007 --verbose
curl -s http://127.0.0.1:30007/v1/models   # id "GLM-5.3-Flash"; note sampling_ranges

The serving package is the capacity-32 one: /models/flare-cache/packages/snapshots/48ee8061…/model.safetensors. The default package (c2a3ac81…) is still in the cache. Tuning receipts: /home/milo/flare/results/cap32-20261021/ on the Station; GSM8K JSON under ~/hermes/research/flare-gb300/results/flare-cap32/.

Receipts: probe JSONL and tool-call responses under /home/milo/flare/results/{flare-glm53-tp1,sglang-glmf-daily}/ on the Station; llm-inference-bench JSON (t1-perf.json, t2-gsm8k200.json) and the aggregation script under ~/hermes/research/flare-gb300/results/ on the M4; Ignition logs /home/milo/flare/run-glm53-tp1-c16-mtp5*.log.