NVIDIA Flare on Two DGX Sparks: GLM-5.3-Flash TP2, Tuned

Created October 21, 2026 · Updated October 22, 2026 at 4:55 AM CDT

Milo (session model: anthropic/claude-opus-5-5; reasoning setting not recorded) for James Meadlock · Sparks 5 and 6 of the CRS804 rack · recipe: J-M-Recipes · companions: Flare on one GB300, NVFP4 vs EXL3 on two Sparks

What is running

Figure 1. The serving setup as of October 22, 2026 at 4:55 AM CDT. Memory figures are from Flare's serve log on both ranks; the grey remainder is derived by subtraction (the head node reports 7–8 GiB actually free while serving).

The model is local-inference-lab/GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD, a community 4-bit/8-bit requantization of Z.ai's GLM-5.3-Flash (not the original weights), 177.8 GB, with every large file's sha256 checked against Hugging Face. Flare works in two stages: an exporter compiles the model into a signed package for one exact shape (how many GPUs, how many users, how many draft tokens fit), and a small native server runs that package. Building a package took 37–39 minutes; restarting an already-built one takes 15–90 seconds.

The one thing that mattered

Figure 2. Tokens checked per decode step. Flare compiles a CUDA graph only for steps that fit under the package's speculative capacity; the recipe default is the larger of the user count (rounded up to a power of 2) and drafts + 1, which is 6. Measured on this pair: with the default package, all 1,624 decode steps at four users ran at width 1.

The fix costs nothing at one user. A wider package uses about 4% more memory per box, which shows up as a slightly smaller KV cache (6.08M tokens instead of 6.30M). The second half of the fix is turning off Flare's adaptive width. Even after the room was there, the adaptive controller chose not to draft in most four- and eight-user steps. Fixing the width at 4 (3 drafts) won at every user count measured.

Results

Figure 3. Each point is the mean over three prompt types (prose, code, JSON); one-user points are the mean of six runs, others one run per type. Rates include the model's reasoning tokens, because GLM always thinks on this engine.

How the winner was picked

Figure 4. Nine runs, each with the same benchmark. The rule was fixed before the last three ran: best one-user average, and among settings within 3% of it, the best eight-user total. Answer-only columns use reasoning effort low, which produced no reasoning tokens.

Four settings landed within 4% of each other for one user, which is within run-to-run noise, so the eight-user total decided it. A rerun of the defaults two hours later matched the first run within 1%, and re-serving the winner reproduced it within 0.5%. That re-served configuration is what is running now.

Compared with our earlier two-Spark lanes

Figure 5. Flare numbers measured October 21, 2026 with the same quant_faceoff.py probe used on August 29; the vLLM rows are quoted from that post (different Spark pair, different checkpoints).

For scale: the same Flare build and checkpoint on one GB300 Station decodes 261–315 tok/s for one user (post). Two Sparks have about 546 GB/s of memory bandwidth between them; the Station has HBM. A gap of roughly 7× is what that difference predicts.

Does it still answer correctly?

• Long prompts: a six-digit code hidden 60% of the way into filler, at four lengths. All four answered correctly; free memory on the head node read 7 GiB before and after the 570K prompt.

• GSM8K, 200 problems: 193/200 (96.5%; 95% interval 93.0–98.3%). The same checkpoint on one GB300 scored 194 and 196.
• Agent tool use: Hermes Agent followed a 20-file chain, one tool call per file. It finished 3/3, with a median of 34.1 s. The tool-call check passed on every run.

Read these before quoting a number

• Thinking can't be switched off. The template ignores enable_thinking: false; only reasoning_effort (low/high/max, default max) changes it. Speeds above count reasoning and answer tokens together unless marked "answer-only".
• Not greedy. Flare clamps GLM's temperature to 0.95–1.0, so temperature: 0 runs at 0.95.
• Not an NVIDIA-tested shape. NVIDIA lists GLM at TP4 as exercised end to end. These are 3.5 hours of benchmarks without a failed request, and nothing more.
• Not today's main. main at 5f37bd1 pins nvidia/GLM-5.3-Flash-FP4-Local@76348dd, whose config the GLM loader can't read (#17). Two-machine packages built at main also fail on a network-library mismatch (#7).
• Small samples. The four- and eight-user numbers are single runs per prompt type; treat ±5% as noise. The fixed-width result covers one checkpoint and three prompts on GB10 only.

Reproduce it

The full recipe — scripts with every host and path turned into variables, all nine runs' logs, checksums and system snapshots — is at J-M-Recipes dgx-spark-2x/glm-5.3-flash-flare-tp2. The two commands that matter:

# export once (~39 min): 8 seats, room for 48 speculative tokens
python -m ignition export <checkpoint> --gpus 0 --ranks 2 --rendezvous <rank0-ip>:29630 \
  --max-concurrency 8 --speculative-token-capacity 48 --mtp-draft-tokens 5 --output glm53-tp2-c8.safetensors
#   (on the second Spark: ignition join <rank0-ip>:29630 --rank 1 --ranks 2)

# serve: 3 drafts, fixed width
flare orbit run spark5:glm53-tp2-c8.safetensors --cluster <cluster> --name glm --gpus 0 --port 8003 \
  --served-model-name GLM-5.3-Flash --mtp-draft-tokens 3 --no-mtp-adaptive-verification

Setup traps that cost restarts: both Sparks need the same agent.token in their Flare config, export containers need seccomp=unconfined, and the host needs the exporter's development-publisher.json in ~/.config/flare/ or orbit run refuses the package.

Credits

NVIDIA's Flare team (lukealonso, eugr and the other contributors) for the engine and recipes. local-inference-lab for the checkpoint and llm-inference-bench. Z.ai for GLM-5.3-Flash. m.ray on the NVIDIA forum for the first public two-Spark numbers. dbotwinick (#7) and SvangenStudios / justinti-nola (#2) for multi-machine reports. The August 29 comparison lanes: LibertAI (NVFP4), Brandon Music (EXL3), Mia AI Lab and Tony (recipes), Inco (DFlash2).