GLM-5.3 744B now runs on all eight Sparks of GhettoRak, using Z.ai's own FP8 weights. Nothing is re-quantized. It is the most faithful copy of the model that exists below BF16 (third-party KLD 0.0223), and it only fits because eight GB10s give it 88 GiB per node. One request at a time it decodes at 23–30 tok/s. Recipe: J-M-Recipes · dgx-spark-8x/glm-5.3-fp8-tp8.
8 × GB10
tensor-parallel 8, one CRS804, 200G RoCE, stock clocks
704 GiB
official FP8, every file SHA-256 checked on every node
30 / 27 / 23
tok/s code / prose / explanation, one request
131K
context, 150–182K-token KV pool
13 min
cold boot (27.5 min stock)
The recipe card, generated from the recipe's measured numbers. Amber = not measured yet.
Why eight nodes
GLM-5.3's official FP8 checkpoint is 703.7 GiB. Four Sparks have about 486 GiB between them, so a 4-node build has to use a 3–4-bit quant. Eight Sparks have about 973 GiB, which leaves room for the real thing: 90.96 GiB per node for the weights and the speculative-decoding (MTP) head, and 9–11 GiB per node for the KV cache.
The trade is context. The 4-bit builds leave ~36 GiB of KV per node and reach 512K; FP8 leaves one ~131K request.
Quality is the other side: on the same comparator (malaiwah/quant-fidelity-suite), FP8 measures KLD 0.0223
against BF16, while the EXL3 and NVFP4 builds that fit a vLLM lane measure 0.045–0.075 (unsloth's Q4 GGUF gets 0.0275, but that is a llama.cpp lane).
The setup
| Weights | zai-org/GLM-5.3 @ ba32f8c1, as shipped. Pulled once, copied to the other seven over the fabric (~13 min), 143/143 files SHA-256-matched to the Hub on every node. |
|---|---|
| Runtime | ciprianveg's gb10-glm-5.2:v19-vision image (a vLLM 0.11.2 fork with B12X sparse-attention kernels for GB10), pinned by digest. TP8, one rank per Spark, no Ray. |
| Launcher | 0xMagnus's FP8 TP8 launcher, pointed at the official weights, with MTP turned back on (adaptive depth 2/4/5) and an 8-thread weight loader. Every change is marked in the script. |
| Fabric | Each Spark's two ~100G PCIe functions on one 200G cable, both used by NCCL. 8-node all-reduce: 20.1 GB/s at 1 GiB, zero switch errors. |
| Defaults | reasoning_effort: low, 131,072 max context, 4 requests at once, bearer key. |
Results
| Test | Result |
|---|---|
| One request, code prompt | 29.9 tok/s (29.4–30.1) |
| One request, prose email | 26.9 tok/s (25.7–26.9) |
| One request, technical explanation | 23.1 tok/s (22.9–23.6) |
| Four requests at once | 40.1 tok/s total, 11.5–14.2 each |
| Find a code hidden in 35K / 70K tokens | pass / pass, in 40 s / 58 s (~1.2K tok/s prefill at 70K) |
| Tool call and tool-result round | pass on both boots |
| Wall power, decode + 70K prefill | 169–189 W per Spark, ~1.5 kW for the rack |
Method: temperature 0, reasoning low, 800 new tokens, streamed; one warm-up request, then the median of three. Decode speed is measured from first to last token. For comparison, 0xMagnus measured 22.8 tok/s with MTP on the uncensored FP8 build on the same image class.
What the optimization hour found
- Boot was the only easy win. Every node reads all 704 GiB to keep its 88, on a single CPU thread. The head took 957 s just to load weights. Eight loader threads cut that to 567 s, and the whole cold boot went from 27.5 to 13.2 minutes (the compile cache from the first boot helped too). Decode did not move.
- The custom NCCL build was already on. The image preloads NCCL 2.30.4 itself, so there was nothing to A/B.
- Light Foundry's biggest lever is not in this image. Their RoCE all-reduce setting took Int4-Int8Mix prose from 28.5 to 38 tok/s; v19 has no such switch. On FP8 it would help less anyway: decode here is mostly reading weights from each node's 273 GB/s memory. A small-message all-reduce probe read 81 µs at 12 KB, which I estimate at ~13–14 ms of an 80–150 ms decode step.
- MTP is already tuned by itself. It guesses 2 tokens ahead on prose (about 2.2 accepted per step) and 4–5 on code (4–5 accepted).
Watch out for
- Reasoning
highwith a small budget returns nothing. With 1,500 max tokens the model spent all of them thinking and returned an empty answer. That happened on the official weights, so it is not an artifact of the uncensored build where 0xMagnus first saw it. Uselow, or givehigh16K+ tokens. Onlylowandhighare honoured; anything else meansmax. - Memory is tight. The head node has about 7 GB of host memory free while serving, so 128K retrieval is untested and the memory setting should not go up.
- Not yet measured: a teacher-forced check that MTP on and off give the same model output on this stack. The weights are byte-identical to the release and MTP verifies every token against the full model, so I expect no change, but the recipe stays experimental until it is measured.
Next
- NVFP4 on the same eight nodes (RadixArk's build, already on disk): more context, worse fidelity. Same tests.
- Pre-sharded weights (
sharded_state) to get the boot under 5 minutes. - The teacher-forced quality gate, then promotion.
Credits
0xMagnus (FP8 TP8 launcher, memory limits, the reasoning traps), ciprianveg (gb10-vllm and the v19 image), wrldsuksgo2mars and the b12x stack (SM121 sparse-attention kernels), local-inference-lab (the vLLM fork), CosmicRaisins (adaptive MTP), Light Foundry (public 8× GB10 GLM-5.3 numbers), eugr (spark-vllm-docker), malaiwah (quant-fidelity-suite), and Z.ai for GLM-5.3 and its FP8 release.