GLM-5.3 is a 744B-parameter mixture-of-experts model: 78 layers, the first three dense, the other 75 each holding 256 small "expert" networks plus one shared expert. For every token, a router in each layer picks just 8 of the 256. NVIDIA's NVFP4 export packs it into 433 GiB (465 GB). The GB300's HBM is 250.7 GiB. So the model is nearly twice too big for the GPU, and the only reason it can run on one Station at all is the 494 GiB of Grace memory welded to the GPU by NVLink-C2C. We have been here before: our slot-cache recipe on vLLM serves the same checkpoint at 54.7 tok/s for one user. TensorFold attacks the same problem with the same basic idea, written from scratch in Zig with its own CUDA kernels, and it handles more than one user much better.
Everything that every token touches stays on the GPU: attention, the three dense layers, the shared experts, the output head and the MTP draft head (whose experts the engine cuts from bf16 to 4-bit at load). That is about 35 GiB as stored. The routed experts are the bulk of the model. Each one is 20.25 MiB (three NVFP4 matrices plus their FP8 scales), 256 of them per layer, 5.06 GiB a layer, 380 GiB in all. All of them live in pinned Grace memory. While the server ran, /proc/meminfo showed 379.8 GiB of shared memory, which matches that arithmetic to within a tenth of a GiB.
The GPU keeps a ring for each MoE layer: 128 slots, each able to hold one expert. That is 9,600 slots, roughly 190 GiB if each slot holds an expert as stored. The startup line says it plainly: 0 experts pinned in HBM, a ring of 128 slots a MoE layer, 4 streams of 32768 tokens. Nothing is permanently resident. The ring is a cache, and Grace keeps the only full copy.
Figure 1. Sizes come from the engine's startup log, the checkpoint's safetensors headers and /proc/meminfo during the October 13 run. "≈190 GiB" for the ring is my arithmetic (9,600 × 20.25 MiB); the engine reports one combined figure, 245.07 GiB.
A decode round walks all 78 layers. At each of the 75 MoE layers, the router looks at every row in the round (one token per conversation, or several when drafting) and picks 8 experts per row. For each pick, the engine checks that layer's ring:
That is why half a layer's experts is enough to get good speed. Routing is not uniform: some experts get picked far more than others, and consecutive tokens tend to reuse some of the same experts. A ring holding the 128 a layer has used recently will catch most picks. It can never catch all of them, so every miss is paid for in C2C time.
Figure 2. The copy kernels and the "Grace experts staged" message are taken from the binary itself. The engine does not report a ring hit rate, so the green/pink mix is illustrative, not measured.
The binary also contains kernels named tf_offload_claim_predict and tf_offload_copy_predicted. From the names, the engine seems to start some copies before the router asks for them. I have not seen the source for this CUDA family (it is not in the public tree yet), so take that as a reading of symbol names, not a description of the algorithm.
GLM-5.3 ships with an MTP ("multi-token prediction") head: one small extra layer trained to guess the next few tokens. TensorFold uses it to propose a chain of guesses (about three a round on average tonight), then runs the full model over the whole chain in one pass and keeps the guesses up to the first one the model disagrees with, plus the model's own next token.
A drafted round costs more than a plain one, because more rows go through every layer. Tonight a plain round took about 21.7 ms and a drafted round about 38.7 ms. But a drafted round committed 2.65 tokens on average, and the extra rows mostly hit the same experts the first row already fetched. So one stream goes from 46 tok/s to 68 tok/s on a 1,024-token prose reply.
Figure 3. Round counts, draft counts and rates come from the server's per-request log lines and streamed timings for the 1,024-token prose prompt; the other three prompts committed 2.5–2.7 tokens a round.
The important part is the rule that drafts may not change the output. Batched math normally does change the output: a GPU summing four rows at once can split the work differently from one row alone, and floating-point rounding then differs in the last bit, which eventually flips a token. TensorFold's exact-decoding rule forbids that. A verify row uses the same group order, the same split of the long dot products and the same rounding it would use alone, and router ties break by a stable expert order. The draft only decides how many rows go into a pass; the model still chooses every token. Each finished request logs an output fingerprint, and the drafted and plain replies matched on all six prompt pairs I tried.
With --parallel 4, TensorFold does not run four conversations side by side. It runs one round at a time with every live conversation's rows in it. That matters a lot for an offloaded model. Within a layer, the experts four conversations need overlap with each other, so the round fetches their union once, and every row that wanted an expert reads the same copy. Four conversations need fewer than four times the misses, and the fixed cost of walking 78 layers is paid once per round instead of four times.
The same exactness rule holds across streams: each conversation's reply must equal what it would have been alone. I checked this with identical prompts (four copies of the same prompt matched the solo reply) and with different prompts (each stream's fingerprint matched across repeats, and the prose and code streams matched their solo runs).
Figure 4. vLLM figures are from our October 1 concurrency card (8 prompts, 512-token replies, 256K context). "My run" is tonight's four different prompts, 256-token replies, 32K context, total tokens divided by wall time; the 1-stream bar is the mean of the prose and code prompts measured the same way. Ash's numbers are from the run notes he sent James.
Our vLLM recipe fell apart at four users (26 tok/s total, 6.6 each), because four unrelated prompts thrashed a slot cache sized for one user and its prompts queued behind each other. TensorFold went the other way: 80 tok/s total, 21–25 for each stream. Each user is slower than alone, but the box does more work. Prompts are quick too: an 8,350-token prompt took 6.0 s to its first token, about 1,400 tokens a second.
James's Station, idle except for this test. Binary sha256 8e72be0a…f806 (tensorfold-native 1.0.8, chip nvidia-sm103), checkpoint GLM-5.3-NVFP4, flags as in Ash's notes: --context 32768 --parallel 4, greedy, thinking on. The model loaded in 187.8 s. Every reply below was still in its thinking phase when it hit max_tokens, so these are reasoning-text speeds.
| Case | Ash's notes | My run, October 13 |
|---|---|---|
| One stream, drafted (default) | 76 tok/s prose, 71 code (256 tokens) | 64.0–68.4 tok/s decode over 1,024 tokens (4 prompts); 62–65 tok/s including first token over 256 |
One stream, "draft": false | 48 tok/s | 44.3–46.2 tok/s (4 prompts) |
| Drafted equals plain? | "the same, token for token" | Yes: 6 of 6 prompt pairs, by full text (256) and fingerprint (1,024) |
| Two streams, total | 84 tok/s | 73.6 tok/s, different prompts (104 with the same prompt twice) |
| Four streams, total | 100 tok/s | 79.9 tok/s, different prompts (134 with the same prompt four times) |
| First token, ~1K-token prompt | about 0.8 s | 0.87 s (1,041 tokens) |
| First token, ~8K-token prompt | about 5.2 s | 6.0 s (8,350 tokens) |
| Four ~1K prompts sent together | all first tokens in about 1.9 s | 1.1, 2.2, 3.3, 4.5 s |
| MTP drafts kept | not stated | 53–61% of drafted tokens on the 1,024-token runs |
My single-stream and concurrency numbers come in 10–20% under Ash's. I do not know why yet: different prompt mixes, reply lengths and timing methods would all move them, and I did not try to reproduce his harness exactly. The "same prompt" rows show how much shared routing helps. Four identical conversations route identically, so the round copies each expert once for all four, and the box reaches 134 tok/s. Real traffic will sit closer to the different-prompts number.
Both engines are built on the same physics: experts in Grace, a per-layer cache in HBM, drafts to get more tokens out of each miss. A few honest differences before anyone reads Figure 4 as a scoreboard:
From Ash's notes, with the box-specific device UUID and paths removed. The GB300 has to be free (the engine takes nearly all of its HBM and about 410 GB of host memory), CUDA 13.2's cuBLASLt has to be on the library path, and the GB300 should be chosen by UUID, because CUDA's device order can differ from nvidia-smi's and the Station's display GPU must not be touched.
export CUDA_VISIBLE_DEVICES=<GB300 UUID from nvidia-smi -L>
export LD_LIBRARY_PATH=/usr/local/cuda-13.2/lib64
./tensorfold-native serve /models/GLM-5.3-NVFP4 \
--host 127.0.0.1 --port 8080 --context 32768 --parallel 4
curl -s 127.0.0.1:8080/health
curl -s 127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"glm","temperature":0,"max_tokens":256,
"messages":[{"role":"user",
"content":"Explain how a refrigerator moves heat out of its cabinet, step by step."}]}'Add "stream": true to stream. The reply's thinking arrives in reasoning_content before content; give it enough max_tokens or it will finish while still thinking (the server warns you). Add "draft": false to decode one token at a time and see the same reply more slowly. /metrics exposes Prometheus counters, including mtp_drafted_total and mtp_accepted_total. It listens only on 127.0.0.1.
capabilities --json); the copy-kernel names (the binary's symbols).