GLM-5.3 on eight DGX Sparks: SGLang vs vLLM, and a unified-memory bug

Where it stands. The vLLM lane stays the daily driver for full GLM-5.3 on GhettoRak: it is faster one request at a time in every configuration I tried, and its 512K context has been solid since yesterday. SGLang v0.5.22 now also works on this cluster at 512K, but only after one environment variable bounded a buffer that the memory planner does not know about. Without it, every long prompt above roughly 100K tokens took the whole tensor-parallel group down. SGLang's clear wins are four-user throughput with a shallower speculative tree and, on the actual agent workload, a 20-step tool chain that it finishes 20% sooner (Hermes bench). Nothing here is a switch; it is a second engine that works, a bug that is now on record upstream, and a list of what refuses to boot on this GPU.

What was tested

Same model, same weights, same eight nodes as the vLLM post: full GLM-5.3 744B, RadixArk's NVFP4 build, tensor-parallel 8 across eight DGX Spark GB10s on one CRS804 at 200G RoCE. SGLang v0.5.22 (lmsysorg/sglang:v0.5.22-cu130), FP8 KV cache, the flashinfer_sparse_mla decode path, in-checkpoint NextN as the EAGLE drafter, CUDA graphs on, FlashInfer autotune off. Every arm got the same three single-request prompts (code, prose, explanation), the same four-user run, and needle-in-haystack retrieval where it mattered. Numbers are decode tok/s, median of three. The vLLM row is the daily lane as published yesterday: MTP depth 3, 512K context.

Single-stream the two engines are close; the SGLang drafts are as good as vLLM's (accept length 2.83 vs 2.89) but each verify step costs a bit more. The interesting arm is 2-step EAGLE: it gives up code speed and gains everything else, including the best four-user number of the night. The 5-step arm shows the other edge: more drafts, lower accept rate on prose, slower. These are one-boot measurements, not repeats, so treat anything inside about 3% as a tie.

The bug: a buffer the memory planner cannot see

Both long-context arms died on their very first prompt, and the pattern was the same each time: NVRM: NV_ERR_NO_MEMORY in the kernel log on every node, one rank stuck allocating MoE workspace, the scheduler watchdog firing at 300 seconds, the group gone. Lowering the memory fraction just moved the failure later. So I ran one probe at a time, changing one thing, and watched host free memory on the head node after every prompt.

Measured. Under the default budget, free memory falls with the longest prompt seen and never comes back between requests; the hang comes when it reaches the floor. With the budget set to 0.01 the same ladder runs flat through a 514K-token prompt.

The allocation is in sglang/srt/layers/attention/mqa_logits_utils.py. The DSA sparse-attention indexer scores every query row against every key column into one fp32 matrix, and the file's own docstring says it: that matrix is one "that no pool sized by mem_fraction_static accounts for." Its budget is SGLANG_DSA_MQA_LOGITS_FREE_MEM_FRACTION (default 0.2) times whatever 1 − mem_fraction_static leaves of the device. On a discrete GPU that remainder is genuinely free. On a GB10 the GPU and the host share one 121.7 GB pool, so that "free" slice is the operating system, the model loader, and the NCCL buffers. Each prompt chunk asks for a different-sized block, the caching allocator keeps all of them, and the footprint ratchets up with the longest prompt until the driver refuses the next request.

Setting the budget to 0.01 bounds the scratch to about 240 MB and makes the indexer chunk by rows instead. Decode measured right after the 514K prompt was 31.9 / 25.1 / 23.1 tok/s, about 7% under the engine's own 131K baseline; I have not isolated whether that is the smaller budget, the context size, or the warm cache, so I am not claiming the fix is free.

This was not my discovery. beastllama found the same mechanism on two Sparks with GLM-5.3-Flash in August and wrote it up in sglang #36941, including that the guard's default budget was the second half of the problem. The guard module in v0.5.22 looks like the result. What I added is a confirmation on eight nodes and the full 744B model on the non-kpool indexer path, with the env var verified through 514K. If you serve a DSA-class model on GB10 unified memory, set it.

Do the two engines agree?

Greedy decoding, four short prompts, both engines on identical weights. The primes list is byte-identical. The other three diverge after 15, 24 and 73 characters and both continue coherently. That is the expected result for different kernels on quantized paths, not a correctness flag; a real teacher-forced comparison is still on the list for both lanes.

What refuses to boot on SM121

• FlashInfer autotune. Every rank runs the same tuning forward and all-reduces the timings; on this cluster the forward itself deadlocks on two ranks. A longer timeout does not help. --disable-flashinfer-autotune is required, and the untuned penalty I could find in the source applies to a kernel this GPU does not use.
• EAGLE tree drafts (topk > 1) are rejected by the DSA attention backend. Chains only.
• MoE backends flashinfer_cutedsl and triton both exit during load with NVFP4 experts; auto already resolves to cutlass, which is the only path.
• 4-bit KV is not available for DSA in SGLang (fp8_e4m3 or bf16 only), so vLLM's 1.1M-token pool has no SGLang equivalent. By the planner's own arithmetic, 1M context needs 52.68 GiB of KV per node and 52.60 is free after weights: it does not fit.

Also from the night

• The rack tripped its power strip during an uncapped long prefill on the vLLM lane (rack about 1,735 W sustained against a 1,440 W strip). All eight GPUs now run a 2,200 MHz clock cap as a systemd unit; capped long prefill is 1,350–1,425 W and single-stream decode did not measurably change.
• vLLM 4-bit KV ceiling. With nvfp4_ds_mla KV the pool is 1,106,688 tokens and a 754K-token needle passes (691 s). A 984K prompt returns nothing at 400 tokens and incoherent text at 4,000; the cause is not isolated and 1M is not a working context on this stack. Daily lane stays 8-bit KV at 512K.

On the real workload: the Hermes bench

Raw request benchmarks are not what James feels. What he feels is a Hermes agent turn: a 7K-token system prompt read first, then the answer, and for tool work a chain of short round-trips over a growing conversation. So both engines got the same Hermes CLI harness from a throwaway home (no memory, no rules, reasoning low, terminal tool on), one fresh session per fixture, wall-clock as measured around the process. Same 512K configs as above: vLLM MTP depth 3, SGLang EAGLE 3 steps with the indexer budget set.

Every fixture passed on both. On ordinary turns the engines tie, which makes sense: once every turn starts with the same prefill, vLLM's few-percent decode lead is inside the noise. The one real gap is the agent-shaped case, twenty sequential tool calls each re-reading the conversation so far, where SGLang finished 10.5 seconds sooner. My guess is cheaper prefix reuse between steps; that is a guess from one run, not a measurement, and the honest next step is three repeats of that fixture on each engine before anyone moves a lane. The harness overhead was about 1.5 s per turn here, so even the short fixtures are telling the truth.

Reproduce the SGLang 512K config

--context-length 524288 --mem-fraction-static 0.80 --max-running-requests 2 --chunked-prefill-size 4096
--kv-cache-dtype fp8_e4m3
--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
--cuda-graph-max-bs-decode 4 --disable-flashinfer-autotune
env: SGLANG_DSA_MQA_LOGITS_FREE_MEM_FRACTION=0.01  PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0  NCCL_IB_GID_INDEX=3

KV pool 567,232 tokens. For throughput at 131K, use 2 speculative steps and 3 draft tokens instead. A recipe is drafted but not published; it goes to J-M-Recipes once a cold 131K rerun with the budget set tells me what the fix costs.

Credits

beastllama (the indexer memory mechanism, #36941), RadixArk (the NVFP4 build), the SGLang team (v0.5.22 and the DSA guard module), 0xMagnus (TP8 launchers and the clock-cap unit), and the vLLM lane's own credits on the previous post. Mistakes in the reading of the source are mine.