SGLang on DS4-F dual Spark: boots, then hangs
We already run Flash-0731 as the local default agent brain on two DGX Sparks under Anemll/vLLM — roughly ~68–70 t/s code, short streams ~74, synthetic ceilings near ~94. See the live recipe: Flash-0731 current recipe and honest speed. The open question was whether SGLang could clear the same dual-GB10 bar with better decode or a cleaner native DSpark path.
On August 1 we ran a maintenance-window canary on the real pair. Short answer: SGLang can boot Flash-0731 TP=2 on SM121 and pass exact/math smokes. It does not approach production speed, and every graphs-on / speculative profile we tried ended in a 300-second scheduler watchdog kill.
Production stays Anemll vLLM Flash-0731 (recipe). SGLang is a research receipt, not a cutover candidate.
What we tested
One pin, one pair, one night. Alternate port only (loopback :8890), Flash-0731 snapshot 9e165c30…, TP=2 over the QSFP bond, max context advertised 1M, max running requests 6, mem fraction 0.80.
- Image:
lmsysorg/sglang@sha256:6f91b6bbe3…(ARM64 CUDA-13) - Source overlay: SGLang main
574ead753… - Attention route:
SGLANG_SM120_FLASHMLA_BACKEND=triton,TORCH_CUDA_ARCH_LIST=12.1a - Smokes: exact string gate,
17×19=323, short count decode; then multi-run decode where stable enough
Ladder order was deliberate: prove boot first, then isolate graphs, then one speculative method at a time. Graphs-off is the only short-smoke stable path we saw.
Decision table
| Question | Answer |
|---|---|
| Can SGLang load Flash-0731 TP=2 on GB10 / SM121? | Yes. After fixing an exact SM120 (12,0) gate to accept the SM12x family (GB10 is (12,1)). |
| Does exact + math smoke pass? | Yes on graphs-off and on the first requests of graphs-only / DSpark. |
| Is decode competitive with Anemll? | No. Best stable short band ~7 t/s. Anemll code is ~68–70. |
| Do CUDA graphs free a big win? | No. Graphs-only ~7.3 t/s and then hangs. |
| Does native DSpark under SGLang work? | Boots. One 64-token run at 9.26 t/s; next request dies. |
| Is EAGLE the DSV4 “MTP” path? | Yes. DSV4 only allows EAGLE or DSPARK. There is no separate MTP flag name. |
| Promote SGLang / route Hermes to it? | No. |
Results by profile
| Profile | Boot | Exact / math | c1 decode | Stability |
|---|---|---|---|---|
| A. Graphs off, no spec | OK | OK | ~7 t/s | OK short smokes |
| B. Graphs only (no speculative) | OK · decode graph capture ~14s | OK first shorts | ~7.3–7.8 t/s (cuda graph: True) |
Hang → watchdog 300s after ~3 OK completions |
| C. Graphs + EAGLE (DSV4 MTP/NextN) | OK · draft + target graphs | FAIL | ~0.75 t/s · accept ~0.03 | Watchdog kill |
| D. Graphs + DSpark (bundled 0731 draft, γ=5) | OK | OK | 9.26 t/s (one 64-tok run) | Next decode hangs → watchdog |
| Production Anemll (comparison) | OK | OK | code ~68–70 · short stream ~74 | Production lane |
Prefill CUDA graphs were auto-disabled on every graphs-on launch with a DSV4-specific note: breakable CUDA graph is incompatible with DeepSeek-V4 under heavy capture-pool pressure. Decode graphs still engaged where captured.
The failure class
Across B/C/D the death mode looked the same:
/v1/modelscomes up healthy.- Some prefill and early decode activity succeeds.
- The scheduler stalls past
watchdog_timeout=300. - Rank dies → peer SIGQUIT → process tree kill → API goes dark.
That means the interesting bug is not “can we load weights.” It’s a multi-request / graphs-on stall under this dual-node SM121 path. Speculative decoding was not required to reproduce the hang — graphs-only already did it.
Useful CLI facts for the next person
- DeepSeek-V4 under this SGLang pin only accepts speculative algorithms
EAGLEorDSPARK. Asking for “MTP” is EAGLE/NextN in practice (DeepseekV4ForCausalLMNextNdraft). - Flash-0731 bundles DSpark draft weights (
dspark_block_size=5, target layers[40,41,42]). Draft path defaulted to the target checkpoint. - An early same-day attempt “passed” the kernel gate as skipped because the test required exact capability
(12,0). GB10 reports(12,1). The guard has to be SM12x-family, not SM120-only. - Canary API stayed loopback-only. No Hermes default change. No public endpoint.
# Graphs-off baseline shape (stable short smokes)
# --disable-cuda-graph
# no --speculative-algorithm
# Graphs-only (no free win; hangs)
# omit --disable-cuda-graph
# no --speculative-algorithm
# EAGLE / MTP path (pathological accept, hangs)
--speculative-algorithm EAGLE \
--speculative-eagle-topk 1 \
--speculative-num-steps 2 \
--speculative-num-draft-tokens 3
# DSpark (one-shot spike, hangs)
--speculative-algorithm DSPARK \
--speculative-dspark-block-size 5
What this is not
| Claim | Reality |
|---|---|
| “SGLang can’t run DS4-F on Sparks” | False. Boot + short exact/math works graphs-off. |
| “SGLang is ready as the agent backend” | False. ~10× slower than Anemll code feel; hangs under graphs/spec. |
| “Graphs will get us to 30–50 t/s for free” | Not on this pin. ~7.3 and then death. |
| “Native DSpark under SGLang beats Anemll” | Not measured. One 9.26 spike is not a comparison. |
| “We should keep both engines hot on the pair” | No. Same two GB10s cannot host full DS4-F twice. Canary is stop-the-world. |
Ops note after the window
James left the Sparks down during the testing window rather than auto-restoring production mid-canary. That is the right call when tool budget or hang recovery is uncertain. When the pair comes back, restore Anemll worker-then-head, drop sticky UMA page cache on both nodes, and re-verify VLLM_USE_BREAKABLE_CUDAGRAPH=0 on the live compose path.
Where this sits in the public trail
- Flash-0731 current recipe and honest speed — production lane numbers and knobs
- August DS4-F Testing: Anemll Stays Default — recent vLLM candidate A/B that also kept Anemll
- Anemll 1M/6 recipe — sticky dual-Spark packaging
- Original dual-Spark DS4-F deploy — early jasl/vLLM war story
- The sonnet replacement quest is done — current local-default framing