August 1, 2026 · Milo H · Funland dual DGX Spark research note

What it takes for SGLang to scream on 2× DGX Spark

Short answer: NOT YET SGLang can boot DeepSeek V4 Flash-0731 TP=2 on dual GB10. It does not scream. Our measured c1 decode is still ~7–9 tok/s with hangs under graphs/spec — against a live Anemll vLLM lane at roughly 50–85+ tok/s. Production stays on Anemll. This post is the honest gap analysis and work plan.

Companion to the short canary writeup SGLang on DS4-F dual Spark: boots, then hangs and the older May failure What Broke. Current production recipe: Anemll 1M/6 / Flash-0731 status.

SGLang no-graph~7 t/s

SGLang + DSpark9.3 t/s*

Anemll vLLM50–85+

Gap~6–12×

*One successful 64-token decode; subsequent requests hung into a 300s watchdog. Not a stable result.

The gap is not “a flag we forgot.” It is kernel path + graphs + spec + stability engineering on SM121.

What “screaming” means here

Not GB300 NVL72 rack scores. Not social-media 95 t/s counting probes. For this fleet, SGLang would be worth promoting only if it clears a local bar:

Gate	Bar	Why
c1 short decode	≥ ~40 tok/s steady, not one lucky run	Must approach Anemll interactive feel
c3 / c4 agent load	No hang, no watchdog, usable aggregate	Hermes is multi-turn + tools
Correctness	Exact smoke + tool parser + no garble	Speed without tools is a demo
Context	At least 128K–384K usable without collapse	1M advertised ≠ 1M healthy
Ops	Clean stop/start, restore path, no 300s death loops	Production is a service, not a lab bench

What we actually measured (August 1, 2026)

Hardware: 2× DGX Spark GB10 (sm_121), QSFP bond0 10.0.0.1↔10.0.0.2, RoCE. Model: deepseek-ai/DeepSeek-V4-Flash-0731 rev 9e165c30…. Image: pinned lmsysorg/sglang digest with candidate source overlay (SM12x Triton FlashMLA path). API: loopback :8890, TP=2, 1M context advertised, max-running-requests=6.

Profile	Boot	Exact / math	c1 decode	Stability
No CUDA graph, no spec	OK	OK / OK	~7 tok/s	Short smokes OK
Graphs on, no spec	OK	OK / OK	~7.3 tok/s	Hang → 300s watchdog
Graphs + EAGLE (DSV4 MTP path)	OK	FAIL / n/a	~0.75 then hang	Accept rate ~0.03; watchdog kill
Graphs + DSpark γ=5	OK	OK / OK	9.26 tok/s (1×64 tok)	Later hangs → watchdog

Important naming note: DeepSeek V4 in this SGLang build only allows --speculative-algorithm EAGLE or DSPARK. There is no separate MTP CLI flag — EAGLE is the NextN/MTP path (DeepseekV4ForCausalLMNextN). DSpark loads a bundled draft (DeepseekV4ForCausalLMDSpark) from the 0731 checkpoint.

What the logs said (not vibes)

Kernel gate first: H=64 / top-k=192 Triton vs PyTorch reference — bilateral PASS on SM121 (after fixing a wrong exact-(12,0) skip guard).
EAGLE: CUDA graphs captured; decode started with accept rate: 0.03 and gen throughput ~0.75 tok/s; then Scheduler watchdog timeout (300s).
DSpark: draft weights bundled; graph capture succeeded; first decode fine; subsequent traffic died the same watchdog way.
Graphs-only: decode graphs ON, prefill graphs auto-disabled (“DSV4 breakable-incompatible”); speed not meaningfully above no-graph.

Why the official “SGLang loves DeepSeek V4” story doesn’t transfer

Upstream SGLang is real and impressive — on the hardware the cookbook actually validates. The official DeepSeek-V4 cookbook and day-0 LMSYS writeups target B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000 paths. Dual DGX Spark GB10 (SM121, aarch64, 1 GPU/node, unified memory) is a different animal.

Claim surface	What it proves	What it does not prove
SGLang DeepSeek-V4 cookbook	Flags, MTP/EAGLE, tool parsers, multi-GPU recipes on datacenter Blackwell/Hopper	Dual GB10 TP=2 is a signed cell
SemiAnalysis / GB300 Dynamo+SGLang	Rack-scale software can move tok/s/MW a lot	Anything about two desk Sparks
mark-ramsey sglang-dgx-spark	SGLang multi-node orchestration on Sparks for smaller models (e.g. GPT-OSS)	DS4-F production path
DSpark feature closed upstream (#29488)	Code path exists	Stable dual-Spark Flash-0731 at Anemll speed
Community “Inkling on 2 Sparks”	Some models work multi-node on this class	“Therefore Flash-0731 must scream”

Open upstream signal that matches our pain: sgl-project/sglang#33134 — Flash-0731 + DSPARK on 2× DGX Spark TP=2, sparse-MLA prefill rejecting topk=192 while config wants 512 and FlashInfer SM120 buckets are 128/512/1024/2048. That is exactly the “shape mismatch / hang / mid-path InternalError” class. Related umbrellas: SM120 DSA enablement, SM121 kernel packaging (#29317), FlashMLA sparse perf PRs.

The gap is a stack, not a single knob

Do these in order. Skipping to “turn on DSpark harder” is how you burn another maintenance window.

1. Kernel / shape correctness on SM121

Status: partially unblocked.

We forced the DSV4 FlashMLA path onto Triton and proved H=64/top-k=192 numerical parity vs torch on both ranks.
Upstream still has SM120/SM121 DSA holes: fixed topk buckets, missing aarch64 wheel reports, open sparse-MLA work.
To scream: either pad/clamp topk to FlashInfer buckets (192→512 with mask), or keep Triton as the default dual-Spark path and make it fast, not just correct.

2. CUDA graphs that do not kill the process

Status: graphs capture; graphs do not buy free tok/s; graphs correlate with hangs.

Prefill graphs auto-disabled for DSV4 in our builds.
Decode graphs enabled still sat ~7 t/s, then watchdog.
To scream: find the post-capture hang (soft watchdog, capture bs tier, draft/target graph mismatch). Until graphs are stable, the only honest baseline is eager decode.

3. Speculative decoding that actually accepts

Status: DSpark is the only interesting path; EAGLE/MTP is currently worse than nothing.

Method	What happened	Implication
EAGLE (MTP)	Accept ~3%, ~0.75 t/s, then death	Broken path on this pair/build — do not chase
DSpark γ=5	One good 9.3 t/s run; then hang	Worth debugging; not a speed miracle yet
No spec	~7 t/s stable short	Honest floor

Community vLLM DSpark recipes (Tony / Mia / Anemll class) already show the dual-Spark ceiling people actually care about. SGLang has to match that stack’s stability first, then race it.

4. Profile / memory / context

Status: 1M advertised is vanity until short c1 is competitive.

GB10 unified memory: every failed canary needs three-layer teardown + drop_caches or the next boot OOMs.
For speed work, drop to 128K–256K and max-running 2–4 until graphs+spec are stable.
Only then climb context. 1M is a late gate, not a first flag.

5. Agent product gates

Status: only trivial exact/math smokes; tools/soak not cleared under graphs/spec.

Must pass: tool-call parser, multi-turn, c4 staggered, 20–50 sequential requests without hang.
Hermes route switch only after that — and only if speed is within striking distance of Anemll.

Staged plan (if we keep investing)

Phase A — stable floor (1–2 maintenance windows). Graphs-off, no spec, 128K/2 or 128K/4. Soak 50 sequential + c4. Goal: prove SGLang can be a boring service at ~7 t/s without dying.
Phase B — make eager fast. MoE backend / Triton FlashMLA / batch-token / mem-fraction sweeps one knob at a time. Goal: ≥15–20 t/s c1 without graphs. If you cannot beat 15 eager, graphs will not save you.
Phase C — graphs isolation. Decode graphs only, soft watchdog first, capture-bs limited to live max-running. Goal: graphs ≥ eager and survive 50 requests.
Phase D — DSpark only (not EAGLE). Fix #33134-class topk/bucket issues; measure accept rate + tok/s. Goal: accept rate healthy (think >0.5 overall) and c1 ≥ 30 t/s without hang.
Phase E — context + agent. 256K → 384K → 1M only after D. Tool harness + Hermes shadow route on alternate model id.
Phase F — promotion bar. Head-to-head vs Anemll on the same prompts: c1/c4, tools, soak, restore time. Promote only if SGLang wins a dimension Hermes cares about (prefix cache reuse, structured output, multi-turn) and is not slower.

Default production decision (unchanged): KEEP ANEMLL / vLLM DS4-F on the dual Spark pair. SGLang remains a research canary with pinned image, alternate ports, and deadman restore.

What not to do

Do not treat “Inkling works on 2 Sparks” as evidence Flash-0731 will.
Do not chase GB300 Dynamo numbers as a dual-Spark roadmap.
Do not enable EAGLE/MTP again until DSpark is stable — EAGLE was strictly worse here.
Do not load a second full dual-Spark model while Anemll is up.
Do not publish “SGLang works for DS4-F” without the stability gates above.

Effort estimate (honest, local)

Outcome	Likely effort	Notes
Stable ~7–12 t/s research endpoint	days	Mostly ops + hang isolation
Stable ≥30 t/s c1	weeks	Needs kernel + graphs + DSpark health
Beat Anemll 50–85+ and promote	weeks–months or never	Requires upstream SM121 DSA maturity; may lose forever to vLLM forks on this silicon

Bottom line

Getting SGLang to boot DeepSeek V4 Flash on two DGX Sparks is a solved-ish canary. Getting it to scream is an open systems project: SM121 sparse attention shapes, graph lifetimes, and speculative acceptance — against a vLLM/Anemll stack that already does the job.

We will keep the canary harness and notes. We will not switch Hermes off Anemll because a server answered MILO_SGLANG_OK at 7 tok/s.