This post is a direct follow-up to yesterday's M2.7 MXFP4 run. After fighting through eight separate gotchas to get a 115 GB MoE model running across two Sparks and measuring 12 t/s at single-request throughput, we asked the obvious question: what does a smaller, faster model look like on the same hardware?
The answer is Qwen3.6-27B-FP8. One Spark. SGLang. NEXTN speculative decoding. Here's what happened.
What Is Qwen3.6?
Qwen3.6-27B is not the same as Qwen3-30B-A3B (the MoE variant). It's a dense 27B-parameter model with a hybrid attention architecture — a mix of standard Gated Attention layers and Gated DeltaNet linear attention layers. The linear attention component is what makes it architecturally interesting: it scales more efficiently with sequence length than standard attention and makes speculative decoding particularly effective.
In FP8, the weights come in at about 27 GB — comfortably within a single GB10 Spark's 119 GiB unified memory pool, with room to spare for KV cache.
The Stack
- Model:
Qwen/Qwen3.6-27B-FP8(27 GB, native FP8) - Container:
scitrera/dgx-spark-sglang:0.5.12— already cached on our Sparks - Backend: SGLang, TP=1, single Spark only
- Speculative decoding: NEXTN (5 steps, 9 draft tokens, topk=1)
- KV cache dtype: fp8_e4m3
- Context length: 65,536 tokens
The launch was straightforward. No Ray cluster, no multi-node setup, no page cache rituals. One docker run, model loaded in under 4 minutes, server up.
docker run -d \
--name qwen36-sglang \
--gpus all \
--shm-size=10g \
--network=host \
-v /home/milo/models:/models \
-e SGLANG_DISABLE_DEEP_GEMM=1 \
-e SGLANG_ENABLE_SPEC_V2=1 \
scitrera/dgx-spark-sglang:0.5.12 \
python3 -m sglang.launch_server \
--model-path /models/Qwen3.6-27B-FP8 \
--served-model-name qwen3.6-27b-mtp \
--context-length 65536 \
--mem-fraction-static 0.75 \
--tp-size 1 \
--host 0.0.0.0 \
--port 8003 \
--kv-cache-dtype fp8_e4m3 \
--page-size 64 \
--cuda-graph-max-bs 64 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--fp8-gemm-backend cutlass \
--speculative-algo NEXTN \
--speculative-num-steps 5 \
--speculative-num-draft-tokens 9 \
--speculative-eagle-topk 1 \
--max-running-requests 8 \
--trust-remote-codeBenchmark Results
Full llama-benchy sweep: pp=[512, 1024, 2048], tg=[128, 256], depth=[0, 1024, 4096, 16384], concurrency=[1, 4, 8], 3 runs each. Completed in approximately 3 hours with no crashes, no OOM, no intervention.
Generation Throughput (depth=0)
| Test | Total t/s | Per-req t/s | Peak t/s |
|---|---|---|---|
| tg128 (c1) | 22.6 | 22.6 | 27.7 |
| tg128 (c4) | 54.3 | 16.2 | 97.7 |
| tg128 (c8) | 95.3 | 15.5 | 170.3 |
| tg256 (c1) | 18.7 | 18.7 | 27.7 |
| tg256 (c4) | 56.1 | 15.3 | 92.0 |
| tg256 (c8) | 97.8 | 13.7 | 161.3 |
Prefill Throughput
| Test | Total t/s | TTFT (ms) |
|---|---|---|
| pp2048 (c1, d=0) | 1,500 | 1,377 |
| pp512 (c1, d=0) | 931 | 562 |
| pp2048 (c4, d=0) | 950 | 8,542 |
| pp2048 (c1, d=16384) | 875 | 21,074 |
Generation at Depth
| Depth | tg128 c1 t/s | tg128 c4 t/s | tg128 c8 t/s |
|---|---|---|---|
| 0 | 22.6 | 54.3 | 95.3 |
| 1,024 | 20.1 | 63.4 | 59.2 |
| 4,096 | 18.7 | 32.7 | 25.4 |
| 16,384 | 19.4 | 8.7 | 7.1 |
The depth story is nuanced. Single-request TG (c1) stays remarkably stable across all depths — 22.6 down to 19.4 t/s even at 16K depth. Speculative decoding continues to work effectively at depth. But concurrent serving at high depth collapses: c4 at d=16384 is 8.7 t/s, c8 is 7.1 t/s. This is expected — 8 concurrent requests each holding 16K+ tokens of KV cache saturates the available memory for KV allocation, forcing sequential processing. For single-user agent workloads this is a non-issue. For multi-user concurrent serving at deep context, you're memory-limited on a single Spark.
The Speculative Decoding Effect
NEXTN at topk=1, 5 steps gives a ~20% burst uplift at c1 (22.6 sustained, 27.7 peak). The peak column in the benchmark represents the best individual run out of 3, and you can see it consistently hitting 27–28 t/s — that's the speculative path firing well. On coding and templated tasks where token prediction is easier, this gap would be larger. On truly creative or random outputs it narrows. For agent workloads (tool calls, structured responses, chain-of-thought) it's a real gain.
Vs. MiniMax M2.7 MXFP4 (Dual Spark)
| Metric | Qwen3.6-27B (1 Spark) | M2.7 MXFP4 (2 Sparks) |
|---|---|---|
| TG c1 (sustained) | 22.6 t/s | 12.4 t/s |
| TG c8 (aggregate) | 95.3 t/s | ~65 t/s |
| TG peak c8 | 170 t/s | ~65 t/s |
| Prefill pp2048 c1 | 1,500 t/s | 1,004 t/s |
| TTFT pp2048 c1 | 1,377 ms | 1,676 ms |
| Sparks used | 1 | 2 |
| Benchmark completion | ~3 hours | Did not complete |
| OOM events | 0 | 1 (aborted first run) |
| Startup time | <4 min | ~4 min (warm cache) |
Qwen3.6-27B wins on every operational metric: faster TG, lower latency, uses half the hardware, benchmarked clean in 3 hours, never crashed. M2.7 is the larger model — more parameters, broader knowledge, likely better at genuinely hard tasks — but for the day-to-day agent workload where you're doing tool calls, reasoning loops, and structured output, Qwen3.6-27B is the right answer on this hardware.
What Spark 2 Is Doing Now
Nothing. It's free. That's the point. Running M2.7 required both Sparks tied together for one endpoint. Running Qwen3.6-27B leaves Spark 2 available for a second model, a different task, or just idling. For a two-Spark lab, single-Spark inference unlocks a lot more flexibility.
Wiring It Into Hermes
Qwen3.6-27B is now our primary candidate for the Hermes agent provider. It has native reasoning support (--reasoning-parser qwen3), native tool calling (--tool-call-parser qwen3_coder), and 22+ t/s interactive response speed. The OpenClaw alias wiring will come in a follow-up post once we've tuned the system prompt for agent use.
Practical Notes
- No page cache ritual needed. At 27 GB, the model fits easily. After a crash, there's enough headroom that
drop_cachesis optional rather than mandatory. - Context window interaction with benchmark depth. We tested depth up to 65,536 but those runs silently dropped — depth + pp exceeds the 65,536 context limit. Usable deep-context benchmark goes up to d=16384 with pp=2048.
- DeepGemm warning is benign. SGLang warns that the FP8 checkpoint's scale format isn't ue8m0 even with
SGLANG_DISABLE_DEEP_GEMM=1. Inference runs correctly regardless. - scitrera/dgx-spark-sglang:0.5.12 is marked experimental (0.5.11 is the stable tag) but ran without incident across a full benchmark sweep.