Atlas + Qwen3.6-35B-A3B on a Single DGX Spark: 74 tok/s, 3× SGLang

Earlier today, Milo published our Qwen3.6-27B-FP8 on SGLang benchmark: 22.8 t/s generation, 170 t/s peak burst with NEXTN speculative decoding. Solid numbers for a single Spark. But while that post was going up, something else landed: HF kernel-builder PR #576, merging SM_121 support for the GB10 GPU, pushed by Azeez at @AtlasInference.

SM_121 is NVIDIA's compute capability for the GB10 chip inside every DGX Spark. It's officially in CUDA 13+. Before this PR, HuggingFace's shared CUDA kernel library couldn't compile for the Spark — every engine fell back to slower generic paths. Now get_kernel() works natively on GB10.

But the real story is what Azeez built on top of that foundation: Atlas, a pure Rust inference engine with hand-tuned CUDA kernels for each (model × hardware) combination. It's not using HF's kernel library at all — it ships its own optimized PTX modules per model architecture. The SM_121 PR was Azeez giving back to the broader ecosystem.

Atlas claims 200+ tok/s on Qwen3.6-35B-A3B on a single DGX Spark. We had to test that.

What We Tested

Hardware: Spark 2 (DGX Spark, GB10, 119 GiB unified, SM 12.1) — previously idle after we moved Qwen3.6-27B to Spark 1
Engine: Atlas (avarok/atlas-gb10:latest Docker image), AGPLv3, open source
Model: Qwen3.6-35B-A3B-FP8 — 35B total, 3B active, GDN + attention + 256-expert MoE
Flags: MTP speculative decoding (K=2), FP8 KV cache with auto high-precision boundary layers, prefix caching, SLAi scheduler

The model is 34.9 GB on disk, 36.6 GB in GPU memory. With 119 GiB unified, that leaves ~83 GiB for KV cache — plenty for the 64K context window we configured.

Launch

Cold start from dropped page caches:

docker pull avarok/atlas-gb10:latest

sudo docker run -d --name atlas \
  --network host --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  avarok/atlas-gb10:latest \
  serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8888 \
    --bind 0.0.0.0 \
    --max-seq-len 65536 \
    --kv-cache-dtype fp8 \
    --kv-high-precision-layers auto \
    --gpu-memory-utilization 0.90 \
    --scheduling-policy slai \
    --enable-prefix-caching \
    --speculative \
    --num-drafts 2 \
    --tool-call-parser qwen3_coder

Startup time: ~30 seconds for the 42-shard fast weight loader (O_DIRECT + pipelined reads). Cold start to first token: ~150 ms.

Benchmark Results

Measured end-to-end through the OpenAI-compatible HTTP API at localhost:8888. Thinking disabled per request via chat_template_kwargs: {"enable_thinking": false}.

Test	TTFT	tok/s	Notes
Short (warm)	25 ms	74.3	"Capital of France?", prefix cache full hit
Long (204 tok)	181 ms	61.2	200-word paragraph, sustained decode
Cold (thinking)	153 ms	64.9	First request, 27 reasoning tokens included
Tool call	—	74.6	calculator("123 + 456") → "579"
Tool call (multi-turn)	—	75.0	Tool result → final answer in one loop

vs. SGLang (from Milo's earlier post)

Metric	SGLang	Atlas	Delta
Model	Qwen3.6-27B-FP8	Qwen3.6-35B-A3B-FP8	Larger MoE
Decode speed	22.8 t/s	61–74 t/s	2.7–3.2×
Peak burst	170 t/s (NEXTN)	—	(not tested)
Prefill	1,500 t/s	—	(not tested)
Hardware	1× Spark	1× Spark	Same
Engine language	Python	Rust	—

Atlas is faster on a larger model. The 2.7–3.2× gap is real and reproducible. We didn't benchmark prefill or peak burst — that'll come in a follow-up.

Tool Calling

Atlas ships with the qwen3_coder tool call parser, backed by XGrammar constrained decoding. OpenAI-compatible tools array works natively — no XML wrapping needed on the client side.

We tested a calculator tool in a multi-turn loop:

Request: "What is 123 + 456?" with calculator tool definition
Model returns: calculator({"expression": "123 + 456"}) — 74.6 tok/s
Tool result: "579" fed back
Model responds: "123 + 456 = 579" — 75.0 tok/s

Clean, fast, no thinking-mode interference during tool calls. The model auto-compresses thinking when tools are present.

Thinking Mode

Qwen3.6-35B-A3B thinks by default. The server logs confirm: thinking_default=true, max_thinking_budget=512. For agentic workloads where you want direct tool calls without reasoning overhead, disable per-request:

"chat_template_kwargs": {"enable_thinking": false}

This is exactly the same pattern we use with MLX Qwen models on the M3 Ultra. Consistent across frameworks.

Gotchas

Page cache is real. After stopping the first Atlas container, 108 GiB appeared "used" — but it was just filesystem cache. drop_caches freed it instantly. Always drop caches before cold starts, same as with SGLang and vLLM on the Sparks.
Model must be pre-downloaded. Atlas doesn't auto-pull from HuggingFace. Run hf download Qwen/Qwen3.6-35B-A3B-FP8 first, then point the container at your HF cache.
Container name conflict. A failed OOM launch leaves the container name reserved. Always docker rm -f atlas before retrying.

Wired into Hermes

After confirming the numbers, we added Atlas as a Hermes provider:

custom_providers:
  - name: spark2-atlas-8888
    base_url: http://192.168.1.12:8888/v1
    api_mode: openai
    models:
      - Qwen/Qwen3.6-35B-A3B-FP8

model_aliases:
  qwen3.6:
    model: Qwen/Qwen3.6-35B-A3B-FP8
    provider: spark2-atlas-8888
    base_url: http://192.168.1.12:8888/v1

Now any agent on the fleet can hit Spark 2 with --model qwen3.6. The Atlas container binds to 0.0.0.0, so Forge, M3 Ultra, M5 Max — any node on the LAN can use it.

What We Didn't Test

Prefill throughput at depth (Milo's post tested 2K/16K context prefill)
Peak burst with speculative decoding (Atlas claims 200+ tok/s on NVFP4 variant)
Concurrent request scaling (the SLAi scheduler should handle this well)
Vision tower (Qwen3.6-35B-A3B has MRoPE-positioned vision — text-only for now)
The NVFP4 variant (RedHatAI/Qwen3.6-35B-A3B-NVFP4) which Atlas claims hits 200+ tok/s

Those are on the list. For now, a single DGX Spark running Atlas + Qwen3.6-35B-A3B at 74 tok/s with working tool calls is a meaningful upgrade from where we were this morning.

What This Means for the Fleet

Spark 2 was a free agent after we consolidated Qwen3.6-27B onto Spark 1. Now it's running a faster engine on a larger MoE model, wired into the Hermes routing layer. The practical effect: any agent on the fleet can route to qwen3.6 and get 61-74 tok/s with tool calling on a 35B MoE — no cloud credits, no API keys, just a Docker container on a $3K desktop box.

The gap between local and cloud inference keeps narrowing. Atlas on a single Spark is now competitive with mid-tier API endpoints for interactive agent work. Not bad for a Tuesday.