July 31, 2026 · updated August 1, 2026 · James Meadlock & Milo · Funland dual DGX Spark · clean rewrite

Flash-0731 on 2× DGX Spark: current recipe and honest speed

Created July 31, 2026 · Last updated August 23, 2026 at 7:10 AM CDT

Do not market “95.9 t/s” as the daily number. We hit 94.3–94.7 on a high-accept count stream after the graph fix — close to 0xSero’s 95.9 probe class — but agent/code work lives in the 47–70 band.

Decision

The second speed fix: verify less as load rises

Speculative decoding has two jobs: the small draft path proposes tokens, then the full model verifies them. K=5 is excellent for a single high-accept request, but asking the full model to verify five draft positions for every request becomes expensive when four to six requests are active. vLLM already has an asynchronous batch-size schedule for this. We use it instead of adding a custom confidence scheduler, extra counters, or CPU/GPU synchronization.

Copy this one-line compose change

Replace the existing SPECULATIVE_CONFIG assignment inside the compose command: | block. The doubled dollar sign is intentional: Docker Compose reduces $$ to $ inside the container.

SPECULATIVE_CONFIG="{\"method\":\"dspark\",\"num_speculative_tokens\":$${MTP_NUM_TOKENS:-5},\"draft_sample_method\":\"probabilistic\",\"num_speculative_tokens_per_batch_size\":[[1,1,5],[2,4,4],[5,6,3]]}";

exec /usr/local/bin/vllm serve ${DSPARK_MODEL:-deepseek-ai/DeepSeek-V4-Flash-0731} \
  ... \
  --speculative-config "$${SPECULATIVE_CONFIG}"

This does not make the drafter exceed K=5. The DSpark draft graph keeps its known-good maximum; the built-in scheduler reduces how many positions the target model verifies at larger active batches. Keep MAX_NUM_SEQS=6 and batch tokens at 8192.

Rollback is one line too

If your matched workload does not improve, restore fixed K=5 by removing the batch-size schedule:

SPECULATIVE_CONFIG="{\"method\":\"dspark\",\"num_speculative_tokens\":$${MTP_NUM_TOKENS:-5},\"draft_sample_method\":\"probabilistic\"}";

Restart through the same launcher and confirm the live process command no longer contains num_speculative_tokens_per_batch_size.

Follow-up diagnostics: cache, batch budget, and first-use JIT

After promoting the 5/4/3 schedule, we attacked three plausible failure modes instead of treating one benchmark win as the end of the work.

Prefix-cache churn did not reproduce

Protocol: A cold → A repeat → unrelated B cold → A again, with unique prefixes and identical chat-template framing. Cache metrics came from /metrics; TTFT came from streamed first content. This did not reproduce the deterministic A→B→A eviction failure we were looking for.

The C4↔C5 K boundary stayed stable

These are aggregate rates across concurrent requests, not per-request decode speed. The mixed case used one shorter branch, so it is a transition/stability test rather than an equal-output throughput comparison. Queues drained and no CUDA, NCCL, engine-death, or preemption condition appeared.

Batch 8216: healthy, measured, rejected

Increasing batch tokens from 8192 to 8216 makes the six-sequence effective scheduler budget land exactly on 8192 after the draft reserve. It was a clean one-axis canary, but the production-shaped probes moved the wrong way.

Decision: restore batch 8192. The restored lane returned a 90.70 t/s best count-300 run, passed tools and Hermes routing, drained its queues, and stayed clean on both TP ranks.

Bounded warmup for first-use JIT

The real operational problem was not steady-state decode. Previously untouched request shapes triggered Triton compilation during inference, turning the first production burst into a latency event.

The production launcher now calls scripts/run-post-readiness-warmup.sh only after /health and the existing minimal chat smoke. The wrapper exercises bounded prefill, C4/K4, and C6/K3 shapes; requires idle queues; scans for fatal runtime errors; and is deliberately fail-open. Set POST_READINESS_WARMUP_ENABLED=0 for an immediate bypass.

Measured speed (August 1, 2026)

Live Funland lane after breakable opt-out: deepseek-v4-flash-0731, Anemll 0.1.1, 1M/6, MTP=5, util 0.85, RoCE TP=2.

# both nodes · .env.dspark
VLLM_USE_BREAKABLE_CUDAGRAPH=0

# docker-compose.dspark.yml environment: (compose does not pass arbitrary .env keys)
VLLM_USE_BREAKABLE_CUDAGRAPH: "${VLLM_USE_BREAKABLE_CUDAGRAPH:-0}"

Also inject the compose line — setting only .env.dspark is not enough on this launcher.

What we run

Base path if you are still on preview DSpark weights: Anemll 1M/6 recipe. Weight-swap on a known-good Anemll dual-Spark checkout; do not change runtime and weights the same weekend.

Topology

Recipe (short)

Weights — download once, rsync once

# Head only — do not dual-WAN ~160 GB
export HF_HUB_OFFLINE=0 TRANSFORMERS_OFFLINE=0 HF_HUB_DISABLE_XET=1
hf download deepseek-ai/DeepSeek-V4-Flash-0731 --max-workers 4

MODEL_DIR=$HOME/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-0731
find "$MODEL_DIR" -name 'model-*-of-*.safetensors' | wc -l   # 48
find "$MODEL_DIR" -name '*.incomplete' | wc -l               # 0

rsync -aH --info=stats2 "$MODEL_DIR/" milo@10.0.0.2:"$MODEL_DIR/"
ssh milo@10.0.0.2 "find $MODEL_DIR -name '*.incomplete' -delete"

Mount the full hub model dir (blobs + snapshots). Snapshot-only binds break HF symlinks.

Checkout + env

cp -a ~/ds4-f-mia-anemll-1m6 ~/ds4-f-mia-anemll-0731
# .env.dspark minimum:
DSPARK_MODEL=deepseek-ai/DeepSeek-V4-Flash-0731
SERVED_MODEL_NAME=deepseek-v4-flash-0731
MAX_MODEL_LEN=1048576
MAX_NUM_SEQS=6
MAX_NUM_BATCHED_TOKENS=8192
GPU_MEMORY_UTILIZATION=0.85          # 0.80 failed 1M KV here
MTP_NUM_TOKENS=5
VLLM_USE_BREAKABLE_CUDAGRAPH=0       # required on Anemll 0.1.1
POST_READINESS_WARMUP_ENABLED=1      # default; set 0 for immediate bypass
DSPARK_VLLM_IMAGE=ghcr.io/anemll/dspark-vllm-gx10:0.1.1
HF_HUB_OFFLINE=1

# also add the compose environment line for BREAKABLE and the
# SPECULATIVE_CONFIG replacement from "The second speed fix" above
rsync -a ~/ds4-f-mia-anemll-0731/ milo@10.0.0.2:~/ds4-f-mia-anemll-0731/

Launch / stop (from head only)

# Clean SHM + page-cache both nodes first; mask Aiden/legacy units
cd ~/ds4-f-mia-anemll-0731
PROJECT_NAME=ds4-f-mia-anemll-0731 WAIT_ATTEMPTS=120 WAIT_SECONDS=15 \
  ./start-deepseek-v4-flash-dspark.sh

./status-deepseek-v4-flash-dspark.sh
# expect: no "Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1"
# expect: GPU KV cache size ~2.2–2.5M · Application startup complete

./stop-deepseek-v4-flash-dspark.sh

Cold bind is often 8–15 minutes. SHM “60 seconds” lines during TileLang/autotune are often non-fatal — wait for startup complete and a live /v1/models. With the hook installed, the launcher then runs the bounded warmup; a warmup failure is logged but does not turn a healthy server startup into a failure.

Hermes wiring

# providers.spark-ds4 sketch
api: http://192.168.1.11:8888/v1
api_key: no-key-required          # not "none"
context_length: 200000            # client advertise; server is 1M
default_model: deepseek-v4-flash-0731
extra_body:
  chat_template_kwargs:
    thinking: false               # JSON bool — string "false" can turn thinking ON

hermes --provider spark-ds4 -m deepseek-v4-flash-0731 -t safe \
  -z 'Reply exactly ROUTE_OK.'

Cloud still wins for vision and picky taste. Text/tools/long agent loops: Sparks first.

Gates before calling it live

curl -s http://127.0.0.1:8888/v1/models | jq .
# id deepseek-v4-flash-0731, max_model_len 1048576

docker logs <container> 2>&1 | grep -E 'BREAKABLE|Breakable CUDA|KV cache size'
# must NOT auto-enable breakable graphs

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-flash-0731","messages":[{"role":"user","content":"Reply exactly PROMOTE_0731_OK."}],"temperature":0,"max_tokens":32,"chat_template_kwargs":{"thinking":false}}'

# tools must return tool_calls[]
# Hermes ROUTE_OK

Rollback to preview weights

cd ~/ds4-f-mia-anemll-0731 && ./stop-deepseek-v4-flash-dspark.sh
# SHM + page-cache both nodes
cd ~/ds4-f-mia-anemll-1m6
PROJECT_NAME=ds4-f-mia-anemll-1m6 WAIT_ATTEMPTS=120 WAIT_SECONDS=15 \
  ./start-deepseek-v4-flash-dspark.sh
# Hermes default_model → deepseek-v4-flash-dspark

Keep both weight trees until you are sure you will not roll back.

Pitfalls

1. Missing VLLM_USE_BREAKABLE_CUDAGRAPH=0 — Anemll auto-enables the slow path. Set env and compose injection.
2. Batch 16384 — measured single-stream regression on count canary. Stay on 8192.
3. Batch 8216 — initialized cleanly but regressed count, code, and structured JSON in the matched canary. It was rejected; stay on 8192.
4. Sticky UMA after stop — free mem can stick ~10–16 Gi with no containers; drop_caches both nodes or reboot. Avoid docker system prune -af (wipes Anemll image).
5. Dual-WAN download — head pull + RoCE rsync only.
6. Snapshot-only mounts — break blob symlinks.
7. GPU_MEMORY_UTILIZATION=0.80 — failed 1M KV here; use 0.85.
8. Gloo on 127.0.0.1 — per-rank VLLM_HOST_IP + bond0 socket IFNAMEs.
9. Hermes api_key = no-key-required, not none.
10. thinking string "false" is not off. Boolean only.
11. Cold start 8–15 min; don’t abort on first SHM timeout during autotune.
12. First-use inference JIT — readiness alone does not compile every production shape. Keep the bounded post-readiness warmup enabled; treat cold-bind proof as pending until observed.
13. Mask Aiden/legacy units before compose.
14. K=7 on this Flash drafter is not free speed — dspark_block_size=5. Keep the draft maximum at K=5; shorten target verification under load with the measured 5/4/3 schedule.
15. Unwatched ≥128k synthetic cold needles on the sole prod head can hard-hang the box. Step long probes; watch the console.
16. Head dies mid-TP — stop orphaned worker on Spark2 immediately; restore from head after clean SHM/page-cache both nodes.
17. Marketing the synthetic ceiling — count-stream ~94 t/s is real but not Hermes chat. Publish code/prose bands.

Attribution

• Mia / @MiaAI_lab — dual-Spark Anemll path and 0731 packaging.
• 0xSero — breakable-graph opt-out / 95.9 probe class (PR #14).
• MiaAI-Lab / Anemll — dspark-vllm-gx10 image and compose.
• Tony / Tech2Wild — 0731 draft-loader / acceptance speed story.
• DeepSeek — weights + encoding.
• vLLM / FlashInfer — under the Anemll image.