July 31, 2026 · James Meadlock & Milo · Funland dual DGX Spark · updated after opt forensics + restore

DS4-F Flash-0731 on a Dual DGX Spark Pair (Recipe)

Official deepseek-ai/DeepSeek-V4-Flash-0731 is the GA Flash weight drop (same structure as DSpark, speculative module attached). Mia (@MiaAI_lab) flagged the HF release the same day it landed — that heads-up is why this pair moved quickly. This Funland recipe runs those weights on two NVIDIA DGX Sparks (GB10) using the dual-Spark Anemll path Mia / MiaAI-Lab and Anemll made practical for GB10 (our 1M/6 writeup): download once, rsync once, RoCE TP=2, Hermes-ready OpenAI API on :8888.

What you get

• OpenAI-compatible API on the head node: http://<spark1>:8888/v1
• Served name: deepseek-v4-flash-0731
• Server context: 1M tokens (Hermes can still advertise a lower client cap, e.g. 200k)
• TP=2 across two Sparks over RoCE / InfiniBand HCAs
• DSpark speculative decode K=5 (probabilistic) via Anemll stack
• Tool calling + DeepSeek-V4 reasoning parser enabled

Prerequisites

This is not a from-scratch vLLM build guide. Start from a working Anemll dual-Spark DSpark checkout, then weight-swap. Changing runtime and weights at the same time is how you lose a weekend.

Topology

Substitute your IPs and NIC names. Wrong VLLM_HOST_IP / socket IFNAME is the classic Gloo-on-127.0.0.1 failure.

Weights: download once, rsync once

Do not dual-WAN pull ~160 GB on both nodes. Pull on the head, rsync the HF hub tree to the worker over RoCE/LAN.

# On Spark1 (head)
export HF_HUB_OFFLINE=0 TRANSFORMERS_OFFLINE=0 HF_HUB_DISABLE_XET=1
hf download deepseek-ai/DeepSeek-V4-Flash-0731 --max-workers 4

MODEL_DIR=$HOME/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-0731

# Verify head
find "$MODEL_DIR" -name 'model-*-of-*.safetensors' | wc -l   # expect 48
find "$MODEL_DIR" -name '*.incomplete' | wc -l               # expect 0
ls "$MODEL_DIR/snapshots"                                    # pin the SHA
test -f "$MODEL_DIR/snapshots/<sha>/config.json"
test -d "$MODEL_DIR/snapshots/<sha>/encoding"

# Rsync full hub model dir (blobs + snapshots) to worker
rsync -aH --info=stats2 \
  "$MODEL_DIR/" \
  milo@10.0.0.2:"$MODEL_DIR/"

# On worker: wipe any leftover .incomplete from a killed dual pull
ssh milo@10.0.0.2 "find $MODEL_DIR -name '*.incomplete' -delete"
ssh milo@10.0.0.2 "find $MODEL_DIR -name 'model-*-of-*.safetensors' | wc -l"  # 48

Mount the full hub model directory (or let Anemll resolve the HF id with HF_CACHE). Snapshot-only bind mounts break relative symlinks into blobs/ and the loader dies with “Invalid repository ID”.

Funland pin: rev 9e165c30e2704aec5d9d593cce3eebd58bbef1cb, ~156 GiB on disk.

Checkout + env

Copy a known-good Anemll DSpark checkout; do not mutate production in place until you trust the swap.

# On head
cp -a ~/ds4-f-mia-anemll-1m6 ~/ds4-f-mia-anemll-0731
cd ~/ds4-f-mia-anemll-0731

# Edit .env.dspark (minimum deltas)
DSPARK_MODEL=deepseek-ai/DeepSeek-V4-Flash-0731
SERVED_MODEL_NAME=deepseek-v4-flash-0731
WORKER_DIR=/home/milo/ds4-f-mia-anemll-0731
WORKER_SCRIPT_DIR=/home/milo/ds4-f-mia-anemll-0731

# Keep the Anemll 1M/6 shape that actually fits this pair
MAX_MODEL_LEN=1048576
MAX_NUM_SEQS=6
MAX_NUM_BATCHED_TOKENS=8192
GPU_MEMORY_UTILIZATION=0.85   # 0.80 was short of KV for 1M on our pair
MTP_NUM_TOKENS=5
DSPARK_VLLM_IMAGE=ghcr.io/anemll/dspark-vllm-gx10:0.1.1
HF_HUB_OFFLINE=1
HF_CACHE=/home/milo/.cache/huggingface
WORKER_HF_CACHE=/home/milo/.cache/huggingface

# Sync checkout to worker
rsync -a ~/ds4-f-mia-anemll-0731/ milo@10.0.0.2:~/ds4-f-mia-anemll-0731/

Env checklist (copy-friendly)

Launch / stop

# Clean both nodes first (Aiden/legacy units masked; SHM + page cache)
# Then from head only:
cd ~/ds4-f-mia-anemll-0731
PROJECT_NAME=ds4-f-mia-anemll-0731 WAIT_ATTEMPTS=120 WAIT_SECONDS=15 \
  ./start-deepseek-v4-flash-dspark.sh

# Status / logs
./status-deepseek-v4-flash-dspark.sh
docker logs -f ds4-f-mia-anemll-0731-vllm-dspark-1

# Stop
./stop-deepseek-v4-flash-dspark.sh

Cold bind is often 8–15 minutes (weight load + TileLang/FlashInfer autotune + cudagraph). SHM “60 seconds” log lines during compile are often non-fatal — wait for Application startup complete and a live /v1/models.

vLLM serve shape (what Anemll runs)

Effective serve (abridged from compose; env-expanded). Useful if you re-implement without their scripts:

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --served-model-name deepseek-v4-flash-0731 \
  --host 0.0.0.0 --port 8888 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --kv-cache-dtype nvfp4_ds_mla \
  --block-size 256 \
  --max-model-len 1048576 \
  --max-num-seqs 6 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --async-scheduling \
  --enable-chunked-prefill \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \
  --tokenizer-mode deepseek_v4 \
  --distributed-executor-backend mp \
  --moe-backend flashinfer_b12x \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --default-chat-template-kwargs '{"thinking":false}' \
  --enable-flashinfer-autotune \
  --nnodes 2 \
  --node-rank $NODE_RANK \
  --master-addr 10.0.0.1 \
  --master-port 25000 \
  ${HEADLESS:+--headless}

Official HF card sometimes shows K=7 greedy on GB300-class gear. We kept Funland-proven K=5 probabilistic on Anemll first. Change one axis at a time if you experiment.

Gates (do these before calling it live)

# 1) models
curl -s http://127.0.0.1:8888/v1/models | jq .
# expect id deepseek-v4-flash-0731, max_model_len 1048576

# 2) exact marker (thinking off)
curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"deepseek-v4-flash-0731",
    "messages":[{"role":"user","content":"Reply exactly PROMOTE_0731_OK."}],
    "temperature":0,"max_tokens":32,
    "chat_template_kwargs":{"thinking":false}
  }'

# 3) tools must return tool_calls[]
# 4) stream completes
# 5) optional Keys (model is argv[3] — default is still the DSpark name):
python3 bench_concurrent.py http://127.0.0.1:8888 1 deepseek-v4-flash-0731

Funland promote gates (2026-07-31): exact PROMOTE_0731_OK, tools PASS, Hermes ROUTE_OK.

Hermes wiring

# config.yaml provider sketch
custom_providers / providers entry for spark-ds4:
  api: http://192.168.1.11:8888/v1
  api_key: no-key-required   # not "none"
  context_length: 200000
  default_model: deepseek-v4-flash-0731
  models:
    deepseek-v4-flash-0731:
      context_length: 200000
      max_output_tokens: 32768
    deepseek-v4-flash-dspark:   # keep for rollback
      context_length: 200000
      max_output_tokens: 16384

# smoke
hermes chat --provider spark-ds4 -m deepseek-v4-flash-0731 \
  -q 'Reply exactly ROUTE_OK.' -Q --toolsets safe

Measured numbers (this pair, same Keys harness)

Post-promote optimization (same day)

After promote we did not thrash the runtime. Goal: make official 0731 as good as possible for daily Hermes on this pair — agent correctness first, Keys second. Sources: HF card, Anemll/howtospark dual recipe, Tony/Tech2Wild Patch3 lineage, Blackwellboy model-serving-minefield (traps 11/28/56–62), NVIDIA Dev forums, SGLang cookbook (no signed dual-GB10 cell), and a Nous Opus-5 second opinion on a measured brief.

Live forensics on the production serve

Ranked next moves

Rollback to DSpark

cd ~/ds4-f-mia-anemll-0731 && ./stop-deepseek-v4-flash-dspark.sh
# SHM + page-cache both nodes
cd ~/ds4-f-mia-anemll-1m6
PROJECT_NAME=ds4-f-mia-anemll-1m6 WAIT_ATTEMPTS=120 WAIT_SECONDS=15 \
  ./start-deepseek-v4-flash-dspark.sh
# Point Hermes default_model back to deepseek-v4-flash-dspark

Do not delete 0731 weights after promote; do not delete DSpark weights until you are sure you will not roll back.

Pitfalls we hit so you do not have to

1. Dual-WAN download — wasteful and contends disk with live benches. Head download + RoCE rsync.
2. Snapshot-only mounts — HF blob symlinks break. Full hub model dir or HF id + shared cache.
3. GPU_MEMORY_UTILIZATION=0.80 — failed 1M KV on this pair; use 0.85.
4. Gloo on 127.0.0.1 — set VLLM_HOST_IP per rank + bond0 socket IFNAMEs.
5. Keys 404 — harness defaults to DSpark model name; pass deepseek-v4-flash-0731 as argv[3].
6. Hermes api_key must be no-key-required, not none.
7. Cold start patience — 8–15 min; do not abort on first SHM timeout log during autotune.
8. Do not thrash Aiden/legacy units under compose — mask them first.
9. Runtime canaries ≠ weight canaries — we previously kept Anemll vs r0b0tlab v0.26 (August DS4-F Testing). This recipe is weights on Anemll.
10. Official K=7 greedy is a separate experiment from K=5 probabilistic — and on this checkpoint dspark_block_size=5, so K=5 is the natural match.
11. thinking string "false" is not off (trap 57). Send a JSON boolean.
12. Unwatched ≥128k synthetic cold needles on sole prod head can hard-hang the box (we power-cycled Spark1). Guarded only.
13. Head-down mid-TP: stop/rm the orphaned worker on Spark2 immediately; restore from head after clean SHM/page-cache both nodes.

What we run right now (Funland end-state)

Attribution

• Mia / @MiaAI_lab — public signal on the official Flash-0731 drop; ongoing dual-Spark / Anemll ecosystem work that this recipe builds on. If you get value from GB10 Flash stacks, follow and credit Mia.
• MiaAI-Lab / Anemll — ghcr.io/anemll/dspark-vllm-gx10 image and dual-node DSpark compose/script shape (1M/6 MTP5 lane). Funland runs a local adaptation; bugs in our env knobs are ours.
• DeepSeek — DeepSeek-V4-Flash-0731 weights and encoding docs on Hugging Face.
• vLLM / FlashInfer — serving and B12X / MLA stack under the Anemll image.