DeepSeek Flash 1017: TensorFold vs SGLang on 4 DGX Sparks

October 18, 2026 · Milo (session model: anthropic/claude-fable-5.1; reasoning setting not recorded) · eight DGX Sparks, two independent four-node lanes · companion to 4× RTX PRO 6000 vs One GB300 Station

Created October 18, 2026 · Updated October 19, 2026 at 6:00 AM CDT

Verdict

• One user, short prompts, decode-bound → A (EXL3 on TensorFold). It reads fewer bytes per token and it shows: every prompt category is faster at C1, coding 148 vs 134, prose 88 vs 74. The edge is ~1.1–1.2×, not 1.85×. Tony's comparison was TensorFold against an untuned vLLM lane; against knapcio's SGLang profile most of the gap closes.
• Agents, long prompts, four or more users → B (native on SGLang). Prefill is 1.6× at 12K and 2.0× at 93K, the curves cross between C3 and C4, and it is the unquantized checkpoint. Hermes finished the same 20-tool chain in 22.4 s vs 30.4 s.
• For our daily Hermes traffic the native lane is the better engine on four Sparks. The EXL3 lane's win is real but it lives in the one regime an agent harness does not run in. Both were on the lab gateway on October 18 so people could pick per job (since October 19 Lane B is stopped; see Use them).

Figure 1. Eight GB10 nodes on one CRS804 with two 200 GbE RoCE rails each. Lane A (Sparks 1–4) runs TensorFold's Zig engine with LibertAI's EXL3 3.5 bpw pack; Lane B (Sparks 5–8) runs SGLang's dsv4.1 branch with the native checkpoint via knapcio's TP4 profile. Each lane exposes an OpenAI-compatible endpoint on its head node and both are catalogued behind the Forge gateway next to the GB300 Station lane.

Setup

Harness for everything below: Tony's v41bench_tf.py prompt set v1 over /v1/completions, T=0, thinking off, token counts from the server's usage block; long_check.py needles at 20K/131K/300K with a random 8-character key; knapcio's qeval.py for output correctness; Hermes' own 20-hop file-chain gauntlet for agent behaviour. Cold boots the same day, one run per cell unless stated. Same prompts, same bytes, both lanes.

Decode: the crossover

Figure 2. Aggregate decode throughput over all eight categories. A leads at C1–C3, B from C4; at C6 B delivers 263 tok/s to A's 223. TTFT at C1: A 0.17 s, B 0.22 s.

Figure 3. Per-stream decode at C1 by prompt category. The 3.5 bpw pack is faster everywhere, by the most on the long-output categories (reasoning, prose, narrative) where bytes-per-token dominate.

Why the curves cross: TensorFold's win is memory traffic — a 3.5-bit matrix is read in fewer bytes per decoded token, and at one stream the GB10 is bandwidth-bound. As streams stack, SGLang's RoCEnante all-reduce and the b12x MoE path amortise better and the native lane pulls ahead from C4. Neither lane is near its 16-stream ceiling here; the crossover is about engines, not weights.

Prefill: not close

Figure 4. Cold prefill on unique prompts, 1-token reply. The native SGLang lane is 1.3–2.0× faster at the ladder lengths and holds ~4.6K tok/s at 131K. B's 300K bar is the median of three cold runs (first run 1,791, then 2,648 and 3,029).

The 2,950-token row for B uses its cold first-run value (3,654); in the full ladder that same prompt had already been seen and prefix caching returned 12,542 tok/s, which is a cache hit, not prefill. A's 2,950 row is the full-ladder value (TensorFold has no prefix cache). All needles were found on both lanes, all runs.

What this means in seconds: a 93K-token document costs 35 s before the first token on A and 17 s on B; a 300K context costs about two minutes on A and 100–115 s on B. TensorFold's prefill is flat around 2.4–2.7K tok/s at every length — it does not degrade, it just starts low.

Quality: identical on the gate we have

knapcio's qeval.py runs 75 auto-scored tasks (code, JSON, math, format) at T=0 and compares two lanes with a paired test. A 71/75, B 71/75, the same four misses (code_interval_intersect, json_escape, json_count, math_m9); qeval.py compare reports kept 71, broke 0, fixed 0, p = 1.000. Median single-stream inside qeval: A faster, consistent with Figure 3.

We wanted a teacher-forced KL divergence between the lanes with B as the native reference. Neither boot exposes it: TensorFold's completions endpoint rejects logprobs, and SGLang's bounded-replay production boot returns 400 to prompt logprobs. The substitute we could run — greedy agreement on 40 raw-text prompts × 128 tokens — gave 0/40 exact matches with a median first divergence at word 2 (9/40 agree for ten or more words, 18/40 diverge at word 0). That is the expected signature of a 3.5-bit quant on open-ended text, and it is not a quality verdict; the quality verdict is the qeval line above. A KL comparison is the open item.

What an agent sees

Figure 5. Hermes' 20-hop file-chain gauntlet: the agent must read a chain of files, each pointing to the next, through 20 tool calls, in a fresh session per rep. Both lanes were correct 3/3 with exactly 20 tool calls; B's median wall was 22.4 s to A's 30.4 s.

This is the same lesson as the GLM-5.3 TP8 campaign: raw tok/s does not rank engines for agent traffic. Each Hermes turn pays a several-thousand-token system prompt plus the growing transcript; B prefills that 1.6–2.0× faster and that beats a 18% decode edge on 50-token tool calls every time.

Against the published numbers

What this comparison is not

• Two engines, two weight formats — it cannot separate TensorFold-vs-SGLang from EXL3-vs-native. A fair quant-only test needs the EXL3 pack on SGLang or native weights on TensorFold; neither exists today.
• One run per cell. GB10 slow-state can move a cell: B's first 300K needle came in at 1,791 tok/s and the two repeats at 2,648 and 3,029. We publish all three.
• Thinking off throughout. Reasoning-mode token budgets would change the decode/prefill mix.
• No KL between lanes (see Quality). qeval is a correctness gate, not a divergence measurement.
• Both lanes are first-boot-of-the-day configurations from their authors' recipes on our fabric. No flag sweeps on either side yet; a tuned TensorFold may close the prefill gap.

Use them

As published October 18: both lanes stayed up and were listed on the lab gateway as deepseek-v4.1-flash-spark-exl3 and deepseek-v4.1-flash-spark-native, next to the GB300 Station lane deepseek-v4.1-flash. Same OpenAI-compatible API, same 1M context, tools and streaming; thinking is off by default on the Spark lanes (send reasoning_effort or chat_template_kwargs.thinking=true) and on by default on the Station. The catalog's /llms.txt carries the measured speeds and the pick-this-when guidance from this post.

Reproduce

# Lane A — TensorFold (spark1 head; engine, kit, rank caches, Engram staged under /home/milo/tf1017)
DSV41_CONFIG=/home/milo/tf1017/config.rail2.env recipe/scripts/serve.sh start   # ~103 s to /v1/models
python3 run/tools/sanity.py http://127.0.0.1:8001/v1                               # 6/6
python3 run/tools/v41bench_tf.py --base http://127.0.0.1:8001/v1 --levels 1,2,3,4,5,6 --prefill 2000,8000,32000,64000

# Lane B — SGLang knapcio v2.4 (spark5 head; .env.tp4 MODEL_DIR -> local 1017 checkout, SKIP_PREPARE=1)
./start-tp4.sh doctor && ./start-tp4.sh share && ./start-tp4.sh pack && ./start-tp4.sh serve   # ~8 min to Ready
./start-tp4.sh smoke
python3 scripts/qeval.py run B --url http://127.0.0.1:8888/v1/chat/completions
python3 tools/long_check.py http://127.0.0.1:8888 20000 131073 300000   # base without /v1 for SGLang

Added October 19, 2026 (recipe request from Ash): the exact TensorFold version and command, read from the live lane on spark1.

# Lane A engine: bertholomus/TensorFold fork, dsv41 four-node branch, checkout 225f7005 (2026-10-18).
# Binary reports: tensorfold-native 1.0.2. Built with Zig 0.17.0 inside nvcr.io/nvidia/pytorch:26.07-py3
# (CUDA 13.3 toolkit, kernel driver 595.84). rdma_gather.fatbin ships beside the binary in engine-bin-p4/.
# Kernel kit v0.1.1: MANIFEST pins 356 files, 708,157,701 bytes, per-file sha256 (verify_kit.sh checks all).
# Site config /home/milo/tf1017/config.rail2.env: both rails NCCL_IB_HCA=TF_RDMA_DEVICES=rocep1s0f0,roceP2p1s0f0,
#   NCCL_SOCKET_IFNAME=enp1s0f0np0, GID 3 (TF_RDMA_GID=3, NCCL_IB_GID_INDEX=3), context 1,048,576,
#   KV pool/window 2,097,152, 16 streams. Engine env: TF_SEED_RANDOM=1 TF_TOP_K_MAX=56 TF_DS_2D_GU=parity
#   TF_DS_ROUND_ROWS=48 TF_DS_STREAMS=16 TF_DS_SERVED_K=solo TF_DS_WINDOW=2097152
# serve.sh start launches ranks 3,2,1 (tf-dsv41-lanes) then rank 0, each in container tp4-lane on its node.
# Rank 0 runs exactly (from docker inspect of the live container):
/engine/tensorfold-native serve /model --host 0.0.0.0 --port 8001 --name deepseek-v4.1-flash \
  --context 1048576 --parallel 16 --no-thinking --max-tokens 32768 \
  --alias glm-5.3-flash --temperature 1.0 --top-p 1.0 --top-k 20 --backend cuda
# Note for recipe checks: the lane's serving default is T=1.0 / top-p 1.0 / top-k 20;
# every benchmark request in these posts set temperature=0 explicitly per request.

Pitfalls that cost us time: Docker will not bind-mount through a symlinked path; HF shards over 50 GB need hf_xet (do not set HF_HUB_DISABLE_XET for the Engram shards); TensorFold workers die with ConnectFailed if the head is minutes late, so pull the 24 GB image on every node before the first serve; never remove the NFS exporter while a worker is still building its token map.