Benchmarks

Benchmark scorecards, methodology notes, throughput reports, and non-comparable result boundaries.

Apple Silicon 512 GB measurements now have their own index: M3 Ultra 512 Benchmarks. GB300 Station numbers live under DGX Station GB300.

• Prefill > Decode: the token volume says so, the tok/s benchmarks don't October 26, 2026 · updated 6:24 AM CDT
  90 days of agent traffic: 13.96 B prompt tokens in, 75.6 M out (185:1). The GB300 DeepSeek-V4.1-Flash lane, 7 days: 53:1 with a 91.6% prefix-cache hit rate and still 4.5 prefilled tokens per token decoded. Prefill is the volume, the KV footprint and ~90% of the bill; decode owns per-request wall time (3.5 s vs 0.24 s) only because the cache carries the rest. The tok/s fixation measures the bounded term.
• One GB300 Station + one RTX PRO 6000 vs four RTX PRO 6000s: same DeepSeek-V4.1-Flash, their harness October 26, 2026 · updated 3:36 AM CDT
  On Fraser Price's own bench.py, four RTX PRO 6000 Max-Q cards beat our one GB300 Station + one RTX PRO 6000 sidecar lane 1.5× at C1 on DeepSeek-V4.1-Flash; we are level at C4, slightly ahead at C16 aggregate, and 2.6× faster on prefill. Block verification is a real but small win (+1–5% C1 by class, no class loses, free); drafting 9 tokens is a 15–20% loss on every class we serve.
• NVIDIA Flare on Two DGX Sparks: GLM-5.3-Flash TP2, Tuned October 21, 2026
  Flare TP2 on Sparks 5–6, with diagrams: defaults 37 tok/s one user and no drafting at four users; 8 seats + capacity 48 + 3 drafts at fixed width → 78 tok/s at four, 107 at eight. 1.7× our vLLM pair's cold prefill, 6.1M-token KV, 570K needle, GSM8K-200 96.5%.
• NVIDIA Flare for agents: two rough edges, measured October 21, 2026
  Two agent-facing rough edges from the four-way run: tool calls arrive all at once (2 deltas in the final ~54 ms vs SGLang's 9 over ~500 ms), and the strict-JSON session-title request gets an instant HTTP 400 on reasoning_effort "none" — 8/8 across two runs, so sessions never get titled. The agent chain itself still ties.
• DeepSeek V4.1 Flash on four DGX Sparks: four engines, one winner October 21, 2026
  Four engines, one model, four Sparks. Winner: the vLLM fork. SGLang is the daily pick for shared use, Flare holds 5.8× the KV cache, TensorFold is the fastest single stream on a 3.5-bit quant.
• Why Four DGX Sparks Talk Slowly in the Middle: NCCL Tuning in Plain English October 26, 2026
  A thread on X showed two NCCL settings that fix a mid-size all-reduce hole on four DGX Sparks. We reproduced it on ours: a 1 MiB exchange went from 1,154 µs to 137 µs (1.36 to 11.5 GB/s) with the 256 MiB peak unchanged at ~22 GB/s. Both of our serving recipes already carried the fix; what changed is our fabric gate, which only looked at big messages. Also measured: the Spark's 10 GbE Realtek port drops packets only under an 850 kpps UDP flood; every TCP test ran 9.41 Gbit/s with zero retransmits, so it stays stock.
• Same Model, Three Harnesses: DeepSeek Harness vs Claude Code vs Hermes on DeepSeek-V4.1-Flash-1017 October 20, 2026
  Same DeepSeek-V4.1-Flash-1017 endpoint, three harnesses, 120 Terminal-Bench 2.0 trials: Pass@1 Hermes 75%, Claude Code 75%, DSH 70% — not distinguishable. Tokens and wall time are where they differ: DSH 1.23× Hermes prompt tokens and fastest lanes, Claude Code 0.74× and fewest output tokens, with a vLLM prefix-cache caveat on its speed.
• DeepSeek Flash 1017: TensorFold vs SGLang vs vLLM on 4 DGX Sparks October 19, 2026
  Third lane on the same four Sparks: a vLLM fork (Christopher Owen's spark-ds41f r6x, TileLang kernels, local build), not stock vLLM, which cannot serve V4.1 on SM121 yet. It leads SGLang at every concurrency (92 vs 85 tok/s at C1, 287 vs 263 at C6), prefills ~6K tok/s cold at 93K and reads a 300K needle in 55 s; TensorFold EXL3 still wins one stream at 100. Why Tony's 48.5 was a kernel-family gap, and nine tuning arms that all lost to Owen's defaults.
• DeepSeek Flash 1017: TensorFold vs SGLang on 4 DGX Sparks October 18, 2026
  Same model, same day, same switch: LibertAI's 3.5 bpw EXL3 pack on TensorFold (Sparks 1–4) against the native MXFP4/FP8 checkpoint on SGLang via knapcio v2.4 (Sparks 5–8). EXL3 wins single-stream decode by 18% (100 vs 85 tok/s); native wins prefill 1.6–2× and the Hermes 20-hop agent gauntlet by 26%, and leads from four streams up. Quality gate identical, 71/75 both. Now extended to a 3-way with a tuned vLLM fork lane; since October 19 Lane B is stopped (kept) because that lane runs on its nodes.
• Watt-hours per solved task: Hermes Agent on Terminal-Bench 2.0, GLM-5.3 on eight DGX Sparks October 18, 2026
  First scored run of a wall-plug energy benchmark for agents: Hermes Agent v0.22.2 driving GLM-5.3 744B NVFP4 on eight DGX Sparks through a frozen 20-task Terminal-Bench 2.0 slice. Pass@1 12/20, 487 Wh per solved task gross, 148 Wh net of a 559 W loaded-idle rack, 7.3 h at c=1. Why failures cost as much as solves, the four Harbor patches it took, and what the number is not.
• 4× RTX PRO 6000 vs One GB300 Station: DeepSeek-V4.1-Flash-1017 on the Same Grid October 17, 2026
  Same 45-case grid on two single-box builds of the native 763B checkpoint: 0xSero’s (sybil-solutions) 4× RTX PRO 6000 (SGLang, DSpark k=5, Engram on NVMe) against one GB300 Station (vLLM, UVA expert offload, pin-hot hook). Station leads single-stream decode 1.25× and prefill 3.4× with TTFT about 3× lower; the four-GPU box leads 0.86× at C4 and 0.56× at C6–C8, where the Station’s k-schedule drops to one draft token — flat k=5 recovers most of that.
• GLM-5.3 on eight DGX Sparks: SGLang vs vLLM, and a unified-memory bug October 17, 2026
  Same 744B model, same eight GhettoRak Sparks, second engine. A 13-arm SGLang v0.5.22 campaign against the vLLM lane: 4–16% slower single-stream, faster at four users with 2-step EAGLE, and every long prompt hung until one env var bounded the sparse indexer's scratch buffer. The bug, the fix, and what refuses to boot on SM121.
• Full GLM-5.3 on eight DGX Sparks: official FP8 and NVFP4 October 16, 2026
  GLM-5.3 744B tensor-parallel 8 across all eight GhettoRak Sparks, two ways. Daily lane: RadixArk NVFP4, 512K context, 25.8–35.7 tok/s single-stream once MTP guesses three tokens ahead, 61.7 at 4 users. Fidelity reference: Z.ai's official FP8, 131K, 23.1–29.9 tok/s, 2.3× closer to BF16. Why the 4-bit build first lost, and recipes for both.
• DeepSeek-V4.1-Flash on four DGX Sparks: stock vs abliterated October 13, 2026
  DeepSeek-V4.1-Flash serves from Sparks 3–6 (TP4, 1M context). Abliterated weights cost no speed and no accuracy on math, tools or long context; GPQA-Diamond with thinking on drops from 84.8% to 80.3% (p = 0.08, one run).
• I went looking for M5 kernel fruit. Apple already ate it. September 29, 2026
  MLX already routes ~98% of M5 prefill compute through NAX kernels; 4-bit GEMM within 3–15% of fp16 at ~57 TFLOPS. No dense-kernel fruit left; headroom is attention at long context and the serving layer.
• DeepSeek-V4-Flash-Vision-Exp on One GB300: Native FP4, SGLang, DSpark, 1M Context September 3, 2026
  Single DGX Station TP=1, no requantization: 2,420 tok/s at C32, exact recall to 810K, zero repetition at C64. Recipe + harness on GitHub.
• Vision-Exp thinking levels on 2× DGX Spark September 1, 2026
  Matched off/low/high/max thinking probes. Effort labels are not a ladder.
• NVFP4 vs EXL3: GLM-5.3-Flash on Two DGX Sparks August 29, 2026
  Matched cold-prefill / decode / concurrency probes of two 4-bit quants on the same Spark pair.
• GLM Flash 5.3 Testing — Dual DGX Spark August 27, 2026
  Day-0 SGLang bring-up of GLM-5.3-Flash NVFP4: GB10 memory trap, 20-hop gauntlet, 24.7 tok/s prose.
• DeepSeek Flash 0731 Testing: weights vs thinking level August 26, 2026
  2×2 A/B on dual Sparks: official vs ablit weights, low vs high thinking. Neither explained the tangents.
• Local LLM Testing — Terminal-Bench 2, June 2026 June 22, 2026
  A narrowed benchmark page focused on one measurement regime: Terminal-Bench core, terminus-2, native tool calling, and comparable local model results.
• GLM-5.2: Terminal-Bench Benchmark — MLX vs GGUF June 22, 2026
  GLM-5.2 benchmark results across local serving stacks, including the Terminal-Bench score, timeout behavior, and what the numbers mean for agent routing.
• DS4-F Under Three Lights: Tool Discipline, Throughput, and Hermes Fit June 25, 2026
  A three-part read on the live DeepSeek V4 Flash route: tool discipline, throughput, and whether the endpoint is a good Hermes fit.
• The Tool-Calling Benchmark: 9 Models, Local vs Cloud April 12, 2026
  Seven models, same 20 prompts, deterministic scoring. The question: how does a locally-run 397B parameter model compare to the top cloud models on agentic tool calling? The answer was surprising.
• MiniMax M2.7 vs Qwen3.5-397B vs Claude Sonnet 4.6: Tool Calling on Apple Silicon April 12, 2026
  Three models, same benchmark. Two run locally on a Mac Studio M3 Ultra. One is Claude Sonnet 4.6 via API. How close can local get to cloud on agentic tool calling?
• Making an Agentic Benchmark Modeled on Doing Agentic Benchmarks April 12, 2026
  Most benchmarks are single-shot snapshots that rot the moment you change hardware or models. Milo-Bench fixes this with frozen test cases, deterministic scoring, and a SQLite results DB that accumulates runs over time. 27 tests across 6 categories, open source.
• Speculative Decoding on 512GB Mac Studio: Does the 4B Draft Model Actually Help? April 12, 2026
  Long reasoning tasks: +58% speedup. Large-context tool calls: -88%, catastrophic. The answer depends entirely on what you are asking the model to do.