Local LLMs

Current fleet, model serving, deployment recipes, and local benchmark notes.

• Local LLM Stack: Current Architecture and Benchmarks May 28, 2026 · updated June 16, 2026
  Fresh live snapshot: DS4-Flash on the dual-Spark cluster is the default local agent path; M3 Ultra is the Qwen3.5-397B test bench; M5 Max runs Qwen3.6 35B, Llama 3B, Qwen3-VL, embeddings, and reranking with measured endpoint speeds.
• July DS4-F Status: Patch3 350K/12 Live July 4, 2026
  Patch3 scheduler fix applied: 350K/12 is now live after winning today’s C1 probe at 56.8 tok/s; patched 1M/6 remains the long-context lane.
• July SGLang Testing: Qwen3.6 on DGX Spark July 1, 2026
  Early July SGLang results for Qwen3.6 on the DGX Spark pair: 27B-FP8 worked but was slow; 35B-A3B-FP8 was much faster and scored better, but both failed the sleeper-injection safety gate.
• June DS4-F Testing June 30, 2026
  Archive summary of the DS4-F work: June Aiden 393K results plus July 1 DSpark speed numbers — current 200K/8 route, c8 209.66 tok/s, and 206.56 tok/s soak.
• June MiMo Testing June 30, 2026
  Updated July 1: MiMo stayed off-route. The upstream reproduce exposed the real gap — our Sparks saw 954K KV tokens versus Tony/Karol's 2.17M+ pool — so DS4-F remains default.
• DeepSeek V4 Flash on Dual DGX Spark: What Broke, and the Recipe That Works May 27, 2026
  149 GB model across two 128 GB nodes. TP=2 over 200 Gbps QSFP56. MTP speculative decoding (1.76× speedup), 200K context, thinking mode, tool calling. Full YAML recipe, the six things that broke, and measured performance — 44.5 tok/s decode, 612K KV cache.
• Local LLM Fleet: June 2026 June 5, 2026
  A dedicated fleet topology snapshot: which boxes run agents, which run inference, and how the local model routes fit together.
• The Lab Bench Report: Our Local LLM Fleet, Measured June 3, 2026
  Echo probes every endpoint on the fleet, measures tokens/sec, catalogs what's broken, and documents everything we built on top of Hermes Agent. Now updated with the dual-Spark DeepSeek V4 Flash cluster (~37 t/s) and the Kimi K2.6 spec-decode results. With architecture diagram.
• Running GLM-5.2 MXFP4 on an M3 Ultra with MLX June 18, 2026
  A practical recipe for loading and serving the 368 GB GLM-5.2 MXFP4 model on a 512 GB M3 Ultra, with the caveats that mattered.
• Qwen3.6-27B: SGLang FP8 + NGRAM vs vLLM NVFP4 + MTP — Two Sparks, Two Stacks May 27, 2026
  We ran the same benchmark on two serving stacks: SGLang FP8 + NGRAM on Spark 1, vLLM NV-FP4 + MTP on Spark 2. NV-FP4+MTP wins single-user throughput by ~2x (23 t/s vs 13 t/s). The gap is almost entirely speculative decoding quality, not quantization.