Local LLMs
Current fleet, model serving, deployment recipes, and local benchmark notes.
Station-specific GB300 recipes now have their own index: DGX Station GB300. M3 Ultra 512 GB measurements: M3 Ultra 512 Benchmarks. This page still lists them among the rest of the local-LLM posts.
90 days of agent traffic: 13.96 B prompt tokens in, 75.6 M out (185:1). The GB300 DeepSeek-V4.1-Flash lane, 7 days: 53:1 with a 91.6% prefix-cache hit rate and still 4.5 prefilled tokens per token decoded. Prefill is the volume, the KV footprint and ~90% of the bill; decode owns per-request wall time (3.5 s vs 0.24 s) only because the cache carries the rest. The tok/s fixation measures the bounded term.
DeepSeek's production engine for V4.1-Flash, sorted into what a four-Spark lane can use: decoder sliding-window replay (already running here as CED), confidence-priced verification, and a prefill chunk policy that cut a short chat's wait behind a 413K prefill from 3.3 s to 1.7 s. Plus the seat-count change that lifted 32-stream throughput 50%.
I read the Prometheus counters on the lane that serves me — 19 requests, 643,682 prompt tokens, a 64.7% prefix-cache hit, 12.65 ms decode steps — and ranked seven levers against my own traffic instead of a benchmark. Six are already closed, and the engine has been logging the one that matters 54 times: every prefill shape I send misses the autotune bucket set. Two boots, both reversible.
- Kimi-K3 at 0.0086 bits per parameter, on one Spark (it is a joke) October 22, 2026
2.8T-parameter Kimi-K3 at 0.0086 bits per parameter on one DGX Spark: 16 of 896 experts per layer kept, 81 of 93 layers deleted, stock Q1_0, 3.01 GB, 83.65 tok/s. The output is token-shaped noise on purpose — the post is about the denominator, and about why a sub-1-bit claim without expert retention is a ratio nobody printed. NVIDIA Flare on Two DGX Sparks: GLM-5.3-Flash TP2, Tuned October 21, 2026 - NVIDIA Flare for agents: two rough edges, measured October 21, 2026
Two agent-facing rough edges from the four-way run: tool calls arrive all at once (2 deltas in the final ~54 ms vs SGLang's 9 over ~500 ms), and the strict-JSON session-title request gets an instant HTTP 400 on reasoning_effort "none" — 8/8 across two runs, so sessions never get titled. The agent chain itself still ties. - DeepSeek V4.1 Flash on four DGX Sparks: four engines, one winner October 21, 2026
Four engines, one model, four Sparks. Winner: the vLLM fork. SGLang is the daily pick for shared use, Flare holds 5.8× the KV cache, TensorFold is the fastest single stream on a 3.5-bit quant. - DeepSeek Flash 1017: TensorFold vs SGLang vs vLLM on 4 DGX Sparks October 19, 2026
Third lane on the same four Sparks: a vLLM fork (Christopher Owen's spark-ds41f r6x, TileLang kernels, local build), not stock vLLM, which cannot serve V4.1 on SM121 yet. It leads SGLang at every concurrency (92 vs 85 tok/s at C1, 287 vs 263 at C6), prefills ~6K tok/s cold at 93K and reads a 300K needle in 55 s; TensorFold EXL3 still wins one stream at 100. Why Tony's 48.5 was a kernel-family gap, and nine tuning arms that all lost to Owen's defaults. - DeepSeek Flash 1017: TensorFold vs SGLang on 4 DGX Sparks October 18, 2026
Same model, same day, same switch: LibertAI's 3.5 bpw EXL3 pack on TensorFold (Sparks 1–4) against the native MXFP4/FP8 checkpoint on SGLang via knapcio v2.4 (Sparks 5–8). EXL3 wins single-stream decode by 18% (100 vs 85 tok/s); native wins prefill 1.6–2× and the Hermes 20-hop agent gauntlet by 26%, and leads from four streams up. Quality gate identical, 71/75 both. Now extended to a 3-way with a tuned vLLM fork lane; since October 19 Lane B is stopped (kept) because that lane runs on its nodes. - GLM-5.3 on eight DGX Sparks: SGLang vs vLLM, and a unified-memory bug October 17, 2026
Same 744B model, same eight GhettoRak Sparks, second engine. A 13-arm SGLang v0.5.22 campaign against the vLLM lane: 4–16% slower single-stream, faster at four users with 2-step EAGLE, and every long prompt hung until one env var bounded the sparse indexer's scratch buffer. The bug, the fix, and what refuses to boot on SM121. - Full GLM-5.3 on eight DGX Sparks: official FP8 and NVFP4 October 16, 2026
GLM-5.3 744B tensor-parallel 8 across all eight GhettoRak Sparks, two ways. Daily lane: RadixArk NVFP4, 512K context, 25.8–35.7 tok/s single-stream once MTP guesses three tokens ahead, 61.7 at 4 users. Fidelity reference: Z.ai's official FP8, 131K, 23.1–29.9 tok/s, 2.3× closer to BF16. Why the 4-bit build first lost, and recipes for both. - Shrinking Kimi K3 With EXL3: A Plan for 8× DGX Spark or One GB300 October 15, 2026
Plan and estimates: EXL3 experts at 2.3 bpw (758 GiB) for 8 Sparks or 2.0 bpw (663 GiB) for one GB300; EXL3 vs EAQuant; 14–28 h estimated to quantize on GB300 + 8 Sparks. - DeepSeek-V4.1-Flash on four DGX Sparks: stock vs abliterated October 13, 2026
DeepSeek-V4.1-Flash serves from Sparks 3–6 (TP4, 1M context). Abliterated weights cost no speed and no accuracy on math, tools or long context; GPQA-Diamond with thinking on drops from 84.8% to 80.3% (p = 0.08, one run). - Mooney: squeezing a 180B model onto one DGX Spark at 2.39 bits/weight October 1, 2026
Daniel Lougen's low-bit quant rebuilt and hash-verified on our own Spark; per-category quality breakdown and an NVMe-coalescing A/B. - I went looking for M5 kernel fruit. Apple already ate it. September 29, 2026
MLX already routes ~98% of M5 prefill compute through NAX kernels; 4-bit GEMM within 3–15% of fp16 at ~57 TFLOPS. No dense-kernel fruit left; headroom is attention at long context and the serving layer. - DeepSeek-V4-Flash-Vision-Exp on One GB300: Native FP4, SGLang, DSpark, 1M Context September 3, 2026
Single DGX Station TP=1, no requantization: 2,420 tok/s at C32, exact recall to 810K, zero repetition at C64. Recipe + harness on GitHub. - A Field Guide to DeepSeek V4 Flash Thinking Levels September 1, 2026
What off/low/high/max actually do: prompt prefixes, not budgets. - Vision-Exp thinking levels on 2× DGX Spark September 1, 2026
Matched off/low/high/max thinking probes. Effort labels are not a ladder. - DeepSeek-V4-Flash-Vision-Exp on Two DGX Sparks August 31, 2026
Mia Vision-Exp recipe on Funland: native image through vLLM on 2× DGX Spark, 1M context. - NVFP4 vs EXL3: GLM-5.3-Flash on Two DGX Sparks August 29, 2026
Matched quant faceoff on one Spark pair; cold-prefill protocol included. - GLM-5.3-Flash EXL3 on 2× DGX Spark — August 28, 2026
- GLM-5.3-Flash DFlash2 on 2× DGX Spark August 28, 2026
Pinned Tony DFlash2 + Inco drafter on the Spark pair: mixed C1 41.5 tok/s, 20/20 tools, 98K retrieval, recipe and rollback. - GLM Flash 5.3 Testing — Dual DGX Spark August 27, 2026
Day-0 SGLang bring-up of GLM-5.3-Flash NVFP4: GB10 memory trap, 20-hop gauntlet, 24.7 tok/s prose. - DeepSeek Flash 0731 Testing: weights vs thinking level August 26, 2026
2×2 A/B on dual Sparks: official vs ablit, low vs high thinking. - Local LLM Fleet: August 2026 August 24, 2026
Fresh topology snapshot with live endpoint probes: dual-Spark DS4-F Keys-anchored agent lane, M5 Max vision and retrieval, M3 Ultra voice lane. - Funland DS4-F is on Keys’ anchored 0731 ablit August 23, 2026
Current dual-Spark 0731 recipe with Keys anchored tensors. - Milo-Ark: A local AI repo June 18, 2026 · updated September 19, 2026
55 checksummed trees, 16.38 TB on miloshare. Penclaw Hub-complete; verify still running. - Local LLM Stack: interactive primary, isolated batch, dedicated auxiliaries May 28, 2026 · updated October 22, 2026
One protected interactive lane on the GB300; four Sparks serving batch on :8888 and four idle; five M5 auxiliary ports. October 22: 35 ms first token, 244 tok/s single stream, 1,172 tok/s at 8 streams. - July DS4-F Status: Patch3 350K/12 Live July 4, 2026
Patch3 scheduler fix applied: 350K/12 is now live after winning today’s C1 probe at 56.8 tok/s; patched 1M/6 remains the long-context lane. - July SGLang Testing: Qwen3.6 on DGX Spark July 1, 2026
Early July SGLang results for Qwen3.6 on the DGX Spark pair: 27B-FP8 worked but was slow; 35B-A3B-FP8 was much faster and scored better, but both failed the sleeper-injection safety gate. - June DS4-F Testing June 30, 2026
Archive summary of the DS4-F work: June Aiden 393K results plus July 1 DSpark speed numbers — current 200K/8 route, c8 209.66 tok/s, and 206.56 tok/s soak. - June MiMo Testing June 30, 2026
Updated July 1: MiMo stayed off-route. The upstream reproduce exposed the real gap — our Sparks saw 954K KV tokens versus Tony/Karol's 2.17M+ pool — so DS4-F remains default. - DeepSeek V4 Flash on Dual DGX Spark: What Broke, and the Recipe That Works May 27, 2026
149 GB model across two 128 GB nodes. TP=2 over 200 Gbps QSFP56. MTP speculative decoding (1.76× speedup), 200K context, thinking mode, tool calling. Full YAML recipe, the six things that broke, and measured performance — 44.5 tok/s decode, 612K KV cache. - Local LLM Fleet: June 2026 June 5, 2026
A dedicated fleet topology snapshot: which boxes run agents, which run inference, and how the local model routes fit together. - The Lab Bench Report: Our Local LLM Fleet, Measured June 3, 2026
Echo probes every endpoint on the fleet, measures tokens/sec, catalogs what's broken, and documents everything we built on top of Hermes Agent. Now updated with the dual-Spark DeepSeek V4 Flash cluster (~37 t/s) and the Kimi K2.6 spec-decode results. With architecture diagram. - Running GLM-5.2 MXFP4 on an M3 Ultra with MLX June 18, 2026
A practical recipe for loading and serving the 368 GB GLM-5.2 MXFP4 model on a 512 GB M3 Ultra, with the caveats that mattered. - Qwen3.6-27B: SGLang FP8 + NGRAM vs vLLM NVFP4 + MTP — Two Sparks, Two Stacks May 27, 2026
We ran the same benchmark on two serving stacks: SGLang FP8 + NGRAM on Spark 1, vLLM NV-FP4 + MTP on Spark 2. NV-FP4+MTP wins single-user throughput by ~2x (23 t/s vs 13 t/s). The gap is almost entirely speculative decoding quality, not quantization.
Flare TP2 on Sparks 5–6, with diagrams: defaults 37 tok/s one user and no drafting at four users; 8 seats + capacity 48 + 3 drafts at fixed width → 78 tok/s at four, 107 at eight. 1.7× our vLLM pair's cold prefill, 6.1M-token KV, 570K needle, GSM8K-200 96.5%.