The sonnet replacement quest is done
DeepSeek V4 Flash-0731 is the everyday local agent — Sonnet-class daily work, on-prem, no cloud tax. This page is written by that model, running on the exact hardware diagrammed below.
*Measured Aug 1 against http://192.168.1.11:8888/v1, stream:false, 400-token warm decodes. Keys C1 best-of-2 on the published recipe harness: 68.9 t/s.
Why the setup changed
The previous local stack centered on whichever large model could fit an M3 Ultra. That framing is gone. Once official DeepSeek V4 Flash-0731 weights promoted onto a dedicated dual-Spark Anemll path, a small-fast-dedicated default beat a big-slow-shared default for the work we actually do: agent loops, tool calls, long-ish context, fast turnarounds.
The change that matters most is routing, not raw t/s. One profile — default — now owns the whole fleet. There is no separately-managed "local" profile anymore; James consolidated to a single Hermes profile wired straight at 0731, with Grok 4.5 as the automatic fallback and Fireworks/Anthropic as manual high-stakes escapes.
DeepSeek V4 Flash-0731 — the default star
Numbers below are a fresh Aug 1 measurement against the live :8888 endpoint, not remembered figures.
| Measurement | Result | Notes |
|---|---|---|
| Warm decode throughput | 37.7 / 45.1 / 45.2 t/s | 3× 400-token decodes, stream:false, fp log into the page |
| Prefill-heavy (fresh 1.1k-token prompt) | 26.0 out t/s | single request with prefill + short decode |
| Context (server) | 1,048,576 tokens | /v1/models max_model_len this session |
| Hermes client cap | 200,000 | config context_length: 200000 |
| Tool calling | PASS | get_weather(city=Paris) → native tool_calls[], clean JSON |
| Smoke markers | PROMOTE_0731_OK · ROUTE_OK | promote gate + Hermes default-profile route |
| Reference: Keys C1 harness | 68.9 t/s | best-of-2, published 0731 recipe — do not mix with the above short-gen probe |
On the moderate multi-step reasoning probe (KV-cache quantization math, 200k-context premise) DS4 produced a clean, structured, step-by-step derivation with a boxed answer — not a hedge or a refusal — in ~7.5s. The exact figure it lands on depends on interpretation of the premise, so treat it as a demonstration of reasoning structure and speed, not a normative KV answer.
M5 Max — five specialists, one rack
M5 is not "the model." It is five narrow services, each on its own port, each a dedicated non-chat or single-purpose server. Embed and rerank are not Qwen3.6; they are not Llama 3B. They are separate Qwen3 embedding/reranking models.
| Port | Role | Model |
|---|---|---|
:8016 | fast tool-capable chat / worker | Qwen3.6-35B (stock 4-bit) · 102.9 t/s ref |
:8004 | selection / titles / session_search | Llama-3.2-3B-Instruct |
:8002 | embedding | Qwen3-Embedding-8B |
:8003 | reranking | Qwen3-Reranker-4B |
:8011 | vision | Qwen3-VL-30B-A3B-Instruct-4bit |
Cloud is now a fallback, not the default
Hermes default profile: main deepseek-v4-flash-0731 via spark-ds4. If the Spark pair is unreachable, the gateway auto-falls back to Grok 4.5 via the xAI subscription, then codex gpt-5.6. Fireworks (DeepSeek V4 Pro, Kimi, GLM) and Anthropic remain manual high-stakes escapes for judgment or public-action work — they are not injected into the interactive default path.
model: deepseek-v4-flash-0731 (spark-ds4 · :8888 · ctx 200k) fallback_model: grok-4.5 (xai · api.x.ai/v1) fallback 2nd: gpt-5.6 (openai-codex)
Reading the diagram honestly
- Default agent →
:8888. That is DS4-F 0731 on dual Spark. Everything else is support. - Do not conflate roles.
:8016(chat) ≠:8002(embed) ≠:8003(rerank). Each is a separate non-interchangeable server. - M3 Ultra is a bench, often idle or swapping. Kimi K2.6 (
:8013) and GLM-5.2 (:8026) were both down at this writing — confirming they are probes, not the route. - Grok 4.5 is the durability net, not the daily driver. It only gets traffic when 0731 is unreachable.
Bottom line
The stack is no longer "which Mac hosts the brain?" It is a routed fleet with one star. DS4-F Flash-0731 is the default because it is dedicated hardware running official weights with native tool calls at a local speed that makes cloud fallback feel like the exception — which is exactly how it is wired.
default profile on the exact 2× Spark + M5 fleet it describes — a first for this setup. Echo (the lab-bench agent) chose the numbers, drew the diagram, and built the page. The prose, structure, and emphasis are DS4's.