Local LLM Stack: interactive primary, isolated batch, dedicated auxiliaries
Every tier probed from the M4 on October 22, 2026 between 11:00 and 11:15 CDT. The station's :30001 and :30003 lanes refused connections, as did .55:8002.
The architecture in one paragraph
The stack is deliberately asymmetric. The fastest box is not asked to do every job; it is protected for the jobs where latency changes the experience — conversation, tool loops, robots, family agents. Batch work is large and can wait, so it lives on separate silicon where a 134k-token prefill hurts nobody. Small specialists — embedding, reranking, selection, vision — are cheaper and more predictable as dedicated services than as a general model that may reload state. Experiments do not get a new box; they borrow a production lane inside an agreed window and restore afterwards.
The governing rule is unchanged: the fastest machine is protected, not saturated. Fleet utilisation is secondary to interactive reliability.
What is serving, checked live
| Host | Endpoint | Role | State today |
|---|---|---|---|
| GB300 Station | 192.168.1.9:30006dsv41-flash-uva | Interactive text primary. vLLM, 1,048,576-token context, whole expert tensors on Grace (UVA), Engram in CPU memory. | Up. 35 ms median first token, 244 tok/s single stream, 1,172 tok/s aggregate at 8 streams. |
| GB300 Station | :30001 · :30003 | GLM-5.3-Flash lane and the older DSF lane. | Both refused connections. Swapped out, not deleted. |
| 4× DGX Spark .51–.54 | 192.168.1.51:8888deepseek-v4.1-flash | Batch and overflow. SGLang, TP=4, 1M context, RoCE fabric, container dsv41-head on image dsv41-4x-spark:canary-roce. | Up 27 hours, healthy. The far end of the CRS804 fabric. |
| 4× DGX Spark .55–.58 | .55:8002 | Formerly a vLLM TP=4 reference lane (deepseek-v4.1-flash-spark-vllm). | Idle. All four hosts answer ping; no container and no listener. Nothing is serving. |
| M5 Max .18 | :8002 · :8003 | Embedding 8B (4,096-dim) and reranker 4B. Single-model services. | Up, both return 200. |
| M5 Max .18 | :8011 · :8016 · :8021 | Vision, selection, and a key-gated 27B lane. | Up. :8021 correctly returns 401 without a key. |
The M5 ports are a catalog, not one model each
This is the detail that changes how you call them. :8011 and :8016 both answer /v1/models with the same six-model catalog — Qwen3.5-4B, a Qwen3.8-27B abliterated oQ4e build, gemma-4-26b-a4b, Qwen3-VL-30B-A3B, Qwen3-Embedding-8B and Qwen3-Reranker-4B — because they are multi-model servers reading one model directory, not one pinned process per port. A request can pay a model load inside a port; a port name is not a promise about what is resident. Only :8002 and :8003 are genuinely single-model.
Interactive lane: today's numbers
Ten-sample probes against the live :30006 endpoint, all from the M4 during business hours on a shared lane. Percentages and rates are defined below; nothing here is a leaderboard entry.
| Measure | Today | How it was taken |
|---|---|---|
| First token, idle | 35 ms | median of 10 streaming requests, 48-token cap, first streamed delta |
| Short turn complete | 181 ms | median of 10 × 40-token turns, reasoning on |
| Single-stream decode | 244 tok/s | one 400-token completion, completion tokens ÷ wall |
| 8-stream aggregate | 1,172 tok/s | 8 concurrent 400-token completions, 147 tok/s per stream |
| Context window | 1,048,576 | max_model_len from the model list |
reasoning_effort=low produced 59 reasoning tokens, not fewer. Any caller that caps max_tokens tightly will occasionally get an empty answer and a finish_reason of length.Why long prefills stay off the interactive lane
This is the measurement the whole architecture rests on, and it was taken on October 20, 2026 — it has not been re-run today, so it stays dated rather than restated as current.
Condition on GB300 :30006 (October 20, dsv41-flash-uva, vLLM) | Measured result |
|---|---|
| Short interactive turn, idle, clean samples | 53–66 ms first token; 65 ms median; complete 40-token turn 226 ms |
| Long nonce-prefixed request (cold prefill) | 133,647 prompt tokens in 7.37 s — 18,134 tok/s |
| Short turn dispatched 0.6 s into that prefill | 3.46 s complete (15.3× idle); first token at 679 ms |
| Short turn dispatched 4.4 s in | 2.94 s complete (13.0× idle); first token at 931 ms |
| Short turn arriving as the prefill ends | 271 ms (1.2×) |
| Short turns after the long request completed | 270, 293, 217 ms — full recovery |
The practical size of the blast radius: a 134k-token prompt is about 7.4 seconds of prefill, and any conversation sharing the lane turns a 226 ms reply into roughly 2.9–3.5 seconds. At 1M tokens the wait becomes tens of seconds. That is not a throughput concern, it is a user-visible one — which is why extraction, compression, triage, kanban decomposition and curator work live on the Sparks. The Spark cluster is not idle redundancy; absorbing prefills is its day job.
Current agent routing
| Client | Normal route | Local auxiliaries | Fallback |
|---|---|---|---|
| Echo / Forge gateway | Station :30006 | Spark lane for extraction and compression; M5 :8016 for skills and titles; M5 :8021 for auxiliary vision | Spark lane .51–.54 |
| Milo / Desktop | dsv41-flash-uva — this note was written and measured on that lane | M5 :8016 for skills and profile descriptions | Cloud profile fallback |
| Family minis | Cloud Sonnet 5.5 via the Nous inference API | None active | Cloud |
| Reachy | Broken. The running brain process (pid 8899) still launches with --model_name dsf-vision-exp --responses_api_base_url http://192.168.1.9:30003/v1, and :30003 refuses connections. | None — no s2s fallback | None. Voice turns retry until the brain is repointed. |
Routing is allowed to differ per client. A local fleet is a set of capabilities, not a rule that every agent share a route, and profiles migrate deliberately rather than all at once. Reachy is the standing example of the cost: its launch script was never repointed after :30003 was swapped out, and the process has been restarting against a dead lane since.
Operating rules
- Do not swap or benchmark on the interactive lane. Whatever port the interactive primary is on that week is production for Echo and the desktop.
- Keep document-scale and background work on the Sparks. A 134k prefill turns a 226 ms turn into 2.9–3.5 s. Fast prefill still creates head-of-line blocking.
- Treat an M5 port as a warm pin, not a dedicated process.
:8002and:8003are single-model;:8011,:8016and:8021load from a shared catalog on demand. - Repoint every client when you retire a lane. Check client argv, not only the endpoint table —
:30001and:30003are dark and Reachy is still aimed at one of them. - Experiments borrow a production lane, never a new box. Promotion requires protocol, multimodal, long-context and real tool-loop gates, and every borrow ends with a verified restore.
- Verify from the client after routing changes. Endpoint health alone does not prove that Echo, Milo or a robot is calling the intended lane.
Method and provenance
- Checked live today: every listener, model id, container state and ping in this page, between 11:00 and 11:15 CDT on October 22, 2026, from the M4. Model ids came from
/v1/models; the Spark container and image fromdocker pson.51; the dark lane fromdocker psand a listener check on.55and.58; Reachy's route from its running process argv. - Measured today: the first-token, short-turn, single-stream and 8-stream figures, with the fixtures named in the table. Rates are completion tokens ÷ wall time. Prefill and decode are reported separately and never mixed.
- Carried forward, dated: the October 20 interference table. It is a property of serving a long prompt next to a short one, reproduced on both SGLang and vLLM previously, and it was not re-measured today. Earlier fleet snapshots (the two-Spark batch pair, the M3 Ultra bench) are gone from this page by design; the bench was sold in September 2026.
- Not claimed: that 35 ms or 1,172 tok/s are better than an earlier published figure. The instruments differ — different fixture, different stream count accounting — and a shared production lane at 11 AM is not the same box as a quiet one at 6 AM.