Local LLM Stack: interactive primary, isolated batch, dedicated auxiliaries

Created Last updated
Living architecture note · James Meadlock & Milo (James's AI agent) · written with gpt-5.6-sol via OpenAI Codex · runtime-managed reasoning
Current call: split the fleet by workload shape. Interactive agent traffic goes to the GB300. Large or background prompts stay on the Spark pair so they cannot stall Reachy or the family agents. Small specialist jobs stay on dedicated M5 services. Experiments stay on the M3 Ultra.
1
interactive primary
1
batch + Echo failover lane
5
dedicated M5 services
1
isolated test bench

Contents

  1. The architecture
  2. What is serving
  3. Why each workload lives there
  4. Why heavy auxiliary work stays on Spark
  5. Current agent routing
  6. Operating rules

The architecture

The stack is intentionally asymmetric. The fastest box is not asked to do every job. It is protected for the jobs where latency changes the experience: conversation, tool loops, robots, and family agents.

Local inference, September 3, 2026 Workload isolation is the architecture CLIENTS Echo · Reachy · family minis Milo local auxiliary calls Different routes, one local fleet DGX STATION GB300 :30003 · dsf-vision-exp · 1M ctx Interactive text + native vision Echo, Reachy, family-agent primary LOW-LATENCY PRODUCTION 2× DGX SPARK :8888 · DSF-Vision · 1M ctx Large and background prompts Echo failover, same model family BATCH + FAILOVER interactive heavy auxiliary work M5 MAX — SPECIALIST AUXILIARIES Five services · each with one stable role :8002 Qwen3 Embedding 8B embed :8003 Qwen3 Reranker 4B rerank :8016 Qwen3.5 4B select :8011 Qwen3-VL 30B local vision :8021 Qwen3.8 27B + MTP dense/VL SMALL JOBS DO NOT EVICT EACH OTHER M3 ULTRA 127.0.0.1:8035 · 262K ctx Ornith-1.5-397B-Q6_K Manual, loopback-only testing Never in a default route ISOLATED BENCH specialist calls Production stays boring: interactive, bulk, specialist, and experimental work remain separate.

What is serving

Every endpoint below was checked live on September 3, 2026. A listener or model-discovery response confirms service state; it is not presented as a fresh quality benchmark.

HostEndpointServing roleWhy it exists
DGX Station GB300:30003
dsf-vision-exp		Interactive text and vision primaryBest latency and throughput; one 1M-context model covers normal agent and robot work.
2× DGX Spark:8888
deepseek-v4-flash-vision-exp		Heavy auxiliary jobs; Echo fallbackSame model family, separate silicon. Large prompts cannot delay production turns on the GB300.
M5 Max:8002 embedding 8BEmbeddingsDedicated forward-pass service; measured at 95 ms in the current check.
:8003 reranker 4BRerankingDedicated scoring service; measured at 143 ms for two documents.
:8016 Qwen3.5-4BSelection, titles, profile descriptions, Echo session searchAlways warm and isolated; five fresh calls measured 165–234 ms, 171 ms median.
:8011 Qwen3-VL-30B-A3BLocal visionVision is available without loading a model into the select process.
:8021 qwen38-27b-ablit-oq4e-mtpKey-gated 27B dense/VL laneA stronger local auxiliary when 4B selection is not enough; one model is currently loaded.
M3 Ultra127.0.0.1:8035
Ornith-1.5-397B-Q6_K		Open test benchLarge-model experiments remain loopback-only and cannot disturb production routing.

Why each workload lives there

Interactive → GB300

Conversation and tool loops care about first-token latency. Reachy and the family agents should not wait behind document-scale prefills.

Bulk/background → Sparks

Extract, compression, triage, kanban decomposition, and curator reviews can be large. Isolation matters more than maximum single-request speed.

Small specialists → M5

Embedding, reranking, and classification are cheaper and more predictable on dedicated processes than on a general model that may reload or evict state.

Experiments → M3

The bench can change models, context settings, and engines without touching an agent's default or fallback path.

The governing rule is simple: the fastest machine is protected, not saturated. Fleet utilization is secondary to interactive reliability.

Why heavy auxiliary work stays on Spark

The tempting simplification was to send Echo's five heavy auxiliary tasks to the GB300 too. The live scheduler test said no.

Condition on GB300 :30003Measured result
Short robot-style turn, idle (5 runs)176–179 ms TTFT; 177 ms median
Nonce-prefixed long request102,734 prompt tokens; 3.10 s TTFT; 33,113 prompt tok/s
Short turn admitted during that prefill2,952 ms TTFT
Short turns after the long request167–178 ms TTFT

The server is configured with --chunked-prefill-size 8192 and --max-running-requests 64, but the concurrent short request still waited almost the full long-prefill window. That turns a normal 177 ms first token into 2.95 seconds.

Therefore: Echo keeps web_extract, compression, triage_specifier, kanban_decomposer, and curator on the Spark pair. The Sparks are not idle redundancy; workload isolation is their day job.

Current agent routing

ClientNormal routeLocal auxiliary routesFallback
Echo / ForgeGB300 dsf-vision-expM5 select for small tasks and session search; Spark pair for the five heavy/background tasks; M5 :8021 for auxiliary visionSpark pair
Reachy + family minisGB300 dsf-vision-expNative vision on the same model; no separate local aux chain requiredFamily minis use cloud Sonnet; Reachy's application fallback is managed separately
Milo / DesktopCloud defaultM5 select for skills, titles, and profile descriptions; Spark for session search; web extraction on xAI; vision on GeminiManaged by the active cloud profile

This difference is intentional. A local fleet is a set of capabilities, not a requirement that every agent use the same route.

Operating rules

Measurement note: the GB300 latency-isolation probe, M5 select calls, embedding call, and rerank call were run during this September 3 revision. Model discovery and listeners were checked across GB300, Spark, M5, and M3 before publication.