New Local LLM Stack

Today we rebuilt the local LLM stack on the M3 Ultra. The old setup was straightforward: one model (DeepSeek V4 Flash) doing everything. The new setup is more interesting: four specialized services, each doing one thing well, with the heavy lifting offloaded to the DGX Spark cluster.

Here is what changed, why, and how it is wired together.

The Architecture

The Stack

Total: 435 GB used, 77 GB free on M3 Ultra (512 GB). The K2.6 instance alone is 366 GB — fitting it required raising the GPU wired-memory limit (see below). All services persist across reboots via macOS LaunchAgents/LaunchDaemons.

Making 366 GB Fit: the Wired-Memory Limit

Loading a 366 GB model on a 512 GB Mac sounds trivial — there's headroom. It isn't. macOS caps how much unified memory the GPU may wire (lock resident for Metal) at roughly 75% of physical RAM by default — about 384 GB on this box. The static weights fit under that, but the dynamic KV cache grows with context, and once the working set crosses the wired ceiling Metal starts paging weights in and out instead of keeping them resident — decode throughput falls off a cliff and the box thrashes.

The fix is one sysctl, raising the wired limit to 480 GB so the full working set stays GPU-resident:

sudo sysctl iogpu.wired_limit_mb=491520

But sysctl doesn't survive a reboot. The clean answer is a LaunchDaemon that re-applies it at every boot:

<!-- /Library/LaunchDaemons/com.echo.iogpu-wired.plist -->
<key>ProgramArguments</key>
<array>
  <string>/usr/sbin/sysctl</string>
  <string>iogpu.wired_limit_mb=491520</string>
</array>
<key>RunAtLoad</key>
<true/>

Owned root:wheel, mode 644, loaded with launchctl load -w. Now the win is permanent instead of evaporating on the next restart. Lesson for the lab notebook: on Apple Silicon, a model that fits in RAM is not the same as a model that serves stably — the wired-memory ceiling, not total RAM, is the real constraint for large-context MLX serving.

DS4 Flash Delegation

The biggest change: DeepSeek V4 Flash no longer lives on the M3 Ultra. We removed the oMLX 4-bit instance from :8020, reclaiming 144 GB RAM and 149 GB disk. Text inference moved to the DGX Spark cluster, where DS4 Flash runs at native FP8 across two GB10 nodes with TP=2 and MTP speculative decoding.

Hermes routes to it via the spark-ds4 custom provider and the ds4 alias. Heavy autonomous work goes through delegate_task with the Spark as the delegation provider — the local agent stays responsive while the Spark crunches through 200K-token reasoning loops.

Hermes Wiring

The config changes were straightforward. Each service is a custom_providers entry named <machine>-<port>:

custom_providers:
     - name: m3ultra-coder-8012
       base_url: http://192.168.1.10:8012/v1
       api_mode: openai
       models:
         - /Users/jamesmeadlock/models/Qwen3-Coder-Next-MLX-4bit
     - name: m3ultra-vision-8011
       base_url: http://192.168.1.10:8011/v1
       api_mode: openai
       models:
         - /Users/jamesmeadlock/models/Qwen3-VL-30B-A3B-MLX-4bit
     - name: kimi-local-8013
       base_url: http://192.168.1.10:8013/v1
       api_mode: openai
       default_model: /Users/jamesmeadlock/models/Kimi-K2.6-mlx-DQ3_K_M-q8
       models:
         - /Users/jamesmeadlock/models/Kimi-K2.6-mlx-DQ3_K_M-q8

With shorthands for quick switching:

model_aliases:
     coder:
       model: /Users/jamesmeadlock/models/Qwen3-Coder-Next-MLX-4bit
       provider: m3ultra-coder-8012
     vision:
       model: /Users/jamesmeadlock/models/Qwen3-VL-30B-A3B-MLX-4bit
       provider: m3ultra-vision-8011
     ds4:
       model: deepseek-v4-flash
       provider: spark-ds4
     kimi:
       model: /Users/jamesmeadlock/models/Kimi-K2.6-mlx-DQ3_K_M-q8
       provider: kimi-local-8013

The vision model is also wired as auxiliary.vision — so anytime Hermes needs to look at an image, it hits the local VLM instead of burning Fireworks credits. The embedding and reranker servers are standalone FastAPI processes with their own dedicated venv. They do not appear in Hermes config; they are called directly by the skill-search MCP server at 192.168.1.10:8002 and :8003.

Why This Stack

Four specialist services plus a heavyweight on call, each doing one thing:

• Coding goes to Qwen3-Coder-Next. It is built for this — 80B of coding knowledge with only 3B active, SWE-Bench scores in the 58–74% range, excellent tool calling, served via Rapid-MLX for continuous batching and persistent prefix cache. DS4 Flash is a generalist; Coder-Next is the coding bench.
• Vision goes to Qwen3-VL. It is not the biggest vision model available, but it is the right one: 30B MoE at 4-bit gives 93 t/s with quality that is indistinguishable from the dense 32B for the kinds of images agents need to parse. (We tested moving vision to Rapid-MLX too — it works but delivers only 12 tok/s vs mlx_vlm.server’s 93 tok/s, so vision stays on mlx_vlm.) The heavier InternVL3-78B would deliver maybe 12–18 t/s for a marginal quality gain that DS4 Flash would compensate for downstream anyway.
• Embeddings + reranking are infrastructure. Together they cost ~6 GB and make skill search work. When they were down, find_skill returned connection-refused errors. Now it returns 179 indexed skills with calibrated relevance scores.
• Text reasoning moved to the Spark cluster. DS4 Flash at native FP8 with MTP on dedicated GPU silicon is faster, more stable, and leaves the M3 Ultra's unified memory free for the specialist models. The old oMLX 4-bit instance on :8020 was a stopgap, not a destination.

Security: Prompt Injection & Quarantine Architecture

This stack serves a single user on a trusted LAN, but Echo and the other agents routinely fetch content from the open web. Any page can embed hidden instructions designed to hijack agent behavior, exfiltrate data, or trigger dangerous tool calls. Research success rates for indirect prompt injection against tool-using agents hit 80–100% in adversarial testing.

"Prompt injection for agents is the SQL injection of AI." Architectural defenses, not patches. No single technique holds up alone.

The Two-Stage Quarantine Pattern

We built a research-with-quarantine skill that enforces structural isolation: a reader agent with no tool access processes untrusted content and outputs structured findings, then the main agent with full privileges acts on the findings only. The main agent never sees raw HTML.

Stage 1: delegate_task(
     goal="research X and return findings as structured JSON",
     toolsets=["safe"]
   )  → structured findings, no side effects possible

   Stage 2: main agent reads findings JSON
   → acts on sanitized data, never sees raw HTML

The safe toolset is the key. It limits the research subagent to web_search, web_extract, browser_navigate, vision_analyze, and image_generate — no terminal, no file writes, no messaging, no delegation. Even if a malicious page fully compromises the reader subagent, the worst it can do is return bad JSON. There is no path from injected HTML to executed shell commands.

Email Quarantine

We also built an email-quarantine skill applying the same two-stage pattern to email processing. A reader subagent with safe toolset (plus himalaya for email access) processes inbound messages and returns structured summaries. The main agent never sees raw email bodies — same isolation principle, different untrusted surface.

Tailscale DS4-F Access Plan

With the Spark cluster stable at 44.5 t/s on DS4 Flash, the next step is sharing access with a few collaborators (Geverson, Bob, Oscar) over Tailscale. Raw vLLM has zero auth — no rate limiting, no attribution, no content filtering, and /metrics wide open. The plan: an nginx auth proxy + Python token-counting sidecar on Forge (:8643).

Architecture

Geverson/Bob/Oscar  ->  Tailscale  ->  Forge (:8643)  ->  Spark 1 (:8000)
                        (encrypted)    +------------+      vLLM DS4 Flash
                                       |  nginx     |      (no auth, raw HTTP)
                                       |  + auth    |
                                       |  proxy     |
                                       +------------+
                                       | Python     |
                                       | sidecar    | -> token counting + logging
                                       +------------+

Why Forge? Already on the tailnet at 192.168.1.19, plenty of spare cycles. The Spark cluster stays LAN-only — external users never touch it directly.

nginx Auth Proxy

• API key auth — each user gets a unique sk-ds4-* key, validated by nginx against a static list. Keys generated with openssl rand -hex 24, distributed via private channel.
• Rate limiting — limit_req at 5 req/min per key, burst 10. One person can't saturate the cluster.
• Endpoint filtering — only /v1/chat/completions passes through. /metrics, /v1/usage, /health blocked.
• Byte-level logging — custom log_format: timestamp, api_key, endpoint, req_bytes, resp_bytes, duration_ms, status. Per-user byte totals as a rough token proxy.

Python Token-Counting Sidecar

Sits between nginx and Spark, extracts real token counts from vLLM responses. ~60 lines with aiohttp. Handles both streaming (SSE) and non-streaming:

1. Forwards request to Spark 1:8000
2. Reads response, extracts usage.prompt_tokens and usage.completion_tokens from the final chunk
3. Logs as JSONL: {"ts":"2026-05-28T14:22:01Z","user":"bob","model":"deepseek-v4-flash","prompt_tokens":2340,"completion_tokens":512,"duration_ms":4521}
4. Passes response through unchanged

An hourly cron job summarizes per-user: request count, total prompt/completion tokens. Byte-level tracking from nginx is a fallback if the sidecar is down.

Tailscale ACL

{
     "acls": [
       {
         "action": "accept",
         "src":    ["tag:trusted-collaborators"],
         "dst":    ["192.168.1.19:8643"],
         "proto":  "tcp"
       }
     ],
     "tagOwners": {
       "tag:trusted-collaborators": ["james@meadlock.com"]
     }
   }

Geverson, Bob, Oscar's tailnet nodes get the tag:trusted-collaborators tag. They can only reach Forge:8643 — Spark:8000 stays LAN-only.

Performance Impact

Risks & Mitigations

• Token-level prompt injection — vLLM's tokenizer maps control tokens in user input to real token IDs (pitfall #19). For single-turn inference this is manageable; for multi-turn agents, a malicious prompt could inject fake system turns. Mitigation: document for users, not fixable without patching vLLM.
• Resource contention — no fairness guarantees in vLLM. First to fill the KV cache wins.
• No content filtering — raw model output. Users are responsible for their own prompts.
• Key sharing — if a key is shared, can't distinguish users. Mitigation: trust + audit log.

Implementation Steps

1. Write Python sidecar (~/.hermes/scripts/ds4-proxy.py)
2. Write nginx config (/etc/nginx/sites-available/ds4-proxy)
3. Test locally: curl -H "Authorization: Bearer sk-ds4-test-xxx" http://localhost:8643/v1/chat/completions
4. Generate 3 API keys, distribute
5. Configure Tailscale ACLs
6. Test from a tagged tailnet node

Routing & Model Dispatch

With four specialized endpoints running across the fleet alongside the dual-Spark DS4 Flash cluster, the question becomes: who handles what? Rather than routing everything through the main reasoning brain, auxiliary tasks fan out automatically based on what they need.

The Routing Table

Ad-Hoc Frontier Mode

For genuinely hard problems — the kind where a local 149 GB FP8 model won't cut it — the brain hot-swaps to a frontier cloud model with a single config change. On Telegram this is just saying "hard mode":

hermes config set default_llm grok-4-0309-reasoning
   hermes config set delegation.model grok-4-0309-reasoning

This is intentionally manual — no auto-escalation, no surprise bills. When the task is done, flip back just as easily:

hermes config set default_llm deepseek-v4-flash
   hermes config set delegation.model /Users/jamesmeadlock/models/Qwen3-Coder-Next-MLX-4bit

The ad-hoc approach keeps things simple: one agent, two brain states, zero infrastructure.