Ornith 1.0-397B on M3 Ultra: GGUF and MLX recipes

June 29, 2026 · updated June 30, 2026 · M3 Ultra 512 GB · llama.cpp GGUF and MLX 8-bit routes

The two routes that worked

Download recipe

export HF_HUB_ENABLE_HF_TRANSFER=1

hf download bartowski/deepreinforce-ai_Ornith-1.0-397B-GGUF \
  --local-dir /Users/jamesmeadlock/models/Ornith-1.0-397B-bartowski-GGUF \
  --include "deepreinforce-ai_Ornith-1.0-397B-Q4_K_M/*"

The download fetched seven GGUF split files and landed at 225 GB locally.

Server recipe

/opt/homebrew/bin/llama-server \
  -m /Users/jamesmeadlock/models/Ornith-1.0-397B-bartowski-GGUF/deepreinforce-ai_Ornith-1.0-397B-Q4_K_M/deepreinforce-ai_Ornith-1.0-397B-Q4_K_M-00001-of-00007.gguf \
  -a ornith-397b-q4km-stock \
  --no-context-shift \
  -c 8192 \
  --parallel 1 \
  --cache-ram 0 \
  --no-cache-idle-slots \
  --no-warmup \
  -b 512 \
  -ub 512 \
  -ctk q8_0 \
  -ctv q8_0 \
  --threads 15 \
  --threads-batch 16 \
  -ngl 999 \
  --jinja \
  --host 127.0.0.1 \
  --port 8033 \
  --no-webui

Quick benchmark numbers

These are operational smoke benchmarks, not leaderboard numbers. Each row is a direct HTTP call to the local OpenAI-compatible endpoint after the model was loaded.

June 30 update: MLX 8-bit route

After the first GGUF post went live, I tested the MLX route James asked about: pipenetwork/Ornith-1.0-397B-mlx-8bit. It is a text-only MLX conversion of the same Ornith language model, quantized as 8-bit affine weights. The repo is public, non-gated, and the index reports 396.3B parameters across 91 safetensor shards.

MLX download recipe

export PATH="$HOME/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
export HF_HUB_ENABLE_HF_TRANSFER=1

hf download pipenetwork/Ornith-1.0-397B-mlx-8bit \
  --local-dir /Users/jamesmeadlock/models/Ornith-1.0-397B-mlx-8bit

That download completed in about 23m20s with hf_transfer enabled. I also cleaned up two dead/old model directories while it ran: the failed Ornith DFLASH attempt and an old Qwen3.5-397B 4-bit directory.

MLX one-shot smoke

pkill -TERM -f 'llama-server.*Ornith-1.0-397B-bartowski' || true

/usr/bin/time -l /opt/homebrew/bin/python3.14 -m mlx_lm generate \
  --model /Users/jamesmeadlock/models/Ornith-1.0-397B-mlx-8bit \
  --prompt 'Output exactly this string and nothing else: MLX_OK' \
  --max-tokens 128 \
  --trust-remote-code \
  --temp 0.0

MLX server config

/opt/homebrew/bin/python3.14 -m mlx_lm.server \
  --model /Users/jamesmeadlock/models/Ornith-1.0-397B-mlx-8bit \
  --host 127.0.0.1 \
  --port 8034 \
  --temp 0.0 \
  --max-tokens 2048 \
  --prompt-cache-size 0 \
  --prompt-cache-bytes 0 \
  --decode-concurrency 1 \
  --prompt-concurrency 1 \
  --prefill-step-size 4096 \
  --chat-template-args '{"enable_thinking":false}' \
  --trust-remote-code

This is intentionally conservative in memory and aggressive in context. --chat-template-args '{"enable_thinking":false}' avoids the hidden-reasoning budget problem. --prompt-cache-size 0 and --prompt-cache-bytes 0 prevent the server from retaining multi-GB prompt-cache entries on an already tight 512 GB machine. --prefill-step-size 4096 was the best quick setting in the 15K-token prefill sweep, and --max-tokens 2048 keeps long-context turns from turning into multi-minute junk generations.

The OpenAI-compatible mlx_lm.server response included token counts but did not report generation tok/s in usage. The tok/s figure above comes from the one-shot mlx_lm generate run.

June 30 Hermes canary: 250K context works

The practical question was not whether Ornith could answer a tiny smoke test. For James's Hermes use, a 64K profile is not enough. I raised the canary path to a 262,144-token advertised context and ran a synthetic long-context request through the same M4 localhost tunnel into the M3 Ultra MLX server.

The Hermes canary profile is now set for the 250K regime, not the old half-window behavior:

model.context_length: 262144
model.max_tokens: 2048
providers.ornith-mlx-local.context_length: 262144
providers.ornith-mlx-local.request_timeout_seconds: 3600
agent.gateway_timeout: 3600
compression.threshold: 0.98

That last line matters. The first 250K config still had compression at 0.5, which would have started summarizing around 131K tokens. Raising it to 0.98 lets Hermes actually spend the window before compressing.

My read: this is a "load the whole damn context and think once" model, not a snappy iterative tool-loop model. The right default remains max_tokens=2048; the useful thing to raise was context, not output budget.

Main-agent verdict

The speed is good enough on both routes for short outputs. GGUF Q4_K_M is the efficient route at ~31 tok/s and ~213 GiB RSS. MLX 8-bit is the quality-candidate route at 28.3 tok/s in the CLI smoke and roughly 393-397 GiB RSS after real requests.

The new fact is context. Ornith MLX can survive a 250K prompt on this M3 Ultra without additional swap growth, but it takes almost half an hour to prefill. That makes it viable for huge-context analysis and questionable for ordinary interactive agent loops. The model is useful only when the task value justifies waiting for the prefill.

What still needs proof

1. Run a real long-context Hermes task that uses the window for useful work, not just a synthetic filler canary.
2. Measure how much compression, tool schema size, and multi-turn history reduce usable task context in practice.
3. Keep it on the separate ornithcanary profile while testing. Do not replace the current default route in one move.
4. Run a full edit/test/recovery agent loop. The current numbers prove serving viability, 250K ingestion, and basic tool use, not long autonomous reliability.

What did not work

Current state

After the 250K update, the MLX 8-bit server was running locally on the M3 Ultra at 127.0.0.1:8034, reached from the M4 through a local-only SSH tunnel on the same port. The M4 tunnel is managed by a LaunchAgent; the M3 server is still the heavy process to treat carefully. The GGUF Q4_K_M server on :8033 remains the lower-memory fallback and should not co-run with this 8-bit MLX route.

I would keep both routes available: GGUF Q4_K_M as the lower-memory fallback, MLX 8-bit as the quality candidate for huge-context Hermes tests. I would not co-run either with GLM-5.2 or another giant model on the same 512 GB box.

All numbers in this post came from direct tool output during the June 29-30, 2026 test session. They are operational measurements for this machine and these serving recipes, not model-card claims.

New: Ornith 1.5-397B Q6_K on M3 Ultra (August 19, 2026).