Hermes Agent embedding/reranking on Mac tested

What Hermes uses these for

Our Hermes setup has a find_skill tool. It embeds the request with Qwen3-Embedding-8B, takes the 15 closest of about 250 skills from a prebuilt index, then asks Qwen3-Reranker-4B to score each name — description pair against the request. The skill index is rebuilt by embedding every skill description. Until today both models lived on the M5 Max; we moved them to the M4 Max so the M5 can become a travel laptop, and used the move to test the options properly.

Setup

Embedding results

The mlx-embeddings script and oMLX run the same mxfp8 weights; their vectors are identical (cosine 1.000), so only one quality run was needed. "Memory" is macOS phys_footprint after the speed run. It leaves out llama.cpp's memory-mapped weights (about 8 GB for 8B Q8_0), and the MLX numbers include MLX's buffer cache, which the old script never empties. The 95% interval is a paired bootstrap over the 300 claims.

Reranking results

Before reranking, the llama.cpp 8B Q8_0 embedder scored 0.953 (Hermes skills) and 0.883 (the same 100 SciFact claims). The per-document script was only timed: it computes the same scores as our batched server, one document at a time.

DGX Spark (GB10)

Same harness and data, client on the Spark, one idle node. llama.cpp is the official b12032 CUDA 13.4 arm64 release build; vLLM is 0.29 in pooling mode with the original bf16 weights (the reranker loaded with Qwen's documented Qwen3ForSequenceClassification override, and the client sends Qwen's reranker prompt).

• vLLM wins reranking outright on the Spark: 128 ms for a Hermes lookup and 704 ms for 20 abstracts, versus 562 ms and 1.35 s for llama.cpp on the same Spark. It batches all 15 pairs into one forward pass on a GPU that is good at exactly that.
• For embeddings it depends on traffic: llama.cpp answers one query in 48 ms (vLLM 73 ms); with 4 clients at once vLLM holds 132 ms while llama.cpp goes to 180 ms, and vLLM embeds documents in bulk 2.6× faster (28.8 vs 11.1 docs/s).
• bf16 weights did not change the quality story: the bf16 embedder scored 0.870 on SciFact-lite (llama.cpp Q8_0: 0.870–0.874), and the bf16 reranker still lowered SciFact from 0.883 to 0.859. The reranker result is the model, not the quantization.
• vLLM takes about 100 s to start versus 2 s for llama.cpp, which matters only if the lane restarts often.

What Hermes power users run

We read the Hermes docs and plugin catalog, GitHub issues and community repos, and recent X threads. Short version:

• Hermes' own retrieval is keyword-based. session_search is SQLite FTS5 with no model calls (docs), and the catalog's skill-retrieval plugin is BM25 with an optional hosted reranker that is off by default.
• Semantic search arrives with a memory provider, each with a small local embedder: Hindsight uses bge-small-en-v1.5 plus a local cross-encoder (Vectorize); qdrant-memory uses Ollama qwen3-embedding:4b (catalog); Nexus uses Qwen3-Embedding-0.6B; lancedb-suite uses Ollama nomic-embed-text; memory-zvec uses Ollama bge-m3; openltm recommends bge-m3 on llama-server --embedding (catalog); the qmd skill downloads embeddinggemma-300M plus qwen3-reranker-0.6b (docs).
• Engines: Ollama is the common default; people who tune their setup run a second llama-server just for embeddings, often on the CPU (example in issue #129323: a 26 MB bge-small-zh-v1.5 on CPU feeding Mem0). We found nobody else running an 8B embedder.
• Rerankers split people. One heavily tuned setup calls a bge-reranker-v2-m3 on llama.cpp "the largest single quality lever" for memory recall (andrewalhaj/custom-hermesagent), and a Hindsight guide pairs Qwen3-Embedding-0.6B with the same reranker (dev.to). Hermes' skill retrieval and hermes-lcm ship reranking off.

That fits our result: a reranker earns its keep behind a weak or noisy first stage (keywords, or hybrid search over messy memory facts). Behind a strong 8B embedder over a few hundred clean skill descriptions it had nothing left to fix.

Five things we learned

1. On an M5, the llama.cpp version is the biggest speed lever. b8980 logs "the tensor API is not supported in this environment - disabling" on the M5 Max; b12032 logs has tensor = true. llama-bench, 8B Q8_0, 512-token prompts: 986 → 2,985 tokens/s on the M5, 924 on the M4 with either build. Homebrew had left us about 3,000 builds behind.
2. Reranking is prompt processing, so batching wins. A Hermes lookup is 15 short pairs of about 100 tokens. The old script ran 15 forward passes. Ours prefills the shared prefix (system + instruction + request) once, copies its cache to every document, and scores all 15 in one batched pass. oMLX also batches and lands in the same place. llama.cpp batches too, but scores each pair as its own sequence, so the shared prefix is computed 15 times.
3. MLX servers need their buffer cache capped. Every rerank request has a new batch shape, so MLX's cache never reuses a buffer and grows. Our first version reached 103 GB on the 128 GB M5; the old per-document script reached 50 GB on the M4 in this run. One mx.clear_cache() per request (plus mx.set_cache_limit) holds ours at about 5 GB with no speed loss.
4. Quantization barely matters for embeddings. Q8_0, mxfp8 and Q4_K_M are within about 0.004 nDCG@10 of each other on SciFact-lite. Model size matters a lot (0.6B: 0.797).
5. Measure the reranker before you pay for it. With a strong 8B embedder in front, the 4B reranker cost time and did not help on either task we ran. That may not hold for other data or a task-specific instruction, but it is cheap to check and we had never checked.

What we run now

Caveats

• This was a quick run (about an hour), not a full study: one speed pass per Mac, SciFact-lite instead of full SciFact, 100 claims for reranking, and only 40 Hermes requests.
• SciFact-lite scores are higher than full-SciFact leaderboard scores because the corpus is smaller. Use them to compare rows here, not against MTEB.
• The M4 Max was not idle: it runs our voice stack and its desktop, and its GPU was busy in the background during the speed pass, so its absolute numbers are pessimistic. Every M4 row saw the same conditions.
• What embedder-only gives up: after we switched, a greeting or off-topic request still returns skills with cosine scores around 0.6. The reranker scored those near zero, which made "no skill fits" easy to detect. Our test sets only contained requests with a correct skill, so they could not measure this.
• The DGX Spark ran each config once on an idle node; the vLLM image is a vLLM 0.29 build we already had on the Sparks, not a stock release.
• The embedder used Qwen's SciFact instruction for queries; the reranker used the default Qwen3 reranker instruction (the instruction llama.cpp's GGUF template hard-codes). As a check we re-ran our reranker with the SciFact instruction on the same 100 claims (on the M4): 0.847, lower than the default's 0.860.

Reproduce

Harness: rigor.py (HTTP, engine-agnostic: quality = nDCG@10 / Recall@100, speed = latency percentiles), run_cfg.py (starts one engine config, waits for readiness, runs suites, records memory, stops it), and rerank_server_fast.py (the prefix-batched Qwen3 reranker with the same /v1/rerank API).

llama-server -m Qwen3-Embedding-8B-Q8_0.gguf --embedding --pooling last -ngl 99 -fa on -c 16384 -np 8 -b 4096 -ub 4096
llama-server -m Qwen3-Reranker-4B.Q8_0.gguf --reranking -ngl 99 -fa on -c 32768 -np 16 -b 4096 -ub 4096
omlx serve --model-dir <dir> --max-concurrent-requests 8 --memory-guard off
python rerank_server_fast.py --model mlx-community/Qwen3-Reranker-4B-mxfp8 --port 8003   # MLX_CACHE_LIMIT_GB=2 default