← al-engr · related: V6 receipt trail
July 9, 2026 · polished July 10, 2026 after the routing call was applied live
Hermes Bench v7 · OAuth / subscription routes only
Which model should Hermes use today?
Route by job, not by brand. Hermes Bench v7 was the hard OAuth agentic reliability cut that turned a Grok canary into a default. The short answer is below; the scorecard and break table are the receipts.
The decision
Use Grok 4.5 as the default autonomous Hermes model today when the OAuth route is healthy. Use Grok 4.3 for cheap/fast background work. Keep GPT-5.5 as the conservative verifier/fallback. Do not promote GPT-5.6 Sol yet.
| Hermes role | Model | Why |
|---|---|---|
| Default autonomous Hermes work | Grok 4.5 | Best default today. Strongest router signal in v7 (3/3), best hidden-pass count (15/18), and repeated ties with GPT-5.5 on prior Hermes agentic gauntlets. |
| Cheap / fast background work | Grok 4.3 | Same 14/18 all-pass tier as GPT-5.5 and Grok 4.5, with the lowest wall time (1484.70s). Do not use it as the primary scheduler-sensitive route yet. |
| Conservative verifier / fallback | GPT-5.5 | Still the safer fallback for hard terminal-style repair and moments when Grok routes show scheduler/API weirdness. Also the better stale/destructive-note lane in earlier V5/V6 work. |
| Candidate only | GPT-5.6 Sol | Not promoted. Launch-day v7 scored below GPT-5.5 (13/18) and failed router 0/3. |
Applied live on July 10, 2026 — this is no longer just a blog conclusion. Default Hermes now runs grok-4.5 on xai-oauth.
| Hermes surface | Live route |
|---|---|
| Main agent | xai-oauth / grok-4.5 |
| Fallback provider | openai-codex / gpt-5.5 |
| Delegation / subagents | openai-codex / gpt-5.5 |
| Approval · MCP · profile_describer · session_search | openai-codex / gpt-5.5 |
| Vision · compression · curator · background_review | xai-oauth / grok-4.5 |
| skills_hub · title_generation | local M5 qwen3.6-35b-stock4 |
Why this is the answer
Top router result in v7: 3/3. Best hidden-pass count: 15/18. Repeatedly tied GPT-5.5 in prior Hermes agentic gauntlets, then earned the default after the OAuth reliability cut.
Tied the 14/18 all-pass tier and finished fastest: 1484.70s. Scheduler was weak, so it is not the primary route.
Steady baseline. Better fit when the task looks like hard terminal repair, or when Grok routes are acting weird. Still the conservative high-stakes lane.
Launch-day route works, but the score did not: 13/18, below GPT-5.5, with router 0/3.
Scorecard
Each model ran 18 attempts: 6 Hermes-shaped task families × 3 attempts. Public tests passed for every model. The differences came from hidden tests and task-specific reliability.
| Model | All-pass | Hidden pass | Public pass | Safety fails | Total wall | Mean attempt |
|---|---|---|---|---|---|---|
| Grok 4.5grok45 · xai-oauth | 14/18 | 15/18 | 18/18 | 0 | 2846.32s | 158.13s |
| Grok 4.3grok43 · xai-oauth | 14/18 | 14/18 | 18/18 | 0 | 1484.70s | 82.48s |
| GPT-5.5gpt55 · openai-codex | 14/18 | 14/18 | 18/18 | 0 | 2089.23s | 116.07s |
| GPT-5.6 Solgpt56 · openai-codex | 13/18 | 13/18 | 18/18 | 0 | 2059.53s | 114.42s |
Where each model broke
The benchmark separated models mostly on router and scheduler behavior. Green means all three attempts passed. Red means none did.
| Task | Grok 4.5 | Grok 4.3 | GPT-5.5 | GPT-5.6 Sol |
|---|---|---|---|---|
| Router policyrouter | 3/3hidden 3/3 | 2/3hidden 2/3 | 1/3hidden 1/3 | 0/3hidden 0/3 |
| Scheduler locksscheduler | 0/3hidden 0/3 | 0/3hidden 0/3 | 1/3hidden 1/3 | 1/3hidden 1/3 |
| Migration safetymigrations | 2/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 |
| Config mergeconfig_merge | 3/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 |
| Payment reconcilereconcile | 3/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 |
| TODO injection guardinjection_guard | 3/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 | 3/3hidden 3/3 |
What this means in practice
- Default session work can run on Grok 4.5 without treating it as a toy route.
- Subagents and high-stakes checks stay on GPT-5.5 so the cheap/fast lane is not also the verifier.
- Scheduler-sensitive work is still a weak spot for both Grok models in this cut; route carefully or verify.
- GPT-5.6 Sol remains interesting, but launch-day Hermes evidence does not support promotion.
What Grok said when asked to review the conclusion
I asked Grok 4.5 to review the same evidence and argue against cheerleading. It agreed with the tiering but sharpened the caveat: this is not a permanent universal winner. It is today’s routing call for Hermes-shaped OAuth/subscription tasks. Milo H and Grok reviewed the same scorecard; the public conclusion is role-based, not a crown.
Do not over-claim this
| Claim | Safe version |
|---|---|
| “Grok 4.5 is always best.” | No. It is the best default for normal Hermes autonomous work today when the OAuth route is healthy. |
| “Grok 4.5 is fastest.” | No. In this v7 run Grok 4.3 was fastest by wall time. |
| “GPT-5.6 is an upgrade.” | Not yet. The launch-day Sol route worked, but it did not beat GPT-5.5 here. |
| “This is pure model quality.” | No. This measures Hermes through OAuth/subscription routes, including provider behavior and latency. |
| “V6 already settled the default.” | No. V6 promoted Grok to real canary work. V7 plus live application is what made it the default autonomous route. |
Protocol in one paragraph
All candidates used Hermes OAuth/subscription routes with direct API-key environment variables stripped. Route proof verified GPT-5.5, GPT-5.6 Sol, Grok 4.3, and Grok 4.5 before scoring. The harness used two-phase interrupted repo fixtures, stale-note injection, public tests, delayed hidden tests, and safety checks for test-file modification. No LLM judge scored the result. Full receipt trail for the earlier V1–V6 work lives in the Grok 4.5 V6 routing writeup.
Artifacts: sanitized JSON results and route proof. Local workspace: ~/benchmarks/hermes-bench-v7-oauth-agentic-reliability-20260710/. Route proof: route_proof_20260709_130036.json. Conclusion reviewed with Grok 4.5 over the same evidence; final edited/deployed by Milo H. Live Hermes routing applied July 10, 2026.