Same Model, Three Harnesses: DeepSeek Harness vs Claude Code vs Hermes on DeepSeek-V4.1-Flash-1017
Rankings at a glance
One row per harness. Pass@1 is the task mean over 20 tasks × 2 reps; everything after the CI is cost or speed. "r1 / r2" columns are the two lane jobs, one per rep, each 20 tasks at four trials in flight.
| harness | Pass@1 | 95% CI | timeouts /40 | engine prompt tok r1 / r2 | engine output tok r1 / r2 | tool turns / trial | agent wall / trial | lane wall r1 / r2 | prefix-cache hit r1 / r2 |
|---|---|---|---|---|---|---|---|---|---|
| Hermes v0.22.2 | 75.0% 30/40 | 57.5– 90.0 | 7 | 16.08M 15.27M | 555K 570K | 29 | 415 s | 48 min 50 min | 97% 96% |
| DSH 0.2.1-rc.2 | 70.0% 28/40 | 50.0– 87.5 | 5 | 21.17M 17.51M | 437K 400K | 29 | 337 s | 38 min 36 min | 96% 96% |
| Claude Code 2.1.303 | 75.0% 29/39 | 57.5– 92.5 | 6 +1 INFRA | 13.39M 9.82M | 364K 268K | 18 | 331 s | 52 min 39 min | 9% 11% |
The three Pass@1 values are statistically one number; with timeouts censored instead of failed they are 86.1% / 77.8% / 86.1%. Token columns are vLLM /metrics deltas over each lane job, so they include everything the harness sent — main loop, subagents, titles, compaction. Tool turns and agent wall are per-trial means from the harness and Harbor (tool turns exclude trials the harness could not report: Hermes writes no session file for a killed trial). The prefix-cache column is the reason Claude Code's wall time is not a clean harness measurement here; see Speed.
Which harness for which job
Most output, most patience
Pick when: you want the broadest stock toolset and are willing to let the agent keep working; it tied for the top Pass@1 (75.0%) and was the only lane with no 0/2 task outside the three nobody solved.
Cost: 31.35M prompt and 1.12M output tokens over two lane jobs — the most output of the three; 29 tool turns per trial.
Trade-off: slowest lanes (48/50 min) and the most cap hits (7); 2 of those still passed the verifier because the solution was on disk when the cap fired.
Fastest lanes, biggest prompt bill
Pick when: wall-clock matters and your server caches prefixes well: 38/36-minute lanes and 337 s per trial, with a 70.0% Pass@1 that is not distinguishable from the others.
Cost: 1.23× Hermes's prompt tokens (38.68M) but only 0.74× its output; ~96% of the prompt was a prefix-cache hit.
Trade-off: on a pay-per-input-token API the prompt volume is the bill; it went 0/2 on two tasks both other lanes solved (build-cython-ext, nginx-request-logging).
Leanest on tokens, slowed by the serving path
Pick when: tokens are the constraint: 0.74× Hermes's prompt tokens, 0.56× its output, 18 tool turns per trial against 29 — for the same 75.0% Pass@1.
Cost: fewest tokens sent, but 9–11% prefix-cache hit on vLLM's /v1/messages, so it recomputed 20.86M of prefill against 1.08M for Hermes.
Trade-off: its lane wall swung 52→39 min between reps; until the Anthropic path caches like the OpenAI paths, its speed on vLLM is a lower bound.
Capability: no separation
Figure 1. Pass@1 per harness with 95% task-cluster bootstrap intervals (10,000 resamples, tasks resampled with both reps). Hollow markers are the timeout-censored sensitivity analysis: 86.1% / 77.8% / 86.1%, CIs [71.9, 97.2] / [58.3, 94.1] / [69.4, 100.0]. Generated from results.json.
The primary statistic is a paired one: for each task, the difference in mean pass rate between two harnesses, averaged over the 20 tasks, tested with an exact sign-flip permutation over all 220 sign assignments, then Holm-corrected across the three pairs. Hermes − DSH is +5.0 points (CI -10.0 to +20.0), Hermes − Claude Code +0.0, DSH − Claude Code -5.0. A secondary McNemar test on (task, rep) pairs agrees (Holm p = 1.0 everywhere).
Censoring timeouts instead of counting them as failures lifts every lane to 78%–86% and still separates nothing. Two Hermes trials were scored as passes after hitting their cap (write-compressor, query-optimize): Harbor runs the verifier on whatever the agent left behind, and the protocol counts the reward, not the exit.
Cost: tokens
Figure 2. Engine-side token counts per lane job from vLLM /metrics deltas taken at lane start and end. The dim part of each prompt bar is prefix-cache hits; the bright part is prompt that had to be prefilled again. Ratios are both reps summed, against Hermes.
The token picture is the cleanest separation in the study. Over both reps DeepSeek Harness sent 38.68M prompt tokens, Hermes 31.35M, Claude Code 23.21M. Output runs the other way from what the prompt numbers suggest: Hermes generated the most (1.12M), DSH 0.84M, Claude Code 0.63M. Claude Code also made the fewest engine requests (357 and 312 per lane job, against 608/599 for Hermes and 531/530 for DSH).
Harness-reported per-trial means tell the same story from the other side: 802K in / 20.4K out for Hermes (n=33, non-timeout trials only), 951K / 20.8K for DSH (n=40), 582K / 16.1K for Claude Code (n=39). In the rep-1 gate, the harness-reported input sums landed within 2–7% of the engine's own prompt-token delta for each lane, which is what lets us trust both sets of counters.
Context, not comparison. Paweł Huryn's Bug Hunt Bench ran the same model through DeepSeek's own API with several harnesses at max effort: DSH 42.7, Pi 42.2, Claude Code 39.1, OpenCode 37.6 planted bugs fixed of 105, with DSH's mean run cost $1.84 against $1.01 for Claude Code (+82%; his note). It is a different suite, LLM-judged, on a different serving stack, so its scores do not line up with ours. The direction of the cost gap does: DSH sent 1.67× Claude Code's prompt tokens here.
Per task
Figure 3. Passes out of 2 reps for every task and harness, with each task's Terminal-Bench 2.0 agent cap in minutes. Amber boxes mark a 0/2 where both other harnesses solved the task at least once. Generated from results.json per_task.
The aggregate gaps come down to a handful of tasks. DSH went 0/2 on build-cython-ext and nginx-request-logging; Claude Code went 0/2 on large-scale-text-editing, which Hermes solved twice. Three tasks — adaptive-rejection-sampler, caffe-cifar-10, gpt2-codegolf — beat all three harnesses in every trial, and they account for 14 of the 18 timeouts. 10 tasks were solved in every scored trial by everyone. With two reps per cell, a single 1/2 is one coin flip; we read the 0/2-vs-2/2 rows as the only per-task signal, and even those are two tasks wide.
Speed, and the Claude Code caveat
Figure 4. Lane wall is Harbor job start to end for 20 tasks at four concurrent trials; agent wall is Harbor's agent-execution time per trial, timeouts included at their cap. Caveat badge values come from the prefix-cache counters in lane-summaries.jsonl.
DSH's lanes were the fastest in both reps (38 and 36 min), Hermes's the slowest (48 and 50). Claude Code's agent time per trial (331 s) is on par with DSH's (337 s), but its lane wall swung from 52 to 39 min between reps, and its mean engine request took 13.3 and 10.5 s against 7.4–7.9 s for the other two.
The reason is on the server, not in the harness. Claude Code talks to vLLM's native Anthropic endpoint, /v1/messages, with no translation shim. On that path vLLM hoists the inline system message Claude Code changes every turn into the leading system prompt, so the prefix differs on every request and the cache misses on the whole conversation — mechanism documented in vllm-project/vllm#58647. We measured it directly: Claude Code's prefix-cache hit rate was 9.4% and 11.0% in the two lane jobs, against 96.7/96.4% for Hermes and 96.0/95.6% for DSH. That changes how much prefill the GPU does, not what the model outputs, so it affects Claude Code's wall time here and not its Pass@1. Read its speed as a lower bound for this harness on a cache-friendly server.
How effort reaches the model
Figure 5. Request shape per harness from a full-body capture proxy on a trivial prompt before the scored runs. Body size includes the system prompt and tool schemas; we did not tokenize the system prompts separately, so bytes are shown instead of tokens.
"Effort=high" is three different fields on three different API surfaces, so we verified it twice. On the wire, a capture proxy showed each harness sending exactly the field above with the value we set, and nothing else changing between low and high. Behaviourally, the engine template rejects medium with HTTP 400 on both OpenAI-style paths, and low vs high on a fixed fixture moved reasoning output from 132 to 249 tokens on /v1/chat/completions and from 120 to 249 on /v1/responses. Claude Code's output_config.effort was confirmed on the wire for both low and high and is validated by the engine on /v1/messages. None of the three sends temperature, top_p or a seed, so all three run on the same server sampling defaults.
Method
| item | what we ran |
|---|---|
| Model / server | DeepSeek-V4.1-Flash-1017 on one DGX Station GB300, vLLM image vllm/vllm-openai:deepseekv41-flash-0909, the v21 recipe from the day-zero post: UVA expert offload to Grace memory, DSpark speculative decoding with the k-schedule (5 draft tokens at 1–8 sequences, 1 above), 1M context, max 24 sequences. No serving changes during the campaign. |
| Harnesses | Hermes Agent v0.22.2 (/v1/responses); DeepSeek Harness 0.2.1-rc.2, headless profile (/v1/chat/completions); Claude Code 2.1.303 (native /v1/messages, no shim). Versions constant across all 120 trials, checked by the gate. |
| What "stock" meant | Fresh home and no user state per trial; shipped built-ins on, subagents as shipped (DSH's subagent and workflow tools, Claude Code's agent tool, Hermes's delegate_task). Uniform exception: web search/fetch disabled on every lane because TB2 solutions are public (Claude Code permission deny list for WebFetch/WebSearch; Hermes web and browser toolsets disabled; DSH ships its web rows off in the headless profile). Hermes also ran with memory, skills, MCP, resume and checkpoints off — a fresh install has no user skills, and its auxiliary calls (titles, compaction) were pinned to the same endpoint. Claude Code kept its [1m] model suffix so it declares the 1M window. |
| Suite | Terminal-Bench 2.0 (laude-institute/terminal-bench-2 at 69671fba) through Harbor 0.24.0 on a 20-core x86 host; per-task agent caps as shipped, no extra wall-clock cap. |
| Fast-tier slice | 20 TB2 tasks with agent caps ≤ 20 min (320 cap-minutes total): 10 carried from the original stratified 20-task slice, 10 new, category-balanced, frozen before any run. |
| Schedule | Lanes sequential within a rep, each lane one Harbor job over 20 tasks at -n 4; lane order rotated across reps (rep 1: Hermes → DSH → Claude Code; rep 2: DSH → Claude Code → Hermes). 2 reps × 3 lanes × 20 tasks = 120 trials, all at effort=high. |
| Concurrency regime | Four trials in flight per lane. On this lane, per-stream decode is roughly 106 tok/s at four streams against 192 at one, so every harness ran slower than it would alone — equally. Cap hits are therefore higher than at c=1 and are reported, not hidden. |
| Gate | A scripted, sealed-outcome gate after rep 1 (rewards masked) checked completeness (60/60), INFRA rate ≤ 10% per lane and ≤ 5 pt between lanes, version constancy, and engine-vs-harness token agreement. It passed: 0 INFRA except one Claude Code AgentSetupTimeoutError on vulnerable-secret; versions constant; token cross-checks within 2–7%. Rep 2 launched automatically. |
| Statistics | Per-task mean difference, exact sign-flip permutation over 20 tasks, Holm over 3 pairs; 95% CIs from a task-cluster bootstrap (10,000 resamples, seed 20261019); McNemar on (task, rep) as secondary; timeout-censored sensitivity. The INFRA trial is excluded, which is why Claude Code is 29/39. |
| Design history | An adversarial methodology review of the first frozen protocol (Opus 5.5) returned SHIP-WITH-CHANGES. Its eight must-fix items — among them hashing every pinned version and config, pinning the TB2 dataset revision, task-blocked paired statistics with Holm, an explicit INFRA classification, and contamination monitoring of the shared lane — were addressed before any scored run. The protocol then went through two more frozen versions (a c=1 hard-task design, then this fast tier); each change got a new version and hash before results were read. |
What this comparison is not
- Not long-horizon. The fast tier excludes every TB2 task with a cap above 20 minutes. That is the regime where planning, subagents and context management are most likely to separate harnesses. We cut to it for time: in the original 20-task slice 6 tasks carried 480 of 755 cap-minutes, and the c=1 pilot below spent 5.8 hours of trial time on its first 22 trials. The pilot's long tasks did not separate the harnesses either, but one trial per cell proves nothing.
- Not a solo-stream speed test. At n=4 every stream decodes at roughly half its single-stream speed. That slows all three harnesses equally, but it makes absolute times and cap hits worse than you would see with one agent on the box.
- Not powered for small gaps. 20 tasks give CIs of ±15–17 points on each lane. A 5-point difference is invisible at this size, in either direction.
- Not Bug Hunt Bench. Huryn's suite is different, LLM-judged and served by DeepSeek's API; we cite it for the direction of cost, never as a score to line up with ours.
- Not a clean speed verdict for Claude Code. Its wall time carries the vLLM
/v1/messagesprefix-cache miss. Its token counts and Pass@1 do not. - One lane, one host, one model. These are three harnesses on DeepSeek-V4.1-Flash-1017 served one way. Nothing here says how they rank on a different model or a hosted API.
The hard-task pilot
Before the fast tier, the first design ran task blocks one trial at a time (c=1), all three harnesses back to back on each task. We stopped it after 22 trials over 7 blocks (5.8 hours of trial time) because the full 180-trial design would not fit the campaign window. It is a labelled pilot: different concurrency, mostly different tasks, one trial per cell, never pooled with the results above.
| task (cap) | Hermes | DSH | Claude Code |
|---|---|---|---|
dna-assembly (30m) | fail · cap hit (1802 s) | pass (348 s) | fail (278 s) |
caffe-cifar-10 (20m) | fail · cap hit (1202 s) | fail · cap hit (1201 s) | fail · cap hit (1203 s) |
compile-compcert (40m) | pass (1811 s) | pass (623 s) | pass (1171 s) |
write-compressor (15m) | pass (855 s) | pass (153 s) | pass (218 s) |
build-pov-ray (200m) | pass (110 s) | fail (167 s) | pass (51 s) |
query-optimize (15m) | pass (428 s) | pass (339 s) | pass (245 s) |
regex-chess (60m) | pass · cap hit (3605 s) | not run | not run |
effort=high cells; agent seconds in brackets. The first block (dna-assembly) also ran each harness at effort=low (DSH fail, Claude Code fail, Hermes fail), archived as out-of-protocol. Two cap hits still scored a pass (regex-chess Hermes; and in the main run, two Hermes trials), for the same reason as above.
Reproduce
Harbor 0.24.0, Terminal-Bench 2.0 at 69671fba, one lane job per harness, run from the Harbor runner container. $EP is your vLLM server; $TASKS is a dataset directory holding the 20 fast-tier tasks. The three -a targets are ours and small: HermesBundle and ClaudeCodeBundle subclass Harbor's installed hermes and claude-code agents and only replace the online install with an upload of a prebuilt install of the pinned version; DshAgent is a custom Harbor adapter around the @deepseek-ai/dsh npm package (headless profile, fresh DSH_HOME per trial). The flags and effort fields below are exactly what the scored runs used.
# Hermes v0.22.2 (/v1/responses, reasoning.effort)
HERMES_EXTRA_CONFIG_JSON='{"agent": {"reasoning_effort": "high", "disabled_toolsets": ["web", "browser"]},
"model": {"default": "dsv41-flash-uva", "provider": "openai-api", "base_url": "'$EP'/v1"}}'
# (auxiliary.vision/compression/session_summary/title/... pinned to the same model and base_url)
harbor run -p "$TASKS" -a hermes_bundle_agent:HermesBundle -m openai/dsv41-flash-uva \
--ak version=v0.22.2 -n 4 -k 1 -y --agent-setup-timeout-multiplier 2
# DeepSeek Harness 0.2.1-rc.2 (/v1/chat/completions, reasoning_effort + thinking:enabled)
harbor run -p "$TASKS" -a dsh_agent:DshAgent -m labvllm/dsv41-flash-uva \
--effort high -n 4 -k 1 -y --agent-setup-timeout-multiplier 2
# Claude Code 2.1.303 (native /v1/messages, output_config.effort; no shim)
CCM='dsv41-flash-uva[1m]'
CAP=effort,xhigh_effort,max_effort,thinking,adaptive_thinking,interleaved_thinking
echo '{"permissions":{"deny":["WebFetch","WebSearch"]}}' > cc-settings-deny.json
harbor run -p "$TASKS" -a cc_bundle_agent:ClaudeCodeBundle -m "$CCM" \
--ak version=2.1.303 --ak config=cc-settings-deny.json --ak reasoning_effort=high \
--ae ANTHROPIC_BASE_URL="$EP" --ae ANTHROPIC_AUTH_TOKEN=dummy --ae ANTHROPIC_API_KEY=dummy \
--ae ANTHROPIC_MODEL="$CCM" --ae ANTHROPIC_DEFAULT_OPUS_MODEL="$CCM" \
--ae ANTHROPIC_DEFAULT_SONNET_MODEL="$CCM" --ae ANTHROPIC_DEFAULT_HAIKU_MODEL="$CCM" \
--ae ANTHROPIC_SMALL_FAST_MODEL="$CCM" --ae CLAUDE_CODE_SUBAGENT_MODEL="$CCM" \
--ae ANTHROPIC_DEFAULT_OPUS_MODEL_SUPPORTED_CAPABILITIES=$CAP \
--ae ANTHROPIC_DEFAULT_SONNET_MODEL_SUPPORTED_CAPABILITIES=$CAP \
--ae ANTHROPIC_DEFAULT_HAIKU_MODEL_SUPPORTED_CAPABILITIES=$CAP \
--ae CLAUDE_CODE_EFFORT_LEVEL=high -n 4 -k 1 -y --agent-setup-timeout-multiplier 2The *_SUPPORTED_CAPABILITIES variables are what let Claude Code send its effort and thinking fields to a model name it does not recognise; we confirmed the field on the wire before the scored runs. The server launch is the v21 command in the day-zero post. Companion reads: DeepSeek-V4.1-Flash-1017 on One GB300 (the lane), Watt-hours per solved task (the Harbor rig and the original 20-task slice).
Credits
- DeepSeek — the model, and DeepSeek Harness (MIT), whose top contributors include @tianyicui, @LegGasai, @imccyu and @CreatixChu.
- Anthropic — Claude Code; top contributors to its public repo include @bcherny, @ant-kurt, @ashwin-ant and @bogini.
- Nous Research — Hermes Agent (MIT); top contributors include @teknium1, @kshitijk4poor, @OutThisLife and @ethernet8023.
- Harbor framework and the Laude Institute — Harbor and Terminal-Bench 2.0 (Apache-2.0); @alexgshaw, @li-boxuan, @TheMikeMerrill, @ibercovich and the task authors. Harbor shipping installed adapters for both Hermes and Claude Code made a same-rig comparison a configuration problem.
- vLLM — vllm-project/vllm (Apache-2.0), for serving OpenAI Chat, OpenAI Responses and Anthropic Messages from one process; and @robertgshaw2-redhat for #58647, which explains the Claude Code cache miss we measured.
- Paweł Huryn — Bug Hunt Bench, whose multi-harness runs on this model prompted the question.