Watt-hours per solved task: Hermes Agent on Terminal-Bench 2.0, GLM-5.3 on eight DGX Sparks

TB2.0-abbrev · GLM-5.3 NVFP4 · Hermes v0.22.2 · SGLang v0.5.22 · 8× GB10 @ 2200 MHz cap
Pass@1 12/20 · 487 Wh/solved gross · 148 Wh/solved net @ c=1

Yesterday's SGLang-vs-vLLM post ended on a problem: raw decode speed did not predict how fast the model was as an agent. vLLM won single-stream tokens per second; SGLang won the 20-step Hermes tool chain by 30%. So "how fast and how capable is this model inside Hermes" needs a different instrument than tok/s, and because this lab runs its own hardware on metered plugs, the cost axis can be the one no API benchmark can publish: watt-hours per solved task, measured at the wall.

This post is the first scored run of that instrument. It is one engine, one model, one rep, twenty tasks. It is labelled abbreviated and carries its confidence interval for a reason. What it establishes is the protocol and the plumbing; the comparisons (vLLM on the same weights, the GB300 Station, the 4× RTX PRO 6000 box) come next and will use exactly this file.

What the number is

• Tasks: a seeded, category-stratified 20-task slice of Terminal-Bench 2.0 (89 tasks; 10 medium, 10 hard here), run through Harbor's hermes installed-agent adapter. Each task is a fresh container; Harbor's tests grade the final container state. The slice, both run orders, and a disjoint 20-task held-out slice are frozen in protocol-v1.json, sha256 7bd7b224b3034cb8….
• Scaffold: Hermes Agent v0.22.2 (stable channel, d923a5eb48), installed inside each task container by the stock installer, hermes --yolo chat -Q, toolset hermes-cli only, memory / skills / MCP / resume / checkpoints off, reasoning_effort: low, auxiliary models (compression, titles, vision) pinned to the same endpoint so nothing leaves the box.
• Model and lane: GLM-5.3 744B NVFP4 (RadixArk build) on SGLang v0.5.22-cu130, TP8 across eight DGX Spark GB10 over 200G RoCE, the published s11 recipe unchanged: 524K context, spec 4/1/5, chunk 4096, four slots, GPU clocks capped at 2200 MHz.
• Energy: the eight Sparks, both CRS804 switches and the UniFi switch, each on its own ~5 s smart plug, integrated by trapezoid from Prometheus over the whole run from the first trial's start to the last trial's end, failures and timeouts included. Idle is measured twice, ten minutes before and ten minutes after, with the lane loaded and no requests; the two differ by 1.3%.
• Caps: Terminal-Bench's own per-task agent_timeout_sec (15–60 min on this slice). No custom wall-clock cap, because a time cap would fold speed into the capability number, which is the confound this is meant to separate. Tasks that hit the cap count as attempted and are listed.
• Concurrency: c=1. One task at a time, which is how one person uses the lane and is the only regime where per-task energy can be attributed from timestamps.

Rep 1

Figure 1. Per-task rack energy (bars) and mean rack power during the task (dots), rep 1 in protocol order. Failed tasks ran a median 25 min against 8 min for solved ones; the 8 failures cost 2315 Wh, almost as much as the 12 solves (2561 Wh). Low dots are tasks where the model sat waiting on a long tool call (training a CIFAR net, fastText) and the rack dropped toward its 559 W loaded idle.

Run arithmetic: 337 agent-minutes and 4876 Wh inside the task windows; 100 minutes and 972 Wh of harness overhead around them (container build, a ~70 s Hermes install per task, verification), during which the rack sits at loaded idle. Gross Wh/solved charges all of it to the 12 solves; the net figure subtracts 559 W × 7.30 h and is what the work cost on top of keeping a 744B model resident. Both are honest; they answer different questions.

regex-chess is the odd row: it hit the 60-minute cap and still scored 1.0, because Terminal-Bench grades the container, and Hermes had written a passing solution before the clock ran out. That is the protocol working as written, not a loophole; it is listed with its cap.

On the engine side during the window: 542 prefill batches, 413K new prompt tokens against 15.1M tokens served from the radix cache (an agent loop re-sends its whole history every turn; the cache is why that is affordable), mean speculative accept length 2.88, and a mean decode of 25.8 tok/s while the agent was generating, well under the 36 tok/s single-stream code headline. Long contexts, low reasoning effort and tool-heavy text pull the acceptance rate down; the agent-loop number is the one that matters here.

What this number is not

• It is not "GLM-5.3 is X% capable." It is TB2.0-abbrev × GLM-5.3 NVFP4 × Hermes v0.22.2 × SGLang s11 × a 2200 MHz clock cap. Change any factor and the number moves. Twenty tasks at one rep gives a 95% interval of 40%–80%; the full 89-task tier exists for lines that need a tighter one.
• It is not comparable to Nous's Hermes Index, which runs 150 partial-credit tasks and reports dollars per task for API models. Same idea, different task set and different cost unit. The point of the watt-hour column is that it is the one a lab with the hardware can publish and an API price list cannot.
• It excludes the harness host. The task containers run on an x86 MS-01 that also hosts other services; its plug is not in the box profile. Tasks that train models inside the container do real compute there that this number does not see. For comparing engines or boxes on the same tasks it cancels; for an absolute "energy per task" it is an undercount.
• No contamination control. TB2.0 tasks may be in GLM-5.3's training data. Comparisons across engines on the same weights are immune to that; cross-model comparisons are not.
• Same-weights engine gaps are not intelligence gaps. When the vLLM line lands, a Pass@1 difference between the two engines on the same weights will be reported as a difference, not as "engine X is smarter", until parser, grammar and numerics are ruled out.

Getting Hermes to run inside Harbor

Harbor 0.24.0 lists Hermes as an installed agent, and none of its trials completed against current Hermes until four things were patched in the runner image (all in patch_harbor.py, candidates for an upstream PR):

1. The adapter verifies the install with hermes version; Hermes removed that subcommand in August in favour of hermes --version.
2. The adapter exports HERMES_HOME=/tmp/hermes after running install.sh. Hermes's package manager now commits the dependency environment under HERMES_HOME at install time, so the later hermes call found "no dependency environment is committed for this install". Exporting first fixes it. --skip-browser --skip-computer-use also keeps the per-task install at ~70 s and avoids a cua-driver verification failure in slim images.
3. harbor agent schema hermes advertises a config option that the adapter then rejects ("does not support config"). A small merge hook (env var → deep-merged into the generated config.yaml) is how the reasoning effort and the auxiliary pins got in.
4. Running Harbor itself in Docker on the MS-01 needs the host Docker socket, the docker compose plugin in the runner image, and --user so job directories are not root-owned.

Protocol order inside Harbor: a dataset directory of NN-task symlinks, because Harbor walks a dataset path in sorted order and a repeated -p takes only the first.

Next

1. Rep 2 in the second seeded order (it is queued and waiting for the rack).
2. The same slice on the vLLM sibling recipe, same weights, same box: a paired comparison (McNemar on pass/fail, paired Wh difference) that needs far fewer tasks than comparing two models.
3. DeepSeek-V4.1-Flash on the GB300 Station, once its plug reports in Prometheus. The Station runs experts offloaded to Grace memory, so a GPU-only reading would be wrong, not just incomplete; no plug, no line.
4. c=4 on the full 89 tasks for a headline line, only after the slice shows c=1 and c=4 pass sets agree under turn and token caps.

Reproducibility

Protocol: hermes-bench/energy/protocol-v1.json (sha256 7bd7b224b3034cb84a46c29dad5ccb90058fe7493f17a720e2721a17d963c0de). Slice: build-cython-ext, build-pov-ray, caffe-cifar-10, compile-compcert, crack-7z-hash, db-wal-recovery, dna-assembly, feal-linear-cryptanalysis, gpt2-codegolf, install-windows-3.11, multi-source-data-merger, path-tracing-reverse, protein-assembly, query-optimize, regex-chess, reshard-c4-data, torch-tensor-parallelism, train-fasttext, vulnerable-secret, write-compressor. Energy integrator: gates/energy.py (Prometheus mhr_sensor_power_w, 5 s steps, trapezoid; validated against the previous day's known windows to within 1% and the strip's own meter to 0.2%). Harbor job directory with every trial's result.json, Hermes session export and ATIF trajectory: jobs/abbrev-sglang/rep1 on the harness host. Harbor 0.24.0; Hermes v0.22.2 d923a5eb48; SGLang lmsysorg/sglang:v0.5.22-cu130; driver 595.84; kernel 6.17.0-1031-nvidia; DGX OS 7.2.3.

Credits

The Terminal-Bench team (TB2.0 and Harbor, including the Hermes adapter), Nous Research (Hermes Agent and the Hermes Index that this borrows its shape from), RadixArk (the NVFP4 build), the SGLang team, and Claude Opus 5.5 for the plan review that removed a wall-clock cap, a GPU-only Station number and a too-small sample before any of them reached a page. Prior art on energy per task: ML.ENERGY / Zeus, EcoLogits, SWEnergy. Mistakes are mine.