Designing an Inference Recipe Generator/Optimizer

In the last three days we published two single-GB300 recipes by hand: GLM-5.3-Flash NVFP4 with DFlash2 and DeepSeek-V4-Flash-Vision-Exp at 1M context. Both followed the same loop: read what the community has found, design a short list of experiments, run them one flag at a time, gate the winner, write down every loser, publish the repo and the post with credit to the people whose work we stood on. The second time through, the loop was obviously a machine we were operating manually. This post is the design for building it.

Two products, not one

The first design decision was noticing that "recipe generator" means two different things, and the split decides what gets open-sourced.

The harness and template are the part someone else can run tomorrow. The loop is the part that makes us fast. Both get published; only the first is a dependency for anyone else.

The two loops

An outer loop that thinks and an inner loop that runs. The outer loop is a frontier model (today Claude Fable 5.1, driven by Milo). The inner loop is a second Hermes agent on a cheaper, fast model (today GPT-5.6-Sol, driven by Miloh) whose only job is to execute experiment cards against the Station and hand back numbers. Between them sits the Hermes Kanban board, which is the handoff, the audit trail, and the human gate, all in one place.

Step 3 is the whole inner loop. The board is the only thing both agents touch; neither reads the other's memory. Cards go right, results go left, and nothing reaches the public without passing the amber gate.

Outer loop: think

1. Research. First source every night is our own ledgers, including do-not-retry.md, so the model doesn't re-propose what already lost. Then the external sources listed below. Nightly is a cheap delta (what changed since yesterday: releases, merged PRs, new checkpoints); weekly is a deep pass on the frontier model.
2. Design. Turn candidates into a ranked queue of experiment cards, each with an estimated cost, the gates it must pass, its rollback, and a "stop if" rule. The queue has to fit the window: a vLLM config change costs about 50 minutes on this box because every materially new config pays a fresh FlashInfer autotune; an SGLang flag relaunch with a warm sweep is closer to 15. Twelve hours is roughly 14 vLLM experiments or 40 SGLang ones, not a hundred of either.
3. Review. Morning after: diff results against the incumbent, update the ledgers, extend do-not-retry.md with the reason each loser lost, and draft the recipe repo, the post, and the cartoon.
4. Publish. Stops at draft-ready. A human approves everything public.

Inner loop: run

1. Claim. One card at a time. The card is a contract (model + revision, image digest, launcher and flag delta, bench contract, gates, rollback container name), never a paragraph to interpret.
2. Stop production. The production container is stopped and renamed as a receipt. Never rm.
3. Run. One delta per relaunch. Warm sweep before measuring; judge at C8 and above, because C1 in a short window is ±12% noise.
4. Gates. Tools-array smoke, needle ladder to the served context, C64 repetition audit, and a real headless agent loop with at least four replayed tool turns. The last one exists because a tools smoke passed while a prompt-encoding bug was silently nesting every argument one level deeper per turn.
5. Restore. A fixed, known-good launch script, then the same gates. If any gate fails, the runner does not troubleshoot; it leaves the receipt, falls back to the previous known-good container, and pages a human. Creative repair happens with someone awake.

Who does what

Two agents on one machine, two profiles, no shared memory. The board is the seam. That's deliberate: it keeps the audit trail in one place, and it means the design doesn't depend on our particular agents. Anyone can put any two profiles in those lanes.

The night

Twelve hours is a shift, not a sprint. It makes the slow experiments routine: the 433 GB full-size GLM-5.3 with expert offload takes about 30 minutes just to place weights before you learn anything, and every vLLM config change pays its autotune. During the window the production endpoint is down and every client that uses it (our agents, a robot, a couple of family machines) rides its configured cloud fallback. The restore step is the real risk, not the experiments: a failed restore means production is still down at breakfast. So restore is boring by design.

What an experiment card is

Conceptually, before schema:

# experiment card (concept; schema TBD)
id:            2026-09-04-03
model:         org/checkpoint @ revision
engine:        sglang | vllm   image: sha256:…
incumbent:     card id this is measured against
delta:         exactly one thing (a flag, a draft model, a quant, a KV dtype)
hypothesis:    one sentence, falsifiable
bench:         throughput contract (see tiers)   prefill ladder: yes/no
gates:         [tools, needle:served_ctx, repetition:C64, agent_loop:4+]
cost_estimate: minutes (autotune-aware)
stop_if:       e.g. "C8 aggregate < 0.9 × incumbent" or "any gate red"
rollback:      container name to restore

A runner reads that and executes. It never gets a paragraph that says "try making it faster."

Benchmarks in the loop

The loop is mostly testing serving configurations, not models. That changes what a benchmark is for. For a flag change, a quality benchmark is wasted time: the weights didn't move. For a quant swap, a drafter, an abliterated checkpoint, or a KV-cache dtype, quality is the whole question. So benchmarks are tiered by what the delta could have broken, and the card says which tier it needs.

Two rules carry over from doing this by hand. Tier 2 results only compare when cut, engine, host, and harness version all match; anything else is a separate datum. And any change that touches weights or KV has to land inside noise of the incumbent at Tier 2 or it doesn't ship, regardless of how fast it is. We don't publish speed bought with quality.

What a recipe must contain

Every recipe repo and post, in its first cut, carries:

• How we made it. Which model designed the experiments, which agent ran them, on what dates, against which image digests and which box. The loop is part of the recipe.
• The launch. Scripts, not prose. Launcher, warmup, bench, restore.
• The loop table. Every config tried, in order, with the number and the verdict. Losers included, with why.
• Cold-prefill table and the throughput contract table at the standard concurrencies.
• The gates and their results.
• The "didn't do" table. What we chose not to try and the reason, so readers don't assume it was overlooked.
• Credits. Quant authors, drafter authors, PR authors, whoever's baseline we reproduced. Nothing here is built alone.
• The cartoon. One per recipe: Milo in a BOFH t-shirt with a different line each time. This is not optional and we will not be taking questions.

Where research comes from

Rules we learned by hand

• The runner's brain is never on the box under test.
• One delta per relaunch. Two changes and you've learned nothing.
• Warm up before measuring, at every batch shape and every prompt-size class. A cold C8 looked like a cliff; it was one-time kernel autotune.
• Judge at C8+. C1 in short windows is noise.
• Never publish numbers from a config you didn't re-measure.
• Rename, don't remove. Stopped containers are receipts and instant rollbacks.
• A tools smoke is necessary, not sufficient. Run a real agent loop with replayed turns before calling anything agent-ready.
• Restore is a fixed script. Unattended troubleshooting is how a 12-hour window becomes a 20-hour outage.
• Keep the losers. The do-not-retry list, with reasons, is worth more than the winner.

Not building

• A generic benchmarking platform. The harness has one contract; that's the point.
• A web UI. The kanban board is the UI.
• Multi-box orchestration. One Station, one card at a time.
• Discord scraping.
• Autonomous publishing. The amber gate stays.

Open questions

1. Time-windowed dispatch. Cards must promote to ready at 22:00 and stop promoting at 08:00, and the restore card must run regardless of what the experiments did. We think that's a cron flipping card states plus an edge-free restore card, but we haven't verified what the kanban scheduler supports natively.
2. Concurrency cap. One GB300 means one running card. If the board can't enforce that per assignee, the runner needs a lock on the box.
3. Role profile vs. named agent. Kanban dispatches by profile name. Whether the runner is "Miloh" or a purpose-built forge-runner profile with only the tools it needs is open. The role profile is the shape anyone else can reproduce.
4. Benchmark floor for greedy equivalence. What match rate should a lossless drafter be held to when batch numerics are in play? We'd like a number from someone who has measured it.
5. Token cost. Nightly frontier research burns budget fast. The nightly delta probably runs on something cheaper; the weekly deep pass stays on the frontier model.

Next

1. Collect input on this post.
2. Verify the three kanban unknowns against the plugin source.
3. Consolidate the five bench scripts we already have (bench.py, prefill.py, needle.py, repaudit.py, spec_metrics.py) into one engine-agnostic harness with the contract baked in.
4. Seed do-not-retry.md from the two existing recipe ledgers.
5. One dry-run night with a single trivial card (re-bench the incumbent) to prove the restore path end to end, before any real queue.

This is a design note; no numbers above are new measurements. Every figure cited (autotune cost, C1 noise, the 30-minute weight placement, the encoding bug) traces to the two linked recipe posts and their repos, where it was measured. Drafted by Milo with claude-fable-5-1 (extended thinking) from a design conversation with James on September 3, 2026; the decisions are his, the diagrams are Milo's. Related: DeepSeek-V4-Flash-Vision-Exp on one GB300 · GB300 GLM 5.3 Testing · GB300 Station day one.