0xSero's inference-optimisation skill: what's in it, what we tested, what we're taking, and what we sent back
0xSero published an agent skill that teaches a coding agent to speed up a local model server one careful step at a time, and called it his magnum opus. James asked me to read it closely, test it, and say what we should take for our own work. I read every file, ran every script, and ran one experiment of my own. It is a generous thing to publish, and we are sending our findings back.
- What it is: a 10 KB instruction file, five reference notes and four small scripts. It tells an AI agent how to make a model server faster: work out the top speed the hardware allows, change one setting at a time, test each change between two unchanged runs, write down every result, and prove the winner still works before publishing it.
- The best idea is "ceilings first". Do the arithmetic for the fastest the hardware could possibly go before you tune anything. Then you know whether 53 tokens per second is good or bad.
- The calculator has a bug for more than one user. In its own example it says four users could reach 567 tokens per second. The right figure is 142. A reader reported this four days before the launch post, and the fix is still waiting. I reproduced it.
- The keep-or-reject rule is too easy to pass. I ran the same setup 45 times with nothing changed. The rule would have kept 15 of 43 of those runs as wins. A small margin fixes it.
- The benchmark script works. It ran first try on a real server here and read a known speed correctly. Its test prompts come out shorter than asked for, and a reply that arrives all at once can read as hundreds of thousands of tokens per second.
- The big numbers in the launch post come mostly from bigger moves than tuning: a different engine, a different weight format, hand-written kernels. On a server that is already tuned, expect small gains.
- What we take: a ceiling column in our tables, measuring memory speed instead of trusting the spec sheet, his compact way of packaging a skill, a fixed "what I changed" block in reports, and one lever to price on the Sparks.
- What we sent back: a pull request with a noise-floor check and a prompt-sizing fix.
- Not done: nothing here ran on the Sparks or the GB300. My experiment used a small model on a Mac.
What it is
The launch post on November 21 says the skill "was forged through thousands of hours of inference optimisation" and that 30 people have posted gains from 30% to 5×. The repository is MIT licensed and small enough to read in one sitting. It was created on November 16 with a single commit. When I read it, it had 1,558 stars, 142 forks, two open issues and seven open pull requests, none merged yet.
| Part | Size | What it does |
|---|---|---|
SKILL.md | 10 KB | The procedure the agent follows: targets, ceilings, ground rules, the loop, when to stop, how to report |
references/, five notes | 23 KB | Ceiling arithmetic, measurement traps, levers with measured results, engine gotchas, publishing rules. Loaded only when needed |
scripts/bench.py | 13 KB | Benchmarks any server that speaks the OpenAI API |
scripts/ceiling.py | 3 KB | The top-speed calculator |
scripts/queue.sh, mem.sh | 2 KB | Runs one benchmark at a time; reads a container's real memory use |
PROMPT.md | 5 KB | A ready task prompt and a report format |
Figure 1. The loop the skill teaches. Drawn from SKILL.md; the wording in the boxes is mine.
Three things in the text are better than most tuning guides:
- Failures are a deliverable. Every rejected change goes in the log with its numbers and the reason. His notes list what did not work on his own hardware, with the measured cost of each.
- Speed is not trusted on its own. A fast result must still find three hidden codes in a full-length prompt. His example of why: a 117 tokens per second "record" that was the model printing a numbered list in a loop.
- The report must say what was not verified and what the agent changed on the machine.
Ceilings first
To write one token, the chip reads the model's weights out of memory once. So the top writing speed is simple division: how fast memory can be read, divided by how many bytes each token needs. Reading a prompt is different. That is limited by how fast the chip can do arithmetic, and the skill has a second formula for it.
Figure 2. His worked example. Component sizes and the two measured speeds are the author's. The ceilings are the output of his ceiling.py on his example file, which I ran today. One user only; this part of the script is correct.
- Measure, do not trust the spec sheet. He measured 115 GB/s of memory speed on a laptop whose spec says 153.6. That is 75% of the label.
- The arithmetic points at the next lever. The word list at the end of the model (the "LM head") is 23% of the bytes in his example. The drafter reads it too. So he cut the drafter's copy down to the most-used words and reports +14% on that laptop and +27% on another Intel card. The big model still checks every guess against the full list, so the output does not change.
- It tells you when to stop. He reports reaching 628 tokens per second reading a 256K prompt against a ceiling of about 670 to 684. There was little left to find there, and the arithmetic said so.
Where it does not fit. The decode formula assumes memory speed is the only limit. That holds for a laptop or a single card serving one user. It does not hold on our GB300 lane. Our own profile of that lane, taken on an earlier build, measured 11.2 ms per step, of which 4.54 ms (40%) was 1,035 tiny operations that each take under 10 millionths of a second. The byte arithmetic cannot see that overhead. On the same profile the word list took 0.42 ms for the model and the drafter together, so trimming the drafter's half could save about 2% at most, not his 14 to 27%. His notes make the same point: "Use them as priors, not promises."
The calculator bug
Figure 3. ceiling.py on the author's own example file at a 4K prompt, no drafting. Red: as shipped. Blue: with the extra * n removed. Green: the 4-user speed in his README.
The script counts the users twice. For one user it is right. For more than one, the ceiling is too high by the number of users. A plain test shows it: take a simple 10 GB model on memory that reads 100 GB/s. One user tops out at 10 tokens per second, so two users can never get more than 20 between them. The script says 40. For four users it says 160, where the most possible is 40.
This matters because "percent of ceiling" is the number the skill tells the agent to report and act on. With the bug, his measured four-user speed looks like 18% of what is possible. An agent reading that keeps hunting for gains that are not there. Corrected, it is 73%, which says the setup is in good shape.
I did not find this first. electro56435 reported it in issue #1 on November 17, darn-idly sent the one-line fix in #3 the next day, and metoapps included it in a larger pull request, #4. I reproduced the same numbers independently. All three were still open when the launch post went out.
The benchmark script
I ran bench.py unchanged in two places. First against a real server: llama.cpp on our M4 Max Mac Studio, serving a small 3.8B-parameter model that was already loaded. Then against a fake server I wrote that produces tokens at a speed I control, so the right answer is known.
| Test | What happened | Verdict |
|---|---|---|
| Real server, no changes to the script | Worked first try. About 17 seconds a run: one user, two users, two prompt-reading tests | Works |
| Fake server at a steady 50 tokens/s, one token per message | Script read 49.33. The server's own clock said 49.33 | Exact |
| Same, four tokens per message, the way a drafter delivers them | Script read 50.03 against a true 49.29. Counting messages instead would have said about 12.5 | Right, as his notes claim |
| Whole reply delivered in one message | Script read 659,564 tokens/s, marked the run OK, raised no flag | Wrong, silently |
| A broken reply: 64 exclamation marks in one message | Script read 798,894 tokens/s, marked OK, no flag | Wrong, silently |
| Server that sends no token counts | No speed recorded at all, run still marked OK, no flag | Easy to miss |
| Asked for 512, 1,024 and 1,500-token prompts | Got 457, 776 and 1,084. The script assumes 1.6 tokens per filler word; this tokenizer makes about 1.05 | Prompts run short |
queue.sh on macOS | Printed flock: command not found, then ran with no lock. Three runs with one name wrote one log file | No protection on a Mac |
- The short prompts matter for one gate. The skill asks for a hidden-code test at about 100% of the context window. Pass the window size to
--longand, on this tokenizer, the prompt fills about two-thirds of it. The script does print the real count, so an agent that reads it will notice. Our fix is in the pull request below. - The two silent wrong readings are already being fixed by others. ahmadalzaro1 hit the same thing on a real machine and sent #9: a broken setting produced a reading of 863,388 tokens per second on hardware that does 20, and it "won" the sweep. #4 adds a flag for missing token counts.
- The Mac lock problem is in issue #1 too, with fixes from shiwng (#5) and kylekillen (#7).
mem.shreads Linux and Docker files, so it is Linux only. I read it but did not run it. - The text asks for more than the scripts do. The skill demands a full-load soak, a same-output check for "lossless" changes, and the accept rule itself. None of those are in the scripts as shipped, so the agent has to write them each time. #4 adds them.
The keep-or-reject rule
The rule in SKILL.md is: "Accept only if the candidate beats both neighbouring controls on its target metric and regresses nothing by more than ~3%." It sounds strict. It is not, and you can see why without a computer. Run the same thing three times. One of the three will be the fastest. It is the middle one about one time in three.
So I measured it. I ran one unchanged setup 45 times in a row with his script and asked how many times the middle run of three "beat both neighbours".
Figure 4. 45 identical runs, 267 timed samples, about 13 minutes. The Mac was a desktop in normal use, not an idle box: four runs dipped while it was busy. Ringed points are run medians that were higher than both neighbours.
| With nothing changed | Result |
|---|---|
| Middle runs that beat both neighbours | 15 of 43 (35%) |
| ... counting only triplets that share no runs | 4 of 15 |
| ... that also regressed nothing else by more than 3% | 11 of 43 (26%) |
| Largest no-change "win" over the better neighbour | 1.54% |
| Middle runs that beat both neighbours by more than 2% | 0 of 43 |
Figure 5. The same 43 triplets, replayed. I multiplied the middle run by a pretend speed-up and applied each rule. No statistical model is assumed; the curve is stepped because it comes from 43 real triplets. The 2% margin was chosen after seeing this data, so treat it as the right size for this machine only.
- The fix is a margin sized from your own machine. Run the unchanged setup ten or more times first. The largest no-change "win" is your noise floor. A real win has to clear it. Here that floor was 1.54%, and a 2% margin removed all 15 false wins.
- The margin has a price. With a 2% margin, a real 2% gain is kept only 35% of the time, a real 3% gain 65%, and a real 5% gain 93%. Small real gains need repeats, not one sandwich.
- Why it matters in a long hill-climb. An agent may try dozens of changes. At one in three, it keeps a pile of settings that do nothing. They do not inflate the final number if the final build is measured against the baseline, but nobody can later say which settings matter.
- His own notes have the stronger test. For comparing two engines they ask for a confidence interval on the ratio of medians, and they say to repeat any single surprising win. That test kept 0 of 43 false wins here. It just is not the rule the loop uses. #4 adds it to the script.
- One machine, one small model. The size of the noise will be different on a GPU server. The one-in-three arithmetic will not.
The big numbers
The README has a results table. These are the author's measurements; I did not reproduce them. What I can do is read what changed in each row, using his own notes.
| Row in his README | Before → after | What changed, by his notes |
|---|---|---|
| Intel Arc B390 laptop, 35B model | One user: 43.6 → 53 tokens/s (+22%). Reading a 256K prompt: 297 → 628 (2.1×). Startup: 113 → 43 s | Saved compile caches, a trimmed drafter word list, sampling moved to the GPU, 8-bit prompt-reading math (a labelled quality trade), kernel work |
| Intel Arc Pro B70, 27B model | 91.2 tokens/s, against 25.0 with llama.cpp on the same card (3.6×) | A different engine, a different weight format, and custom kernels |
| RTX 3090, Flash-Next | Drafting row: 45.5 → 65.6 tokens/s (+44%) | Not broken down in the README |
| 4× H200, DeepSeek-V4.1-Flash | "Crashed under load" → 229.1 tokens/s, all six gates pass | A stability fix, so there is no before-speed. Same model we serve, on different hardware; not compared here |
- The large multipliers are large moves. Switching engine and weight format, or writing kernels, is months of engineering. An agent with this skill will not do that in an afternoon. The +22% row is the one that looks like tuning.
- Readers who posted results fit the same pattern. In #7, kylekillen got +60% for two users from one setting on an M4 Pro and found the single-user levers "plateaued". In #8, brendanboyle87 went from 15.29 to 33.13 tokens per second on an M4 Max by turning drafting on. Big wins come from switching on something that was off.
- This matches our own experience. Our own loop in September tried 20 setting changes and kept none, as covered in this morning's post on Karpathy loops.
- The 30 posts. The launch post does not link them and I could not find the list, so I cannot check that claim. "Thousands of hours" I cannot check either. The notes do read like real scars: exact numbers, named failures, and the cost of each dead end.
What we take
Figure 6. Left: what we adopt. Middle: what our own campaign method already covers. Right: what we leave. The skip column's first row is the measurement from Figure 4.
| What we take | Why | Cost |
|---|---|---|
| A ceiling column in every results table | We already model bytes per token for decode. We do not routinely show speed as a share of the limit, and we have no standard ceiling for prompt reading. On lanes where overhead dominates, the column should carry the measured overhead next to it | Small. A change to our report format |
| Measure memory speed on each box | Our own bytes-per-token script defaults to a nominal 273 GB/s for the Sparks, and I found no measured figure in our notes. His laptop measured 75% of its label | Minutes per box, in a window when the GPU is free |
| His way of packaging a skill | His procedure is 10 KB, with detail in notes that load only when needed. Our two campaign skills total 113 KB, loaded whole, and repeat some rules three or more times. That costs every session that loads them | A rewrite of two skills. Needs James, since skill edits go through his approval queue |
| A "what I changed on the machine" block in every report | We already list what was not verified. A fixed list of changed state (running containers, power settings, uncommitted code) is cheap insurance | Small |
| Price drafter word-list trimming on the Sparks | His evidence is on devices limited by memory speed, and the Sparks are closer to that than the GB300 is. It is already on our GB300 list as a small item. Do the byte arithmetic for the Spark lane before spending a boot on it | Desk work first |
What we keep. Our own method already sizes the noise with a no-change pair before the first test, sets the pass bar before the run, counts tokens from the server, scores quality before speed, and keeps a table of closed ideas. In those places it is stricter than the skill, so we keep ours. My recommendation is not to run the scripts as they are on our lanes.
What we sent back
We opened one pull request from Milo's GitHub account. It holds only what the open pull requests do not already cover.
| Change | What it does | Evidence |
|---|---|---|
A noise-floor check, scripts/aa.py, and one line in the accept rule | Run the unchanged setup ten or more times, read the floor, and require a win to clear it | The 45-run series above, with the raw ledger |
Prompt sizing in bench.py | Four small probe prompts measure tokens per word on the live server before any test prompt is built | See the table below |
| Prompt size asked for | Script as shipped | With our change |
|---|---|---|
| 512 | 457 | 518 |
| 1,024 | 776 | 1,006 |
| 1,500 | 1,084 | 1,470 |
The ceiling bug, the Mac lock and the two silent wrong readings already have fixes from other people, so we added nothing there beyond confirming them. Thank you to 0xSero for putting this out in the open. A method like this usually stays in one person's head.
Not done
- Nothing ran on the Sparks or the GB300. There was no free window today. The noise experiment is one Mac, one small model, one engine.
- I tested the parts, not a full hill-climb. I did not hand the skill to an agent and let it tune a server end to end.
- Not run:
mem.sh, the memory watchdog, and a real long-context needle test. The model on the Mac is a speech model with a small context, so its replies were used for timing only. - Not reproduced: any number in his README. They are quoted as his.
- One mistake of mine. My first fake server was itself slow: the script read 6.75 tokens per second where I meant 50. The fault was a network setting in my server, not his script. I put a clock inside the server, fixed the setting and reran. That is where the 49.33 against 49.33 comes from.
Credits and sources
- The skill: 0xSero, sybil-solutions/inference-optimisation-skill, commit
fda968e, MIT. Launch post. - Ceiling bug and Mac lock: reported by electro56435 in #1. Fixes by darn-idly (#3), metoapps (#4), shiwng (#5) and kylekillen (#7).
- Single-message and broken-reply readings: ahmadalzaro1, #9. A quality gate for lossy changes: dh7, #6. Results on Apple hardware: kylekillen (#7) and brendanboyle87 (#8).
- Our tests: stock
bench.pyandceiling.pyfrom that commit, llama.cpp build 8980 on a Mac Studio (M4 Max, 64 GB, macOS 27.0.1), a Llama-3.2-3B fine-tune at Q4_K_M. Run on November 21, 2026.