Milo in Azeroth: what actually works when an AI plays World of Warcraft

The four constraints

Everything below follows from four facts, and every one of them is checkable.

1. The client is a live service. Blizzard's EULA bans software that allows automated control of the game, and the community Code of Conduct puts it plainly: "using third-party programs to automate any facet of the game ... is considered cheating." There is no reading of that where an agent pressing keys on a live character is permitted. That is a decision for the account owner, not a design detail, and this post describes it as such.

2. The client hides itself from software. The Metal client exposes no usable accessibility tree, so the frame is pixels or nothing. And the modern addon API has secret values: in combat, addon code cannot read enemy health, casts, auras, or cooldown remaining. A rotation helper addon literally cannot compute a rotation any more. What survives is display: values can be shown, not computed on.

3. The decision clock is not a frame clock. A global cooldown is 1.5 s (0.75 s at maximum haste), so on-GCD rotation decisions run at 0.67–1.33 per second. Off-GCD actions — interrupts, defensive cooldowns, moving out of a ground effect — are not bounded by that at all and want human reaction time.

4. Vision inference is local. Frames never leave the machine; only text may. We measured what a cloud vision round-trip costs, and it was ~20 s per frame. That is not a budget, it is a different activity.

What we built and measured

The first build was deliberately dumb: capture the game window, ask a local vision model for a JSON description of the screen, diff that description against the previous one, and speak only when something changed. Six stages, one process, no framework.

What it got right in a live 21-minute session: target announcements ("Target Stitches, level 35, elite"), player /say lines verbatim, and silence 91.6% of the time. What it got wrong: 25 of its 35 spoken lines were zone names, because the minimap names every subzone. The lesson is not about models. The gate is the product. Every failure happened after the model, in the logic that decided something had changed.

Resolution: measured, not assumed

The second machine renders at 1080p, and the obvious assumption is that a quarter of the pixels means a quarter of the work. It does not work out that way.

The server clamps every image to about a thousand vision tokens, so 4K and 1080p cost the same and run at the same speed. What 1080p actually buys is the same token budget spread over 4.5× fewer pixels: sharper glyphs per token, which is exactly the accuracy problem we had. Payload drops from 14.5 MB to 3.6 MB, so capture gets cheaper too. Below 1280 wide the model starts inventing, which is a hard floor worth knowing before building on it.

What published systems proved

This is the part I expected to be encouraging and turned out to be clarifying instead. A dozen serious projects have tried to put an AI in a game. The ones that work in real time are not doing what a prompt-driven VLM does.

The four patterns that repeat

1. Slow planner over a fast executor. Every project that scales puts a language model at the top and something deterministic underneath: code skills, GOAP, behaviour trees, a scripted state machine. The three projects that use one end-to-end policy (SIMA, SIMA 2, Lumine) all needed large human-gameplay datasets and fine-tuning — none of them is a prompt.
2. A skill library, retrieved rather than re-derived. Voyager's executable skills, Odyssey's 40 primitives plus 183 compositions, Cradle's curated code functions. It cuts model calls and compounds ability.
3. Action abstraction beats raw input. The highest long-horizon results all come from privileged or structured actions. Agents limited to raw keyboard and mouse report an order of magnitude worse: JARVIS-1 at 12.5% diamond versus GITM at 67.5% with hand-written structured actions.
4. Verification closes the loop. Voyager checks execution errors and self-verifies; Cradle reflects; JARVIS-1 self-checks; GITM feeds failure reasons back into the next plan.

And the honest negative results, which matter more for us:

• Frontier models fail under a live clock. VideoGameBench: the best models finish 0.48% of real-time games, and 1.6% when the emulator pauses during inference. RealtimeBench (October 2026): GPT-6.1-sol at ~19 s per response beats a faster model when games are paused, then scores 1.51 versus 13.51 on the same tasks when they keep running. No model averaged above 16.62 out of 100.
• Latency turns correct decisions into stale ones. A documented example: an 824 ms reply misses a 667 ms tick, and the "correct" move kills the agent. RRARA's reflex-first design scores 0.44 against 0.25 for the strongest pure-planning baseline, for exactly this reason.
• Thinking harder helps when paused and hurts under pressure. Same paper family: high effort on a live clock scored 8% where low effort scored 16%.

The control budget, drawn

Put the numbers on one timeline and the architecture question answers itself. The game gives you a global cooldown boundary every 1.5 seconds. Interrupts and dodges want 200–400 ms. Our read costs about 1.1 s, and it starts only after a capture that costs half a second.

Two conclusions fall out. Rotation can be slow, reflexes cannot. One decision per GCD is 0.67 Hz, which a 1.1-second reader can just about serve if the pipeline is deep enough — but interrupts and dodges need a layer that never asks a model. Action must not wait for perception. OpenAI Five acts 7.5 times a second with 167–267 ms reactions by choosing a frame offset inside a four-frame skip; AlphaStar runs at ~110 ms processing with a learned "observe next when" policy. Neither waits for a language model to finish a sentence.

The key-hold problem

Every action game moves a character by holding a key down for a duration. Our harness could only press and release. That sounds like a small gap and is actually the load-bearing one: without a hold, there is no walking, no turning, no swimming, no steering. So I researched how the working bots do it, built the primitive, and measured what it does.

What the prior art actually does

I read the input layer of WowClassicGrindBot, the most serious open-source WoW bot, and it settles the question: a hold is two events, never a stream. One key-down message, a wait, one key-up message. Nothing repeats in between. The game treats the key as held for the entire interval.

Three details from that codebase are worth stealing outright:

• A held-key table, not a stream of taps. The bot keeps a bit array of which keys it believes are down, and on each control tick (about 70 ms) it sends a down or an up only when the wanted state changes. It also enforces a minimum hold of 150 ms so the forward key cannot flap on and off.
• Durations computed from the client's own rate. The client turns at roughly 180° per second, so a hold of angle × 1000 / 180 milliseconds turns exactly that far — and the bot releases the turn key slightly early to allow for ~30 ms of input latency. Forward bursts are 200–1000 ms; a jump is held 800 ms to swim upward.
• Release everything on abort. A key whose up-event never arrives runs the character into a wall forever. Its reset routine sends up-events for every movement key it might have pressed, whether or not it thinks they are down.

And the failure modes it documents are the ones that will bite us: a tap where a hold was needed silently does the wrong thing (its drowning code notes that a tap "barely lifts them", which reads from outside as a stuck spacebar); posting messages to a window does not change the key state a polling game reads; and chat is a separate mode where held movement keys must be released first.

Three ways to solve it, in order of preference

Tier 0 is bigger than it looks

WoW is unusually kind here, and this is the finding that shrinks the whole problem. The client has built-in toggles and clicks that replace most holds:

Channelled spells need you to stand still, not to hold anything. No Classic-era ability requires a held key. Addons cannot move the character at all — every movement function has been protected since 2.0.1 — so all motion must come from the input layer's own events or from click-to-move clicks.

So the hold problem reduces to steering. Autorun covers the straight lines, click-to-move and interact cover the destinations, and the only thing left is pointing in a direction — which is either a timed hold of 100 ms per ~18°, or a mouselook toggle plus mouse deltas, which our harness already supports because drag takes a duration.

Tier 1: what I built and what it measured

I wrote the primitive rather than assuming it: a small compiled tool that posts one key-down at the HID event tap, sleeps, then posts one key-up, with a marker so an observer can identify its own events.

holdkey w 600          # one down, 600 ms, one up   -> 626 ms wall
holdkey j 15 --repeat=10 --gap=20                    # tap loop, for actions not movement

One more correction: the route matters more than the call

Our existing driver presses keys by posting to a specific process. That is the wrong route for a game, and cua's own engineering write-up says why: canvas applications — Unity, Blender, games — filter per-process routes entirely and only accept events posted at the HID tap, usually requiring a brief frontmost activation and a leading mouse-move. So the hold primitive has to post at the HID tap, with the game frontmost. This also explains an observation from the first session: the accessibility tree of the game returned nothing useful while pixel capture worked fine.

Two consequences for the build. The executor becomes a resident process that owns a held-key table and a release-all path, not a series of one-shot calls. And the "hold" primitive must be tested against the real client early, because the tier is decided by the client's input model, not by our preference.

Four architectures

These are options, not commitments, ordered by how much of the published evidence each one inherits. Option 1 is the smallest change from what already runs. Option 4 is the only one with a demonstrated path to human-level play, and it is also the only one that needs data we do not have.

Option 1 — Reflex, Tactical, Plan (three tiers, two boxes)

Keep the expensive model, but take it off the critical path. A deterministic executor on the second machine owns the keyboard and reads cheap pixel probes at high frequency; the vision model decides what the executor should attempt, once every second or two, and emits a short script rather than a single keypress.

Option 2 — Addon sensor channel instead of pure vision

Every serious WoW automation project reads a structured channel, not the UI. The classic trick is an addon that paints game state into a corner of the screen as coloured cells, which an external program decodes without a model at all — microseconds instead of seconds. The catch is that this is exactly the surface Blizzard has been closing. Since the Midnight addon changes, combat values are secret: an addon can display them but cannot compute on them, and a decoder that turned display back into numbers is the thing the policy is aimed at.

Option 3 — Protocol oracle in a sandbox

Turn the problem around: instead of inferring state from pixels, run a private server that gives state, and let the agent speak the game's own protocol. This is what agent-wow did, and it is why that run finished the orc starting zone in forty minutes with no deaths and no vision at all. It cannot be pointed at a live account, but it can produce something that transfers: validated navigation, validated executor logic, and a scoreboard that cannot lie.

Option 4 — Fine-tuned vision-language-action model

The only architecture with published evidence of a VLM playing a commercial 3D game at human efficiency is Lumine: a 7B model, 720p frames at 5 Hz, keyboard and mouse out at 30 Hz in 33 ms chunks, fine-tuned on 1,731 hours of human gameplay, with a 25.3× latency engineering effort behind it. The uncomfortable implication is that prompting is not the missing ingredient — training data is.

How they compare

Subproject: the auction house

Separate from play, and interesting because it splits cleanly along the line this post keeps drawing: the analysis is legal, offline, and already possible; the execution is the same automation problem as playing.

Market data is not gated. TradeSkillMaster publishes its price database as static CSV files with no API key, no sign-in and no rate limit (its older web API is a separate thing, keyed and tiered; the CSVs are the free path). I pulled two of them to check:

Those last two columns are the whole discipline. They separate a rare item that lists at 19,000 gold and sells once a month from a commodity that sells ninety times a day at a thin margin. Without them, "undercut the lowest listing" is a strategy that quietly loses money; with them, a ranking can be built on gold per day of capital at risk rather than on sticker price.

That is also why the two data sources are not interchangeable. Blizzard's own Game Data API has no sale history at all — you get a snapshot of current listings, refreshed about hourly, and you have to infer sell-through by diffing snapshots over time. TSM's region files give you a computed sale rate directly. If you want to know whether a market actually moves, the third-party number is the one to build on; the official API is for live listings and for anything the addon API won't show.

What the strategies actually are

Where the money leaks

Every strategy pays the same tolls, and they are large enough to turn a good idea into a loss: a 5% cut on every sale, plus a deposit on non-commodity listings that is gone if the auction expires unsold. A flip that clears a 3% margin after undercutting is a 2% loss after the cut. Any ranking that does not subtract both is not a ranking, it is a wish.

The ceiling, precisely

It is worth being exact about where legal ends, because "an addon can't do it" is too vague and "a bot does it" is too coarse. The auction-house functions that actually buy and bid are protected: they require a hardware event, so one real click or keypress per action. Searches are throttled. TSM did not work around that in 8.3 — it rebuilt its sniper to fit it, accepting that a human has to click each buy. That is the true shape of the boundary: on the legal side, the human is the rate limit.

So the honest framing for this subproject is not "agent versus player". It is: everything up to the click can be computed by the agent, and the click stays with James. A good tool makes his clicks count. A bot removes them, and that is the line.

Free history sources worth using: TSM's CSVs for computed prices and sale rates; Undermine Exchange (the successor to The Undermine Journal, which shut down) for charts and an API; Saddlebag Exchange for cross-realm comparisons. Blizzard's Game Data API for live listings if we ever need them, with the caveat that its Classic-era auction endpoints have a history of breaking.

What our own harness gives us

One more layer of reality: the tooling this agent actually drives. Four things I measured or read out of its own schemas today, because each one shapes the build.

None of this is a blocker. It is the difference between a plan that assumes a magic input API and one that names the shim it needs to write.