August 2, 2026 · James Meadlock & Milo · Funland lab · mlxfast challenge notes

Created August 2, 2026P26-08-02T08:20:00-05:00">August 2, 2026 at 8:20 AM CDT

Trying the mlxfast challenge: speeding up AI on a Mac

A beginner-friendly field report. No leaderboard win yet — but we learned a lot about how modern on-device inference is already optimized, and how easy it is to fool yourself with noisy benchmarks.

What is mlxfast, in plain English?

Imagine a cooking contest where every chef gets the same recipe book and the same ingredients. You are not allowed to invent a new dish. You have to make the exact same meal — same bites in the same order — but finish faster.

That is mlxfast.

• The meal = the next token an AI model would produce (greedy / temperature-zero behavior).
• The kitchen = Apple Silicon (Metal GPU + unified memory), not NVIDIA GPUs.
• The recipe = Poolside Laguna XS 2.1, a medium-size MoE language model shipped in a compact 4-bit format called NVFP4.
• The stopwatch = official ranking on a self-hosted M5 Max 128 GB machine. Your laptop numbers are practice only.

The public score blends two speeds:

score = decode_speedup^0.75 × prefill_speedup^0.25

Prefill is “read the whole prompt once.” Decode is “emit the next token, one at a time.” Decode is weighted more heavily because chat feels slow when each word crawls out.

One more hard rule: this track is serial. You may not cheat by drafting future tokens or running speculative multi-token tricks. Each one-token decode request is allowed to advance exactly one position. That keeps the contest honest about “single-step inference work.”

What we set up

We did this on a Mac Studio-class M4 Max with 64 GB unified memory. That is enough to load the ~21.6 GB text tower plus working room (the docs say roughly 36 GB practical minimum; the ranked box has 128 GB and stays more comfortable).

Useful local knobs we hit immediately:

export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
export MLXFAST_LOCAL_COOL_GATE=0          # desk GPU often won't sit ≤40°C
export MLXFAST_LOCAL_ALLOW_GOLDEN_DRIFT=1 # M5 goldens can near-tie differently on M4
./benchmark.sh --local-iterate

What “already optimized” looks like

We turned on the challenge’s fusion tracer and watched the worker. On the stock main tree, twenty-four fused paths announced themselves during the timed window — attention fusions, MoE gather/QMV fusions, residual+norm+router fusions, lm_head pruning, packed scale layouts, and more.

In beginner terms: most of the obvious “glue many small ops together” work is already done in upstream main. If you walk in expecting a free +20% from one clever idea, you will be disappointed.

The editable surface is also broader than a typical “tune configs” contest. You can touch:

• the Swift runtime model path,
• the offline weight transform,
• and listed vendored MLX Metal kernels.

But you still have to pass hidden correctness gates and a paired timed measurement on M5. Bigger wins must often be chunked across submissions because a single run’s acceptance band caps about +5% versus the pinned calibration reference.

Experiment 1: a tiny kernel tweak (negative result)

Using a coding agent (Nous-hosted DeepSeek V4 Flash as a helper while our local Spark lane was briefly down), we tried a reversible, default-off micro-opt on the gated NVFP4 attention output-projection kernel:

• drop a dead +0.0 seed in the first qdot group, and
• fold the E4M3 scale sign the same way sibling MoE kernels already do.

It compiled. Fusion traces proved the kernel path actually fired (not a silent fallback). Then we measured n=3 off vs on.

Verdict: revert. The change was honest, narrow, and reversible — and still not a win. That is a good outcome. Publishing only green numbers teaches the wrong lesson.

How we decided what “counts” as a win

Before chasing more flags, we measured a clean-main noise floor (n=5):

We set a local kill bar of roughly 3.3% mean decode improvement (about 2× standard deviation, floored at 1%), with multi-run interleaved A/B. Anything smaller is “maybe weather.”

Experiment 2: async scheduling (the schedule of GPU work)

Even when every math kernel is fused, the CPU still has to feed the GPU. mlxfast can insert process-once async boundaries during a decode step — not to compute future tokens, but to overlap already-built work. Think of it as prepping the next pan while the current one sizzles, without changing the recipe.

On a warm thermal plateau (~60°C), interleaved n=3:

Two takeaways:

1. Async is load-bearing. Turning it off hurts a lot.
2. ladder1 looked better when hot — but absolute speeds were far worse than the cool baseline (~0.008). Heat can change which schedule wins. We are not promoting a default change until we re-measure on a cool machine (ideally with fan control via the challenge’s tools/fan-control.sh, which needs an smc binary we do not have installed yet).

What we did not do (yet)

• Submit to the official Yukon / M5 leaderboard (needs the separate mlxfast CLI + account).
• Edit ranked Metal _nax kernels aimed at M5 specifically.
• Claim a production speedup for anything beyond “async off is bad.”

A mental model if you want to try yourself

1. Get green setup first. Xcode, Metal toolchain, weights hash, both binaries, local iterate.
2. Trace before you invent. See which fusions already fire.
3. Measure a noise floor on unmodified main.
4. Change one axis. Prefer default-off flags or tiny reversible patches.
5. Prove the path ran (trace / log), then multi-run A/B.
6. Respect the band. Large honest wins may need multiple ranked submissions.
7. Remember the authority machine. M4 is a practice piano. M5 is the recital.

# directional local loop we used
cd ~/code/forks/mlxfast-challenge
export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
export MLXFAST_LOCAL_COOL_GATE=0
export MLXFAST_LOCAL_ALLOW_GOLDEN_DRIFT=1
./setup.sh                    # once
./benchmark.sh --local-iterate
DARKBLOOM_TRACE_FUSION=1 ./benchmark.sh --local-iterate   # see what fires

Overnight follow-up (through morning)

Cooler interleaved A/B of async schedules (n=5 each arm when complete):

Cool-regime absolute speeds acceptable: True. Promote criterion met: False.

Default patch: skipped.

Extra env probes vs control (negative = faster):

• C_suffix_affine: -0.53%
• C_control: +0.00%
• C_async_ladder1: +0.62%
• C_async_at_comment: +1.79%
• C_async_off: +14.77%

M5 Max baseline (ranked silicon class)

We brought the same tree up on the lab M5 Max 128 GB (.18). Unlike M4, goldens match with no drift flag.

Env ablations on cool M4 confirmed the big load-bearing defaults: turning off decode async costs ~+15% decode; turning off lm_head prune costs ~+4.5%. Other knobs (packed scales, QMV R1, prefill async density, ladder1 vs default) sat inside noise. Next real wins need Metal/runtime work measured on this M5 loop, not more flag thrash.

Bottom line

mlxfast is a great way to learn real on-device inference constraints: memory traffic, kernel fusion, correctness under near-ties, thermal gates, and paired scoring. Our first pass did not produce a keepable speedup — and that is fine. The useful artifacts are the setup path, the noise floor, the fusion inventory, and two clean negative/conditional results:

• a tempting Metal micro-opt that did nothing good, and
• an async schedule candidate that needs a cool re-test before anyone changes defaults.

If you are new to this space: start by reproducing the baseline. Resist the urge to “optimize” until you can explain your measurement error bars. The challenge’s main tree is already a high bar.

Written from Funland lab notes (August 1–2, 2026). Local timings are M4 Max directional estimates under golden-drift allowance; official mlxfast ranking is M5-only. No leaderboard submission from this writeup.