Fabian: this is the write-up from putting #2756 / #2760 on three machines we already have. You said additional devices would be useful. We had an M3 Ultra (your reference chip), an M4 Max, and an M5 Max.
Same checkpoint you tweeted, True2456/Qwen3.8-27B-AWQ-4.85bpw, under oMLX 0.6.1. Prefill only. Decode stayed on GPU.
16k PP. Dual-die, two physical ANEs. This is the chip the kernel was written for.
16k PP. Single 16-core ANE, slower GPU. Still a win.
16k PP. Single ANE, much faster GPU/NAX. Hybrid is net-negative.
Isolation, not the in-app bench. omlx serve --no-cache, localhost, temperature 0, max_tokens=1, identical English prompts (not the Qwen pad token). ANE knobs matched your isolated config: seq 2048, MLP 0.53 / 64 layers, GDN 0.50 / 48 layers, dual_ane=true.
Two measured runs after a load/compile warmup, except M4 which is one measured run after warmup. PP tok/s is prompt_tokens / wall on an uncached request. Prefix cache was off on purpose. With it on, a 4k prompt came back as cached_tokens=4096 and a fake 4.7 s “prefill.”
| Host | Chip | RAM | macOS | ANE |
|---|---|---|---|---|
| Studio | M3 Ultra | 512 GB | 26.5.1 | two physical instances |
| MBP | M4 Max | 64 GB | 26.6.1 | one 16-core ANE |
| MBP | M5 Max | 128 GB | 26.4.1 | one 16-core ANE (ANEHWDevice=1) |
All three logs said the same thing, including the single-ANE boxes:
Eagerly compiled 64 MLP and 48 GDN procedures
into two instance-pinned ANE programs (sequence_length=2048)| Host | 4k GPU | 4k ANE | 8k GPU | 8k ANE | 16k GPU | 16k ANE | 32k GPU | 32k ANE |
|---|---|---|---|---|---|---|---|---|
| M3 Ultra | 456 | 530 (+16%) | 452 | 541 (+20%) | 438 | 520 (+19%) | 408 | 483 (+18%) |
| M4 Max | 261 | 316 (+21%) | not run | 248 | 299 (+20%) | not run | ||
| M5 Max | 885 | 582 (−34%) | 843 | 567 (−33%) | 775 | 533 (−31%) | 718 | 537 (−25%) |
Medians of two uncached runs (M4 is n=1 after warmup). M3 16k/32k deltas sit next to the 0.6.1 oQ4e release table (+17.7% / +18.9%), not the tweet isolation microbench (+35.6% @ 16k). Same direction, smaller, because this is a full request rather than the layer-body bench. M5 GPU 16k at 775 tok/s also matches the True2456 card (765 tok/s, ANE off), so the GPU baselines are not a measurement error.
Nothing crashed. No jetsam. First generated token stayed stable GPU vs ANE (We / The). Hermes on the M4 and embed/rerank/vision on the M5 stayed up through compile.
Stock brew install omlx 0.6.1 does not ship the kernels. The ANE settings accept. There is no qwen35_prefill _ext.so until brew reinstall --with-custom-kernel.
That option needs the Metal toolchain component, not just Xcode.app. On the Ultra, Xcode 26.6 was present and metal was on PATH. First build still failed with cannot execute tool 'metal' due to missing Metal Toolchain. xcodebuild -downloadComponent MetalToolchain (17F109) fixed it. The M5 already had the component and built clean. The M4 xcode-select pointed at Command Line Tools; brew’s cmake sandbox could not find metal even with DEVELOPER_DIR set and Xcode.app installed. We transplanted the M5-built .so / .dylib / .metallib. qwen35_ane_available() returned true and the path compiled. Treat the M4 numbers as “feature ran,” not “the brew recipe is portable.”
Enabling the flag unloads the model. The next request pays eager compile. Observed load+compile: Ultra 29 s, M5 57 s, M4 77 s. In the ballpark of the PR (27–44 s on the reference box).
No physical ANE probe. Single-ANE M4 and M5 still compiled two instance-pinned programs. On the M4 that still helped. On the M5 it made prefill slower. M5 GPU/NAX prefill is already about 1.8× the Ultra GPU, so the hybrid split is the wrong split.
--no-cache is required for an honest PP A/B. Default prefix cache will lie.
On the Ultra the feature does what you said. About +18–20% PP from 8k to 32k. On the M4 (single ANE, slower GPU) it still helps about +20%. On the M5 (single ANE, much faster GPU) it should stay off.
If you want a follow-up we can do oQ4e-mtp, a 0.53/0.53 in-app bench, or a single-ANE-forced Ultra ablation so you can see the two-program path with one instance tied off.
Earlier oMLX note on this site: DSv4-Flash on the M3 Ultra. Source: jundot/omlx#2756 (Fabian / onthehub97), hardened in #2760, shipped in 0.6.1. Tweet: @onthexitter69.