Third pass at oMLX PR #3059 on the M3 Ultra 512 GB, at Fabian's request: run with profiling enabled and attach logs, on the head that adds the projections that profiled positive in isolation. Head under test: 304bb447 — five commits past round 2's 19fdb656, adding shared down projections (65% ANE), grouped wo_a (50% ANE), tail padding, profiler coverage, and tuning changes. The load log now reports 174 ANE procedures across 6 programs (was 88 / 2).
19fdb656 measured +7.1% / +7.0%, and official fp8 drops to roughly break-even (+0.9–1.3%). The profiler itself costs about another point when enabled. The isolated-positive projections appear to lose their margin in-model — the profiler data below says the stalls are on the input side, not in the ANE work.
[benchmark-ane-profile] output plus benchmark result JSONs, for both the profiled (v5) and unprofiled (v6) runs at 304bb447.
Same protocol throughout: built-in throughput benchmark (code_python, TG=128, greedy), 16k/32k, fresh server + wiped SSD cache per config, PR-default ANE settings, stock config (no local patches since round 1's fix went upstream). GPU baselines re-measured at this head and statistically unchanged (≤0.5 tok/s from round 2's).
| Checkpoint | Prompt | GPU PP | 19fdb656 (r2) | 304bb447 | 304bb447 + profiler |
|---|---|---|---|---|---|
| 0731-oQ2.5e | 16k | 598.9 | +7.1% | +4.6% | +3.3% |
| 0731-oQ2.5e | 32k | 582.1 | +7.0% | +3.8% | +3.7% |
| 0731 fp8 | 16k | 612.8 | +3.8% | +1.3% | +0.4% |
| 0731 fp8 | 32k | 595.2 | +3.0% | +0.9% | +0.6% |
All MTP-off; fp8 MTP-on rows in the profiled run track the same pattern (+3.7%/+3.8% raw, i.e. flat vs MTP-off round 2). Round-2 percentages are against that round's own fresh baselines, which match this round's within noise. Every ANE-on config verified active: effective_step=4096, full 4096-token chunks, 174 procedures compiled.
Representative 16k figures from the fp8 profiled run (per-op means, [benchmark-ane-profile]):
| Category | ops | input_ready ms | ANE region ms | gap_before ms | ANE duty |
|---|---|---|---|---|---|
deepseek_down | 126 | 2.4 | 2.8 | 0.1 | 1.2% |
deepseek_query | 126 | 11.0 | 8.9 | 0.6 | 3.9% |
deepseek_wo_a | 126 | 32.7 | 12.5 | 1.6 | 5.7% |
deepseek_mlp | 126 | 60.9 | 6.0 | 17.1 | 2.6% |
wo_a waits ~33 ms in input_ready for ~12 ms of ANE work; the shared-expert mlp waits ~61 ms (plus a 17 ms gap_before) for ~6 ms of ANE work. down is the only new category whose input arrives promptly (2.4 ms), consistent with it consuming the mlp's own output in place.expected_operations slightly above successful_operations (129 vs 126 at 16k) and large shape_rejections counts (~5.6k) — these are the sub-4096 tail chunks correctly falling through to GPU, but the counter naming may read as errors. Zero runtime_failures everywhere.If these numbers replicate on the reference Ultra, per-projection enable flags (or tuner-driven selection) would let the round-2 subset stay default while down/wo_a remain opt-in per host. Happy to run any specific settings combination — the harness here is fully scripted and takes ~12 minutes per configuration pair.
onthehub97/omlx @ 304bb447, kernels rebuilt from scratch. Production oMLX untouched; bench on :8022.OMLX_ANE_PROFILE=1; v6 = ANE-on repeats without the profiler to separate instrumentation cost from the offload change. Both in the attached tarball.Jundot/DeepSeek-V4-Flash-0731-oQ2.5e and official deepseek-ai/DeepSeek-V4-Flash-0731.Round 4 (tuner run — it disables wo_a and beats every prior round): oMLX PR #3059 Round 4.
Series: round 1 (fix discovery, final) · round 2 (fix upstream, best numbers) · this post. Sources: PR #3059, Fabian's profiling request.