oMLX PR #3059 Round 2: Chunking Fix Confirmed Upstream

Created August 23, 2026

Follow-up to this morning's field test of oMLX PR #3059 (DeepSeek-V4 hybrid ANE prefill). Fabian pushed two commits within a couple of hours of the reports and asked for a retest, so: same box (M3 Ultra 512 GB), same harness, fresh matrix at the new head.

What changed in the branch

• 19fdb656 “Chunking fix” — engine startup now floors the scheduler's prefill_step_size at the compiled ANE sequence length, exactly the fix from round 1. The docs now name effective_step=4096 / chunk_tokens=4096 as the activation signal and note the ane_full_tiles counter isn't reliable for DeepSeek yet. Ships with 81 lines of new tests.
• 6f848656 — combines wq_a, wkv, compressor, and sparse-indexer projections into one hybrid dispatch (previously only wq_b and the stacked indexer wq_b were offloaded). This is the “offload more projections” part.

Results at head 19fdb656

Same protocol as round 1: built-in throughput benchmark (code_python, TG=128, greedy), 16k and 32k prompts, fresh server and wiped SSD cache dir per configuration, PR-default ANE settings. Fresh GPU baselines too, since the combined-dispatch commit touches the model patch.

Round-1 column is this morning's run at f77c2ba with the equivalent chunk fix applied locally. GPU baselines are statistically unchanged between rounds (within 0.7 tok/s), so the combined dispatch did not regress the plain path.

Reading the deltas

• The fix works stock. Load log reports Eagerly compiled 88 DeepSeek ANE procedures (43 shared experts, 2 attention-input stacks, 43 query projections, 21 with the indexer wq_b stacked in), and every ANE-on config shows effective_step=4096 with full 4096-token chunks. No local edits of any kind this round.
• The combined dispatch is a small positive on this box. Four of six cells improved vs round 1 (up to +0.7 pts on oQ2.5e 16k and fp8 MTP-on 32k), two dipped slightly. Averaged across the matrix it's roughly +0.2 pts — in the direction of the theory, but this benchmark's run-to-run spread is of the same order, so we'd call it “neutral to slightly positive,” not a confirmed few-percent bump.
• The headline number is unchanged in character: the oQ2.5e quant gains ~7% prompt processing at long context; the official fp8 release gains ~3–4%, consistent with its expert path streaming more weight bytes and competing with the ANE for unified-memory bandwidth.

Protocol

• M3 Ultra 512 GB, macOS 26.5.1. Isolated venv build of onthehub97/omlx @ 19fdb656, custom kernels rebuilt from scratch after the branch update. Production oMLX (brew 0.6.3rc2, :8020) untouched; bench on :8022.
• Checkpoints: Jundot/DeepSeek-V4-Flash-0731-oQ2.5e (public, non-MTP) and deepseek-ai/DeepSeek-V4-Flash-0731 (official fp8). Eight configs attempted; oQ2.5e MTP rows skipped (mtp_compatible: false, its -mtp variant is still gated on HF).
• Activation verified per config from serve logs before trusting any number — the round-1 lesson, now also in the PR's own docs.

Round 3 (profiled, new offloads regress here): oMLX PR #3059 Round 3.

Round 1 (final, read-only): oMLX PR #3059: DeepSeek-V4 ANE Prefill on the M3 Ultra. Sources: PR #3059, Fabian's retest request.