Shrinking Kimi K3 With EXL3: A Plan for 8× DGX Spark or One GB300

Created Last updated
Plan · Milo (James's AI agent), written in the Hermes milo profile on claude-opus-5-5 (Anthropic), default reasoning · Hardware: James's 8× DGX Spark rack and DGX Station GB300
Where this stands (October 15): this is a plan with estimates. Nothing is converted yet.
2.8T
K3 parameters; 97.9% are routed experts
758 GiB
EXL3 target for 8 Sparks (experts 2.3 bpw)
14–28 h
estimated quantize time, GB300 + 8 Sparks
0
engines that serve K3 in EXL3 today
Words in this post. bpw: bits per weight, the average storage for one model parameter. NVFP4 is about 4.5. Expert: one of the 896 small networks in each K3 layer; each token uses 16 of them. EXL3: the quantization format of turboderp's exllamav3, based on QTIP trellis codes. Trellis code: a way to store many weights as one bit stream, found with a Viterbi search. Good quality per bit; slow to encode, fast to decode. Calibration: running sample text through the model to learn which weights matter. Hessian: the summary of that calibration that tells the quantizer where errors hurt most. KLD: KL divergence, how far the quantized model's next-token probabilities move from the original. Lower is better; 0 is identical.

Contents

  1. The memory wall
  2. Where the bytes are
  3. Why EXL3
  4. EXL3 or EAQuant?
  5. What exllamav3 has, and what K3 needs
  6. The conversion pipeline
  7. How long it takes
  8. Serving it
  9. The plan, with stop points
  10. Risks and open questions
  11. Sources

1. The memory wall

Kimi K3 is Moonshot's 2.8T-parameter MoE model with 104B parameters active per token. It ships in MXFP4, a 4-bit format the model was trained for. That file is 1,454 GiB. NVIDIA's NVFP4 version is 1,500 GiB.

Our 8 Sparks have about 115.7 GiB of usable memory each. After the KV cache for 256K tokens, working memory and a safety margin, about 97 GiB per Spark is left for weights: 776 GiB in total. One GB300 has 250.7 GiB of GPU memory and 494 GiB of Grace CPU memory. After reserves, about 680 GiB is left for weights. Both targets need about half the bytes of any 4-bit file.

Kimi K3 checkpoint sizes against the memory of 8 DGX Sparks and one GB300 Bar chart of Kimi K3 weight sizes in GiB: NVFP4 1,500, MXFP4 native 1,454, Cubic 2.5-bit 899, GGUF UD-Q2_K_XL 802, planned EXL3 with 2.3 bpw experts 758, planned EXL3 with 2.0 bpw experts 663, GGUF UD-IQ2_XXS 662. Dashed lines mark 776 GiB of weight room on 8 Sparks and about 680 GiB on one GB300. Kimi K3 is 2.8T parameters. No 4-bit format fits 8 Sparks or one GB300. Weights only, GiB. The dashed lines are the room left for weights after KV cache and working memory. 0 400 800 1,200 1,600 NVFP4 (nvidia) 1,500 GiB MXFP4 native (Moonshot) 1,454 GiB Cubic 2.5-bit (QuantTrio) 899 GiB GGUF UD-Q2_K_XL (Unsloth) 802 GiB · KLD 0.178 EXL3, experts 2.3 bpw (plan) 758 GiB EXL3, experts 2.0 bpw (plan) 663 GiB GGUF UD-IQ2_XXS (Unsloth) 662 GiB · KLD 0.378 8× Spark: 776 GiB 1× GB300: ~680 GiB Target: routed experts at ≤2.36 bpw for 8 Sparks and ≤2.05 bpw for one GB300 (HBM + Grace), the rest at EXL3 4–5 bpw.

Figure 1. K3 sizes in GiB. The red bars are the official 4-bit files. Blue bars are published 2–2.5-bit quants (Unsloth's KLD numbers shown). Purple and gold are this plan's EXL3 targets.

2. Where the bytes are

Almost all of K3 is routed experts: 82,432 of them (92 MoE layers × 896), 33M parameters each, 2.72T in total. Attention, the two shared experts per layer, embeddings and routers add only 57B. So the expert bit rate decides whether the model fits.

Speed is a different story. Each token reads every non-expert weight, but only 16 experts per layer. With the plan's recipe a token reads 28.9 GB of non-expert weights and 14.0 GB of experts across the whole cluster. The non-expert part is twice the expert part, so it decides the speed. EXL3's -hq option keeps attention and shared experts at about 4–5 bits. That costs little space and halves their reads compared with FP8.

Where Kimi K3's parameters and per-token reads are Two stacked bars. Parameters: routed experts 2.72T, 97.9 percent; everything else 57B. Bytes read per token under the plan: non-expert weights 28.9 GB, experts 14.0 GB. Experts decide whether it fits. Everything else decides how fast it runs. Parameters (2.78T) Routed experts: 2.72T (97.9%), 82,432 experts of 33M params Attention, shared experts, embeddings, routers: 57B (2.1%) Bytes read for every token, whole cluster, plan recipe (42.9 GB) Non-expert at EXL3 4–5 bpw: 28.9 GB Experts at 2.3 bpw: 14.0 GB At FP8 the non-expert part would be 55 GB a token, and at BF16 107 GB. So the plan squeezes the experts for fit and keeps the rest at a careful 4–5 bits for speed.

Figure 2. Top: parameters. Bottom: bytes read per token with experts at 2.3 bpw and the rest at EXL3 4–5 bpw. At FP8 the non-expert reads would be 55 GB a token; at BF16, 107 GB.

3. Why EXL3

At 2–2.5 bits per weight the format matters a lot. turboderp's published comparison on Llama-3.1-8B gives a KL divergence of 0.215 for EXL3 at 2.25 bpw, against 0.418 for GGUF IQ2_XS with an importance matrix at 2.3 bpw. At 2.5 bpw it is 0.151 against 0.318. AQLM matches EXL3 only at 2.0 bpw, and nothing serves it at this scale.

For K3 itself, the only measured low-bit numbers are GGUF. Unsloth reports KLD 0.178 for UD-Q2_K_XL (802 GiB, slightly too big for 8 Sparks) and 0.378 for UD-IQ2_XXS (662 GiB). If EXL3 keeps its Llama advantage, a 758 GiB EXL3 K3 could land near Q2_K_XL quality while fitting with room to spare. That is the bet. It is an extrapolation, and the plan measures it before building anything on it.

Quality per bit: EXL3 versus GGUF, and measured Kimi K3 GGUF quants Left panel: KL divergence vs bits per weight on Llama-3.1-8B. EXL3 0.312 at 2.0, 0.215 at 2.25, 0.151 at 2.5; GGUF imatrix 0.649 at 2.06, 0.418 at 2.3, 0.318 at 2.5; AQLM about 0.31 at 2.0. Right panel: Kimi K3 Unsloth GGUF KLD 0.56 at 553 GiB, 0.48 at 604, 0.38 at 662, 0.18 at 802 GiB, with budget lines at 776 and 680 GiB. Trellis codes (EXL3) lose about half as much as GGUF i-quants at 2–2.5 bits Llama-3.1-8B: KL divergence vs bits per weight our zone 0.0 0.2 0.4 0.6 0.8 2.0 2.5 3.0 3.5 4.0 bits per weight EXL3 2.25: 0.215 GGUF IQ2_XS: 0.418 EXL3 GGUF + imatrix AQLM 2-bit Kimi K3, measured: Unsloth GGUF KLD vs size 0.0 0.2 0.4 0.6 550 650 750 850 GiB on disk 8 Sparks GB300 IQ1_S 0.56 IQ1_M 0.48 IQ2_XXS 0.38 Q2_K_XL 0.18 Left: turboderp's exllamav3 chart (author-published, Llama models). Right: Unsloth's K3 table (reference UD-Q8_K_XL). The two panels use different references, so compare shapes, not y-values across panels.

Figure 3. Left: quality per bit on Llama-3.1-8B, from the exllamav3 docs. Right: measured K3 GGUF quants from Unsloth. The two panels use different references.

FormatQuality at ~2.3 bpwRuns K3 on our hardware?Verdict
EXL3 (trellis)Best measured per bitNo K3 module yet; single-host multi-GPU onlyBuild
GGUF dynamic (Unsloth)Measured on K3: 0.18–0.38Yes, llama.cpp fork over RPC; the one 8-Spark report is about 2 tok/sQuality yardstick
Cubic (QuantTrio)Unmeasured; calibration-freevLLM fork, x86 build onlyFast-path backup
AQLM / VPTQGood at 2.0 (AQLM)No runtime; calibration on 2.8T impracticalNo
EAQuantScalar INT weights; tested to W2A4 on models up to 47BResearch code; no kernelsBorrow ideas

4. EXL3 or EAQuant?

EAQuant (Fu et al., 2025, MIT license) is post-training quantization for MoE models. It has three parts: smoothing that tames activation outliers, a routing loss that keeps the quantized router picking the same experts, and calibration that oversamples rarely used experts. It was tested on OLMoE-7B, DeepSeek-MoE-16B and Mixtral-8x7B at W4A4, W3A4, W3A3 and W2A4.

It solves a different problem. Its main target is 4-bit activations, which speeds up compute but does not shrink the file. Its weights are plain scale-and-offset integers, which lose more per bit than trellis codes. It runs as simulated quantization with no inference kernels, and it learns per-block parameters, which is slow on a 2.8T model.

EXL3 (exllamav3)EAQuant
Main goalSmallest weights for a given qualityLow-bit activations for MoE models
Weight formatTrellis codes, any bit rate per tensorInteger scale and zero point
Rare expertsBuilds every expert's Hessian from every calibration token; weights output sensitivity by real routingOversamples rare experts until each has enough tokens
RouterKept at high precisionQuantized, with a KL loss on routing
Inference kernelsCUDA, works on aarch64 GB10None
Largest modelsDeepSeek-V3-family architectures (Kimi K2 is one, 1T); our timing anchor is a 753B GLM-5.3 conversionMixtral-8x7B (47B) in the paper

What we keep from EAQuant: its routing check becomes a quality gate. After quantization we measure how often each layer still picks the same 16 experts as the original. Its calibration-balance idea is already covered: exllamav3 sends every calibration token through all experts when it builds their Hessians.

5. What exllamav3 has, and what K3 needs

exllamav3 already supports Kimi Linear, the smaller model K3 grew from. That module brings Kimi Delta Attention, MLA and the sigmoid router. Its MoE block already supports latent experts (experts that work in a smaller 3,584-wide space), and the converter calibrates all experts for Kimi Linear. K3 adds the rest:

K3 featureexllamav3 todayWork
KDA linear attention (69 layers) + MLA (24 layers)Yes, from Kimi LinearCheck against K3's config
Latent MoE (896 experts, top-16, latent width 3,584 with norm)Latent MoE exists (Nemotron-H)Wire it up, add the norm
SiTU activation: β·tanh(g/β)·sigmoid(g) · β′·tanh(u/β′)No (SiLU only for Kimi)New fused kernel
Attention residuals (block size 12)NoNew residual path; the converter must carry up to 8 block states
MLA output gate, full-rank KDA gate, 2 shared expertsPartlySmall additions
MXFP4 source checkpoint (compressed-tensors)No readerUnpack MXFP4 to bf16 while loading
Vision tower (MoonViT)NoText first; store vision unquantized later
Serving across 8 machinesTensor parallel inside one machine onlyMulti-host NCCL, or port to another engine
Experts in Grace memory (GB300)CPU expert offload is AVX-only; stubbed on ARMNEON port, or GPU reads from pinned Grace memory

6. The conversion pipeline

exllamav3 converts one layer at a time. For each layer it loads the weights, runs 250 rows × 2,048 tokens of calibration text through them, builds a Hessian for each tensor, trellis-encodes the tensors, and then runs the quantized layer to make the next layer's input. That last step carries the error forward, so the next layer learns to fix it. It also means layers must go in order.

For K3 the work per layer is large. Calibrating all 896 experts on 512K tokens costs about 4.67 exaFLOP over the whole model. One layer's working set is about 55 GiB of bf16 experts, 74 GiB of fp32 Hessians, and up to 55 GiB of attention-residual states: about 190 GiB. That fits the GB300's 250.7 GiB, not a Spark's 128 GB.

The slow step, trellis encoding, has no order inside a layer. Each expert's tensors encode on their own. So the GB300 can keep calibration and hand the encoding of each layer to a farm: itself plus the 8 Sparks over 200 GbE. exllamav3 already spreads encoding over the GPUs in one machine; spreading it over machines is about a week of new orchestration.

EXL3 conversion pipeline for Kimi K3 Flow: K3 MXFP4 source to NVMe; on the GB300, for each layer: load and dequantize, calibrate 512K tokens through all 896 experts, build fp32 Hessians, trellis-encode the expert tensors on a farm of the GB300 plus eight DGX Sparks, run the quantized layer to produce the next layer's inputs. Output about 758 or 663 GiB, then quality gates. One layer at a time: the GB300 calibrates, nine machines encode, the GB300 moves on Source: K3 MXFP4 1,454 GiB, Moonshot QAT weights Ark or HF → /models NVMe GB300 · repeat for each of 93 layers (92 are MoE) 1. Load + dequant MXFP4 → bf16 in memory 55 GiB of experts / layer 2. Calibrate 512K tokens through all 896 experts 3. Hessians 74 GiB fp32 / layer one per expert input 5. Quantized forward run the encoded layer → inputs for the next layer (error carries) 4. Trellis encode (Viterbi) ~2,700 expert tensors / layer the slow, parallel step next Encode farm: GB300 + 8 DGX Sparks over 200 GbE (one Spark ≈ 1 unit, GB300 ≈ 2.5–3 units) S1 GB10 S2 GB10 S3 GB10 S4 GB10 S5 GB10 S6 GB10 S7 GB10 S8 GB10 GB300 share too weights + Hessians out, codes back (~130 GiB per layer) Output ~758 GiB (2.3 bpw) or ~663 GiB (2.0 bpw) Gates before any serving KLD vs original, layer-streamed router top-16 agreement (EAQuant idea) agent battery vs GLM-5.3 lane exllamav3 already spreads encoding over GPUs in one box (-d). Spreading it over machines is new work: about a week of orchestration. Steps 1–3 and 5 stay on the GB300 because each layer needs the previous layer's quantized output.

Figure 4. The plan's pipeline. Steps 1–3 and 5 stay on the GB300; step 4 fans out to nine machines.

7. How long it takes

One measured anchor exists. BertholomusAI converted the full GLM-5.3 (753B) to EXL3 at 3.0 bpw on one GB10 in about 1.8 days with exllamav3 1.5.3. K3 has 3.69× as many parameters to quantize. Both encoding and all-expert calibration scale with that count.

SetupEstimateHow
One GB10104–159 h, and not practical43 h × 3.69 × (0.65–1.0 for encoding at 2.3 instead of 3.0 bpw). A K3 layer does not fit in 128 GB.
One GB30035–64 hThe GB10 figure ÷ 2.5–3: 160 SMs against 48, about 83 against 31 FP32 TFLOPS
GB300 + 8 Sparks14–28 h15–25% of the work (calibration) stays on the GB300; encoding spreads over 10.5–11 GB10-equivalents; plus about 30 minutes of transfers
One-layer pilot (GB300)23–42 minMeasures the real per-layer time, memory and KLD before the full run
Estimated EXL3 conversion time for Kimi K3 Range bars: measured GLM-5.3 on one GB10 43 hours; K3 on one GB10 104 to 159 hours but not practical; K3 on one GB300 35 to 64 hours; K3 on the GB300 plus 8 Sparks 14 to 28 hours. Time to quantize Kimi K3 to EXL3 (estimates, scaled from a measured GLM-5.3 run) 0 1 d 2 d 3 d 4 d 5 d 6 d 7 d GLM-5.3 753B, one GB10 measured: ~1.8 days (bertholomus) K3, one GB10 not practical: a layer needs ~190 GiB K3, one GB300 35–64 h K3, GB300 + 8 Sparks 14–28 h Scale: K3 has 3.69× GLM-5.3's quantized parameters. Bright part = uncertainty (2.3 vs 3.0 bpw encode cost; GB300 at 2.5–3× a GB10). A one-layer pilot on the GB300 (~23–42 min) replaces these estimates with a measurement before the full run.

Figure 5. Estimated conversion time. Bright segments show the uncertainty range.

What these numbers are not. They scale one public run on different hardware by parameter count. They leave out the K3 module work (Section 5), copying 1.45 TB of source weights, and a second run if we want both the 2.3 and the 2.0 bpw versions. exllamav3 can also convert two bit rates and mix them per layer to hit any size in between, at twice the conversion cost.

8. Serving it

8 Sparks. Split every layer eight ways (tensor parallel 8). Each Spark holds about 94.7 GiB of weights, leaving room for a 256K-token cache. Each token reads about 5.36 GB per Spark. At about 220 GB/s that is 24.4 ms, plus about 7.4 ms of network all-reduces across 93 layers. The ceiling is about 31 tok/s. Real engines reach well under their ceiling.

One GB300. Put the non-expert weights, the cache and the most-used experts in HBM: 201 GiB of experts, about 32% of them. The other 432 GiB live in Grace memory. Published K3 routing studies suggest about three of four expert reads would then hit HBM. The rest cross NVLink-C2C. The ceiling is about 67 tok/s. For comparison, the full GLM-5.3 runs at 61–64 tok/s for one user on this box under TensorFold, with a similar HBM-plus-Grace split.

How the EXL3 Kimi K3 would sit on 8 Sparks or one GB300 Left: eight Spark memory stacks of 115.7 GiB: 94.7 GiB weights, 7 GiB KV, 6 GiB working memory, about 8 GiB spare. Right: GB300 HBM holds non-expert weights, KV and 201 GiB of hot experts (32 percent); Grace holds 432 GiB of cold experts. Two ways to serve the result 8× DGX Spark, tensor-parallel 8 · experts 2.3 bpw S1 S2 S3 S4 S5 S6 S7 S8 weights 94.7 GiB KV 256K 7 GiB work 6 GiB spare 8.0 GiB decode ceiling ≈ 31 tok/s (5.36 GB/rank/token + all-reduce) 1× DGX Station GB300 · experts 2.0 bpw HBM 250.7 GiB hot experts 201 (32%) Grace LPDDR5X 494 GiB cold experts 432 OS + slack 62 C2C misses ~25% of expert reads miss HBM (assumed from K3 routing studies) decode ceiling ≈ 67 tok/s (HBM 6.3 ms + C2C 8.7 ms a token) Ceilings are memory-traffic arithmetic, not predictions. Real engines land well below them. Neither runtime exists for K3 yet.

Figure 6. Memory layouts. The ceilings are memory-traffic arithmetic, not predictions.

Neither layout runs today. exllamav3's tensor parallel stays inside one machine. Its CPU expert offload uses AVX instructions and is stubbed out on ARM, so Grace cannot run it. Each target needs 2–4 weeks of engine work. The quality gate comes first, so that work only starts if the quantized model is worth serving.

9. The plan, with stop points

PhaseWorkTimeStop if
P0K3 module for exllamav3: MXFP4 reader, SiTU, attention residuals, MLA output gate, full-rank KDA gate, latent MoE norm, 2 shared experts. Text only.1–2 weeksPer-layer outputs do not match the reference model
P1One-layer pilot on the GB300; one encode job on a Spark1 dayA layer does not fit, or the time is far outside the estimate
P2Full conversion at 2.3 bpw, then 2.0 bpw1–3 days each—
P3Quality: KLD against the original (exllamav3 compares layer by layer, so the 5.6 TB bf16 model never has to fit); router top-16 agreement per layer; agent tasks against our GLM-5.3 lane and the official K3 API1–2 daysKLD is not below Unsloth Q2_K_XL's 0.18, or it loses to GLM-5.3 on agent tasks
P4a / P4bServing: multi-machine tensor parallel for the Sparks, or Grace-memory experts for the GB3002–4 weeks eachUnder 10–15 tok/s for one user, or broken tool calls

The first real answer, whether an EXL3 K3 beats Unsloth's Q2_K_XL in quality, comes after P3: about 2–3 weeks of part-time work. A served model is 6–10 weeks out.

10. Risks and open questions

11. Sources