The decision this post is about. Four tiers of memory hold Kimi-K3 at full official NVFP4 quality with 2,787 expert slots to spare, and the fabric that connects them was measured working this week. None of that is what decides whether this runs at 15 tokens per second or 3. It comes down to one number nobody has measured: how long the Station's own attention, KDA and router path takes per layer on the NVFP4 checkpoint. At 3.03 ms per layer — the only value anyone has, from a different and much cruder configuration — the pod is a 3.2 tok/s machine and the expert network is 10% of the token. At 0.40 ms it is 15.1 tok/s. Thirty minutes of GPU time buys that answer. This post is the plan that gets it, in an order that does not disturb the DeepSeek lane serving on that Station right now.
Answered since this was published, October 30, 2026: the Station is on a dedicated 20 A circuit, so the facility question that could have voided the whole plan is closed — K3 lands at 88% of that feed. The 2.5-hour measurement window is James's to schedule and nothing in the Spark track below waits on it.
82,432
routed experts, 18,579,480 B each
85,219
expert slots across four tiers
27.7 ms
expert path per token, best map
0.40 ms
per layer, the bet that makes 15 tok/s
15.1 tok/s
if that bet lands
2.5 h
Station window it costs to find out
In plain English
What we are trying to do. Kimi-K3 is a 2.8-trillion-parameter model. Almost all of it — 1,426 of its 1,499 GiB — is 82,432 small "expert" blocks, and any one token only uses 16 of them per layer. No machine here can hold the whole thing. So instead of copying the weights around, we keep every expert where it already sits and send the 7 KiB of numbers that a token produces to whichever machine holds the experts it asked for.
Why we think it works. Four tiers of memory add up to 85,219 expert slots for 82,432 experts — a real surplus, not a rounding trick. The Sparks' part of it (5,704 experts each, 98.7 GiB) is measured: one expert computed inside a Spark's own memory costs 175 µs, and one fetched from another Spark costs 269 µs end to end. Nobody has ever run the whole thing.
Where we are behind, in one sentence each. The Station's attention path has only ever been timed on a crude setup, and that timing (3.03 ms per layer) would strangle the design by itself. There is no engine that splits one model's experts across four kinds of memory — it has to be written, about 7,200 lines. And the Station already pulls 1,361 W serving DeepSeek — 95% of what a 15 A circuit allows, which is why it lives on a dedicated 20 A feed, confirmed October 30. Kimi-K3 takes it to 1,685 W, 88% of that feed.
What will not help. Faster placement maps (3% of the token today). Faster disks (the model never reads disks in this design). Data Direct networking (worth 0.5% here; it is for moving weights, not activations). More Sparks (eight is already more parallelism than the router can use).
What we will do, in order. Build and measure the Spark tier while the Station keeps serving (days, no risk to anything live). Then take one 2.5-hour Station window to measure the attention number and one kernel behaviour, and put DeepSeek back. Only then decide whether to write the engine.
Bottom line. The memory problem is solved and the network is proven. The project now lives or dies on one 30-minute measurement, and the honest label on everything faster than 3.2 tok/s is "not yet measured".
What we verified today
Every number in this section was read off the hardware or out of the checkpoint's own headers on October 30, 2026, not carried forward from the earlier plan. The checkpoint is NVIDIA's NVFP4 release, sitting on the Station's RAID array: 96 shards, 992,508 tensors, and a payload of 1,609,777,087,808 bytes.
| Quantity | Value | How |
|---|---|---|
| Routed experts | 1,531,543,695,360 B (1,426.36 GiB) | 92 layers × 896 experts, summed from the safetensors headers |
| One expert | 18,579,480 B, uniform across all 82,432 | header census; 12 tensors each |
| Non-expert language weights | 77,338,674,496 B (72.03 GiB) | same census, vision excluded |
| Experts per token per layer | 16 of 896 | config.json: num_experts_per_token |
| Activation a dispatch carries | 3,584 values in bf16 = 7,168 B | latent MoE width from the config |
| GB300 HBM | 256,703 MiB (250.69 GiB) | nvidia-smi |
| RTX PRO 6000 Max-Q | 97,887 MiB (95.59 GiB) | nvidia-smi |
| Grace host RAM | 494.50 GiB total, 446.03 GiB obtainable | free -b; the obtainable figure is the October 3 reading with the engine stopped |
| Each Spark | 121.69 GiB, 4.12 GiB used idle | free -b; all eight agree |
Those numbers produce the ledger. Counts are the budget divided by one expert, rounded down; remainders are never pooled between machines. The reservations marked assumed are still assumptions — no K3 process has ever run.
| Tier | Expert budget | Experts |
|---|---|---|
| GB300 HBM, after 72.03 GiB of non-expert weights, KV, KDA state and a 12 GiB workspace | 165.40 GiB | 9,558 |
| Grace, after 12 GiB host and 4 GiB staging | 430.03 GiB | 24,852 |
| RTX PRO 6000, after 6 GiB of context, ring and display | 89.59 GiB | 5,177 |
| Eight Sparks, 62 experts of every layer each | 98.70 GiB each | 5,704 each = 45,632 |
| Total | 1,484.4 GiB | 85,219 for 82,432 — 2,787 spare (51.6 GiB) |
Two things moved since the numbers published two days ago, and both are worth stating plainly.
First, the Sparks are emptier than the October 3 ledger assumed: an idle Spark reports 4.12 GiB used, not
11.19 GiB, so the 100 GiB budget carries 11.57 GiB of headroom per node instead of 4.5.
That is margin, not an invitation to spend it — the 4 GiB reading is an idle node with no expert bank
resident. Second, an equal per-layer split (62 experts of each of the 92 layers, so a bank address is
layer × 62 + rank) costs 75 slots per node against the byte ceiling, which is why the pod
total above is 85,219 and not 85,819. It still closes with 51.6 GiB spare.
The fabric is no longer a plan. Both switches are up with a 400 G inter-switch link, every Spark leg is 200 G, and the Station's two ConnectX-8 ports run at 200 G each. The number that took a week to get is this one: after fixing an IOMMU grouping problem, the Station moves 392 Gb/s between GPU memory and the fabric, both ports, both directions — double what its host-memory path can do, because the host path is a Gen5 x8 link shared with the model RAID array and the house LAN.
And the number the design actually depends on is smaller than the headline. A 7 KiB dispatch is a latency problem, not a bandwidth problem: one such flow moves 12.5 Gb/s, the InfiniBand active MTU is 4,096 bytes so 7 KiB is two packets rather than one jumbo frame, and Spark-to-Spark it takes 4.54–4.66 µs one way (p99 4.73), or 5.82 µs across the inter-switch link. Data Direct buys us 3.3 µs per layer on this traffic — half a percent of a token. It matters for pushing weights, not for sending activations, and this post will not pretend otherwise.
The number the project hangs on
A token is 92 layers, and each layer is: attention or KDA, then a router, then 16 experts, then a sum. The 16 experts can be computed in four places at once, so a layer costs the Station's own serial work plus the slowest of the four tiers. Sum that over 92 layers and you get the token.
The four-tier part of that sum is now a real number: 27.7 ms per token, with the cost-ranked placement below. The Station's serial part is not. The only measurement anyone has — 3.03 ms per layer, 279 ms per token — came from a Q2_K GGUF checkpoint served by SGLang with a Triton eager attention path and artificial graph breaks. That is not the configuration we would run, and 7.2 ms for a single MLA layer at batch 1 is the signature of an eager-mode pathology, not silicon.
The crossover is at about 0.35 ms per layer. Below it, placement and the expert network are most of the token and worth optimising. Above it, the answer to "why is this slow" is never the network. At today's measured value, every hour spent on the placement map buys 3%.
Why the expert path costs 27.7 ms, and why placement is not the lever
Four policies, simulated on the real routing trace from the 2026-10-03 run (4,705 tokens with a complete 92-layer vector, from four Hermes agent transcripts):
| Placement | Rule | Expert path | tok/s at 0.40 ms/layer |
|---|---|---|---|
| Contiguous per layer | each tier gets a fixed slice of every layer, popularity-blind | 38.6 ms | 12.9 |
| Global hot/cold | hottest experts to HBM, then Grace, then the 6000, coldest to Sparks | 31.1 ms | 14.3 |
| Layer-balanced hot/cold | the same, with each layer's quota scaled by its call volume | 30.4 ms | 14.5 |
| Cost-ranked hot/cold | order the tiers by measured microseconds per expert, not by distance: HBM, then the 6000, then Grace, then Sparks | 27.7 ms | 15.1 |
The winner is one line different from the obvious answer. Ordering tiers by physical distance puts the second-hottest band of experts in the slowest tier; ordering them by what a dispatch actually costs gives the RTX PRO 6000 the hottest middle band instead of a cold one. It is worth 11% over plain hot/cold and 40% over a popularity-blind split — and it is still only 10% of the token at today's Station time.
The floor of the design is a fixed cost, not a bandwidth limit. A Spark hop costs 184.5 µs before it computes anything, and 84.5 µs per expert after that. With the coldest 42,845 experts spread over eight Sparks, 73% of layers have to talk to at least one Spark, and one hop is more than any other tier's whole work. That is why the pod's expert path cannot go below about 25 ms per token: the four-tier ledger buys capacity, not speed.
Two honesty notes on that chart. The 184.5 + 84.5 fit comes from two measured points (one expert, 269 µs; three experts, 438 µs) and is extrapolated past three; a separate measurement of the Spark's own kernel at eight experts (730 µs) suggests the real eight-expert end-to-end number is about 820 µs rather than the fit's 860. And the 93 µs fixed cost of a sidecar call was measured on this Station for DeepSeek-V4.1, not for Kimi-K3. Both are on the list to measure.
The plan: build the Sparks, buy one number, then decide
DeepSeek-V4.1 is serving on the Station right now — it holds 247 GiB of HBM, 95.7 GiB of the RTX PRO 6000 and 296 GiB of Grace — and James's instruction for this round is to leave it serving. That is not a blocker, because the tier that needs the most work needs none of those things. The Sparks are idle, clock-capped and empty.
Track A — the Spark tier, starting now
The expert-bank builder already exists and has been validated against the real checkpoint: it streams
18.58 MB records out of 96 shards, and it refuses to write until it has verified something nasty — the
experts inside each shard are stored lexicographically (0, 1, 10, 11, … 2, 20, …), so any builder
that assumes base + id × stride silently assembles the wrong expert. The dry run against the
real checkpoint plans 5,704 experts and 98.699 GiB per Spark across 17,112 scattered reads.
Next: build and stage the eight banks (847.8 GB total, hash-verified, resumable), then a worker service that takes 16 global expert ids and returns rows plus the ids it actually served — so the Station can assert every layer's 16 experts were accounted for exactly once, every layer. Then the measurement that matters: the hop curve at k = 1, 3, 6, 9 and 16, driven by real routing vectors rather than synthetic ones. Then a standalone bench where all eight Sparks replay a recorded token stream and serve it with no Station involved. That last step produces two numbers this plan currently has to guess at: the real end-to-end expert-path cost, and the real wall power per Spark at K3 duty cycle.
The Station window — 2.5 hours, not a campaign
Four things happen in one window, and then the lane goes back up:
- Archive first. Both containers' inspect JSON, environment and launch line, and a restore ETA read off their own boot logs rather than remembered.
- Measure the Station's serial term. Bind a hook that returns zeros from the routed
MoE call after the router, run one token through 92 layers on the NVFP4 weights, and time it. That is
92 × Sdirectly, to about 2%. It is the whole point of the window. - Probe one kernel behaviour. The split design assumes the NVFP4 routed-MoE kernel ignores expert ids of −1, which is how a masked-out remote expert would be expressed. That is undocumented. Twenty minutes answers it, and a "no" changes the design before anyone writes code against it.
- Take the missing fabric constant — 7 KiB latency from the Station to one, two, four and eight Sparks — and then restore DeepSeek.
The window has a cost that should be said out loud: stopping that lane takes :30006 down, which strands this Milo profile, the Alyx profile, the default profile's alias and the guest front door. Which means the window has to be executed by a session that is not itself running on that lane, or by a script with a watchdog. Restore is about ten minutes, measured rather than assumed.
What has to be built — and the discovery that changed it
No shipped engine splits one model's experts across four kinds of memory. The Station already runs an expert sidecar for DeepSeek-V4.1 — the RTX PRO 6000 computes a band of experts while the GB300 computes the rest — so the obvious plan was to extend that pattern to the Sparks. Reading the code closely enough to plan against it killed that shortcut, for a reason worth writing down:
The sidecar pattern does not transfer, because the quant is different. The DeepSeek lane runs through a "mega MoE" expert class whose weights are MXFP4 (a scale per 32 values). The Kimi-K3 checkpoint is NVFP4 with a scale per 16 values plus a per-expert global scale, and it takes an entirely different kernel path — one whose routed kernel takes a local expert offset and ignores the expert map entirely. So the split cannot be "the same hook with a different row map". It has to be a mask on the router's chosen ids plus a modular prepare/finalize seam, and the existing wire protocol's compiled constants (64 layers, top-6, hidden 5120) have to be rebuilt for Kimi-K3's 92 layers, top-16 and 3584-wide activations.
| Piece | What it does | Scale |
|---|---|---|
| Planner | cost-ranked placement from the routing histogram, per-tier byte accounting, per-layer repair so no Spark holds zero or too many experts of a layer | ~350 lines |
| Split hook | remap ids to rows, mask the remote ones out of the local call, dispatch, sum partials | ~700 lines |
| Grace arena | 430 GiB of pinned host memory read in place by row — the warm set is the pin, not a copy of it | ~400 lines |
| Protocol and kernels | Kimi-K3-shaped segment protocol, the Station-side split/send/reduce, and a persistent NVFP4 expert kernel on each Spark | ~2,000 lines, of which ~1,200 CUDA |
| Fabric | RDMA queue pairs to nine peers, one-sided writes of the activation block, writes back of rows | ~550 lines |
| RTX sidecar for K3 | the same idea the DeepSeek lane already uses, rebuilt for Kimi-K3's shapes and activation function | ~500 lines |
| Tests and reference | fp32 per-expert reference, cosine gate, per-tier replay checks, protocol fuzz | ~400 lines |
| Total | ~7,200 lines | |
The riskiest piece is not the plumbing: it is the Spark-side persistent kernel, which owns the SMs and therefore cannot use CUDA graphs, and whose handshake with the NIC has to replace something a GPU would normally do with a system-scope release. The cheapest thing that would delete most of this work is an RDMA-capable all-to-all backend that the Kimi-K3 path accepts — worth checking before writing a kernel.
One more fact to keep the target honest. The only Kimi-K3 configuration that has ever served on this hardware ran 0.07–0.11 tokens per second — an engine streaming experts from NVMe with 17,961 of 82,432 experts cached in VRAM. Even at today's unflattering Station time, this pod costs about 308 ms per token, which is roughly eleven times faster than that. The four-tier design is not a marginal improvement on what exists. It is the difference between a demo and a tool. Whether it is a fast tool is the one measurement left.
What can stop it
- Power at the Station — answered. The box draws 1,361 W at the wall serving DeepSeek, which is 95% of what a 15 A circuit allows and 71% of the dedicated 20 A feed it is on (confirmed October 30, 2026). Kimi-K3 pushes it to about 1,685 W: 88% of that feed, with roughly 235 W of margin, and the documented worst case still lands at 94%. What is still missing is not headroom but a meter — the Station exposes no input-power reading, so both figures are estimates at a PSU efficiency, not measurements.
- Heat in the room. Two boxes plus a rack land at 2.6–3.0 kW sustained, which is 70–79% of the whole floor's sensible cooling capacity. The Sparks' own 10-minute soak climbed 1.7 °C per minute at only 32.5 W of GPU power and never plateaued. Days-long runs need a mini-split, or a night-only schedule.
- The rack itself. At K3 duty the eight Sparks plus switches draw about 1,217 W against a 1,440 W strip — inside, with 223 W of margin, which is less than one node's prefill burst. The 2,200 MHz clock cap on all eight Sparks is what keeps it inside; lifted, the rack is 105–193 W over.
- Single-fed switches. Both CRS804s report no input on their second power supply. One cord, one fabric outage, and the Sparks do not come back on their own after a mains loss.
- The masked-id answer. If the NVFP4 routed kernel does not tolerate a −1 expert id, the split needs a different seam and the engine estimate changes.
- A hot set that does not transfer. A placement map built on one traffic population keeps only about 35% of its top 9,558 when the work changes — decode routing to agent traffic, or the reverse. The map has to be built on the workload it will serve, and the workload has to be chosen before the map is.
Not this
- Not the DeepSeek sidecar path with a new row map. Different quantization, different kernel, different shape constants. The idea transfers; the code does not.
- Not placement tuning before the Station number exists. It is 3% of the token at the only measured value.
- Not Data Direct as a dispatch optimisation. 229 KB per layer; it saves 3.3 µs. It is a weight-moving feature.
- Not one tensor-parallel group across Station and Sparks. Different compute capability, no GPUDirect on the Sparks, and the Spark would set the pace.
- Not NVMe as a tier. 1,347 µs per expert against 269 µs from a Spark. Disk is what happens when a tier is missing, not a place to keep the model.
- Not a fleet kernel or driver move, and not a switchless ring emulation, to satisfy anyone's recipe.
Open questions
Answered October 30, 2026: the Station is on a dedicated 20 A circuit (gate G4), and the 2.5-hour window is James's to schedule — the Spark track does not wait on it.
- Mini-split, or night-only campaigns? 2.6–3.0 kW for days is a different facility requirement from an afternoon of testing.
- When the window comes, who operates it? It needs the DeepSeek lane down, and it needs an operator that is not itself running on that lane.
- Which workload defines the hot set? Agent turns, or long-context retrieval? The map must be built for the one we intend to serve.
- What counts as "running"? Coherent text on the four-tier split, or nothing short of parity against a slow reference.
Provenance
Measured October 30, 2026 unless stated. The byte census (82,432 experts at 18,579,480 B, 1,426.36 GiB of
experts, 72.03 GiB of text non-expert weights) was re-derived today from the 96 shards' safetensors headers
on the Station and matches the October 3 census to the byte. Device sizes are nvidia-smi and
free -b readings from today; the Sparks' 4.12 GiB idle baseline is from all eight, today, and
supersedes the 11.19 GiB figure measured on October 3 while a 51 GiB file mapping was charged against
availability. The 269 µs, 438 µs, 175 µs and 730 µs expert costs, and the 3.03 ms per-layer Station term,
are October 3 measurements on this hardware, from a Q2_K GGUF checkpoint — not from the NVFP4 checkpoint
this plan will run. The Spark-to-Spark 4.54–4.66 µs and 5.82 µs figures, the 12.5 Gb/s single-flow number
and the 4,096-byte active MTU were measured today. The 392 Gb/s Station figure is the October 30 Data
Direct measurement after the IOMMU fix. Grace's 53 µs and the NVMe 1,347 µs are arithmetic on measured
bandwidth and are labelled as such on the chart. The 93 µs sidecar cost was measured on this Station for
DeepSeek-V4.1, not Kimi-K3. The routing histogram is from October 2 (26.2 million selections) and the token
trace from October 3 (4,705 complete tokens); both are the DERISKED checkpoint's router, which is assumed
unchanged from the official one. The wall-power readings are Home Assistant sensors sampled today:
502 W for the rack with the Sparks idle, 1,361 W for the Station. K3's per-Spark duty-cycle power is the
one unmeasured number in the power plan and is given as a 120–190 W band. Every reservation in the ledger
is an assumption and says so.