GhettoRak's second switch: the goal is Kimi-K3 running

The second MikroTik CRS804 arrives tomorrow. It is not an upgrade to the rack's speed. It is the rack's first fabric path to the GB300 Station, and the opening move in the only more-than-one-terabyte model this lab can physically hold. This page is the plan: the goal, what the switch buys, two things it does not, and the six experiments in the order the goal needs them. Nothing here has been measured yet except the numbers labelled measured below.

The correction first, because it is the part that matters. The second switch does not add KV cache to the GLM-5.3 lane, and it does not make a Spark-only TP8 group faster. A tensor-parallel group replicates MLA KV on every rank, so a ninth rank does not enlarge the pool. And the port arithmetic forces a 4+4 split behind a single 400G link, which puts a new hop in the middle of every cross-switch collective. What the switch actually unlocks is the Station on the fabric — and with it the Kimi-K3 pod, the only checkpoint here that needs more than a terabyte.

The goal: Kimi-K3 running

Updated October 28, evening. James set the goal for the second switch: get Kimi-K3 running. Everything below is now either a step toward that or secondary to it.

Kimi-K3 is Moonshot's 2.8-trillion-parameter open model. In NVIDIA's NVFP4 release it is 1,499.5 GiB, and 1,426 GiB of that is routed experts: 896 small experts per layer, 16 chosen per token. No box here holds it. The Station plus eight Sparks does, by about 4.88 GiB (where a terabyte comes from). The design that uses that memory is a split by role: the Station runs attention and keeps the KV cache and the 72.9 GiB of non-expert weights; each Spark holds a share of the experts and runs them itself. Per layer, the Station sends a token's 7 KiB activation to the Spark that owns the expert and gets 7 KiB back. Weights never cross the wire. That is why the Station needs a fabric port, and why this switch is the prerequisite.

Running is reachable. Fast is not proven. The network is not the worry: the wire is about 11 µs of a 438 µs remote branch. The worry is the Station's own work per token. On October 3 we measured it on the only Kimi-K3 lane that boots on the Station today — a 2-bit GGUF with Triton attention, no CUDA graphs: 279 ms per token across 92 MoE layers, a 3.6 tokens-a-second ceiling before a single expert leaves the box. Most of that is the 24 full-attention layers at about 7.2 ms each. The NVFP4 attention path has not been measured. Until it is, 15 tokens a second is a target, not a forecast.

What it costs: every Spark and the whole Station for the duration. Both Spark serving lanes go down, and the Station stops whatever it is serving. Staging 1,426 GiB of experts into Spark memory took 1.02 GB/s over the old 10G management path, about 25 minutes for the full set; the fabric should cut that, not yet measured.

What it is not: a compressed Kimi-K3 squeezed onto the eight Sparks alone. That needs about 2.3 bits per expert, is a real quality loss, and does not need this switch. It stays a separate idea.

What the switch buys, and what it does not

One CRS804 has four QSFP56-DD cages. Each one breaks out to 2 × 200G, so one switch feeds exactly eight Sparks and nothing else — which is why all eight have been on a single switch since October 16. The Station has been on 10G management, outside the fabric, because there was no cage left for it.

Adding a second switch does not give the fabric more Spark ports. The Station needs a cage on each switch to reach every Spark in one hop, and the two switches need a cage to talk to each other. Four cages minus those two leaves two for Sparks per switch: 4 + 4. That is the whole design, and it is forced.

• Buys: a 200G fabric path from the Station to all eight Sparks, 400G aggregate at the Station across the two switches. Unblocks the expert/attention split that the Kimi-K3 pod needs, and unblocks the multi-node constants the MoE placement solver has been waiting on.
• Buys: weight staging over the fabric between Station and rack instead of through 10G management.
• Does not buy: any KV cache for GLM-5.3. That is a config change, not hardware (see the KV question).
• Does not buy: TP8 speed. It costs a little, and experiment E1 is there to price it instead of assuming.
• Does not buy: GB300 and GB10 in one serving group. Why not one big group.

The topology, and the hop it adds

The rack was built on one backplane. After the split, a TP8 all-reduce sends roughly half its traffic across the 400G link, at 2:1 oversubscription. The band that matters is not the big-message number: a tensor-parallel step carries tokens × hidden × dtype bytes per collective, and this rack's model has hidden 5,120 in bf16, so 10 KiB per token. A 50–200 token batch is 0.5–2 MiB per collective. That is exactly the band where a badly configured fabric falls over.

We already have the control. Measured on Sparks 5–8 on a single switch on October 26, with NCCL 2.30.7, 20 warm-up and 100 timed iterations: default NCCL gives 1.36 GB/s at 1 MiB (1,154 µs per call) but 22.07 GB/s at 256 MiB. With NCCL_PROTO=^LL128 and NCCL_MAX_NCHANNELS=8 — the pair both production lanes already carry — the same 1 MiB cell measures 11.50 GB/s (137 µs) and 256 MiB measures 22.78 GB/s. The pool gate that came out of that run: 1 MiB ≥ 9 GB/s and 256 MiB ≥ 22 GB/s under the lane's own environment.

We priced the 2:1 trunk before buying the switch: the September comparison put the inter-switch hop at about 1 µs, under 0.5% per token, so what remains is bandwidth, not latency. The control curve above, and the NCCL pair behind it, are written up in NCCL tuning in plain English.

E1 re-runs that exact curve on the 4+4 topology and compares. My prediction, written down before the measurement: the 512 KiB–2 MiB cells get worse, because the half of each collective that crosses the link now shares 400G instead of a switch backplane. If it is worse by more than about 10%, the honest conclusion is that CRS804-2 is a Station switch, not a Spark switch — keep tensor-parallel lanes at four Sparks per switch, where they never cross the link, and publish the cross-switch number as a regression rather than a topology change.

Where a terabyte comes from

Eight Sparks give 925.6 GiB of usable memory. The Station adds 250.7 GiB of HBM — reported by the driver as 256,703 MiB — and 494 GiB of Grace host memory. Reserve a working set for the OS and the engines and the honest total is about 1,576 GiB. Kimi-K3 in NVFP4 is 1,499.5 GiB, of which 1,426.4 GiB is routed experts and 72.9 GiB is everything else: attention, shared experts, embeddings, the head, vision.

The pass was done properly, expert by expert, from the safetensors index rather than from the model card: at 18,579,480 bytes per expert, the per-node budgets close with 282 spare experts, 4.88 GiB. Thin. It closes only because the measured Spark ceiling is what it is; rounding each Spark's budget down to a round 100 GiB opens 1,790 homeless experts, and a 30 GiB workspace growth does it too.

Note what this chart also says: GLM-5.3 does not need the Station's memory at all. FP8 is 703.8 GiB, which is 88.0 GiB per rank at TP8 and fits — with 8.8 GiB per rank left for KV, which is exactly why that lane serves 131K context and not more. NVFP4 is 433 GiB on disk, 57.1 GiB per rank, and leaves 37.8 GiB per rank for KV. Both already fit eight Sparks. The Station's contribution to GLM-5.3 is compute, not capacity, and that is a different experiment.

The KV question

Here is the number that governs everything else. The daily GLM-5.3 NVFP4 lane measured 57.13 GiB of weights and 37.82 GiB of KV per rank at 512K context, and that 37.82 GiB is 720,384 tokens. Divide: 56.4 KB per token, per rank. 78 layers of MLA with a 512-wide latent plus 64 rope dimensions, fp8, plus the sparse-attention indexer's side cache.

Three consequences, in order of how much they cost:

1. A ninth rank does not help. The pool lives on each rank; the Station would hold its own copy for its own shard, or hold nothing. There is no version of this where the Station's 250 GiB becomes KV for the Spark group.
2. A one-million-token request still does not fit. 1M tokens at 56.4 KB is 52.5 GiB on the rank holding it, against a 37.8 GiB pool. That matches the earlier campaign result exactly: 52.68 GiB required against 52.60 GiB available, so 1M was rejected at mem-fraction 0.88, and about 783K tokens per stream is the ceiling with speculation off.
3. Data-parallel attention is the real lever, and it is free. --enable-dp-attention --dp-size 8 splits the batch across eight ranks instead of splitting each tensor. Each rank keeps its own 720K-token pool and serves its own requests: 5.76M tokens of aggregate capacity, and roughly eight concurrent long streams where there are about one and a half today. The cost is that attention weights replicate per rank — a few GiB — so per-rank KV drops slightly. Nothing was measured here yet; it is experiment E4.

What would give a longer single request is context parallelism, splitting one sequence across ranks. I checked the engine build in the serving image today: there is no dcp_size flag in it. So on this engine, "1M context on eight Sparks" is not a tuning problem, it is a feature that is not there.

Six experiments

Order, now set by the Kimi-K3 goal: E0 first, then E2 and E3 (the Station link and its constants), then the Kimi-K3 build. E1 is off Kimi-K3's critical path — the pod talks Station to Spark, one hop on each switch, and never needs the inter-switch link — but it decides how the Spark serving lanes come back. E4 is now secondary and runs only if it does not delay Kimi-K3.

E5 is the one that makes the terabyte real, and it is not a tomorrow experiment. Projecting from published numbers rather than measured: 1,426 GiB of experts cannot be read per token from anywhere — the winning designs read a router-selected slice per layer and place experts so the remote fan-out is balanced. On the Station plus eight Sparks the placement accounting closes with 4.88 GiB spare, and every MoE layer then pays a fixed cross-node charge that this fabric has never had measured. That number is what E3 produces and what E5 consumes. If the remote branch comes in near the 81 µs that this pool measured for a 12 KB eight-rank all-reduce, the per-layer budget has room; if it lands at multiples of it, the model is a proof of fit, not a lane.

Why not one big tensor-parallel group

The obvious reading of "more than a terabyte across the Station and the Sparks" is one group with nine ranks. It is the wrong answer, for three reasons that compound:

• The slowest rank sets the step. A GB10 sustains roughly 273 GB/s of memory bandwidth; the GB300 in the Station is an order of magnitude beyond that. Every step waits on the Sparks, so the Station's share of the weights is dead weight.
• GB10 has no GPU-direct. Cross-node collectives are host-bounced, which is why this pool's small-message all-reduce is measured in tens of microseconds for a message that fits in a pocket.
• Different silicon, different kernels. The two boxes are different architectures, and the quantized kernel paths are not the same code. A single group would have to be the intersection of what both can run fast, which is neither one's best path.

The version that works is a split by role, not by rank: the Station keeps the non-expert 72.9 GiB and the KV, the Sparks hold experts, and per-layer traffic is a fixed charge rather than a per-token byte cost. That is a custom engine, and it is the difference between a 4-week project and a wasted afternoon.

What can stop it tomorrow

1. Power. The strip feeding the rack is full, and it is a 15 A / 120 V device: 1,440 W continuous. The capped rack measured ~1.35 kW on a long prompt with one switch; a CRS804 draws about 56 W. That is a ~1.4 kW rack against a 1,440 W line, before the second switch's own feed. Measured per-node basis: 135 W per Spark decoding capped, 147 W on a long prefill capped, 176 W stock. Split the load or don't run the long-context rungs.
2. Cable reach. The Station breakout on hand is 1 m. Passive 200G DAC is good to about 3 m, so 1 m only works if the Station is beside the rack. Measure before mounting anything.
3. The Station is busy. It is running a split-KV promotion bracket right now, with 366 GiB of Grace pinned. The fabric work does not need it; the Station's own legs get plugged in when that campaign ends.
4. The clock cap stays. Every rung above runs with the 2,200 MHz GPU cap in force, verified from the driver under load rather than from the advertised maximum.

Provenance

Published numbers on this page are measured, and each one is dated: the eight-node fabric and power figures are from the GhettoRak build page (October 16, 2026); the GLM-5.3 per-rank weights and KV pool are from that lane's own logs (October 16); the NCCL curve is the October 26 run on Sparks 5–8; the memory totals, the KV per-token arithmetic and the DP-attention finding were read from the live boxes on October 28. The Kimi-K3 checkpoint size and its expert accounting are from the pod's own capacity pass (October 3). Everything about the new topology, the 4+4 split, the pass bars, and experiments E0–E5 is plan. The Kimi-K3 rungs, remote-hop timings and the 279 ms Station serial term are from the pod's October 3 runs; the 25-minute staging figure is arithmetic on the measured 1.02 GB/s: predicted, not measured. Where a figure is an estimate it says so.

Anyone copying this: the port arithmetic is the part that transfers. Four cages per switch, break out two legs each, and count what the inter-switch link and the Station cost you before you decide how many Sparks go where.