GhettoRak: an 8-node DGX Spark cluster

Created August 22, 2026 Updated October 16, 2026 at 10:50 PM CDT

by James & Milo
Earlier versions: Milo on grok-4.6, claude-opus-5-5, grok-4.7 and claude-fable-5-1 (reasoning modes not recorded unless noted). October 2 photos and brevity rewrite by Milo · claude-fable-5-1 / Anthropic · extended thinking. October 12 fabric bring-up and October 13 rewrite by Milo · claude-opus-5-5 / Anthropic · reasoning mode not recorded. October 16 power-trip, clock-cap, photo and CAD v4 update by Milo · claude-opus-5-5 / Anthropic · reasoning mode not recorded.

Eight DGX Sparks on a kitchen wire rack, wired into a 400G MikroTik switch with one 200G cable per Spark. No 19-inch cabinet, no NVIDIA Spectrum switch. This page is the build as it stands, how to copy it, and what is left.

Where it is, October 16, 2026. All eight Sparks are on the 200G fabric of one CRS804. 8-node NCCL all_reduce: 20.13 GB/s bus bandwidth at 1 GiB (23.49 at 32 MiB), zero switch errors, switch CPU 1%. The rack now serves the full GLM-5.3 744B tensor-parallel 8: daily lane RadixArk NVFP4 (512K context, 25.8–35.7 tok/s single-stream with three-token MTP, 61.7 at 4 users), with the official FP8 (131K, 23.1–29.9 tok/s) one relaunch away. Those speeds are at stock clocks. That evening a long-prompt test held the rack at ~1.73 kW at the wall for about eight minutes, and the power strip feeding the whole rack tripped. Since 21:31 CDT every Spark runs a 2,200 MHz GPU clock cap: the same prompt now holds the rack at ~1.35 kW and finishes, with no decode loss. Power. DeepSeek-V4.1-Flash on Sparks 3–6 is stopped. Still to come: the second CRS804, then the GB300 Station. Next steps.

Sparks, one per shelf

8 / 8

on the 200G fabric

20.1 GB/s

8-node all_reduce busbw, 1 GiB

1 + 1

CRS804: installed + on order

1.72 → 1.35 kW

long-prompt mean at the wall, stock → capped

2,200 MHz

GPU clock cap, all eight

Next steps

#	Step	Waiting on
1	Sparks 1–2 onto the switch. Cage 4, QSFP port 0 on each Spark (all eight must match). They come off their direct-attach link (the DS4 pair), so plan a short window if anything is running on them. Done October 16: cage 4, bond0 removed, both PCIe functions on 10.0.0.N / 10.0.1.N.	—
2	8-node all_reduce and the first all-eight power run. Done October 16: 20.13 GB/s busbw; 169–189 W per Spark, ~1.5 kW rack under GLM-5.3 FP8 prefill. Temperatures not yet logged. Later that evening a 1M-context long-prompt run reached ~1.73 kW and tripped the strip; see Power.	—
3	Power headroom. Clock cap on all eight, done October 16: the same long prompt now averages ~1.35 kW; peak 1,425 W over a 16-minute ~984K prompt. Next check: the same load with CRS804-2 added (one CRS804 draws ~56 W). We expect the capped rack to stay in a sane envelope; a second wall outlet is the fallback if it sits at the 1,440 W line.	CRS804-2
4	Second CRS804. Split the Sparks four per switch, join the switches with the 400G DAC already on hand.	Switch delivery
5	GB300 Station onto switch 2: one breakout, a 200G leg into each of its two ports (cable on hand). Last, once the rack stops moving.	Steps 1–4
6	RouterOS update (on 7.24.2) in a quiet window.	—

Network

Each Spark has two QSFP56 cages on one ConnectX-7, sharing about 200G of PCIe. One cable per Spark. Plugging the second cage does not make 400G. A CRS804 has four 400G QSFP56-DD cages, each split to 2 × 200G with a breakout DAC, so one switch takes exactly eight Sparks and nothing else. That is why there will be a second one: to bring the Station onto the same fabric.

In Linux the one cable shows up as two devices, enp1s0f0np0 and enP2p1s0f0np0: two PCIe functions feeding the same 200G port, not two rails. One alone measured 111.9 Gb/s with ib_write_bw; it takes both to fill the port. Each gets its own subnet, and NCCL is told to use both.

As built, October 16

The fabric on October 16, 2026: one CRS804, all eight Sparks live. Sparks 1 and 2 moved from their direct-attach pair link to cage 4 that morning. The port-to-Spark map was read off the switch, not traced by hand.

CRS804 cage	Leg -1	Leg -5	DAC	Link
1	Spark 8	Spark 7	1 m	200G, FEC91
2	Spark 6	Spark 5	1 m	200G, FEC91
3	Spark 4	Spark 3	2 m	200G, FEC91
4	Spark 1	Spark 2	2 m	200G, FEC91

Results

Test	Result
Jumbo ping, 8972 B, don't-fragment, every pair, both subnets	60 of 60 (six nodes); 112 of 112 (all eight, October 16)
RDMA write, one PCIe function at a time (`ib_write_bw`)	111.9 Gb/s each
NCCL all_reduce, 2 nodes, 1 GiB	23.08 GB/s busbw
NCCL all_reduce, 6 nodes, 1 GiB	22.75 GB/s busbw ≈ 182 Gb/s
NCCL all_reduce, 8 nodes, October 16	20.13 GB/s busbw at 1 GiB; 23.49 at 32 MiB; 5.4 at 8 MiB
Switch after ~76 GB through each port	0 errors, 0 drops, 0 pause frames; CPU 0%. Eight-node run: no new errors, CPU 1%

A public 6-Spark CRS804 build reported 184 Gb/s. Method: torch.distributed all_reduce, one rank per Spark in NVIDIA's PyTorch 25.09 container (NCCL 2.27.7), float32, 5 warm-up + 20 timed, slowest rank counts, busbw = 2(n−1)/n × bytes ÷ time. NCCL's log shows both PCIe functions in use over RoCE v2 with GPU Direct RDMA (DMABUF) enabled. Stack: RouterOS 7.24.2, DGX OS 7.2.3, kernel 6.17.0-1031-nvidia, ConnectX-7 firmware 28.45.4028.

Recipe

Switch. One bridge holding the eight lane-master legs with hardware offload, speed forced, run lossy. On 7.24.2, QoS trust and PFC were already off on every leg (/interface/ethernet/switch/qos/port/print to check). Management stays on the RJ45 port, outside the bridge.

/interface/bridge/add name=fabric
:foreach p in={"qsfp56-dd-1-1";"qsfp56-dd-1-5";"qsfp56-dd-2-1";"qsfp56-dd-2-5";"qsfp56-dd-3-1";"qsfp56-dd-3-5";"qsfp56-dd-4-1";"qsfp56-dd-4-5"} do={
  /interface/ethernet/set [find name=$p] auto-negotiation=no speed=200G-baseCR4 fec-mode=fec91 l2mtu=9216 mtu=9000
  /interface/bridge/port/add bridge=fabric interface=$p hw=yes
}

Sparks. One netplan file per Spark (N = Spark number), NetworkManager renderer, no gateway. First delete the auto-created DHCP profiles ("Wired connection N") that DGX OS leaves on both fabric devices. Apply one Spark at a time and reboot the first to prove it persists.

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp1s0f0np0:   { addresses: [10.0.0.N/24], mtu: 9000, dhcp4: no, dhcp6: no, accept-ra: no, link-local: [] }
    enP2p1s0f0np0: { addresses: [10.0.1.N/24], mtu: 9000, dhcp4: no, dhcp6: no, accept-ra: no, link-local: [] }

NCCL. NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0, NCCL_SOCKET_IFNAME=enp1s0f0np0, NCCL_IB_ROCE_VERSION_NUM=2. Pin the socket interface, or with host networking NCCL will pick Docker or Tailscale for bootstrap.

Target layout, two switches

Switch	Cage 1	Cage 2	Cage 3	Cage 4
CRS804-2 (14 in)	Spark 1 + 2 (2 m)	Spark 3 + 4 (2 m)	400G to CRS804-1	GB300 Station, 2 × 200G
CRS804-1 (10 in)	Spark 5 + 6 (1 m)	Spark 7 + 8 (1 m)	400G to CRS804-2	free

The Station's ConnectX-8 cages are QSFP112 (4 × 100G lanes); the CRS804's are 8 × 50G. A plain cable cannot do one 400G link between them. A Spark-style breakout can: each 200G leg runs 4 × 50G, which the Station's cages accept, so the Station gets 2 × 200G from one switch cage. That is spec-supported but not yet tested on this Station. A true 400G per port needs DR4 optics and fiber, about ten times the price; only worth it if 2 × 200G turns out to be the limit.

The rack

A Shelving Inc SC121854-5, 12″ × 18″ × 54″, built sideways so 18 inches is the depth: a CRS804 is 387 mm deep with its PSU handles and 391 mm clears between the collars. Eight extra SI-S1218C decks make thirteen shelves. Posts uncut, 5″ threaded casters (3″ rubber ones rolled hard on carpet).

Rules: fronts face you, all I/O exits the rear, one Spark per shelf (two will not fit side by side: 340 mm needed, 238 mm clear). Blue nylon zip ties through the mesh hold everything; nothing printed holds anything down. No extra fans. Do not back the gold intake into a wall.

The stack

OpenSCAD isometric of GhettoRak, October 16 layout: a bare top shelf, eight gold DGX Sparks on separate chrome wire shelves on a 4-inch pitch zip-tied with blue ties, then two grey CRS804 switches and a white UniFi switch with its brick on the same 4-inch pitch, and on the base a tied row of eight power bricks beside two white Tapo P316M strips, uncut posts on 5-inch casters. — OpenSCAD, October 16 layout (v4). Top to bottom: top shelf at 50″, Spark 1–8 on a 4″ pitch (46″ → 18″), CRS804-2 at 14″, CRS804-1 at 10″, UniFi Pro XG 8 PoE at 6″ (switches also on 4″), bricks and two P316M strips at 1″. The grey vertical between the switches is the 400G inter-switch DAC. Source: inference-box-ghettorak.scad.

Front view of GhettoRak: bare top shelf, eight Sparks stacked on wire with gold intakes facing the operator, two switches, UniFi, bricks and strips. — Front. Sparks face the operator.

Right view: 18-inch depth holds both CRS804s; Sparks occupy only the front 150 mm; Cat6a and DACs run down the rear. — Right. 18 in is depth so the CRS804 fits. Cat6a drops down the rear to the UniFi.

Photos

The rack as it stands on October 16, 2026. Photos by James. HEIC→JPEG and resized; the Matter setup-code labels on the power strips are pixelated in the two front views. Otherwise unaltered.

GhettoRak from the front on October 16: a tall chrome wire rack on casters with one gold DGX Spark per shelf, a green clip label reading Spark 1, empty decks below the stack, then switches, power bricks and a white power strip on the lower decks, held with blue zip ties. — Front, 9:35 PM. One Spark per shelf, gold intakes toward you; Spark 1 carries the green label. Switches, bricks and strips below.

GhettoRak from the front left: the stack of gold DGX Sparks on chrome wire shelves with the green Spark 1 label, open decks below, and the switch and power decks near the casters. — Front, from the left, 9:35 PM.

GhettoRak from the side: Sparks edge-on on each shelf, their leads strapped to a rear post with black hook-and-loop wraps, long black braided cables hanging in loops down the right side, white gear held with blue ties on the lower decks. — Side, 2:57 PM, during the GLM-5.3 bring-up. Leads strapped to the post; the fabric cables loop down the rear.

GhettoRak from the rear: heavy black braided cables strapped to both uprights drape in large loops in front of the stacked units, with white patch cords, a white switch with status LEDs, and power bricks on the lower shelves. — Rear, 9:36 PM, where all the I/O exits. Not pretty, and the cables have slack to spare.

The lower decks of GhettoRak: a white switch zip-tied to a wire shelf, power bricks with coiled white cords beneath it, and a black unit on the deck below. — Lower decks, 9:33 PM: switches, bricks and cabling under the Spark stack.

A white Tapo power strip tied along the front of a lower wire shelf, black and white plugs in several outlets, small blue LEDs lit, cable bundles and blue zip ties around it. — A Tapo P316M on the bottom deck, 9:34 PM. Strip-into-strip puts the whole rack behind one of these, and that is what tripped. See Power.

Top-down OpenSCAD view of the bottom deck: a tied column of eight black bricks on the left; two white Tapo P316M strips side by side on the right with six outlets each, labelled INT and EXT, and a cord from the external strip to the internal one. — Bottom deck plan, front at the bottom. Bricks left; Internal strip inboard, External at the posts. Outlet spacing is schematic.

OpenSCAD isometric of the bottom deck showing the tied brick block and the two white P316M strips with outlet recesses and blue status LEDs, cords exiting to the rear. — Bottom deck, isometric (September 30 model).

Power

Two Tapo P316M strips, six metered outlets each. External plugs into the wall and feeds Sparks 8 and 7, the UniFi, both CRS804s, and the Internal strip, which carries Sparks 6 → 1. A strip into a strip on purpose: all six Sparks downstream read as one number, and the switches stay up when Spark outlets are cut. Apple Home for control, Home Assistant over Matter for numbers, Prometheus and Grafana for history.

Apple Home screen titled GR Strip Internal, 6 On, with six yellow outlet tiles labelled Spark6, Spark5, Spark4, Spark3, Spark2, Spark1. — Apple Home, internal strip: one tile per Spark. Outlet 1 = Spark 6 because of how the cords fall.

October 16: the strip tripped

Whole-rack wall power from the Tapo outlet meters, Prometheus, 30-second steps. The red band is the time the rack was dark.

At 20:16 CDT a long-prompt retrieval test on the 1M-context GLM-5.3 lane drove all eight Sparks to ~192–207 W each at the wall. The rack held at ~1.7 kW (last full sample 1,734 W: Internal strip 1,236, Spark 7 200, Spark 8 194, CRS804 56, UniFi 49). At about 20:24 every outlet meter on GhettoRak went silent in the same minute, switches included. Forge, the NAS and the GB300 stayed up. James found the strip tripped.

Strip-into-strip means the External P316M carries the whole rack. A 15 A strip is 1,800 W at 120 V and 1,440 W for continuous load (the 80% rule), so the rack's real ceiling is one 15 A strip, whatever the wall circuit is rated. Eight minutes at ~1.7 kW was well past the continuous line.

This was predictable. On October 14, a 4-node test (DeepSeek-V4.1-Flash TP4 on Sparks 3–6) measured a long-prompt Spark at 176 W at the wall and projected ~1.5 kW for all eight plus switches, over the continuous rating. The clock cap was planned but not yet installed. GLM-5.3 TP8 drew more than that projection: about 200 W per Spark.

Recovery. When the strip came back, Sparks 3–8 booted on their own. Sparks 1 and 2 stayed off, drawing 1.2 W at the plug, until their power buttons were pressed. Nothing auto-starts: containers have no restart policy, and the fabric config is persistent. After power-up: both links per node at 200G with MTU 9000, the jumbo-ping mesh (112/112), then relaunch.

The clock cap

Since 21:31 CDT on October 16, every Spark runs gb10-clocklock.service (0xMagnus's unit), which applies nvidia-smi -lgc 0,2200 at boot. The driver log reports gpuClkMax 2200 on all eight, and Spark 1 holds 2,184–2,190 MHz under load. To undo: sudo systemctl disable --now gb10-clocklock && sudo nvidia-smi -rgc. GB10 has no nvpmodel and no nvidia-smi -pl, so a clock lock is the only power lever.

Same lane before and after: GLM-5.3 744B NVFP4, tensor-parallel 8, 4-bit KV cache, 1M context, three-token MTP. The same ~514K-token retrieval prompt ran in both.

October 16, all eight Sparks	Stock clocks	Cap 2,200 MHz
Rack at the wall, ~514K-token prompt	1,717 W mean (1,682–1,734); tripped the strip after ~8 min	1,345 W mean (1,293–1,403) over 7 min
Per Spark at the wall	194–203 W	150–157 W
~514K-token retrieval (513,814 tokens)	never finished: power lost	pass, 464 s
Decode, one request: code / prose / explanation	34.5 / 27.9 / 25.8 tok/s	34.5 / 30.7 / 25.4 tok/s
Decode, 4 requests at once, total	58.4 tok/s	61.5 tok/s

−22% at the wall and no decode loss. That matches the October 14 4-node test (DeepSeek-V4.1-Flash TP4: −16% at the wall, decode 203.8 vs 202.8 tok/s). Prefill cost is not isolated yet: the stock run of this prompt never finished. An earlier stock run on the 8-bit-KV lane took 414 s, but that is a different KV format.

Margin is thin. Capped, the ~514K prompt peaked at 1,403 W. A ~984K-token prompt then ran 16 minutes on the capped rack and peaked at 1,425 W, within ~15 W of the strip's 1,440 W continuous line, with no trip and no guard abort. (It failed the retrieval check: a 4-bit KV quality question, not a power one.) A wall-power guard polls the outlet meters every 15 s and kills the test client if the rack stays above 1,400 W for a minute. The next test is the second CRS804 (~56 W more). We expect the capped rack to stay in a sane envelope; if it sits at the line, a second wall outlet is the fallback.

Idle. All eight idle with their 200G links up: ~448 W for the rack at the wall. A Spark idles at ~43 W with its ConnectX-7 powered and ~26 W with it powered off. That was the "unexplained" 20 W gap between Sparks 1–2 and the rest on September 30, settled by an on/off A/B on Spark 8 on October 14.

Budget. Nameplate (8 × 240 W bricks, the UniFi's 210 W, two switches) is more than 15 A can carry. Planning around the wall circuit was not enough: the limit that bit was the strip. Capped, the rack peaks within ~15 W of 1,440 W; we will measure again once CRS804-2 is in, and split Sparks 7–8 and the switches onto a second wall outlet only if needed.

Clock facts. On Spark 3 (bf16 8192² matmul), stock is ≈ 63 TFLOPS at ~94 W GPU rail, and locking 1800 MHz gives 51 TFLOPS at ~54 W. GB10 has no nvpmodel and no nvidia-smi -pl, so a clock lock is the only power lever. Wall power runs 3–4.5× the GPU-rail reading (CPU, LPDDR5X, ConnectX-7, brick losses), so do not size power from nvidia-smi. A GPU stuck near 611 MHz is the USB-C PD negotiation bug: cold-drain the brick and update PD firmware.

Software

All eight Sparks run the same stack. Record it with any recipe measured here.

Item	All eight
Kernel	`6.17.0-1031-nvidia`, boot default; 7.0.0-1019 installed as manual fallback
Driver	`595.84`, prebuilt module, no DKMS
Held	Kernel and driver on `apt-mark hold`; neither `apt upgrade` nor DGX Dashboard can move them
Firmware	`fwupdmgr`: nothing pending
Clocks	Capped since October 16: `gb10-clocklock.service` runs `nvidia-smi -lgc 0,2200` at boot; driver reports `gpuClkMax 2200` on all eight. Results before 21:31 CDT October 16 are stock clocks.
Check	bf16 8192² matmul: 62.6–64.2 TFLOPS on every Spark tested

Why not kernel 7.0. DGX OS 7.6.0 ships 7.0.0-1019. NVIDIA's advisory tells multi-node RoCE users to hold off, and the bug thread shows RDMA registration failing under two-node load. On ours it also left CmaTotal at 0. Move when NVIDIA ships a real fix, and test one pair first.

Buy

Qty	What	SKU
1	12″d × 18″w × 54″h chrome wire, 5 shelves. Build sideways.	SC121854-5
8	Extra 12 × 18 shelf + clips (thirteen decks)	SI-S1218C
1	5″ polyurethane threaded casters, 4-pack. Threaded, not stem.	SI-APC5WB-4
2	MikroTik CRS804-4DDQ-hRM. The second one is for the Station.	CRS804 DDQ
4	MikroTik-coded QSFP-DD → 2× QSFP56 200G breakout DAC, one per Spark pair: 2× 2 m (Sparks 1–4), 2× 1 m (Sparks 5–8)	NADDOD `Q2Q56-400G-CUx`
1	400G QSFP-DD ↔ QSFP-DD passive DAC, 1 m, switch to switch	NADDOD `QDD-400G-CU1`
1	Breakout DAC for the Station (2 × 200G)	NADDOD `Q2Q56-400G-CU1`
2	Tapo P316M smart strip, 15 A, per-outlet metering	Tapo P316M
1	UniFi 10G switch, 8× RJ45 + 2× SFP+, fanless. Management only.	USW-Pro-XG-8-PoE
8	Cat6a patch: 6× 1 m, 2× 1.5 m	—
12	1 ft C5 → 5-15P cords for the bricks	CableCreation
1	Blue nylon zip ties, long enough to go over a Spark and through the mesh	—

Do not buy

Taller posts. 54″ uncut is enough; 64″+ ship in two pieces.
NVIDIA InfiniBand OSFP splitters: wrong cage for the CRS804.
400G QSFP112 DACs for Sparks or for the Station-to-switch link: 100G lanes, nothing on a CRS804 to meet them.
DR4 optics for the Station before 2 × 200G is shown to be the bottleneck.
0.5 m breakouts. We could not make one work on this rack; 1 m is the shortest in use.
Extra fans, 10-inch wall cubes, or a UPS inside the frame.
Printed clips or keepers for the Sparks. Zip ties do the job.

Prints

Labels and cable hooks only. The green Spark 1 clip is measured to this Shelving Inc lip: inference-box-label-clip.scad plus inference-box-label-measured.scad in the same folder. PLA, on end, 5 mm brim, no supports. One dimension (A6) is still a guess, so the file's MEASURED flag is false. For DAC and USB-C slack, these wire-shelf hooks in PETG (link, do not rehost).

Assembly order: posts and rings; bricks and strips on 1″; UniFi on 6″; switches on 10″ and 14″, fans and IEC toward you; Sparks 8 → 1 bottom up from 18″ to 46″, zip-tying as you go; bare top shelf at 50″. Every Spark and switch shelf is 4″ apart; DACs and Cat6a last.