Networking DGX Station and DGX Sparks with MikroTik CRS804

This page is the recipe. The build itself (shelves, photos, power) is on the GhettoRak page.

The five steps. Each one ends with a check.

Parts

Use copper (DAC) cables, not optics, at these lengths. We could not make a 0.5 m breakout work. One switch is enough for four Sparks plus the Station, or for eight Sparks.

1. Cabling

As built. Sparks 1–4 and the Station on switch 2, Sparks 5–8 on switch 1.

• Each breakout cable has one 400G end for the switch and two 200G ends. Each 200G end goes to one Spark. On the switch, the two ends show up as lanes 1 and 5 of that cage (for example qsfp56-dd-1-1 and qsfp56-dd-1-5).
• The Station uses one breakout for both of its ports, both on cage 4 of the same switch.
• Put the Station and the Sparks it talks to most on the same switch. Crossing to the other switch costs about 1 µs (2.46 vs 3.50 µs) and no measurable bandwidth: four Sparks forced to cross on every hop matched four on one switch.
• Label both ends of every cable. Then read the mapping back from the switch rather than trusting a drawing.

Check: link light on every leg.

2. Switches

RouterOS 7.25 on both switches. Management stays on the copper ports. The fast ports get their own bridge, and only that bridge may use the switch chip. If the factory bridge keeps hardware offload, the fabric bridge falls back to the CPU and is slow.

# management: keep the factory bridge for ether ports, offload off
/interface/bridge/port/set [find bridge=bridge] hw=no
/interface/bridge/set bridge name=mgmt

# fabric bridge on the fast ports
/interface/bridge/add name=fabric protocol-mode=rstp igmp-snooping=no
:foreach p in={"qsfp56-dd-1-1";"qsfp56-dd-1-5";"qsfp56-dd-2-1";"qsfp56-dd-2-5"} do={
  /interface/ethernet/set [find name=$p] auto-negotiation=no \
    speed=200G-baseCR4 fec-mode=fec91 l2mtu=9216 mtu=9000
  /interface/bridge/port/add bridge=fabric interface=$p hw=yes
}

# cage 3: the 400G cable to the other switch
/interface/ethernet/set [find name=qsfp56-dd-3-1] auto-negotiation=no \
    speed=400G-baseCR8 fec-mode=fec91 l2mtu=9216 mtu=9000
/interface/bridge/port/add bridge=fabric interface=qsfp56-dd-3-1 hw=yes

# ECN on the default traffic class
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=1] ecn=yes

• On the Station's switch, add qsfp56-dd-4-1 and qsfp56-dd-4-5 to the same loop.
• ECN lets the switch warn senders before its queue overflows. With it on, traffic into the Station lost 85% fewer packets (4.24 M → 0.65 M in one test). Spark-to-Spark speed did not change. We left flow control (pause frames) off: turning it on gained nothing.
• Second switch: give it its own name, limit SSH/WinBox to your admin machines, and plug its management port into switch 1's.

Check: /interface/bridge/port/print shows the H (hardware) flag on every fabric port, and /interface/ethernet/monitor qsfp56-dd-1-1 once shows 200G, link ok.

3. Sparks

One 200G cable shows up on a Spark as two interfaces, one per half of the card. Give each half its own subnet. N is the Spark's number. Delete the auto-created "Wired connection" profiles first, then add one netplan file with no gateway:

# /etc/netplan/40-cx7-fabric.yaml
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp1s0f0np0:
      addresses: [10.0.0.N/24]
      mtu: 9000
      dhcp4: no
      dhcp6: no
      accept-ra: no
      link-local: []
    enP2p1s0f0np0:
      addresses: [10.0.1.N/24]
      mtu: 9000
      dhcp4: no
      dhcp6: no
      accept-ra: no
      link-local: []

The Sparks link at 200G with auto-negotiation left on, so nothing else is needed. Because both halves sit on one network, each must answer ARP only for its own address. Our Sparks already had these set system-wide; confirm yours, and add them to a file in /etc/sysctl.d/ if not:

sysctl net.ipv4.conf.all.arp_ignore net.ipv4.conf.all.arp_announce net.ipv4.conf.all.rp_filter
# want 1, 2, 2

Check: both interfaces up, 200G, MTU 9000. Keep the Sparks on kernel 6.17 for now: NVIDIA's advisory tells multi-node RoCE users to hold off on 7.0.

4. Station

The Station's ConnectX-8 will not link while it auto-negotiates against the switch's fixed 200G. Force it, and set RS FEC to match. Addresses go in netplan (same shape as the Sparks, host number .9: 10.0.0.9 on enP1p3s0f0np0, 10.0.1.9 on enP1p3s0f1np1). Netplan cannot store the ethtool part, so a small boot service does it:

# /etc/systemd/system/station-fabric-link.service
[Unit]
Description=Force CX8 fabric ports to 200G-CR4 / RS-FEC, set ARP rules
Wants=sys-subsystem-net-devices-enP1p3s0f0np0.device sys-subsystem-net-devices-enP1p3s0f1np1.device
After=sys-subsystem-net-devices-enP1p3s0f0np0.device sys-subsystem-net-devices-enP1p3s0f1np1.device
Before=NetworkManager.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/ethtool -s enP1p3s0f0np0 autoneg off speed 200000 duplex full lanes 4
ExecStart=/usr/sbin/ethtool --set-fec enP1p3s0f0np0 encoding rs
ExecStart=/usr/sbin/ethtool -s enP1p3s0f1np1 autoneg off speed 200000 duplex full lanes 4
ExecStart=/usr/sbin/ethtool --set-fec enP1p3s0f1np1 encoding rs
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.arp_ignore=1
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.arp_announce=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.rp_filter=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.arp_ignore=1
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.arp_announce=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.rp_filter=2

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now station-fabric-link.service

Then fix the GPU path. The card has a direct path to the GB300's memory, called Data Direct. Out of the box, a PCIe isolation setting (ACS) puts the card and the GPU in different groups, and GPU-memory transfers fail with error 524. NVIDIA's tool finds and fixes it (NVIDIA's Station guide):

sudo rdma_topo check              # look for FAIL lines about ACS / iommu_group
sudo rdma_topo write-grub-acs     # writes /etc/default/grub.d/config-acs.cfg
sudo update-grub
# then restart the Station

Check: after the restart, rdma_topo check prints only OK lines, and both ports come back at 200G by themselves.

5. Test

Jumbo pings between every pair, on both subnets. -M do forbids splitting, so a wrong MTU anywhere fails loudly:

ping -c 3 -M do -s 8972 10.0.0.2
ping -c 3 -M do -s 8972 10.0.1.2

Eight Sparks plus the Station gave 128 of 128. Then RDMA bandwidth, one pair at a time (ib_write_bw from perftest; on the Station, a CUDA build adds the GPU flags):

# receiver (Spark 1, first half of the card)
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d rocep1s0f0
# sender (another Spark)
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d rocep1s0f0 10.0.0.1
# Station sender, from GPU memory through Data Direct
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d mlx5_0 \
  --use_cuda_bus_id=<GPU PCI address> --use_cuda_dmabuf --use_data_direct 10.0.0.1

Then NCCL, with these settings on every Spark:

NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_SOCKET_IFNAME=enp1s0f0np0
NCCL_IB_ROCE_VERSION_NUM=2
NCCL_IB_GID_INDEX=3
NCCL_PROTO=^LL128
NCCL_MAX_NCHANNELS=8

Pin NCCL_SOCKET_IFNAME, or NCCL may start up over Docker or Tailscale. All eight Sparks, four per switch, reached 24 GB/s bus bandwidth in all_reduce at 256 MiB.

What speed to expect

Measured on this fabric, October 29–30, 2026. Gb/s; divide by 8 for GB/s.

Why: the links inside each machine, not the 200G cables, set most of these limits.

• A Spark is a 185 Gb/s machine, not 200. Its card reaches the chip through two PCIe Gen5 x4 links of about 112 Gb/s each. One flow on one half stops near 112.
• The Station from ordinary memory stops near 231 Gb/s for both ports together (211–215 inbound without ECN, 228 with). Both ports share one PCIe Gen5 x8 link to the Grace CPU. One port alone runs at 196, its full line rate.
• The Station from GPU memory runs 392 Gb/s both ways through Data Direct, after the step-4 fix. Software must ask for it: NCCL 2.27.3 or newer, UCX 1.20 or newer (NVIDIA).
• Copies are usually the slowest thing. rsync or scp over ssh is limited by one CPU core doing encryption, under 10 Gb/s. Run several copies at once, or use the rsync daemon. For hundreds of GB into one Spark, its SSD is the limit: about 7 GB/s for the first ~50 GB, then about 1.7 GB/s. Serve big model files from one place (NFS or RDMA) instead of copying them to every Spark.

When something is wrong

Versions on this build: RouterOS 7.25 on both switches; Sparks on DGX OS 7.2.3, kernel 6.17.0-1031-nvidia, ConnectX-7 firmware 28.45.4028; Station ConnectX-8 firmware 40.47.1088. More from NVIDIA: Spark clustering, connecting two Stations. From MikroTik: Quality of Service.