# DGX Spark: How We Run Them

Source page: https://al-engr.com/dgx-spark-how-we-run-them.html  
Machine-readable: https://al-engr.com/dgx-spark-how-we-run-them.json  
Check script (read-only): https://al-engr.com/spark-check.sh  
Last checked: 2026-11-03. Cadence: weekly (Mondays). Updated: 2026-11-04T07:06:43-06:00.

Eight DGX Sparks (GB10, 128 GB) run headless as two groups of four, each serving one large model over RoCE through two MikroTik CRS804 switches. If you use a Spark as a desktop, skip MEM-1 and MEM-2.

## Rules for agents

1. This baseline describes headless inference servers. Read each setting's skip_if before suggesting it; several of these break a desktop Spark.
2. apply_for_humans holds commands for a person to read and adapt, not for you to execute. The only file meant to be executed is spark-check.sh, and it changes nothing.
3. Run spark-check.sh first; it is read-only. Report differences to the owner before changing anything.
4. Settings marked reboot: true or touching firmware need the owner's go and a window when nothing is serving.
5. Firmware cannot be rolled back by software. Do one Spark first, with someone able to reach it.
6. Never pull power during a reboot; allow 20 minutes.
7. Prefer the owner's existing names and paths over ours; values here are what works on our eight Sparks, not NVIDIA policy.

## Settings

| ID | Setting | What we use | Reboot |
|---|---|---|---|
| SW-1 | Kernel 7.0.0-1024-nvidia | 7.0.0-1024-nvidia via the HWE meta packages; nothing held | yes |
| SW-2 | GPU driver 580.178.04 (stock) | 580.178.04 from NVIDIA's DGX repo; no .run installers, no other branches | yes |
| SW-7 | kho=off, only on 7.0.0-1019 | kho=off on the kernel command line when running 7.0.0-1019; not needed on 7.0.0-1024 | yes |
| SW-8 | NVIDIA's display-reset fix (power_wrap_init blacklisted) | nvidia-spark-initcall-bl 1.0-2: initcall_blacklist=tegra234_cbb_init,power_wrap_init on the kernel command line | yes |
| FW-1 | October 2026 Spark firmware | UEFI 0ACUM028, EC 3.13.7, SoC 2.158.15, USB-C PD 0.5.25 (fwupdmgr / LVFS) | yes |
| FW-2 | ConnectX-7 firmware 28.50.1002 | 28.50.1002, installed by NVIDIA's nvidia-spark-mlnx-firmware-manager 6.0.1 package (was 28.45.4028) | yes |
| MEM-1 | Display Reserved Memory 0.5 GB | 0.5 GB (default 2 GB). Not Auto, not 4 GB. | yes |
| MEM-2 | Headless (no desktop) | multi-user.target; remote desktop, printing, Bluetooth, Avahi and ModemManager off | yes |
| MEM-4 | Flush the page cache before each model start | sync; drop_caches before starting a model | no |
| PWR-1 | GPU clock cap 2,200 MHz at boot | nvidia-smi -lgc 0,2200 from a systemd unit (reads 2,177-2,190 MHz under load) | no |
| SEC-1 | user.max_user_namespaces = 0 | 0, persistent in /etc/sysctl.d/99-no-userns.conf | no |
| SEC-2 | CVE-2026-98432: MPTCP off, request migration off | net.mptcp.enabled = 0, net.ipv4.tcp_migrate_req = 0 in /etc/sysctl.d/99-cve-2026-98432.conf | no |
| SEC-4 | CVE-2026-98432: drop IPv4 TCP packets carrying IP options | own nftables table, loaded by its own oneshot unit | no |
| SEC-5 | DGX Dashboard auto-update off | /opt/nvidia/dgx-dashboard/settings.json: "update": {"enabled": false} | no |
| NET-1 | Fabric: two interfaces, two subnets, MTU 9000, ARP pinned | enp1s0f0np0 = 10.0.0.N/24, enP2p1s0f0np0 = 10.0.1.N/24, MTU 9000 end to end; arp_ignore=1, arp_announce=2, rp_filter=2 | no |
| NET-4 | NCCL settings on every Spark | NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 NCCL_NET=IB NCCL_SOCKET_IFNAME=enp1s0f0np0 NCCL_IB_ROCE_VERSION_NUM=2 NCCL_IB_GID_INDEX=3 NCCL_PROTO=^LL128 NCCL_MAX_NCHANNELS=8 | no |
| NET-5 | Switch: ECN on, RDMA in its own queue, no PFC | RoCEv2 (UDP 4791) in its own traffic class sharing 50/50 with everything else; ECN on; no PFC | no |

### SW-1 Kernel 7.0.0-1024-nvidia

- What we use: 7.0.0-1024-nvidia via the HWE meta packages; nothing held
- Why: On our headless Sparks it passed RDMA, NCCL (with and without 100 GiB of GPU memory held), the real 4-Spark model with a bit-identical answer check, and a 6 h soak with 0 resets; since then 0 GPU firmware timeouts or Xid errors in 38 boots across all eight. That is evidence for headless servers only: Sparks with displays attached (especially 4K) reset on 1024 until NVIDIA's SW-8 fix, and we never tested a 4K display or HDMI.
- Applies to: all
- Reboot: yes · downtime: one reboot
- Source: https://al-engr.com/testing-dgx-spark-kernels.html
- spark-check.sh rows: SW-1, SW-4, SW-5, SW-6

Check (expect: 7.0.0-1024-nvidia):

```bash
uname -r
```

Apply (for humans, not for agents to execute):

```bash
sudo apt-get update && sudo apt-get dist-upgrade   # after the dry run in ROUTINE-1
```

Undo: Boot the previous kernel from GRUB; keep it installed until the new one has run a week.

### SW-2 GPU driver 580.178.04 (stock)

- What we use: 580.178.04 from NVIDIA's DGX repo; no .run installers, no other branches
- Why: Stock driver; the only one with prebuilt modules for 6.17.0-1032, 7.0.0-1019 and 7.0.0-1024. Prebuilt modules pin one exact driver version, so a kernel move is a driver decision.
- Applies to: all
- Reboot: yes · downtime: one reboot
- Source: https://al-engr.com/testing-dgx-spark-kernels.html
- spark-check.sh rows: SW-2, SW-3

Check (expect: 580.178.04 (both lines)):

```bash
nvidia-smi --query-gpu=driver_version --format=csv,noheader; modinfo -k $(uname -r) -F version nvidia
```

Apply (for humans, not for agents to execute):

```bash
# comes with the DGX OS packages; before rebooting into any kernel:
modinfo -k <kernel> -F version nvidia   # must print your driver version
```

Undo: n/a

### SW-7 kho=off, only on 7.0.0-1019

- What we use: kho=off on the kernel command line when running 7.0.0-1019; not needed on 7.0.0-1024
- Why: On 7.0.0-1019 without it, NCCL cannot register memory across Sparks (ibv_reg_mr ... Cannot allocate memory) and multi-Spark models do not start. perftest still passes, so test with NCCL.
- Applies to: multi-Spark RDMA on 7.0.0-1019
- Reboot: yes · downtime: one reboot
- Source: https://al-engr.com/testing-dgx-spark-kernels.html#kho
- spark-check.sh rows: SW-7

Check (expect: kho=off (on 7.0.0-1019 only)):

```bash
grep -o 'kho=off' /proc/cmdline
```

Apply (for humans, not for agents to execute):

```bash
sudo apt-get install nvidia-spark-grub-kho && sudo update-grub && sudo reboot
```

Undo: sudo apt-get remove nvidia-spark-grub-kho && sudo update-grub
Skip if: you are not on 7.0.0-1019

### SW-8 NVIDIA's display-reset fix (power_wrap_init blacklisted)

- What we use: nvidia-spark-initcall-bl 1.0-2: initcall_blacklist=tegra234_cbb_init,power_wrap_init on the kernel command line
- Why: NVIDIA's package changelog (Nov 1, 2026, urgency high): the MediaTek power_wrap driver drives power management this platform does not use, and enabling it has been correlated with spontaneous resets while a 4K display is attached. This is the fix for the 7.0.0-1024 reboot loops people reported with monitors. Our headless Sparks never showed it; we take it anyway because it is NVIDIA's stock package. Side effect on our Sparks: RDMA latency went from 3.35 µs back to 2.50 µs (the 7.0.0-1019 level) on both groups of four.
- Applies to: all; most important with a display attached
- Reboot: yes · downtime: one reboot
- Source: https://forums.developer.nvidia.com/t/dgx-spark-repeatedly-rebooting-after-most-recent-update/387397
- spark-check.sh rows: SW-8

Check (expect: initcall_blacklist=tegra234_cbb_init,power_wrap_init):

```bash
grep -o 'initcall_blacklist=[^ ]*' /proc/cmdline
```

Apply (for humans, not for agents to execute):

```bash
sudo apt-get update && sudo apt-get install nvidia-spark-initcall-bl   # 1.0-2 or later; runs update-grub
sudo reboot
```

Undo: sudo apt-get install --allow-downgrades nvidia-spark-initcall-bl=1.0-1 && sudo reboot (1.0-1 is still in NVIDIA's repo; not recommended)

### FW-1 October 2026 Spark firmware

- What we use: UEFI 0ACUM028, EC 3.13.7, SoC 2.158.15, USB-C PD 0.5.25 (fwupdmgr / LVFS)
- Why: Needed for the 0.5 GB display memory option (MEM-1). Also fixes fans stopping in headless mode.
- Applies to: all
- Reboot: yes · downtime: one reboot (several minutes while it flashes)
- Source: https://al-engr.com/testing-dgx-spark-kernels.html#now
- spark-check.sh rows: FW-1

Check (expect: contains 0ACUM028):

```bash
sudo dmidecode -s bios-version
```

Apply (for humans, not for agents to execute):

```bash
sudo fwupdmgr refresh && sudo fwupdmgr upgrade
sudo reboot   # flashes during this reboot; do not cut power
```

Undo: No software undo. Flash one Spark first, with someone able to reach it.

### FW-2 ConnectX-7 firmware 28.50.1002

- What we use: 28.50.1002, installed by NVIDIA's nvidia-spark-mlnx-firmware-manager 6.0.1 package (was 28.45.4028)
- Why: Comes with the normal DGX OS updates. The package flashes the network card during apt install; it takes effect after the next reboot and is verified on that boot. On our Sparks: RDMA and NCCL unchanged (within 1%).
- Applies to: all
- Reboot: yes · downtime: one reboot
- Source: https://al-engr.com/dgx-spark-how-we-run-them.html#updates
- spark-check.sh rows: FW-2

Check (expect: 28.50.1002):

```bash
sudo mstflint -d 0000:01:00.0 q | grep 'FW Version'
```

Apply (for humans, not for agents to execute):

```bash
# arrives with: sudo apt-get dist-upgrade   (watch for nvidia-spark-mlnx-firmware-manager in the dry run)
sudo reboot
```

Undo: Save the old image first: cp /opt/nvidia/spark-mlnx-firmware-manager/firmwares/fw-ConnectX7.bin ~/fw-ConnectX7-28.45.4028.bin (before updating); flash it back with mstflint.

### MEM-1 Display Reserved Memory 0.5 GB

- What we use: 0.5 GB (default 2 GB). Not Auto, not 4 GB.
- Why: +1.6 GiB of GPU-claimable memory on every Spark (117.4 → 119.0 GiB, headless). Auto sizes from whatever monitor is attached at boot (1 GB with a 4K monitor). 0.5 GB still shows a text console. Needs FW-1.
- Applies to: headless or text-console Sparks; desktop users with a 4K screen should use 1 GB or more
- Reboot: yes · downtime: one reboot
- Source: https://al-engr.com/testing-dgx-spark-kernels.html#now
- spark-check.sh rows: MEM-1, MEM-3

Check (expect: ends in 00 02 00 00 (512 MiB)):

```bash
sudo od -An -tx1 /sys/firmware/efi/efivars/NvidiaDispFrmStatus-284e19b9-235f-4aab-ae4c-190456d1bd76
```

Apply (for humans, not for agents to execute):

```bash
V=/sys/firmware/efi/efivars/NvidiaDispFrmSizeRequest-284e19b9-235f-4aab-ae4c-190456d1bd76
sudo chattr -i $V 2>/dev/null
printf '\x07\x00\x00\x00\x01' | sudo tee $V > /dev/null     # last byte: 0=Auto 1=0.5GB 2=1GB 3=2GB 5=4GB
sudo reboot
```

Undo: Same command with last byte 3 (2 GB), then reboot.
Skip if: you use the Spark as a desktop or plan to attach a 4K display

### MEM-2 Headless (no desktop)

- What we use: multi-user.target; remote desktop, printing, Bluetooth, Avahi and ModemManager off
- Why: About +0.4 GiB of GPU-claimable memory and fewer background processes. NCCL speed did not change (A/B tested).
- Applies to: servers only
- Reboot: yes · downtime: one reboot (or isolate)
- Source: https://al-engr.com/testing-dgx-spark-kernels.html#now
- spark-check.sh rows: MEM-2

Check (expect: multi-user.target):

```bash
systemctl get-default
```

Apply (for humans, not for agents to execute):

```bash
sudo systemctl set-default multi-user.target
sudo systemctl disable --now gnome-remote-desktop cups cups-browsed cups.socket cups.path bluetooth avahi-daemon avahi-daemon.socket ModemManager
```

Undo: sudo systemctl set-default graphical.target, re-enable the units, reboot
Skip if: you use the Spark as a desktop

### MEM-4 Flush the page cache before each model start

- What we use: sync; drop_caches before starting a model
- Why: NVIDIA's unified-memory advice: the file cache otherwise competes with the GPU for the same memory.
- Applies to: all
- Reboot: no · downtime: none
- Source: https://al-engr.com/testing-dgx-spark-kernels.html#now
- spark-check.sh rows: not checkable from the host by spark-check.sh; use the check command above

Check (expect: Cached small right before a model start):

```bash
grep -E '^(MemAvailable|Cached)' /proc/meminfo
```

Apply (for humans, not for agents to execute):

```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
```

Undo: n/a

### PWR-1 GPU clock cap 2,200 MHz at boot

- What we use: nvidia-smi -lgc 0,2200 from a systemd unit (reads 2,177-2,190 MHz under load)
- Why: GB10 has no power-limit setting; the clock is the only power lever. On our rack a long prompt at stock clocks averaged 1,717 W at the wall and tripped a 15 A strip; capped: 1,345 W (-22%) with no loss in decode speed.
- Applies to: all
- Reboot: no · downtime: none
- Source: https://al-engr.com/inference-box.html#power
- spark-check.sh rows: PWR-1

Check (expect: active; about 2190 MHz under load, max still reads 3003):

```bash
systemctl is-active gb10-clocklock; nvidia-smi --query-gpu=clocks.gr,clocks.max.gr --format=csv
```

Apply (for humans, not for agents to execute):

```bash
sudo tee /etc/systemd/system/gb10-clocklock.service > /dev/null <<'EOF'
[Unit]
Description=GB10 GPU clock cap
After=nvidia-persistenced.service
Wants=nvidia-persistenced.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/nvidia-smi -lgc 0,2200
ExecStop=/usr/bin/nvidia-smi -rgc

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now gb10-clocklock
```

Undo: sudo systemctl disable --now gb10-clocklock && sudo nvidia-smi -rgc
Skip if: you need every bit of compute speed and have checked your wall power and cooling

### SEC-1 user.max_user_namespaces = 0

- What we use: 0, persistent in /etc/sysctl.d/99-no-userns.conf
- Why: Blocks the first step of a public local root exploit that works on these Ubuntu 24.04 kernels. Docker GPU containers and RoCE still work (tested). Software that needs rootless containers or browser sandboxes will break.
- Applies to: all
- Reboot: no · downtime: none
- Source: https://al-engr.com/testing-dgx-spark-kernels.html#security
- spark-check.sh rows: SEC-1

Check (expect: 0):

```bash
sysctl -n user.max_user_namespaces
```

Apply (for humans, not for agents to execute):

```bash
echo user.max_user_namespaces=0 | sudo tee /etc/sysctl.d/99-no-userns.conf
sudo sysctl -p /etc/sysctl.d/99-no-userns.conf
```

Undo: sudo rm /etc/sysctl.d/99-no-userns.conf && sudo sysctl -w user.max_user_namespaces=<previous value>
Skip if: you run rootless containers (podman), Flatpak or snaps with sandboxes, or a browser on the Spark

### SEC-2 CVE-2026-98432: MPTCP off, request migration off

- What we use: net.mptcp.enabled = 0, net.ipv4.tcp_migrate_req = 0 in /etc/sysctl.d/99-cve-2026-98432.conf
- Why: Remote use-after-free in the IPv4 TCP handshake (CVSS 9.8, no login needed). Remote code execution has been reported against MPTCP listeners; Go 1.24+ programs listen over MPTCP by default. Canonical lists linux-nvidia-7.0 on 24.04 as Vulnerable (checked 2026-11-03). Restart MPTCP-listening services after setting it.
- Applies to: all, until a fixed linux-nvidia-7.0 kernel ships
- Reboot: no · downtime: none
- Source: https://ubuntu.com/security/CVE-2026-98432
- spark-check.sh rows: SEC-2, SEC-3

Check (expect: 0 and 0):

```bash
sysctl -n net.mptcp.enabled net.ipv4.tcp_migrate_req
```

Apply (for humans, not for agents to execute):

```bash
printf 'net.mptcp.enabled = 0\nnet.ipv4.tcp_migrate_req = 0\n' | sudo tee /etc/sysctl.d/99-cve-2026-98432.conf
sudo sysctl -p /etc/sysctl.d/99-cve-2026-98432.conf
sudo systemctl restart dgx-dashboard tailscaled   # and any other Go service
```

Undo: Remove the file once Canonical marks linux-nvidia-7.0 fixed and you run that kernel.

### SEC-4 CVE-2026-98432: drop IPv4 TCP packets carrying IP options

- What we use: own nftables table, loaded by its own oneshot unit
- Why: Blocks the trigger described in the upstream fix (commit e3f33b0: the freed memory is the IPv4 options saved from a SYN). Normal traffic never carries IP options; RoCE is UDP and unaffected. Do NOT load it through the stock nftables.service: its default config flushes the whole ruleset, including Docker's rules. Verified on ours: a normal connect still reaches ssh, both model APIs and the metrics port, a connect whose SYN carries a Record-Route option is dropped, over the LAN, the fabric and Tailscale. Mask nftables.service so nobody enables it later.
- Applies to: all, until a fixed linux-nvidia-7.0 kernel ships
- Reboot: no · downtime: none
- Source: https://ubuntu.com/security/CVE-2026-98432
- spark-check.sh rows: SEC-4

Check (expect: ip protocol tcp ip hdrlength != 5 counter ... drop):

```bash
sudo nft list table ip cve_2026_98432
```

Apply (for humans, not for agents to execute):

```bash
sudo tee /etc/nft-cve-2026-98432.nft > /dev/null <<'EOF'
table ip cve_2026_98432
delete table ip cve_2026_98432
table ip cve_2026_98432 {
	chain prerouting {
		type filter hook prerouting priority -300; policy accept;
		ip protocol tcp ip hdrlength != 5 counter drop
	}
}
EOF
sudo tee /etc/systemd/system/cve-2026-98432-nft.service > /dev/null <<'EOF'
[Unit]
Description=CVE-2026-98432 mitigation: drop IPv4 TCP with IP options (own nft table)
DefaultDependencies=no
Wants=network-pre.target
Before=network-pre.target
After=local-fs.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f /etc/nft-cve-2026-98432.nft
ExecStop=/usr/sbin/nft delete table ip cve_2026_98432

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now cve-2026-98432-nft.service
sudo systemctl mask nftables.service
```

Undo: sudo systemctl disable --now cve-2026-98432-nft.service

### SEC-5 DGX Dashboard auto-update off

- What we use: /opt/nvidia/dgx-dashboard/settings.json: "update": {"enabled": false}
- Why: No one-click kernel, driver or firmware change under a running model. The cost: you now have to run updates yourself. We fell 418 updates behind (273 security) before we started doing it monthly (ROUTINE-1). Turn it off only if you will do that.
- Applies to: anyone who runs updates by hand
- Reboot: no · downtime: none
- Source: https://al-engr.com/testing-dgx-spark-kernels.html#now
- spark-check.sh rows: SEC-5

Check (expect: "enabled": false):

```bash
cat /opt/nvidia/dgx-dashboard/settings.json
```

Apply (for humans, not for agents to execute):

```bash
printf '{\n  "update": {\n    "enabled": false\n  }\n}\n' | sudo tee /opt/nvidia/dgx-dashboard/settings.json
sudo systemctl restart dgx-dashboard.service
```

Undo: set enabled to true and restart dgx-dashboard
Skip if: you will not run updates by hand every month

### NET-1 Fabric: two interfaces, two subnets, MTU 9000, ARP pinned

- What we use: enp1s0f0np0 = 10.0.0.N/24, enP2p1s0f0np0 = 10.0.1.N/24, MTU 9000 end to end; arp_ignore=1, arp_announce=2, rp_filter=2
- Why: One 200G cable appears as two interfaces (one per half of the card). At MTU 1500, RDMA between two Sparks ran about 5% slower (185 vs 196 Gb/s).
- Applies to: multi-Spark
- Reboot: no · downtime: brief link reset
- Source: https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html
- spark-check.sh rows: NET-1, NET-2, NET-3

Check (expect: 9000 9000; 1 2 2):

```bash
cat /sys/class/net/enp1s0f0np0/mtu /sys/class/net/enP2p1s0f0np0/mtu; sysctl net.ipv4.conf.all.arp_ignore net.ipv4.conf.all.arp_announce net.ipv4.conf.all.rp_filter
```

Apply (for humans, not for agents to execute):

```bash
# full netplan file and switch config:
https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html
```

Undo: n/a
Skip if: you run a single Spark

### NET-4 NCCL settings on every Spark

- What we use: NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 NCCL_NET=IB NCCL_SOCKET_IFNAME=enp1s0f0np0 NCCL_IB_ROCE_VERSION_NUM=2 NCCL_IB_GID_INDEX=3 NCCL_PROTO=^LL128 NCCL_MAX_NCHANNELS=8
- Why: Pin NCCL_SOCKET_IFNAME or NCCL may start over Docker or Tailscale. NCCL_NET=IB makes a broken RDMA path fail loudly instead of falling back to TCP (1.17 GB/s instead of ~23, no error). The last two fix medium messages: 1 MiB all_reduce on 4 Sparks 1.36 -> 11.5 GB/s (found by 0xCS).
- Applies to: multi-Spark
- Reboot: no · downtime: model restart
- Source: https://al-engr.com/nccl-tuning-4x-dgx-spark-plain-english.html
- spark-check.sh rows: not checkable from the host by spark-check.sh; use the check command above

Check (expect: the seven variables above):

```bash
env | grep ^NCCL_   # inside your serving container
```

Apply (for humans, not for agents to execute):

```bash
# pass with -e to docker run / your launcher
```

Undo: n/a
Skip if: you run a single Spark

### NET-5 Switch: ECN on, RDMA in its own queue, no PFC

- What we use: RoCEv2 (UDP 4791) in its own traffic class sharing 50/50 with everything else; ECN on; no PFC
- Why: Without the RDMA queue an RDMA pile-up starved TCP to 0.002-0.015 Gb/s; with it TCP kept 41-94 Gb/s. ECN cut packet loss into the Station 85%. PFC hurt here.
- Applies to: switched multi-Spark
- Reboot: no · downtime: forwarding pauses briefly when queues change
- Source: https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html
- spark-check.sh rows: not checkable from the host by spark-check.sh; use the check command above

Check (expect: RDMA counted in queue 3):

```bash
(on the switch) /interface/ethernet/print stats  -> tx-queue3-packet rising during RDMA
```

Apply (for humans, not for agents to execute):

```bash
# MikroTik CRS804 commands: https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html
```

Undo: n/a
Skip if: you run a single Spark or no switch

## Routines

### ROUTINE-1 Monthly updates (four Sparks first)

1. Dry run on every Spark: sudo apt-get -s dist-upgrade | grep -E '^(Inst|Remv)'. Read every Remv line and any linux-image, linux-modules, nvidia-driver or libnvidia line. A kernel or driver move is a separate, tested decision.
2. Look for firmware and boot changes inside packages (names with firmware, initcall, grub, kho) and read their install scripts before saying go: nvidia-spark-mlnx-firmware-manager flashes the ConnectX-7 during install. Run sudo fwupdmgr get-updates for system firmware.
3. Four Sparks first: stop the model, sudo apt-get dist-upgrade, reboot. Check kernel, driver, both fabric links, clock cap, then an NCCL all_reduce sweep. Restart the model and check it answers.
4. Then the other four.
5. Treat every reboot as a test that your boot-time settings survive: after it, run spark-check.sh and read the first 90 seconds of the kernel log (journalctl -b -k) for Xid, rpcRecvPoll, Oops.
6. Allow 20 minutes per reboot before assuming a Spark is dead, and know who can reach it physically. Do not pull power.

### ROUTINE-2 After any change

1. bash spark-check.sh --table on every Spark: everything ok or n/a.
2. Jumbo ping between every pair on both subnets: ping -c 3 -M do -s 8972 10.0.0.N and 10.0.1.N.
3. NCCL all_reduce across sizes, not just perftest: perftest passed 72/72 on a kernel where NCCL failed every run.

### ROUTINE-3 After a power cut

1. Some Sparks stay off until their power button is pressed (2 of 8 on ours).
2. Nothing starts by itself on ours (containers have no restart policy, on purpose): check both fabric links on every Spark before relaunching a multi-Spark model.

## What we decided not to do

- Remove snapd or the desktop packages (tens of MB; makes DGX OS updates harder).
- Turn off the DGX Dashboard itself (13 MB; useful). Turn off only its auto-update.
- Change swap, THP or CPU governor (already right on DGX OS).
- Set display memory to 4 GB, or Auto with a monitor attached.
- Use 64K-page kernels.
- Install drivers off the DGX repo (.run installers, other branches).
- Turn on PFC on the switch (it hurt here).
- Load an nftables rule through the stock nftables.service (its config flushes Docker's rules).

## Open issues

- OPEN-1 CVE-2026-98432 not fixed in linux-nvidia-7.0: Mitigated (SEC-2, SEC-4). Canonical lists linux-nvidia-7.0 on 24.04 as Vulnerable as of 2026-11-03. Watch: `https://ubuntu.com/security/CVE-2026-98432`. Next: When marked fixed: move to that kernel with ROUTINE-1, then remove SEC-2 and SEC-4.
- OPEN-2 Rare slow shutdown: 1 of 13 recent reboots on one Spark: nvidia-persistenced stuck in the display driver (nvkms_close); shutdown took 7.5 min, reboot ~15 min. 0 of 13 on three others. Its next restart, the same day, took about 2 minutes. Watch: `journalctl -b -1 -k | grep nvkms_close`. Next: Wait it out; do not pull power.
- OPEN-3 7.0.0-1024 resets with displays attached: NVIDIA shipped nvidia-spark-initcall-bl 1.0-2 (SW-8). Other 1024 reset reports on the forum involved a self-patched driver module or a different driver (595.99.02); none matched our stock headless setup. Watch: `https://forums.developer.nvidia.com/c/accelerated-computing/dgx-spark-gb10/dgx-spark-gb10/721`. Next: Rollback trigger for us: one unexplained reset or GPU firmware timeout on a stock headless Spark.
- OPEN-4 Fans ignore CPU heat until about 80 °C; no owner fan control on EC 3.13.7: Measured on one Spark on 2026-11-04: 20 busy cores, no model, reached 80 °C in 4 s and held near 94 °C with fans about 3,100 rpm; nvidia-smi read 62 °C at most. The ACPI fan setting is accepted and ignored. Partial workaround: cap the Cortex-X925 cores at 2,808 MHz (79.7 °C after 20 min, 16.5% less CPU work). We do not run the cap; our Sparks serve models on the GPU. Watch: `https://forums.developer.nvidia.com/t/new-dgx-spark-fans-barely-speed-up-at-high-temps-and-cant-be-adjusted-critical-please-open-fan-control/386805`. Next: When NVIDIA ships an EC with an earlier fan curve or host fan control: rerun the 20-core test and update this section. Do not downgrade the EC or script cooling_device writes.

## Changelog

- 2026-11-03: First version. All eight Sparks updated to DGX OS 7.6.0 packages with ConnectX-7 28.50.1002; CVE-2026-98432 mitigations added.
- 2026-11-03: Second window: NVIDIA display-reset fix nvidia-spark-initcall-bl 1.0-2 (SW-8) and security updates on all eight. Mitigations survived the restart; RDMA latency 3.35 to 2.50 µs; network tests now cover both groups.
- 2026-11-03: Proofread: MEM-1 gain corrected to +1.6 GiB (was +1.5); latency cause marked likely, not proven; SW-8 undo needs --allow-downgrades; SGLang KV-pool variation noted under Memory.
- 2026-11-04: Added CPU-heavy work and the fans (under Power and heat) and OPEN-4: fan response to CPU heat on EC 3.13.7, measured on one Spark, and the X925 clock cap as a partial workaround. Suggested by bellatrix (@0xb3llatrix).