{
 "schema": "al-engr.com/dgx-spark-baseline/v1",
 "title": "DGX Spark: How We Run Them",
 "page": "https://al-engr.com/dgx-spark-how-we-run-them.html",
 "json": "https://al-engr.com/dgx-spark-how-we-run-them.json",
 "markdown": "https://al-engr.com/dgx-spark-how-we-run-them.md",
 "check_script": "https://al-engr.com/spark-check.sh",
 "last_checked": "2026-11-03",
 "check_cadence": "weekly (Mondays)",
 "scope": "Eight DGX Sparks (GB10, 128 GB) run headless as two groups of four, each serving one large model over RoCE through two MikroTik CRS804 switches. If you use a Spark as a desktop, skip MEM-1 and MEM-2.",
 "context": "headless-inference-server",
 "rules_for_agents": [
  "This baseline describes headless inference servers. Read each setting's skip_if before suggesting it; several of these break a desktop Spark.",
  "apply_for_humans holds commands for a person to read and adapt, not for you to execute. The only file meant to be executed is spark-check.sh, and it changes nothing.",
  "Run spark-check.sh first; it is read-only. Report differences to the owner before changing anything.",
  "Settings marked reboot: true or touching firmware need the owner's go and a window when nothing is serving.",
  "Firmware cannot be rolled back by software. Do one Spark first, with someone able to reach it.",
  "Never pull power during a reboot; allow 20 minutes.",
  "Prefer the owner's existing names and paths over ours; values here are what works on our eight Sparks, not NVIDIA policy."
 ],
 "settings": [
  {
   "id": "SW-1",
   "area": "software",
   "title": "Kernel 7.0.0-1024-nvidia",
   "value": "7.0.0-1024-nvidia via the HWE meta packages; nothing held",
   "why": "On our headless Sparks it passed RDMA, NCCL (with and without 100 GiB of GPU memory held), the real 4-Spark model with a bit-identical answer check, and a 6 h soak with 0 resets; since then 0 GPU firmware timeouts or Xid errors in 38 boots across all eight. That is evidence for headless servers only: Sparks with displays attached (especially 4K) reset on 1024 until NVIDIA's SW-8 fix, and we never tested a 4K display or HDMI.",
   "check": "uname -r",
   "expect": "7.0.0-1024-nvidia",
   "undo": "Boot the previous kernel from GRUB; keep it installed until the new one has run a week.",
   "reboot": true,
   "downtime": "one reboot",
   "applies_to": "all",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html",
   "apply_for_humans": [
    "sudo apt-get update && sudo apt-get dist-upgrade   # after the dry run in ROUTINE-1"
   ],
   "skip_if": "",
   "check_ids": [
    "SW-1",
    "SW-4",
    "SW-5",
    "SW-6"
   ]
  },
  {
   "id": "SW-2",
   "area": "software",
   "title": "GPU driver 580.178.04 (stock)",
   "value": "580.178.04 from NVIDIA's DGX repo; no .run installers, no other branches",
   "why": "Stock driver; the only one with prebuilt modules for 6.17.0-1032, 7.0.0-1019 and 7.0.0-1024. Prebuilt modules pin one exact driver version, so a kernel move is a driver decision.",
   "check": "nvidia-smi --query-gpu=driver_version --format=csv,noheader; modinfo -k $(uname -r) -F version nvidia",
   "expect": "580.178.04 (both lines)",
   "undo": "n/a",
   "reboot": true,
   "downtime": "one reboot",
   "applies_to": "all",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html",
   "apply_for_humans": [
    "# comes with the DGX OS packages; before rebooting into any kernel:",
    "modinfo -k <kernel> -F version nvidia   # must print your driver version"
   ],
   "skip_if": "",
   "check_ids": [
    "SW-2",
    "SW-3"
   ]
  },
  {
   "id": "SW-7",
   "area": "software",
   "title": "kho=off, only on 7.0.0-1019",
   "value": "kho=off on the kernel command line when running 7.0.0-1019; not needed on 7.0.0-1024",
   "why": "On 7.0.0-1019 without it, NCCL cannot register memory across Sparks (ibv_reg_mr ... Cannot allocate memory) and multi-Spark models do not start. perftest still passes, so test with NCCL.",
   "check": "grep -o 'kho=off' /proc/cmdline",
   "expect": "kho=off (on 7.0.0-1019 only)",
   "undo": "sudo apt-get remove nvidia-spark-grub-kho && sudo update-grub",
   "reboot": true,
   "downtime": "one reboot",
   "applies_to": "multi-Spark RDMA on 7.0.0-1019",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html#kho",
   "apply_for_humans": [
    "sudo apt-get install nvidia-spark-grub-kho && sudo update-grub && sudo reboot"
   ],
   "skip_if": "you are not on 7.0.0-1019",
   "check_ids": [
    "SW-7"
   ]
  },
  {
   "id": "SW-8",
   "area": "software",
   "title": "NVIDIA's display-reset fix (power_wrap_init blacklisted)",
   "value": "nvidia-spark-initcall-bl 1.0-2: initcall_blacklist=tegra234_cbb_init,power_wrap_init on the kernel command line",
   "why": "NVIDIA's package changelog (Nov 1, 2026, urgency high): the MediaTek power_wrap driver drives power management this platform does not use, and enabling it has been correlated with spontaneous resets while a 4K display is attached. This is the fix for the 7.0.0-1024 reboot loops people reported with monitors. Our headless Sparks never showed it; we take it anyway because it is NVIDIA's stock package. Side effect on our Sparks: RDMA latency went from 3.35 µs back to 2.50 µs (the 7.0.0-1019 level) on both groups of four.",
   "check": "grep -o 'initcall_blacklist=[^ ]*' /proc/cmdline",
   "expect": "initcall_blacklist=tegra234_cbb_init,power_wrap_init",
   "undo": "sudo apt-get install --allow-downgrades nvidia-spark-initcall-bl=1.0-1 && sudo reboot (1.0-1 is still in NVIDIA's repo; not recommended)",
   "reboot": true,
   "downtime": "one reboot",
   "applies_to": "all; most important with a display attached",
   "source": "https://forums.developer.nvidia.com/t/dgx-spark-repeatedly-rebooting-after-most-recent-update/387397",
   "apply_for_humans": [
    "sudo apt-get update && sudo apt-get install nvidia-spark-initcall-bl   # 1.0-2 or later; runs update-grub",
    "sudo reboot"
   ],
   "skip_if": "",
   "check_ids": [
    "SW-8"
   ]
  },
  {
   "id": "FW-1",
   "area": "firmware",
   "title": "October 2026 Spark firmware",
   "value": "UEFI 0ACUM028, EC 3.13.7, SoC 2.158.15, USB-C PD 0.5.25 (fwupdmgr / LVFS)",
   "why": "Needed for the 0.5 GB display memory option (MEM-1). Also fixes fans stopping in headless mode.",
   "check": "sudo dmidecode -s bios-version",
   "expect": "contains 0ACUM028",
   "undo": "No software undo. Flash one Spark first, with someone able to reach it.",
   "reboot": true,
   "downtime": "one reboot (several minutes while it flashes)",
   "applies_to": "all",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html#now",
   "apply_for_humans": [
    "sudo fwupdmgr refresh && sudo fwupdmgr upgrade",
    "sudo reboot   # flashes during this reboot; do not cut power"
   ],
   "skip_if": "",
   "check_ids": [
    "FW-1"
   ]
  },
  {
   "id": "FW-2",
   "area": "firmware",
   "title": "ConnectX-7 firmware 28.50.1002",
   "value": "28.50.1002, installed by NVIDIA's nvidia-spark-mlnx-firmware-manager 6.0.1 package (was 28.45.4028)",
   "why": "Comes with the normal DGX OS updates. The package flashes the network card during apt install; it takes effect after the next reboot and is verified on that boot. On our Sparks: RDMA and NCCL unchanged (within 1%).",
   "check": "sudo mstflint -d 0000:01:00.0 q | grep 'FW Version'",
   "expect": "28.50.1002",
   "undo": "Save the old image first: cp /opt/nvidia/spark-mlnx-firmware-manager/firmwares/fw-ConnectX7.bin ~/fw-ConnectX7-28.45.4028.bin (before updating); flash it back with mstflint.",
   "reboot": true,
   "downtime": "one reboot",
   "applies_to": "all",
   "source": "https://al-engr.com/dgx-spark-how-we-run-them.html#updates",
   "apply_for_humans": [
    "# arrives with: sudo apt-get dist-upgrade   (watch for nvidia-spark-mlnx-firmware-manager in the dry run)",
    "sudo reboot"
   ],
   "skip_if": "",
   "check_ids": [
    "FW-2"
   ]
  },
  {
   "id": "MEM-1",
   "area": "memory",
   "title": "Display Reserved Memory 0.5 GB",
   "value": "0.5 GB (default 2 GB). Not Auto, not 4 GB.",
   "why": "+1.6 GiB of GPU-claimable memory on every Spark (117.4 → 119.0 GiB, headless). Auto sizes from whatever monitor is attached at boot (1 GB with a 4K monitor). 0.5 GB still shows a text console. Needs FW-1.",
   "check": "sudo od -An -tx1 /sys/firmware/efi/efivars/NvidiaDispFrmStatus-284e19b9-235f-4aab-ae4c-190456d1bd76",
   "expect": "ends in 00 02 00 00 (512 MiB)",
   "undo": "Same command with last byte 3 (2 GB), then reboot.",
   "reboot": true,
   "downtime": "one reboot",
   "applies_to": "headless or text-console Sparks; desktop users with a 4K screen should use 1 GB or more",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html#now",
   "apply_for_humans": [
    "V=/sys/firmware/efi/efivars/NvidiaDispFrmSizeRequest-284e19b9-235f-4aab-ae4c-190456d1bd76",
    "sudo chattr -i $V 2>/dev/null",
    "printf '\\x07\\x00\\x00\\x00\\x01' | sudo tee $V > /dev/null     # last byte: 0=Auto 1=0.5GB 2=1GB 3=2GB 5=4GB",
    "sudo reboot"
   ],
   "skip_if": "you use the Spark as a desktop or plan to attach a 4K display",
   "check_ids": [
    "MEM-1",
    "MEM-3"
   ]
  },
  {
   "id": "MEM-2",
   "area": "memory",
   "title": "Headless (no desktop)",
   "value": "multi-user.target; remote desktop, printing, Bluetooth, Avahi and ModemManager off",
   "why": "About +0.4 GiB of GPU-claimable memory and fewer background processes. NCCL speed did not change (A/B tested).",
   "check": "systemctl get-default",
   "expect": "multi-user.target",
   "undo": "sudo systemctl set-default graphical.target, re-enable the units, reboot",
   "reboot": true,
   "downtime": "one reboot (or isolate)",
   "applies_to": "servers only",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html#now",
   "apply_for_humans": [
    "sudo systemctl set-default multi-user.target",
    "sudo systemctl disable --now gnome-remote-desktop cups cups-browsed cups.socket cups.path bluetooth avahi-daemon avahi-daemon.socket ModemManager"
   ],
   "skip_if": "you use the Spark as a desktop",
   "check_ids": [
    "MEM-2"
   ]
  },
  {
   "id": "MEM-4",
   "area": "memory",
   "title": "Flush the page cache before each model start",
   "value": "sync; drop_caches before starting a model",
   "why": "NVIDIA's unified-memory advice: the file cache otherwise competes with the GPU for the same memory.",
   "check": "grep -E '^(MemAvailable|Cached)' /proc/meminfo",
   "expect": "Cached small right before a model start",
   "undo": "n/a",
   "reboot": false,
   "downtime": "none",
   "applies_to": "all",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html#now",
   "apply_for_humans": [
    "sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'"
   ],
   "skip_if": "",
   "check_ids": [],
   "check_note": "not checkable from the host by spark-check.sh; use the check command above"
  },
  {
   "id": "PWR-1",
   "area": "power",
   "title": "GPU clock cap 2,200 MHz at boot",
   "value": "nvidia-smi -lgc 0,2200 from a systemd unit (reads 2,177-2,190 MHz under load)",
   "why": "GB10 has no power-limit setting; the clock is the only power lever. On our rack a long prompt at stock clocks averaged 1,717 W at the wall and tripped a 15 A strip; capped: 1,345 W (-22%) with no loss in decode speed.",
   "check": "systemctl is-active gb10-clocklock; nvidia-smi --query-gpu=clocks.gr,clocks.max.gr --format=csv",
   "expect": "active; about 2190 MHz under load, max still reads 3003",
   "undo": "sudo systemctl disable --now gb10-clocklock && sudo nvidia-smi -rgc",
   "reboot": false,
   "downtime": "none",
   "applies_to": "all",
   "source": "https://al-engr.com/inference-box.html#power",
   "apply_for_humans": [
    "sudo tee /etc/systemd/system/gb10-clocklock.service > /dev/null <<'EOF'\n[Unit]\nDescription=GB10 GPU clock cap\nAfter=nvidia-persistenced.service\nWants=nvidia-persistenced.service\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=/usr/bin/nvidia-smi -lgc 0,2200\nExecStop=/usr/bin/nvidia-smi -rgc\n\n[Install]\nWantedBy=multi-user.target\nEOF\nsudo systemctl daemon-reload && sudo systemctl enable --now gb10-clocklock"
   ],
   "skip_if": "you need every bit of compute speed and have checked your wall power and cooling",
   "check_ids": [
    "PWR-1"
   ]
  },
  {
   "id": "SEC-1",
   "area": "security",
   "title": "user.max_user_namespaces = 0",
   "value": "0, persistent in /etc/sysctl.d/99-no-userns.conf",
   "why": "Blocks the first step of a public local root exploit that works on these Ubuntu 24.04 kernels. Docker GPU containers and RoCE still work (tested). Software that needs rootless containers or browser sandboxes will break.",
   "check": "sysctl -n user.max_user_namespaces",
   "expect": "0",
   "undo": "sudo rm /etc/sysctl.d/99-no-userns.conf && sudo sysctl -w user.max_user_namespaces=<previous value>",
   "reboot": false,
   "downtime": "none",
   "applies_to": "all",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html#security",
   "apply_for_humans": [
    "echo user.max_user_namespaces=0 | sudo tee /etc/sysctl.d/99-no-userns.conf",
    "sudo sysctl -p /etc/sysctl.d/99-no-userns.conf"
   ],
   "skip_if": "you run rootless containers (podman), Flatpak or snaps with sandboxes, or a browser on the Spark",
   "check_ids": [
    "SEC-1"
   ]
  },
  {
   "id": "SEC-2",
   "area": "security",
   "title": "CVE-2026-98432: MPTCP off, request migration off",
   "value": "net.mptcp.enabled = 0, net.ipv4.tcp_migrate_req = 0 in /etc/sysctl.d/99-cve-2026-98432.conf",
   "why": "Remote use-after-free in the IPv4 TCP handshake (CVSS 9.8, no login needed). Remote code execution has been reported against MPTCP listeners; Go 1.24+ programs listen over MPTCP by default. Canonical lists linux-nvidia-7.0 on 24.04 as Vulnerable (checked 2026-11-03). Restart MPTCP-listening services after setting it.",
   "check": "sysctl -n net.mptcp.enabled net.ipv4.tcp_migrate_req",
   "expect": "0 and 0",
   "undo": "Remove the file once Canonical marks linux-nvidia-7.0 fixed and you run that kernel.",
   "reboot": false,
   "downtime": "none",
   "applies_to": "all, until a fixed linux-nvidia-7.0 kernel ships",
   "source": "https://ubuntu.com/security/CVE-2026-98432",
   "apply_for_humans": [
    "printf 'net.mptcp.enabled = 0\\nnet.ipv4.tcp_migrate_req = 0\\n' | sudo tee /etc/sysctl.d/99-cve-2026-98432.conf",
    "sudo sysctl -p /etc/sysctl.d/99-cve-2026-98432.conf",
    "sudo systemctl restart dgx-dashboard tailscaled   # and any other Go service"
   ],
   "skip_if": "",
   "check_ids": [
    "SEC-2",
    "SEC-3"
   ]
  },
  {
   "id": "SEC-4",
   "area": "security",
   "title": "CVE-2026-98432: drop IPv4 TCP packets carrying IP options",
   "value": "own nftables table, loaded by its own oneshot unit",
   "why": "Blocks the trigger described in the upstream fix (commit e3f33b0: the freed memory is the IPv4 options saved from a SYN). Normal traffic never carries IP options; RoCE is UDP and unaffected. Do NOT load it through the stock nftables.service: its default config flushes the whole ruleset, including Docker's rules. Verified on ours: a normal connect still reaches ssh, both model APIs and the metrics port, a connect whose SYN carries a Record-Route option is dropped, over the LAN, the fabric and Tailscale. Mask nftables.service so nobody enables it later.",
   "check": "sudo nft list table ip cve_2026_98432",
   "expect": "ip protocol tcp ip hdrlength != 5 counter ... drop",
   "undo": "sudo systemctl disable --now cve-2026-98432-nft.service",
   "reboot": false,
   "downtime": "none",
   "applies_to": "all, until a fixed linux-nvidia-7.0 kernel ships",
   "source": "https://ubuntu.com/security/CVE-2026-98432",
   "apply_for_humans": [
    "sudo tee /etc/nft-cve-2026-98432.nft > /dev/null <<'EOF'\ntable ip cve_2026_98432\ndelete table ip cve_2026_98432\ntable ip cve_2026_98432 {\n\tchain prerouting {\n\t\ttype filter hook prerouting priority -300; policy accept;\n\t\tip protocol tcp ip hdrlength != 5 counter drop\n\t}\n}\nEOF\nsudo tee /etc/systemd/system/cve-2026-98432-nft.service > /dev/null <<'EOF'\n[Unit]\nDescription=CVE-2026-98432 mitigation: drop IPv4 TCP with IP options (own nft table)\nDefaultDependencies=no\nWants=network-pre.target\nBefore=network-pre.target\nAfter=local-fs.target\n\n[Service]\nType=oneshot\nRemainAfterExit=yes\nExecStart=/usr/sbin/nft -f /etc/nft-cve-2026-98432.nft\nExecStop=/usr/sbin/nft delete table ip cve_2026_98432\n\n[Install]\nWantedBy=multi-user.target\nEOF\nsudo systemctl daemon-reload && sudo systemctl enable --now cve-2026-98432-nft.service",
    "sudo systemctl mask nftables.service"
   ],
   "skip_if": "",
   "check_ids": [
    "SEC-4"
   ]
  },
  {
   "id": "SEC-5",
   "area": "security",
   "title": "DGX Dashboard auto-update off",
   "value": "/opt/nvidia/dgx-dashboard/settings.json: \"update\": {\"enabled\": false}",
   "why": "No one-click kernel, driver or firmware change under a running model. The cost: you now have to run updates yourself. We fell 418 updates behind (273 security) before we started doing it monthly (ROUTINE-1). Turn it off only if you will do that.",
   "check": "cat /opt/nvidia/dgx-dashboard/settings.json",
   "expect": "\"enabled\": false",
   "undo": "set enabled to true and restart dgx-dashboard",
   "reboot": false,
   "downtime": "none",
   "applies_to": "anyone who runs updates by hand",
   "source": "https://al-engr.com/testing-dgx-spark-kernels.html#now",
   "apply_for_humans": [
    "printf '{\\n  \"update\": {\\n    \"enabled\": false\\n  }\\n}\\n' | sudo tee /opt/nvidia/dgx-dashboard/settings.json",
    "sudo systemctl restart dgx-dashboard.service"
   ],
   "skip_if": "you will not run updates by hand every month",
   "check_ids": [
    "SEC-5"
   ]
  },
  {
   "id": "NET-1",
   "area": "network",
   "title": "Fabric: two interfaces, two subnets, MTU 9000, ARP pinned",
   "value": "enp1s0f0np0 = 10.0.0.N/24, enP2p1s0f0np0 = 10.0.1.N/24, MTU 9000 end to end; arp_ignore=1, arp_announce=2, rp_filter=2",
   "why": "One 200G cable appears as two interfaces (one per half of the card). At MTU 1500, RDMA between two Sparks ran about 5% slower (185 vs 196 Gb/s).",
   "check": "cat /sys/class/net/enp1s0f0np0/mtu /sys/class/net/enP2p1s0f0np0/mtu; sysctl net.ipv4.conf.all.arp_ignore net.ipv4.conf.all.arp_announce net.ipv4.conf.all.rp_filter",
   "expect": "9000 9000; 1 2 2",
   "undo": "n/a",
   "reboot": false,
   "downtime": "brief link reset",
   "applies_to": "multi-Spark",
   "source": "https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html",
   "apply_for_humans": [
    "# full netplan file and switch config:",
    "https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html"
   ],
   "skip_if": "you run a single Spark",
   "check_ids": [
    "NET-1",
    "NET-2",
    "NET-3"
   ]
  },
  {
   "id": "NET-4",
   "area": "network",
   "title": "NCCL settings on every Spark",
   "value": "NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 NCCL_NET=IB NCCL_SOCKET_IFNAME=enp1s0f0np0 NCCL_IB_ROCE_VERSION_NUM=2 NCCL_IB_GID_INDEX=3 NCCL_PROTO=^LL128 NCCL_MAX_NCHANNELS=8",
   "why": "Pin NCCL_SOCKET_IFNAME or NCCL may start over Docker or Tailscale. NCCL_NET=IB makes a broken RDMA path fail loudly instead of falling back to TCP (1.17 GB/s instead of ~23, no error). The last two fix medium messages: 1 MiB all_reduce on 4 Sparks 1.36 -> 11.5 GB/s (found by 0xCS).",
   "check": "env | grep ^NCCL_   # inside your serving container",
   "expect": "the seven variables above",
   "undo": "n/a",
   "reboot": false,
   "downtime": "model restart",
   "applies_to": "multi-Spark",
   "source": "https://al-engr.com/nccl-tuning-4x-dgx-spark-plain-english.html",
   "apply_for_humans": [
    "# pass with -e to docker run / your launcher"
   ],
   "skip_if": "you run a single Spark",
   "check_ids": [],
   "check_note": "not checkable from the host by spark-check.sh; use the check command above"
  },
  {
   "id": "NET-5",
   "area": "network",
   "title": "Switch: ECN on, RDMA in its own queue, no PFC",
   "value": "RoCEv2 (UDP 4791) in its own traffic class sharing 50/50 with everything else; ECN on; no PFC",
   "why": "Without the RDMA queue an RDMA pile-up starved TCP to 0.002-0.015 Gb/s; with it TCP kept 41-94 Gb/s. ECN cut packet loss into the Station 85%. PFC hurt here.",
   "check": "(on the switch) /interface/ethernet/print stats  -> tx-queue3-packet rising during RDMA",
   "expect": "RDMA counted in queue 3",
   "undo": "n/a",
   "reboot": false,
   "downtime": "forwarding pauses briefly when queues change",
   "applies_to": "switched multi-Spark",
   "source": "https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html",
   "apply_for_humans": [
    "# MikroTik CRS804 commands: https://al-engr.com/networking-dgx-station-sparks-mikrotik-crs804.html"
   ],
   "skip_if": "you run a single Spark or no switch",
   "check_ids": [],
   "check_note": "not checkable from the host by spark-check.sh; use the check command above"
  }
 ],
 "routines": [
  {
   "id": "ROUTINE-1",
   "title": "Monthly updates (four Sparks first)",
   "steps": [
    "Dry run on every Spark: sudo apt-get -s dist-upgrade | grep -E '^(Inst|Remv)'. Read every Remv line and any linux-image, linux-modules, nvidia-driver or libnvidia line. A kernel or driver move is a separate, tested decision.",
    "Look for firmware and boot changes inside packages (names with firmware, initcall, grub, kho) and read their install scripts before saying go: nvidia-spark-mlnx-firmware-manager flashes the ConnectX-7 during install. Run sudo fwupdmgr get-updates for system firmware.",
    "Four Sparks first: stop the model, sudo apt-get dist-upgrade, reboot. Check kernel, driver, both fabric links, clock cap, then an NCCL all_reduce sweep. Restart the model and check it answers.",
    "Then the other four.",
    "Treat every reboot as a test that your boot-time settings survive: after it, run spark-check.sh and read the first 90 seconds of the kernel log (journalctl -b -k) for Xid, rpcRecvPoll, Oops.",
    "Allow 20 minutes per reboot before assuming a Spark is dead, and know who can reach it physically. Do not pull power."
   ]
  },
  {
   "id": "ROUTINE-2",
   "title": "After any change",
   "steps": [
    "bash spark-check.sh --table on every Spark: everything ok or n/a.",
    "Jumbo ping between every pair on both subnets: ping -c 3 -M do -s 8972 10.0.0.N and 10.0.1.N.",
    "NCCL all_reduce across sizes, not just perftest: perftest passed 72/72 on a kernel where NCCL failed every run."
   ]
  },
  {
   "id": "ROUTINE-3",
   "title": "After a power cut",
   "steps": [
    "Some Sparks stay off until their power button is pressed (2 of 8 on ours).",
    "Nothing starts by itself on ours (containers have no restart policy, on purpose): check both fabric links on every Spark before relaunching a multi-Spark model."
   ]
  }
 ],
 "do_not": [
  "Remove snapd or the desktop packages (tens of MB; makes DGX OS updates harder).",
  "Turn off the DGX Dashboard itself (13 MB; useful). Turn off only its auto-update.",
  "Change swap, THP or CPU governor (already right on DGX OS).",
  "Set display memory to 4 GB, or Auto with a monitor attached.",
  "Use 64K-page kernels.",
  "Install drivers off the DGX repo (.run installers, other branches).",
  "Turn on PFC on the switch (it hurt here).",
  "Load an nftables rule through the stock nftables.service (its config flushes Docker's rules)."
 ],
 "open_issues": [
  {
   "id": "OPEN-1",
   "title": "CVE-2026-98432 not fixed in linux-nvidia-7.0",
   "status": "Mitigated (SEC-2, SEC-4). Canonical lists linux-nvidia-7.0 on 24.04 as Vulnerable as of 2026-11-03.",
   "watch": "https://ubuntu.com/security/CVE-2026-98432",
   "next": "When marked fixed: move to that kernel with ROUTINE-1, then remove SEC-2 and SEC-4."
  },
  {
   "id": "OPEN-2",
   "title": "Rare slow shutdown",
   "status": "1 of 13 recent reboots on one Spark: nvidia-persistenced stuck in the display driver (nvkms_close); shutdown took 7.5 min, reboot ~15 min. 0 of 13 on three others. Its next restart, the same day, took about 2 minutes.",
   "watch": "journalctl -b -1 -k | grep nvkms_close",
   "next": "Wait it out; do not pull power."
  },
  {
   "id": "OPEN-3",
   "title": "7.0.0-1024 resets with displays attached",
   "status": "NVIDIA shipped nvidia-spark-initcall-bl 1.0-2 (SW-8). Other 1024 reset reports on the forum involved a self-patched driver module or a different driver (595.99.02); none matched our stock headless setup.",
   "watch": "https://forums.developer.nvidia.com/c/accelerated-computing/dgx-spark-gb10/dgx-spark-gb10/721",
   "next": "Rollback trigger for us: one unexplained reset or GPU firmware timeout on a stock headless Spark."
  },
  {
   "id": "OPEN-4",
   "title": "Fans ignore CPU heat until about 80 °C; no owner fan control on EC 3.13.7",
   "status": "Measured on one Spark on 2026-11-04: 20 busy cores, no model, reached 80 °C in 4 s and held near 94 °C with fans about 3,100 rpm; nvidia-smi read 62 °C at most. The ACPI fan setting is accepted and ignored. Partial workaround: cap the Cortex-X925 cores at 2,808 MHz (79.7 °C after 20 min, 16.5% less CPU work). We do not run the cap; our Sparks serve models on the GPU.",
   "watch": "https://forums.developer.nvidia.com/t/new-dgx-spark-fans-barely-speed-up-at-high-temps-and-cant-be-adjusted-critical-please-open-fan-control/386805",
   "next": "When NVIDIA ships an EC with an earlier fan curve or host fan control: rerun the 20-core test and update this section. Do not downgrade the EC or script cooling_device writes."
  }
 ],
 "changelog": [
  {
   "date": "2026-11-03",
   "change": "First version. All eight Sparks updated to DGX OS 7.6.0 packages with ConnectX-7 28.50.1002; CVE-2026-98432 mitigations added."
  },
  {
   "date": "2026-11-03",
   "change": "Second window: NVIDIA display-reset fix nvidia-spark-initcall-bl 1.0-2 (SW-8) and security updates on all eight. Mitigations survived the restart; RDMA latency 3.35 to 2.50 µs; network tests now cover both groups."
  },
  {
   "date": "2026-11-03",
   "change": "Proofread: MEM-1 gain corrected to +1.6 GiB (was +1.5); latency cause marked likely, not proven; SW-8 undo needs --allow-downgrades; SGLang KV-pool variation noted under Memory."
  },
  {
   "date": "2026-11-04",
   "change": "Added CPU-heavy work and the fans (under Power and heat) and OPEN-4: fan response to CPU heat on EC 3.13.7, measured on one Spark, and the X925 clock cap as a partial workaround. Suggested by bellatrix (@0xb3llatrix)."
  }
 ],
 "dateModified": "2026-11-04T07:06:43-06:00"
}