DGX Spark: How We Run Them

by Milo · text and diagrams: anthropic/claude-fable-5.1 (reasoning high); measurements, settings and fact-check: claude-opus-5-5 · CPU-heat section (November 4): claude-opus-5-5, reviewed by anthropic/claude-fable-5.1 · checked weekly

Created November 3, 2026 · Updated November 4, 2026 at 7:06 AM CST · Last checked November 3, 2026

What we run

Eight DGX Sparks (GB10, 128 GB of unified memory each) sit on one wire shelf we call GhettoRak. They work as two groups of four. Each group runs one large model, DeepSeek-V4.1-Flash, split across its four Sparks (tensor-parallel 4). The four talk over RoCE, which is RDMA over ordinary Ethernet: the network cards copy data straight between the Sparks' memories without going through the CPU. Sparks 1–4 run SGLang; Sparks 5–8 run a vLLM fork. Both lanes are shared with a few friends through our own gateway. Two MikroTik CRS804 switches carry the traffic, with one 200G cable per Spark and one 400G cable between the switches. Nobody sits at these machines. If you use a Spark as a desktop, skip the headless and display-memory settings (MEM-1 and MEM-2); the rest still applies. The physical build is in the GhettoRak post.

Our settings at a glance

This is the whole baseline in one table, verified on all eight Sparks on November 3, 2026; each ID links to its detail card.

Every row has a check command with its expected output, an apply command and an undo in the detail cards further down, so you can confirm a Spark matches before you change anything.

Memory

A Spark has one pool of 128 GB that the CPU and the GPU share. NVIDIA calls this unified memory. The useful consequence is that a model can use almost all of it. The awkward consequence is that everything else on the machine takes memory out of the same pool: the slice the firmware sets aside for the display, the desktop and its background services, and the file cache Linux fills as it reads files. On a normal computer those costs come out of system RAM and the GPU keeps its own. Here they come out of the model.

So we take three of them back. We measured how much GPU memory one process can claim on a Spark (cudaMemGetInfo after a cache flush, on 7.0.0-1024 with driver 580.178.04): 117.0 GiB stock, with the desktop on and the default 2 GB display reservation; 117.4 GiB headless; 119.0 GiB headless with the display reservation cut to 0.5 GB. That is +2.0 GiB per Spark and +16 GiB across our eight. The display setting is the big one, about four times what turning off the desktop is worth. The measurement and its chart are in the kernel post.

The 0.5 GB option needs the October 2026 firmware (FW-1); older UEFI menus only offer 2 GB or 4 GB. With the new firmware you set it from Linux by writing one EFI variable, no keyboard at the box (MEM-1). We pick 0.5 GB rather than Auto because Auto sizes the reservation from whatever monitor is attached at boot, and a 4K monitor makes it 1 GB. 0.5 GB still shows a text console, which is all a headless Spark needs. If you run a desktop with a 4K screen, use 1 GB or more and skip MEM-2 too.

The third one is the file cache. NVIDIA's own advice for unified memory is to flush it before starting a model, and we do that before every model start (MEM-4): sync; echo 3 > /proc/sys/vm/drop_caches. Otherwise the cache sits on memory the model is about to ask for.

What the extra memory buys depends on your engine. An engine that sizes its KV cache as a share of total memory gets a bigger cache with no config change. An engine with a fixed KV size just gets headroom. Both of ours came back with correct answers after each change. One catch with the share-of-memory kind: the file cache comes back while the model loads. Our SGLang lane sizes its KV cache from free memory right after loading about 76 GB of weights per Spark, and across 11 starts with identical settings the pool ranged from 4.66 to 5.93 million tokens. It tracked the engine's own free-memory reading every time, to within 0.2 GB. We think leftover file cache from the weight load explains most of that swing and are testing a flush at that exact moment. If your pool size jumps between restarts, compare free memory at load time before blaming an update.

Power and heat

A GB10 has no power-limit setting. nvidia-smi -pl is unsupported, so the GPU clock is the only power lever you have. Out of the box the GPU runs up to about 2,400 MHz (the default application clock is 2,418 MHz; the rated maximum is 3,003 MHz). Generating text is limited mostly by memory speed, not GPU clock, so the top of that range costs power and heat and buys very little.

We learned this from a tripped power strip. During a long prompt at stock clocks, each Spark drew about 200 W at the wall; the rack averaged 1,717 W and tripped its 15 A strip after about eight minutes. A 15 A strip at 120 V is 1,800 W, and 1,440 W for a load that runs for minutes (the 80% rule). The strip, not the wall circuit, was the limit. With a 2,200 MHz cap the same kind of run averaged 1,345 W, finished, and lost nothing in decode speed; that is −22% at the wall. The worst capped peak we saw was 1,425 W, still under the line. Idle, with all eight 200G links up, the rack sits around 448 W.

Two things about measuring. nvidia-smi reports the GPU rail only; on these boxes the wall number runs 3–4.5× what it shows. Measure at the wall. We use smart strips with per-outlet meters, and we plan around the strip's 1,440 W continuous line, not its 1,800 W label.

Others go lower than we do. One two-Spark sweep on NVIDIA's forum found the best energy per token around 1,400–1,800 MHz for about 3% less decode speed. That is their measurement, not ours; we have not tested below 2,200.

The cap does not survive a reboot unless something sets it again, so it lives in a small systemd unit that runs nvidia-smi -lgc 0,2200 at every boot (PWR-1). Under load the clock reads 2,177–2,190 MHz. One harmless message: when you swap GPU drivers, the unit's stop step logs "Driver/library version mismatch" once at that shutdown; the cap comes back on the next boot.

CPU-heavy work and the fans

If you use a Spark for CPU work, like builds or a lot of containers, rather than for a model on the GPU, it gets hot before the fans react. bellatrix (@0xb3llatrix) asked us to cover this, so we measured it on one of our Sparks with all 20 cores busy and no model running.

On the October firmware (EC 3.13.7, see FW-1) the fans never stop, but they sit at their idle speed, about 2,050 rpm, until the hottest sensor on the chip passes about 80 °C. Then they step up to about 2,520 rpm, and to about 3,090 rpm near 90 °C. So the fans do respond to CPU heat; they just start late, and they do not wait for the GPU. On stock settings the chip passed 80 °C in 4 seconds and 90 °C in 32 seconds, then held near 94 °C for the rest of a ten-minute run with the fans at about 3,100 rpm. The fast cores slowed a little, from about 3.89 to about 3.85 GHz. That is the fans at roughly 70% of the 4,350 rpm the fan controller reports as its maximum (figure from the NVIDIA forum thread, where maxeffort measured the same fan steps on another Spark).

Why it is easy to miss: nvidia-smi reports the GPU only, and it read at most 62 °C the whole time. Watch the chip's hottest sensor instead (value in thousandths of a degree):

cat /sys/class/thermal/thermal_zone0/temp

What you can do on this firmware:

• There is no working fan setting. Linux lists both fans as cooling devices you can set from 0 to 100 (/sys/class/thermal/cooling_device0 and 1). On EC 3.13.7 the value is accepted and the fans do not change. Do not script it anyway: forum owners report that on older EC firmware the same setting takes over the fans, and low values stop them.
• Cap the ten fast cores. GB10 has ten Cortex-X925 cores (up to 3.9 GHz) and ten Cortex-A725 cores (up to 2.8 GHz). Capping the X925 cores at 2,808 MHz kept the same 20-core load at 79.7 °C after 20 minutes, still creeping up slowly, and the fans never left idle. It costs CPU speed: 16.5% less work done than stock over a long run of a fully CPU-bound loop. Mixed container loads should lose less; we did not measure them. It is a partial fix: about 14 °C cooler, parked just under the point where the fans would start helping. Forum users suggested this idea before us (card.ps, and vedrin.jeliazkov, who runs 2.3 GHz with the schedutil governor); we only tested 2.8 GHz with the stock governor.

Apply (lasts until the next reboot):

for p in /sys/devices/system/cpu/cpufreq/policy*; do
  [ "$(cat $p/cpuinfo_max_freq)" = 3900000 ] && echo 2808000 | sudo tee $p/scaling_max_freq >/dev/null
done

Check (expect 2808000 on all 20 lines; stock shows ten lines of 3900000):

grep -H . /sys/devices/system/cpu/cpufreq/policy*/scaling_max_freq

Undo:

for p in /sys/devices/system/cpu/cpufreq/policy*; do
  cat $p/cpuinfo_max_freq | sudo tee $p/scaling_max_freq >/dev/null
done

The forum thread also reports a fix that downgrades only the fan firmware (the EC) to 3.5.8, blocks its updates, and then sets the fans from Linux. We have not tried it and do not recommend it: it is a firmware downgrade, and we have not worked out which part of the October firmware our 0.5 GB display setting (MEM-1) depends on. Our own Sparks serve models on the GPU, and we do not run the CPU cap. What we measured is one Spark, one run of each, room temperature not recorded, with a synthetic CPU load rather than real containers. Tracked as OPEN-4.

Security

There are two unfixed kernel bugs on these Sparks as we write this. They are different bugs with different doors, and one of our settings does nothing for the other, so here they are separately.

Bug 1: local root through user namespaces (SEC-1). A public exploit gives any logged-in user root on Ubuntu 24.04 kernels, including the Spark kernels; none of the kernels we tested has the fix. Its first step is to create an unprivileged user namespace. Setting user.max_user_namespaces=0 blocks that step. On ours, Docker containers with the GPU and multi-Spark RoCE still work with it set. It will break rootless containers and some browser sandboxes, which is a fair trade on a server and may not be on a desktop.

Bug 2: CVE-2026-98432, remote, no login (SEC-2 and SEC-4). This one is a use-after-free in the kernel's IPv4 TCP handshake. The trigger is a SYN packet carrying IPv4 options (Record Route, Timestamp) followed by a retransmission. Any open IPv4 TCP port will do, and no login is needed. Canonical rates it CVSS 3.1 9.8 / CVSS 4.0 9.2. Remote code execution has been reported against hosts with an MPTCP listener (the details are not published), and Go 1.24+ programs listen over MPTCP by default, which is why we restart the DGX Dashboard and Tailscale after setting it. Before we did anything, our Sparks had net.mptcp.enabled=1, no rule dropping IP-option packets, and a SYN with a Record-Route option reached spark8's port 22. user.max_user_namespaces=0 does nothing for this bug.

Credit to 0xCS for flagging that the Spark kernel is not fixed yet. The Canonical tracker (checked 2026-11-03) lists linux-nvidia-7.0 on 24.04 noble as Vulnerable; fixes for the standard kernels are in the proposed pocket, and our 7.0.0-1024 changelog does not mention the CVE.

Two settings close it from both sides. SEC-4 is an nftables rule that drops IPv4 TCP packets whose header length is not 5, which is the same as saying "carries IP options". Normal traffic never carries them, and RoCE is UDP, so it is untouched. SEC-2 turns MPTCP off (the reported path to remote code execution) and pins net.ipv4.tcp_migrate_req=0 (the second path the fix describes). SEC-4 is the one that blocks the trigger itself: the upstream fix shows that the memory freed too early is the IPv4 options saved from a SYN, and with SEC-4 no SYN carrying options gets in. SEC-2 closes the two follow-on paths in case one ever did. We applied both to all eight on 2026-11-03 at 09:56–09:58 CST, restarted the Dashboard, our Prometheus exporter and tailscaled, and then checked: a normal connect still reaches ssh (22) and both model APIs (8888 and 8002); a connect whose SYN carries a Record-Route option is dropped, on the LAN, on the fabric addresses and over Tailscale; Docker's firewall rules were unchanged (31 before and after); the kernel's count of MPTCP sockets was 0; both models answered.

One trap worth its own sentence: do not load the rule through the stock nftables.service. The default /etc/nftables.conf starts with flush ruleset, which would delete Docker's rules along with everything else. Our rule lives in its own table, loaded by its own oneshot unit, at prerouting priority −300 so it runs before anything else looks at the packet. The unit file is in the SEC-4 card. We also mask nftables.service, so nobody can turn it on later and wipe the rule. The next reboot, in a second update window the same day, was the real test: the rule and both settings came back on all eight Sparks, and the probe was still dropped.

We will remove SEC-2 and SEC-4 only after a fixed Spark kernel ships, we have moved to it with the monthly routine, and Canonical's tracker marks linux-nvidia-7.0 fixed. Until then they stay; so far nothing we run has broken because of them.

Network

The full how-to, with the netplan file and the switch commands, is in Networking DGX Station and Sparks with MikroTik CRS804, and the NCCL part is in NCCL tuning in plain English. This section is only what we run and why, in the order it matters.

One cable, two interfaces (NET-1). A Spark's 200G cable shows up in Linux as two network interfaces, one per half of the ConnectX-7 card. We give each half its own subnet, set MTU 9000 on every port on the path (at 1500, RDMA between two Sparks ran about 5% slower, 185 vs 196 Gb/s), and pin the ARP settings so each half answers only for its own address.

Seven NCCL settings (NET-4). NCCL is NVIDIA's library for combining partial results across GPUs; when one model runs on four Sparks, it is what carries the all-reduces between them, many times per token. We pass the same seven variables to every engine container. Two of them are safety, not speed: NCCL_SOCKET_IFNAME pinned, or NCCL may start over Docker or Tailscale; and NCCL_NET=IB, or a broken RDMA path falls back to TCP silently (1.17 GB/s instead of about 23, with no error). The last two fix medium messages: a 1 MiB all_reduce on four Sparks went from 1.36 to 11.5 GB/s, which is 8.4×. Credit to 0xCS for those.

Switch (NET-5). ECN on; RDMA traffic (RoCEv2, UDP 4791) in its own queue sharing bandwidth 50/50 with everything else; no PFC, which hurt here. Without the separate queue an RDMA pile-up starved TCP to 0.002–0.015 Gb/s; with it, TCP kept 41–94 Gb/s.

Test with NCCL, not just perftest. After any network change we run jumbo pings between every pair on both subnets, then ib_write_bw, then an NCCL all_reduce sweep across sizes. The last one is the gate. On the broken 7.0.0-1019 kernel, perftest passed 72 of 72 runs while NCCL failed every one.

How we update

Automatic updates are off on our Sparks, including the DGX Dashboard's one-click update (SEC-5). The reason is simple: a kernel, driver or firmware change should never happen under a running model, and a kernel change is also a driver decision, because NVIDIA's prebuilt GPU modules pin one exact driver version. So we run updates by hand, once a month, taking one group of four down at a time.

The routine in plain words. First a dry run on every Spark, reading every line that would remove a package and every line that touches a kernel or GPU driver package; those are separate, tested decisions, not part of the monthly pass. Then we look for firmware that arrives inside ordinary packages, and run fwupdmgr get-updates for system firmware. Then four Sparks first: stop their model, update, reboot, check kernel, driver, both fabric links and the clock cap, run the NCCL sweep, restart the model and make sure it answers. Only then the other four. And we allow 20 minutes per reboot before worrying.

The first window (November 3, 2026), as a worked example

This page started on the day we ran the routine for the first time, November 3, 2026, so here is what it looked like. The embarrassing part first: we had turned automatic updates off and never run them by hand. Sparks 1–2 had 418 pending updates (273 from the security pocket) and still reported DGX OS 7.5.0; Sparks 3–8 had 225–231 pending (170 security on spark5) and reported 7.6.0. The dry run moved no kernel and no GPU driver package and removed nothing, so this was a normal monthly pass; 226–442 packages upgraded per Spark.

The dry run did turn up one thing worth knowing about: a package called nvidia-spark-mlnx-firmware-manager (6.0.1) flashes the ConnectX-7 network card firmware, 28.45.4028 → 28.50.1002, during apt install. The new firmware takes effect after a reboot, and the updater verifies it on the next boot. That is the card all multi-Spark traffic runs over, so it got the four-first treatment: spark5 alone, then 6–8, then the network tests before we touched Sparks 1–4.

Everything within 1%, which we call no difference. Both model servers came back and answered; the vLLM lane on 5–8 was ready 92 s after start. Downtime per group of four was about 35–40 minutes including the network test. All eight now show DGX OS 7.6.0 packages and 0 pending updates.

One Spark gave us the 20-minute rule. spark8 took 7.5 minutes to shut down: nvidia-persistenced hung in the display driver (nvkms_close in nvidia_modeset) and systemd eventually forced the reboot. Its full reboot took about 15 minutes against about 2 for the others. Counting the boots since, it has happened on 1 of spark8's last 13 boots and on none of Sparks 5–7's. If we had pulled power at minute five we would have turned a slow reboot into an interrupted one. So: wait 20 minutes before assuming a Spark is dead, and do not pull power. We are watching it as an open issue below.

Same day, second window

Two hours after the first pass, a new NVIDIA package showed up: nvidia-spark-initcall-bl 1.0-2 (SW-8). It stops the kernel from starting a MediaTek power-management driver (power_wrap_init) that this platform does not use. NVIDIA's changelog says that driver has been correlated with spontaneous resets while a 4K display is attached. That matches the 7.0.0-1024 reboot loop people reported on the forum with monitors plugged in. Our headless Sparks never had it: 0 GPU firmware timeouts and 0 Xid errors in 38 boots on 1024 across all eight. We took it anyway because it is a stock NVIDIA package, and because the reboot doubled as a test we had not done yet: do our boot-time settings survive a reboot?

Same routine, Sparks 5–8 first. Each Spark got 22 packages, nothing removed, and no kernel or driver change. After every reboot the check script showed every row as ok, the CVE rule and both of its settings were back, a SYN with IP options was dropped over the LAN and over Tailscale, and the first 90 seconds of the kernel log had no Xid, GPU firmware timeout or Oops lines. Every reboot took about 2 minutes, spark8 included. This time we also ran the network tests on Sparks 1–4, which the morning window had skipped.

The surprise was latency. On 7.0.0-1024 our RDMA latency had been 3.35 µs in four separate test runs (3.33–3.39), against 2.50 µs on 7.0.0-1019. With power_wrap_init switched off it is back to 2.50 µs on Sparks 5–8 and 2.51 µs on Sparks 1–4. So that driver was most likely also adding about 0.85 µs to every small RDMA message on 1024. We have not isolated it with an A/B (that one package alone), so treat the cause as likely, not proven. Bandwidth did not change, and we have not yet measured whether model speed did. Fourteen other packages changed in the same window, but none of them touch networking. The 179.1 Gb/s cell is inside the spread that cell has always shown (174.8–192.5 Gb/s). One more lesson: downloading the test image left 12–18 GiB of file cache, so flush the cache before any big allocation test.

ROUTINE-1 Monthly updates (four Sparks first)

1. Dry run on every Spark: sudo apt-get -s dist-upgrade | grep -E '^(Inst|Remv)'. Read every Remv line and any linux-image, linux-modules, nvidia-driver or libnvidia line. A kernel or driver move is a separate, tested decision.
2. Look for firmware and boot changes inside packages (names with firmware, initcall, grub, kho) and read their install scripts before saying go: nvidia-spark-mlnx-firmware-manager flashes the ConnectX-7 during install. Run sudo fwupdmgr get-updates for system firmware.
3. Four Sparks first: stop the model, sudo apt-get dist-upgrade, reboot. Check kernel, driver, both fabric links, clock cap, then an NCCL all_reduce sweep. Restart the model and check it answers.
4. Then the other four.
5. Treat every reboot as a test that your boot-time settings survive: after it, run spark-check.sh and read the first 90 seconds of the kernel log (journalctl -b -k) for Xid, rpcRecvPoll, Oops.
6. Allow 20 minutes per reboot before assuming a Spark is dead, and know who can reach it physically. Do not pull power.

ROUTINE-2 After any change

1. bash spark-check.sh --table on every Spark: everything ok or n/a.
2. Jumbo ping between every pair on both subnets: ping -c 3 -M do -s 8972 10.0.0.N and 10.0.1.N.
3. NCCL all_reduce across sizes, not just perftest: perftest passed 72/72 on a kernel where NCCL failed every run.

ROUTINE-3 After a power cut

1. Some Sparks stay off until their power button is pressed (2 of 8 on ours).
2. Nothing starts by itself on ours (containers have no restart policy, on purpose): check both fabric links on every Spark before relaunching a multi-Spark model.

Every setting, in detail

One card per setting: the value, why, a check command with its expected output, the apply command, the undo, whether it needs a reboot, and where it came from.

What we decided not to do

• Remove snapd or the desktop packages (tens of MB; makes DGX OS updates harder).
• Turn off the DGX Dashboard itself (13 MB; useful). Turn off only its auto-update.
• Change swap, THP or CPU governor (already right on DGX OS).
• Set display memory to 4 GB, or Auto with a monitor attached.
• Use 64K-page kernels.
• Install drivers off the DGX repo (.run installers, other branches).
• Turn on PFC on the switch (it hurt here).
• Load an nftables rule through the stock nftables.service (its config flushes Docker's rules).

Fewer changes is a feature here: every item we leave at the DGX OS default is one less thing to re-check after each monthly update and one less way our Sparks differ from the ones NVIDIA tests.

Open issues we are watching

Things that are not fixed yet, what we do about them in the meantime, and what we are waiting for.

How this page stays current

We re-check this page every week against our eight Sparks, using the same read-only check script linked at the top, and against NVIDIA's DGX Spark release notes, known issues and forum advisories, plus Canonical's CVE tracker for the Spark kernel. When something changes we update the settings and the machine-readable copy, add a changelog line here, and bump the Last checked date at the top. If the Last checked date is more than a couple of weeks old, assume we missed a week and check the sources yourself. Start date: November 3, 2026.

• 2026-11-03: First version. All eight Sparks updated to DGX OS 7.6.0 packages with ConnectX-7 28.50.1002; CVE-2026-98432 mitigations added.
• 2026-11-03: Second window: NVIDIA display-reset fix nvidia-spark-initcall-bl 1.0-2 (SW-8) and security updates on all eight. Mitigations survived the restart; RDMA latency 3.35 to 2.50 µs; network tests now cover both groups.
• 2026-11-03: Proofread: MEM-1 gain corrected to +1.6 GiB (was +1.5); latency cause marked likely, not proven; SW-8 undo needs --allow-downgrades; SGLang KV-pool variation noted under Memory.
• 2026-11-04: Added CPU-heavy work and the fans (under Power and heat) and OPEN-4: fan response to CPU heat on EC 3.13.7, measured on one Spark, and the X925 clock cap as a partial workaround. Suggested by bellatrix (@0xb3llatrix).

Related posts

• GhettoRak: an 8-node DGX Spark cluster: the build, including the power section that produced the clock cap.
• Testing DGX Spark Kernels: the kernel, driver, memory and security measurements this page's settings rest on.
• Networking DGX Station and DGX Sparks with MikroTik CRS804: the fabric how-to, netplan and switch QoS.
• Why Four DGX Sparks Talk Slowly in the Middle: NCCL Tuning in Plain English: the seven NCCL settings and what each one does.