Testing DGX Spark Kernels
On November 1, James told someone on X not to take the current DGX Spark kernels, for three reasons: 7.0.0-1024 reboot-loops Sparks, 7.0.0-1019 broke multi-node RoCE, and his own Sparks were "still on 6.17.0-1032". He wanted to check that advice properly rather than repeat it, so we took four Sparks out of service, installed both kernels, and are running the same test battery on each.
- "Mine are on 6.17.0-1032" was wrong. All eight of our Sparks were on 6.17.0-1031. 1032 is the "known good" kernel named in NVIDIA's forum thread about the 1019 regression.
- "1019 broke multi-node RoCE" was true in September, but it is out of date. On 1019 with NVIDIA's
kho=offhotfix (shipped as thenvidia-spark-grub-khopackage), our four-node RoCE and NCCL runs had 0 failures, including with 100 GiB of GPU memory held resident, which is the condition the September bug needed. Throughput was the same as the 6.17 control (NCCL ~186 Gb/s on 4 nodes). 1024 setsKHOoff in the kernel config itself. - "1024 is reboot-looping Sparks" is not tested yet. The only change in 1024 since 1023 is a USB-C (xHCI) controller fallback, and the first reboot reports involved a USB-C monitor. Our headless 1024 run is scheduled for tonight.
- The security advice holds. None of the kernels we looked at carries a fix for the public root exploit (ZDI-CAN-33367). Setting
user.max_user_namespaces=0blocked user-namespace creation and did not break Docker GPU containers or RoCE NCCL on our boxes.
Bottom line so far: moving from 6.17 to 7.0.0-1019 (with the stock 580 driver and kho=off) changed nothing we could measure. The one difference: 6.17.0-1032 showed about 0.5 µs higher RDMA latency than 6.17.0-1031 or 7.0.0-1019.
Status
| Arm | Kernel | Driver | State |
|---|---|---|---|
| K0 | 6.17.0-1031 | 595.84 | done: our fleet before the test |
| S0 | 6.17.0-1032 | 580.178.04 | done: same-driver control, 6 h soak, 0 resets |
| S1 | 7.0.0-1019 (kho=off) | 580.178.04 | running: fabric + lane done, soaks in progress |
| S1x | 7.0.0-1019 (kho=off removed) | 580.178.04 | queued: reproduce the September bug |
| S2 | 7.0.0-1024 | 580.178.04 | queued: today's stock Dashboard kernel |
The claims, one by one
| Claim in the post | What we found | Verdict |
|---|---|---|
| "mine are still on 6.17.0-1032" | All eight Sparks ran 6.17.0-1031-nvidia (uname -r). 1032 was in the archive but not installed. | Wrong |
| "1019 broke multi-node RoCE" | Real in September: NVIDIA forum thread 383023, ibv_reg_mr_iova2 ENOMEM once a model is resident, with a staff hold notice posted September 13. NVIDIA then shipped the kho=off hotfix. With it, 1019 passed every RDMA and NCCL run on 2 and 4 nodes, with and without 100 GiB resident. | Out of date (repro without kho=off pending) |
| "7.0.0-1024 is reboot-looping Sparks" | Forum threads 387397 (31 posts) and 387401 report resets after the 1024 update; NVIDIA staff say they are investigating. 1024's only change vs 1023 is the NVDA8000 xHCI "Power Wrap" fallback (LP #2171975). | Pending (tonight) |
| "public root exploit works on Ubuntu 24.04" | Sysdig/TuxCare: ZDI-CAN-33367 roots 24.04; the fix is not merged upstream. No candidate kernel's changelog has a fix. | True |
| "set user.max_user_namespaces=0" | It blocks unshare -U. Docker GPU containers and 2-node RoCE NCCL still work (details below). We did not run the exploit. | Reasonable, low cost here |
How we test
Sparks 5–8 (four GB10 nodes, ConnectX-7 on both rails through one MikroTik CRS804 switch) were taken out of service; Sparks 1–4 kept serving. Each arm boots all four nodes into one kernel. Test kernels boot with grub-reboot, so a crash falls back to 6.17.0-1032 on its own, and an unexpected fallback is itself a detected reset.
Why the driver changed. Prebuilt NVIDIA kernel modules pin one exact driver version, and one host has one driver. Our boxes ran 595.84, which has no module for any 7.0 kernel. Only the stock DGX OS 7.6.0 driver, 580.178.04, has modules for a 6.17 kernel (6.17.0-1032) and for both 7.0.0-1019 and 7.0.0-1024. So all comparison arms (S0, S1, S2) run 580.178.04, and only the kernel changes between them. K0 is kept as a reference for the stack we actually ran.
| Test | What it does |
|---|---|
| Boot facts | Kernel, cmdline (kho=off), driver, MemTotal/MemAvailable, Xid count, CX-7 firmware, HCA state |
| RDMA | perftest ib_write_bw/ib_read_bw (1 MiB, 4 QPs, 10 s) and ib_write_lat (8 B), per rail, node pairs 5↔6 and 7↔8, 3 reps, with and without a 100 GiB GPU allocation held on every node |
| NCCL | Our own torch.distributed all_reduce sweep (float32, 1 MiB–4 GiB, peak bus bandwidth), 2 and 4 nodes, both rails, 3 reps, with and without 100 GiB held. This is not nccl-tests. |
| Real workload | DeepSeek-V4.1-Flash-1017 TP4 on the vLLM fork (Christopher Owen's spark-ds41f r6x): time to ready, C1 and C16 decode (3 reps), 32K/131K prefill, and a teacher-forced numerics check against a saved baseline (45 segments, 5,485 positions) |
| Single Spark | llama-bench (llama.cpp 785322d) on gpt-oss-120b MXFP4, -fa on -ub 2048, pp2048/tg32 at depth 0/4K/16K, 3 reps; bf16 GEMM peak; device copy bandwidth |
| Soak | 2 h under steady 4-way load, then 4 h idle with no monitor attached. Every node is polled each minute for boot_id/kernel/uptime/Xid. |
Fabric and single-node tests ran in NGC PyTorch 25.09 (CUDA 13.0) on every arm, because 26.07 (CUDA 13.3) targets a newer driver. That choice matters: on the same box and kernel, bf16 GEMM peaked at 91.2 TFLOPS in 26.07 and 61.1 TFLOPS in 25.09. Compare containers before you compare kernels.
Results so far
K0 here is our pre-test stack measured in the same 25.09 container as the S arms (for the lane rows, the container doesn't matter because the vLLM image brings its own). Values are means, with min–max across reps in brackets.
| Metric | K0 6.17.0-1031 · 595.84 | S0 6.17.0-1032 · 580.178.04 | S1 7.0.0-1019 · 580.178.04 |
|---|---|---|---|
| GPU Xid errors this boot (4 nodes) | 0 | 0 | 0 |
| MemAvailable after boot | 117.1 GiB | 117.4 GiB | 117.5 GiB |
| RDMA write, per rail, 100 GiB held | 111.86 Gb/s | 111.86 Gb/s | 111.86 Gb/s |
| RDMA write latency (8 B, typical) | 2.50 µs | 3.04 µs (3.02–3.09) | 2.51 µs |
| RDMA failures (all runs) | 0 | 0 | 0 |
| NCCL all_reduce peak, 4 nodes | 184.3 Gb/s | 186.4 Gb/s | 186.0 Gb/s |
| NCCL all_reduce peak, 4 nodes, 100 GiB held | 184.6 Gb/s | 186.6 Gb/s | 186.3 Gb/s |
| NCCL all_reduce peak, 2 nodes, 100 GiB held | 189.7 Gb/s | 185.3 (179.3–191.2) | 184.8 (177.2–190.7) |
| NCCL failures (all runs) | 0 | 0 | 0 |
| DSV4.1 TP4 lane ready | 137 s | 127 s | 125 s |
| DSV4.1 C1 decode | 94.0 tok/s | 93.0 tok/s | 93.1 tok/s |
| DSV4.1 C16 aggregate | 480.9 tok/s | 484.6 tok/s | 477.7 tok/s |
| Numerics vs saved baseline | PASS, |Δlogp| 0.0 | PASS, |Δlogp| 0.0 | PASS, |Δlogp| 0.0 |
| gpt-oss-120b pp2048 / tg32 | 2,611 / 60.1 tok/s | 2,604 / 59.0 tok/s | running |
| bf16 GEMM peak / copy bandwidth | 61.1 TFLOPS / 246.7 GB/s | 61.1 TFLOPS / 246.5 GB/s | running |
| Loaded soak (2 h, 4 streams) | — | 0 resets; 4,040 requests, 0 errors | running |
| Idle soak (4 h, headless) | (3+ days uptime before test) | 0 resets | pending |
Reading it: RDMA throughput sits at the same per-rail ceiling on every arm. That ceiling is the PCIe Gen5 x4 link each ConnectX-7 port hangs off, not the kernel. NCCL differences are within run-to-run spread. The model produced bit-identical log-probabilities on all three stacks. The one measured difference is the ~0.5 µs higher RDMA latency on 6.17.0-1032. It is consistent across reps and goes away on 1019 with the same driver, but it is too small to show up in any workload number above.
Exploit mitigation on a Spark
With user.max_user_namespaces=0 set at runtime on two nodes (restored afterwards):
| Check | Result |
|---|---|
unshare -Ur true (what the exploit needs first) | Blocked (No space left on device = limit 0) |
Docker container with GPU (--gpus all) | Works |
| 2-node NCCL over RoCE in containers | Works (175.9 Gb/s peak, short sweep) |
systemd-run --user | Works |
| User namespace inside a container | Blocked (Docker's default seccomp blocks it anyway) |
| Firefox snap headless screenshot | Inconclusive: it fails on these boxes even without the sysctl |
This setting breaks things that rely on unprivileged user namespaces, such as rootless Docker/Podman and some browser sandboxes. We did not run the exploit itself; this only shows the step it depends on is blocked. Note that Ubuntu's AppArmor user-namespace restriction (on by default) is not enough: Sysdig reports the proof of concept gets past it.
What this is not
- Four Sparks, one switch, one site. Your RoCE topology (direct cable, ring, other switch) may behave differently.
- We test the 4K-page
nvidiaflavour the Dashboard installs, notnvidia-64k. - "No reset in 6 hours" is a soak result, not proof of stability. The 1024 reports include resets within minutes of boot, which a soak would catch.
- We test stock 580.178.04 because it is the only driver that can boot every kernel compared here. Our production stack before the test was 595.84; on the workloads above the two drivers measured the same.
What's left
| Step | Expected |
|---|---|
| S1 (1019) soaks + single-Spark numbers | ~12:45 PM CST today |
S1x: 1019 with kho=off removed. Does the September ibv_reg_mr ENOMEM come back? That would show kho=off is the fix. | ~1:30 PM CST |
| S2: 7.0.0-1024, full battery + 6 h soak headless | ~8:20 PM CST |
| 1024 with a USB-C monitor attached (tests the xHCI hypothesis) | after S2, needs hands on the box |
| Test scripts published as a repo so any Spark owner can run the same battery | with the final update |
We'll update this post when each of these finishes, and James will post the correction on X.