DGX Station GB300: How We Run It

by Milo · text and diagrams: anthropic/claude-fable-5.1 (reasoning high); measurements, settings, test plans and fact-check: claude-opus-5-5 (test plans and results also reviewed by Fable 5.1) · checked weekly

Created November 4, 2026 · Updated November 4, 2026 at 9:47 PM CST · Last checked November 4, 2026

What we run

One box, one model, all day.

A DGX Station GB300 is built around one GB300 module: NVIDIA's Grace CPU and a Blackwell Ultra GPU in one package, joined by a fast link called NVLink-C2C.

The GPU has its own memory, HBM (high-bandwidth memory). On our box 250.7 GiB of it is visible, not the 288 GB on the spec sheet. The Grace CPU adds 494.5 GiB that Linux sees as ordinary memory and the GPU can reach over that link.

Serving a model has two phases. Prefill is reading the prompt. Decode is writing the answer, one token at a time.

The one PCIe slot holds an RTX PRO 6000 Blackwell Max-Q (96 GB, 300 W), installed October 21, 2026. It is a sidecar: the model's experts that don't fit in HBM (the "cold" experts) run on the RTX instead of being read from Grace memory.

Two 200 Gb/s ports go to our MikroTik switches and on to our Sparks. One 1,600 W supply runs it from a dedicated 120 V / 20 A circuit.

It serves DeepSeek-V4.1-Flash on port 30006 in a lane we call Pg61, built on Garner McCloud's dgx-station-recipes plus our patches. Earlier posts cover the card install and what the sidecar buys.

Which Station? Six makers, one software image

Every maker ships the same software. Pick the box for hardware and support.

NVIDIA doesn't sell the DGX Station itself. Six makers build it: ASUS, Dell, Gigabyte, HP, MSI and Supermicro. Ours is an MSI, bought through the integrator Exxact.

We read Canonical's public OEM archive for all six on November 4, 2026. Every image depends on the same NVIDIA kernel, driver and GB300 tuning packages. The maker adds EULAs, a logo, and sometimes a wallpaper or a login banner.

What differs is outside the image: chassis and cooling, firmware, add-on apps, support, and power-supply notes. HP requires a 200–240 V circuit; the other five are rated for 115 V and up.

Not DGX OS, and we didn't reinstall

Our image is the maker's Ubuntu, not NVIDIA's DGX OS, and we're keeping it.

There is no /etc/dgx-release on this box; MSI shipped it that way. We looked at DGX OS and said no, for three reasons: the Station isn't on its supported list, it ships an older driver and kernel than we already run, and NVIDIA's own reimage procedure pulls from the maker's repositories anyway.

Some Station images do carry a dgx-release file, which NVIDIA's tools call "Software 2.0". Ours doesn't, and that matters for one power feature below.

Our settings at a glance

The whole baseline in one table, verified on our Station on November 4, 2026. Each ID links to its detail card.

Every row has a check command, its expected output, an apply and an undo in the cards below. Confirm a Station matches before you change anything.

Software

Stock kernel and driver, nothing held.

Kernel and GPU driver both come from Canonical's NVIDIA archive, and no package is held. The policy since October 26 is no holds: our recipes have to work on other people's machines, and a hold drifts us off the track everyone else is on.

Containers name each GPU by UUID through CDI (SW-6), which matters when the two GPUs are different kinds.

What NVIDIA's tuning packages set

Eighteen small NVIDIA packages own the kernel command line. We change none of it (SW-4).

In plain words: the kernel stops shuffling memory between the CPU's memory regions, and stops zeroing every page it hands out. The platform's power-meter cap is turned on; the Power section is about that cap. Two ports on two subnets each answer only for their own address. The driver stays loaded between jobs.

Updates are split, and the split is NVIDIA's

Security fixes install themselves. Kernel and driver moves wait for a window we plan (ROUTINE-1).

unattended-upgrades is on for the security pocket. NVIDIA's own blacklist package keeps every kernel and driver package out of that run and leaves a login message saying so.

The November 4 window moved the kernel from 7.0.0-1019 to 7.0.0-1024. The driver didn't change, and 1019 stays installed as the fallback.

Memory

Size models against 250.7 GiB and 494.5 GiB, not the spec sheet.

The GB300's memory runs in what the driver calls CDMM (coherent memory, driver mode), the stock setting. Linux sees Grace memory only; the driver manages the GPU's HBM itself.

A model that fits "in 288 GB" on paper does not fit in HBM here. Anything past 250.7 GiB comes from Grace memory over NVLink-C2C or, in our lane, from the RTX sidecar.

There is no swap, and we want none: a model server pins large host buffers, and swapping one would be a stall, not a slowdown (MEM-2).

Power

The 1,250 W cap costs us almost nothing, 240 V wouldn't change that, and the RTX stays at 300 W.

The rule

The Station has one 1,600 W supply and a fixed 1,600 W budget. NVIDIA's Station development guide splits it by the add-in card's rated power: no card or 75 W and under, the GB300 module gets all 1,600 W; a 76–300 W card, 1,250 W; over 300 W, 950 W. Line voltage is not an input.

Our box reports exactly that: module 1,250 W, RTX 300 W. The 50 W left over goes to nobody.

120 V or 240 V?

On this box, 240 V is not a speed upgrade.

The makers rate the supply at its full 1,600 W from 115 V up. The 1,300 W band is 100–114 V: Japan's mains, or an outlet sagging under load. A healthy US 120 V outlet is in the full band. 240 V buys half the current and some margin against sag, not speed.

HP is the exception: HP says the ZGX Fury "requires a dedicated 200–240V AC circuit" and is not meant for a standard 120 V outlet.

The claim that 240 V helps comes from Wendell of Level1Techs, who said in his review of this same MSI WS300 that 240 V gives "a few percent more performance". He ran his unit on 240 V, so 120 V was never on his bench. Power sloshing on his image, or outlet sag, would explain it.

Ours is a static split on a healthy outlet, so we measured.

What the cap costs

Nothing on decode, under 1% on prefill. Measured on our lane on November 4, 2026.

We lowered the GB300's module cap step by step on the live serving lane and measured throughput at each step, over seven traffic shapes, with the pass/fail bar written down first (PWR-1). At the stock 1,250 W the cap was never active during decode, and active under 9% of the time in prefill.

Below the cap, prefill loses about 0.5% per 50 W; decode doesn't notice until 950 W. The firmware won't let us measure above 1,250 W with this card, so 1,600 W is a bound, not a measurement: about 0.2–0.5% on the hungriest prefill shape, nothing on decode.

This is lane-specific. On October 22 our older lane hit the cap in 86–89% of prefill seconds and lost up to 4.9% of clock, with a different engine, prompts and sampling. Your lane will differ. Measure it before you decide the cap matters.

Can software raise it?

Not here.

nvidia-smi lowers the module limit fine. Asking for more is accepted as a request, and the limit stays at 1,250 W.

NVIDIA's own power-sloshing daemon, vsloshd, moves budget between the module and the card as load changes. NVIDIA documents it only for its DGX OS "Software 2.0" image. It isn't packaged for any maker's image, and there is no public word that it is coming (OPEN-3).

Running the card at 325 W

About 1% faster. We kept 300 W.

The card's limit can be set from 250 to 325 W, so we tried the top with the module held at 1,250 W (PWR-2). Mixed 8-user traffic and prefill each came out about 1% faster; 32-user decode was within noise. In the mixed test that is four more finished requests in 140 seconds.

Prefill was meant to be the unchanged control and moved too, so we ran a proper control that evening, pass rule set beforehand. Single-user decode, where the card's own limiter barely engages, didn't move. Mixed traffic repeated at about +1%, every 325 W run faster than every 300 W run. The gain is the card.

325 W did not raise the peak combined draw of the two GPUs. The card ran 88–89 °C at both settings, about 4 °C below its maximum operating temperature. About 1%, at the edge of what the test can see, isn't worth running the card above its rated class with that little margin.

The circuit

Alone on a dedicated 20 A circuit at 120 V, no power strip. That is arithmetic, not preference.

On October 22 the GPUs peaked at 1,547 W DC together. With fans, pumps, drives and the network card on top, that is about 15.0–15.6 A at 120 V against the 16 A continuous limit of a 20 A circuit (PWR-3). The arithmetic is in the October 22 post.

Storage

OS on the factory mirror, weights on local NVMe, nothing served from the NAS.

The operating system is on md0, a RAID1 mirror of two 2 TB drives. Right now it runs on one disk: the second stopped appearing on the October 26 boots, and the check script flags it (STO-1, OPEN-1). A replacement is on the way.

Model weights live on /models, a RAID0 stripe of two 8 TB drives in the M.2 slots that shipped empty. RAID0 is fine here because this is a cache, not a vault: every weight set also lives on our NAS.

We never serve from the NAS mount. Pinning weights over CIFS faults them in 512 KB at a time and takes hours; the local NVMe loads the same weights in about 7–8 minutes (STO-2).

Build and repair arrays by /dev/disk/by-id and serial number, never by nvme number. The November 4 reboot was a live example: the OS disk moved from nvme1n1 to nvme2n1. The drive install is in the M.2 post.

Network

Two 200G ports, two subnets, jumbo frames. The switch side lives in its own post.

Two ConnectX-8 ports at 200 Gb/s each, MTU 9000, each on its own subnet. The ARP and reverse-path settings that make that work come from NVIDIA's package, not from us (NET-1).

The two ports let the Station join our Sparks over RoCE, which is RDMA over ordinary Ethernet: the network cards copy data between machines' memories without the CPU in the path. There is also a 1 GbE LAN port and Tailscale.

Security

One open kernel bug, mitigated two ways. ssh by key only.

CVE-2026-98432 is a use-after-free in the kernel's IPv4 TCP handshake. Any machine that can reach any open IPv4 TCP port can trigger it; no login is needed. Canonical lists the Station's kernel family as Vulnerable, with a fix in progress for 7.0.0-1025. Our 1024 does not have it (OPEN-2).

We applied the same two mitigations as on our Sparks, each approved by James. SEC-1 turns off two kernel features through sysctl. SEC-2 is a firewall rule that drops the one kind of packet that triggers the bug. Normal traffic never carries it, and RoCE is UDP, so nothing we use is affected.

ssh (SEC-3): the factory image allowed password logins. One file turns them off. A password attempt now gets "Permission denied (publickey)"; key login works.

Still different from the Sparks, as facts rather than plans: user namespaces are enabled here (the Sparks run 0), and ufw is inactive. The box is reachable on the LAN and over Tailscale only.

How we update

One check before a kernel reboot matters more than everything else in the routine.

NVIDIA's prebuilt GPU modules are tied to one exact kernel. On day one this box booted a kernel without its NVIDIA modules. ROUTINE-1 has the check that prevents it.

The November 4 kernel window was the first under this guide: ssh back six minutes after the reboot, lane dark about 15 minutes in all. The lane now restarts itself after a boot (SRV-1); that unit failed its first real reboot on a boot-time race, and the fixed version hasn't been through one yet (OPEN-4).

After a power cycle the front button can stay dead while the BMC boots, and the box can take about five minutes to answer ssh. The lane then takes about six more minutes to serve.

ROUTINE-1 Kernel and driver updates (a planned window)

1. Dry run: sudo apt-get -s dist-upgrade | grep -E '^(Inst|Remv)'. Read every linux-image, linux-modules and nvidia line.
2. Keep the current driver packages for rollback (dpkg-repack them), and check free space on /.
3. Stop the model. Install. Before rebooting, confirm the NVIDIA kernel-modules package for the NEW kernel is installed (linux-modules-nvidia-595-open-<new kernel>). Day one booted a kernel without it.
4. Reboot. Allow ~5 minutes for ssh. Check: both GPUs in nvidia-smi, driver string, CDMM still driver, MemTotal, md0, regenerate the CDI spec (SW-6).
5. Start the model, check it answers, run station-check.sh.

ROUTINE-2 After a power cut or reboot

1. If the front button does nothing, the BMC is still booting. Wait; don't hold the button.
2. The lane starts itself (SRV-1). Allow about 15 minutes from power-on: about 2 for the box to answer ssh after a reboot (longer after a power cut), 4 more while the autostart waits for the GPUs to settle, ~6 for the lane to load. If it is still dark after 30 minutes, read the newest autostart-*.log.
3. Run station-check.sh. Look at STO-1 (both OS disks) and the security rows.
4. A power cut is an unclean shutdown: check dmesg for filesystem recovery on / and /models.

ROUTINE-3 Before running a different model

1. Write a MAINTENANCE flag (or NO-AUTOSTART) so a reboot doesn't bring the old lane back on top of you.
2. Stop the lane with its own down command, so its watchdog stops too.
3. Stage the new weights on /models (local NVMe). Never serve from the NAS mount.
4. Drop the page cache before starting a big model, then start it.

Every setting, in detail

One card per setting: the value, why, a check command with its expected output, the apply, the undo, whether it needs a reboot, and where it came from.

What we decided not to do

• Reinstall DGX OS: the Station isn't on DGX OS 7's supported list, 7.6.0 is older than the factory image, and the supported reinstall is the maker's image.
• Hold the kernel or driver: the stock blacklist already keeps them out of unattended upgrades; holds just drift you off everyone else's track.
• Try to raise the GB300 module limit with nvidia-smi: it takes the request and keeps enforcing 1,250 W.
• Set the add-in card to 325 W for speed: we measured about +1% and kept 300 W.
• Write your own power-sloshing daemon: on this box the module can't be raised from software, and the measured upside is a few tenths of a percent.
• Rewire to 240 V for speed: it buys current margin, not tokens per second (HP owners: HP requires 200–240 V).
• Pull the add-in card to get 1,600 W for the GB300: our lane would lose its sidecar for a few tenths of a percent of prefill.
• Put a 600 W RTX PRO 6000 Workstation Edition in the slot: a card rated above 300 W drops the GB300 to 950 W.
• Build or repair RAID by /dev/nvme number: the numbers reshuffle between boots. Use /dev/disk/by-id and the serial.

Each of these looked like a fix at some point. The measurements and archive reads above are why it isn't one, on this box, as of November 4, 2026.

Open issues

Hands-on videos, by maker

Where someone has had a Station on a bench. We confirmed the titles on November 4, 2026 and have not re-watched these for this post; the notes say what each one is, not whether it is right.

Everything else we've collected (talks, articles, benchmarks) is on our GB300 links page.

How this page stays current

We re-check this page every Monday against our Station with the read-only station-check.sh linked at the top, and against NVIDIA's Station documentation and Canonical's CVE tracker for linux-nvidia-7.0. When something changes we update the settings and the machine-readable copy, add a changelog line here, and bump the Last checked date.

If that date is more than a couple of weeks old, assume we missed a week and check the sources yourself. The November 4 run of the check script, after the kernel update: every row ok except STO-1, the degraded mirror. Start date: November 4, 2026.

• 2026-11-04: Kernel 7.0.0-1019 to 7.0.0-1024 (driver unchanged); lane autostart fixed after its first real reboot. Earlier the same day: first version. Measured the GB300 power cap and the RTX 300 vs 325 W choice; applied CVE-2026-98432 mitigations, key-only ssh and a lane autostart; documented the degraded OS mirror.

Related posts

• DGX Spark: How We Run Them: the sister guide for the eight Sparks, same format.
• GB300 with RTX 6000, and 120V Power: the full 120 V versus 240 V argument and the October 22 power trace.
• Adding an RTX PRO 6000 Blackwell Max-Q to the DGX Station GB300: the card install and how we first found the 1,250 W cap.
• DeepSeek-V4.1-Flash on a GB300 + RTX PRO 6000 Sidecar: what the sidecar is and what it buys.
• Filling the empty M.2s on an Exxact GB300 Station: the /models RAID0 build and the by-id lesson.
• Networking DGX Station and DGX Sparks with MikroTik CRS804: netplan and switch configuration.
• GhettoRak: an 8-node DGX Spark cluster: the rack the Station's two ports connect to.
• GB300 links: everything we've read and watched about the GB300 Station.