DGX Spark thermal shutdown: how to tell whether heat switched your GB10 off
One of our three NVIDIA GB10 machines, the DGX Spark class of box, has switched itself off six times since July, and the system log never said why. This is how we found out what killed it each time, what the numbers look
One of our three NVIDIA GB10 machines, the DGX Spark class of box, has switched itself off six times since July, and the system log never said why. This is how we found out what killed it each time, what the numbers looked like, and the small free tool that came out of it.
What a GB10 power-off looks like
-
journalctl -b -1simply ends mid-activity. There is no shutdown in it: a hard power-off never gives journald the time to write its last minutes. -
last -xstill lists the previous boot as still running, because nothing ever recorded its end. - The hardware watchdog did nothing. A watchdog reboots a machine that hangs; this one had no power.
- After one of them, the clock came back with a date weeks in the past.
So the one question you have after the machine comes back, why did it switch off?, has no answer in the logs.
Check the temperature now
Before any tooling, these two commands show what the machine is doing:
# every thermal zone, in millidegrees; the hottest one is the number that matters
cat /sys/class/thermal/thermal_zone*/temp
# GPU temperature, power, utilisation and SM clock
nvidia-smi --query-gpu=temperature.gpu,power.draw,utilization.gpu,clocks.sm --format=csv
The trouble is that by the time you look, the machine is either fine or off.
A flight recorder: one line a minute, written through to disk
The fix for an empty journal is a record that survives a power-off: one line a minute, flushed to disk with fsync before the next one, so the minute before the power-off is already on the disk when the power goes.
2026-11-19T08:43:21Z hot=95 zones=95,86,95,85,94,86,87 gpu=82C,49.5W,96%,2437MHz load=15.26,13.76,12.00 mem=113342/123970MB
When the machine is back, the gaps in that record are the power-offs, and the minutes before each gap say what happened. A restart someone asked for leaves a clean shutdown in wtmp, so it can be told apart from a power-off.
What it showed on our machine
191,624 readings from 5 July to 18 November, ten gaps:
CLEAN 3, COOL 1, HEAT 6
Three were restarts we had asked for. One was a stop at a cool 85 °C. Six were power-offs with the hottest sensor at 95 to 98 °C in the minutes before. The most recent one:
2026-11-14 10:28 UTC dark for 31 min
HEAT: the hottest sensor read 98 C in the last 10 minutes. The GPU was nearly idle (0% at 500 MHz), so the heat came from the CPU cores, not from GPU work.
10:26 UTC hottest 96 C gpu 78 C 9.0 W 0% 500 MHz load 5.05 mem 78,140/124,545 MB
10:27 UTC hottest 98 C gpu 78 C 8.9 W 0% 500 MHz load 2.54 mem 77,387/124,545 MB
10:28 UTC hottest 95 C gpu 78 C 9.1 W 0% 500 MHz load 8.46 mem 78,090/124,545 MB
Two things surprised us.
It switched off well below the limit it reports. The ACPI critical trip point on this machine reads 104.8 °C. The record never saw 100 °C; every power-off came at 95 to 98 °C. A reading a minute misses short peaks, so the chip may have touched more, but the reported trip point is not a safe target.
The GPU was not always the heater. The September power-offs came with the GPU at 96% load. The last three came with the GPU idle at 500 MHz, because a heat guard had already paused the GPU work: the CPU cores alone kept the chip at 95 to 98 °C.
How hot does it run normally?
Over the last 30 days, the hottest sensor's median was 89 °C:
85 C or more: 73.5% of minutes
90 C or more: 47.9% of minutes
95 C or more: 18.0% of minutes
97 C or more: 8.8% of minutes
100 C or more: 0.0% of minutes
That table is the one to set alarms from. Our first heat alarm was at 88 °C, which turned out to be this machine's normal temperature, so it rang all day and was ignored.
What helped
-
Cap the GPU clock, because you cannot cap the power. On GB10,
nvidia-smi -planswers N/A.nvidia-smi -lgc 0,2000works: in July it took our hottest zone from 91 °C to 77 to 83 °C within seconds. Apply it before Docker and your model server start (their cold start at full power is the moment that crashed us), and check it under load: with the cap on, the clock still reads about 2,400 MHz when the GPU is idle, and holds at 1,976 MHz under load. - Pause the heavy jobs before the limit, and resume them only once the machine has cooled, not the moment it drops a degree.
- Look at the airflow first. This machine's cooling was two to three times worse than its siblings' at the same load. Software kept it useful; it did not fix it.
The tool
We published the recorder as gb10-blackbox: one Python file with no dependencies, a systemd timer and an optional service for the clock cap. MIT licence.
git clone https://github.com/lightsoutfactory/gb10-blackbox
sudo gb10-blackbox/install.sh # a reading every minute
sudo gb10-blackbox/install.sh --clock-cap 2000 # and the GPU clock cap at every boot
gb10-blackbox why # after a power-off: every gap and its verdict
gb10-blackbox bands # how hot the machine runs
The whole story of this machine, including the log line that blamed memory for weeks, is in our field note: Our GB10 machine kept switching itself off. The log blamed memory; it was heat. The heat guard that pauses jobs before the machine gets that hot is part of Factory Floor, our kit for keeping a local-AI box running with nobody watching it.
If your GB10 box has switched itself off, run the recorder for a week and post what why says. We would like to know whether other machines stop at the same temperatures.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.