A single-board computer running headless in a closet, a greenhouse, or a remote sensor post has one failure mode that hurts more than any other: the silent hang. The board is powered, the link light may even blink, but the workload stopped hours ago and nobody is there to cycle power. A hardware watchdog exists precisely for this situation. It is a countdown timer in silicon that reboots the machine unless software pets it on schedule. Systemd can drive that timer for you with a few lines of configuration, turning an unreachable brick back into a node that recovers on its own. This guide covers the system-wide watchdog, per-service watchdogs, and the testing discipline that keeps the watchdog from becoming a source of surprise reboots.
What the watchdog actually guarantees
A watchdog timer makes one narrow promise: if the software responsible for petting it stops doing so, the machine reboots. It does not detect wrong answers, wedged peripherals, or a network stack that is alive but useless. It detects the absence of a heartbeat. That is exactly the right tool for the classic SBC failure, where a runaway process or a deadlocked kernel path leaves the system powered but unresponsive. The petting normally happens inside systemd itself, which refreshes the hardware timer on a fixed interval as long as the init system is still scheduling. If systemd itself wedges, the timer expires and the board power-cycles. Understand this boundary before you configure anything, because a watchdog cannot fix a service that is running but producing garbage. It only fixes a system that stopped running at all.
Enabling the system-wide watchdog
The system-wide setting lives in the systemd system configuration file, where two options matter. RuntimeWatchdogSec sets the hardware timeout while the system is up, and ShutdownWatchdogSec sets it during shutdown and reboot, which catches the case where a hung shutdown would otherwise sit forever. A common starting point on modest SBC hardware is thirty to sixty seconds for the runtime timer. That is long enough to ride out a slow SD card stall or a heavy I/O burst, and short enough that a real hang costs you a minute of downtime rather than a drive to the site. After editing the configuration, reboot and confirm the timer is armed by checking the kernel messages or the watchdog device status. If the board reboots in a loop immediately after enabling the watchdog, the timeout is shorter than the board needs to finish booting, which is the single most common misconfiguration on slow storage.
Pointing systemd at the right device
Most SBC system-on-chip designs expose a hardware watchdog device that the kernel presents as /dev/watchdog, and systemd uses it automatically when the runtime watchdog is enabled. Some boards need a kernel module loaded first, and a few expose multiple watchdog devices where only one is wired to actually reset the board. Check which device your kernel registered and confirm it is the one that triggers a real reset, because petting a watchdog that cannot reboot the board is pure theater. The WatchdogDevice option lets you name the device explicitly when the default choice is wrong. Do this verification early, while you have console access, not after the node is mounted somewhere inconvenient. A watchdog you have never seen fire is a watchdog you do not actually have.
Per-service watchdogs for critical workloads
The system-wide timer protects against a dead init system, but your important service can hang while everything else looks fine. Systemd lets individual services opt into their own watchdog with two settings in the unit file. WatchdogSec sets how often the service must check in, and Type=notify (or the notify-aware variants) gives the service the channel to do it. A service that supports the notification protocol sends a periodic heartbeat, and if the heartbeat stops, systemd restarts just that service instead of rebooting the whole machine. This is the right granularity for a sensor collector, a camera pipeline, or a gateway process. Set the interval with real headroom above the service's worst normal pause. A collector that occasionally blocks for twenty seconds on a slow sensor read needs a watchdog interval comfortably above that, or normal operation will look like a hang and you will get restarts during the busiest, most important moments.
Testing without learning the hard way
A watchdog configuration is only done when you have watched it fire on purpose. The safe test is to trigger the exact failure the watchdog is meant to catch and confirm the recovery. For the system timer, you can simulate a wedged init path in a controlled maintenance window and watch the board reboot on schedule. For a service watchdog, stop the heartbeat in the service (many daemons have a debug flag or you can simply block their event loop in a test build) and confirm systemd restarts the service within the configured interval plus a small margin. Time both. Write down the observed reboot and restart latencies, because those numbers are your real recovery budget, not the configured values. Test on the exact hardware and storage you will deploy, since a timeout that behaves on a fast bench card can false-trigger on the slower card in the field unit.
Failure modes the watchdog creates
Every safety mechanism adds its own ways to fail, and watchdogs are no exception. The classic one is the too-aggressive timeout that reboots a healthy node during a long but legitimate stall, such as a filesystem check after an unclean shutdown or a huge package upgrade. Give boot and shutdown generous margins, and remember that shutdown watchdogs exist so a hung unmount does not trap the board in limbo. Another is the service that pets the watchdog from a thread that stays alive while the actual work thread is deadlocked, which produces a perfectly petted timer on a completely stuck workload. Pet from the code path that proves real progress, like the loop that successfully completes a sensor read or a queue drain, not from a timer callback that fires regardless. Finally, in virtualized or containerized environments the hardware device may be emulated or absent, so confirm what a timeout actually does there before depending on it.
A watchdog turns your worst failure mode, the silent hang, into a scheduled reboot and a log entry. Enable the system timer with a conservative interval, verify the device can actually reset the board, add per-service watchdogs on the workloads that matter, and test each one by firing it deliberately. Do that, and the node in the far closet starts taking care of itself.
Related Guides
- Watchdog Timer Service Setup for Remote Pi Nodes: A Raspberry Pi deployed as a remote sensor node, a camera, or an environmental monitor will eventually hang. It might be a kernel panic, a wedged
- MQTT Broker Heartbeat Optimization on Pi: MQTT's keep-alive heartbeat is the quiet mechanism that holds an IoT system together: clients ping the broker on a schedule, and the broker uses
- Optimizing eMMC Storage Write Lifecycles on Pi: Raspberry Pi boards used as IoT devices often boot from eMMC storage, either on compute modules or via eMMC-to-SD adapters, and eMMC flash wears out
- Setting Up Read-Only Root Filesystems on Remote Pi: A Raspberry Pi deployed remotely, a sensor node, a sign controller, a gate monitor, will eventually lose power without warning, and every unclean