A Raspberry Pi deployed as a remote sensor node, a camera, or an environmental monitor will eventually hang. It might be a kernel panic, a wedged driver, or a runaway process eating all memory; the cause hardly matters because you cannot reach the power button. The hardware watchdog timer exists for exactly this situation: a countdown circuit that reboots the Pi unless software resets it regularly. If the software hangs, the resets stop, the timer expires, and the Pi reboots itself. Setting up the watchdog as a proper service is the difference between a remote node that recovers alone and one that waits for a site visit.
How the Hardware Watchdog Works
The Pi's SoC includes a watchdog peripheral: a hardware counter that counts down from a programmed timeout, typically tens of seconds. Software must periodically "pet" or "feed" the watchdog, resetting the counter, to prevent it from reaching zero. When it reaches zero, the hardware asserts a reset and the Pi reboots, no software involvement required. This is the key property: because the reset is generated in hardware, it works even when the kernel is completely wedged, as long as the hang does not also stop the watchdog clock itself. The watchdog cannot fix the underlying bug, but it converts every hang from a permanent outage into a reboot, and for remote nodes that transformation is everything. Choose the timeout thoughtfully: too short and normal load spikes cause spurious reboots; too long and the node sits dead for minutes before recovering. Tens of seconds is the usual sweet spot.
Enabling the Watchdog Device
The watchdog hardware is exposed to Linux as a device node, and enabling it is a configuration step. On Raspberry Pi OS, this typically involves loading the watchdog kernel module and ensuring the device node exists. The modern approach uses systemd's built-in watchdog support: the system manager itself can pet the hardware watchdog, which means even a userspace hang that takes down your application still gets recovered, because systemd keeps running and keeps petting, or conversely, if systemd itself wedges, the petting stops and the hardware resets. Configure the watchdog timeout in the system configuration and enable it so it starts at boot. Verify the device node is present and that the timeout is actually programmed: a watchdog configured but not running is the classic false sense of security.
The Service That Pets the Watchdog
Beyond systemd's own petting, the robust pattern is a dedicated watchdog service that also monitors application health. A simple daemon, or a systemd service with watchdog notification support, pets the hardware timer on a schedule comfortably inside the timeout, say every few seconds for a tens-of-seconds timeout. But mere petting only proves the pinger is alive. The stronger design has the service check the application's vitals before petting: is the sensor process running, is it producing fresh data, is the network interface up, is disk space sane. If any check fails, the service first attempts a targeted recovery, restarting the failed component, and only if recovery fails does it stop petting and let the hardware reboot. This layered approach avoids rebooting for problems a service restart would fix, while guaranteeing that a total wedge still recovers.
Application-Level Heartbeats
For the health checks to mean anything, the application must report its health honestly. The pattern is a heartbeat: the application writes a timestamp or increments a counter in a known location, a file, shared memory, or a systemd notification, every cycle of its main loop. The watchdog service reads the heartbeat and verifies it is fresh. A process that is running but stuck, spinning in a deadlocked thread or blocked on a hung I/O call, stops updating its heartbeat, and the watchdog catches what a simple process-alive check would miss. Design the heartbeat into the application's main loop from the start; retrofitting it later is possible but the discipline of writing it alongside the loop logic produces more honest health signals. Also heartbeat the critical peripherals: a sensor read that suddenly returns the same value forever is a different failure than a dead process, and the watchdog service should know the difference.
Testing the Recovery Path
An untested watchdog is a rumor. Test deliberately: simulate each failure mode and verify recovery. Kill the application process and confirm the service restarts it. Wedge the application, by blocking its heartbeat without killing it, and confirm the escalation to reboot. Load the system heavily and confirm no spurious reboots under legitimate load. Test across actual reboots: does the watchdog configuration persist, does the service start at boot, does the first pet happen before the first timeout. Log every watchdog-triggered reboot distinctly, so you can distinguish "the watchdog saved us" from unexplained restarts when reviewing logs later. A remote node whose watchdog has never fired in testing is a node whose recovery path is unverified; the first real hang is a bad time to discover a misconfiguration.
The watchdog timer is the remote node's immune system: hardware that reboots what software cannot fix, driven by a service that checks real health before petting. Enable the hardware, build the layered service, heartbeat honestly, and test every failure mode. Then the node can hang, and you will read about it in the logs instead of driving to it.
Related Guides
- Configuring Systemd Watchdog Timers on SBC Nodes: A single-board computer running headless in a closet, a greenhouse, or a remote sensor post has one failure mode that hurts more than any other: the
- Setting Up Read-Only Root Filesystems on Remote Pi: A Raspberry Pi deployed remotely, a sensor node, a sign controller, a gate monitor, will eventually lose power without warning, and every unclean
- MQTT Broker Heartbeat Optimization on Pi: MQTT's keep-alive heartbeat is the quiet mechanism that holds an IoT system together: clients ping the broker on a schedule, and the broker uses
- Optimizing eMMC Storage Write Lifecycles on Pi: Raspberry Pi boards used as IoT devices often boot from eMMC storage, either on compute modules or via eMMC-to-SD adapters, and eMMC flash wears out