Hardening Unattended Raspberry Pi Edge Nodes: Watchdog, fail2ban, nftables, and the Mistakes That Take Down DNS

Two Raspberry Pi 4Bs (starter set I used*) run AdGuard Home and Unbound for my entire home network, as an active/passive pair behind Keepalived. If you’re setting up edge nodes today, the Raspberry Pi 5* is the better buy. These are physical boards on a shelf, not VMs or LXCs: no Proxmox snapshot, no PBS backup, no terraform destroy && apply to recover from a bad state. If one hangs hard at 2am, nobody notices until someone’s phone can’t resolve a hostname.

This is the hardening pass that closed every gap I found in that setup: a hardware watchdog for total-freeze recovery, fail2ban for the one SSH-exposed surface, an nftables host firewall that stays out of the way of Docker’s own iptables rules, log size caps against slow SD-card death, and a DNS health check that works even on the day the rest of the monitoring stack is offline. That turned out to be exactly the day it mattered.

View the complete homelab infrastructure source on GitHub 🐙

Why “It’s Just DNS” Needs More Hardening, Not Less

The instinct with a small, single-purpose device is to leave it alone: fewer moving parts, fewer ways to break it. That’s backwards for a device with no operator watching it and no automated recovery path. A k3s pod that crashes gets rescheduled in seconds. A Raspberry Pi that hard-hangs stays hung until a human walks over and pulls the power.

Everything below closes that gap: detect failure independently, recover from total freezes without intervention, and don’t introduce a new failure mode along the way.

Hardware Watchdog: Recovering From a Hang Software Can’t See

Docker restarts a crashed container. A kernel deadlock is different: the whole system stops responding, nothing crashes, nothing logs anything, and nothing is left running to notice the problem or act on it.

The Broadcom SoC in a Raspberry Pi has a hardware watchdog timer, a circuit that resets the board if it isn’t periodically “petted.” As long as something pets it, the system is presumed alive. If petting stops because the kernel is deadlocked and nothing can run, the watchdog fires and resets the board.

# /boot/firmware/config.txt
dtparam=watchdog=on

# /etc/systemd/system.conf
RuntimeWatchdogSec=15s
RebootWatchdogSec=10min

RuntimeWatchdogSec=15s arms the hardware watchdog with a 15-second timeout, and systemd pets it at half that interval while the system is healthy. If systemd itself stops running (the deadlock case this exists for), the pets stop and the watchdog circuit force-resets the board. RebootWatchdogSec=10min is a second, independent safety net: if a reboot hangs somewhere in shutdown, the watchdog fires after 10 minutes instead of leaving the board stuck mid-reboot indefinitely.

The config.txt change only applies at boot, so this needs a reboot. I gated that reboot behind an explicit flag (rpi_optimize_reboot, default false) rather than auto-rebooting a DNS server as a side effect of an Ansible run.

fail2ban: The One Exposed Surface

These Pis are reachable from the entire server VLAN, and the Keepalived VIP gives them a single, stable address that’s an obvious target for anything scanning the network. The only network-facing attack surface that matters here is SSH.

# /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true
port = ssh
filter = sshd
maxretry = 5
findtime = 10m
bantime = 1h

Five failed attempts within ten minutes ban the source IP for an hour. fail2ban only watches the sshd auth log and has no interaction with the DNS path (AdGuard, Unbound, Docker). That isolation matters. A jail watching the wrong log file or using the wrong filter is a self-inflicted outage on a box where outages are expensive. Scoping it to one well-understood log source limits the blast radius of a fail2ban mistake to SSH access, never DNS.

The nftables Trap: Don’t Touch /etc/nftables.conf

This part could have caused the exact outage the rest of this hardening pass exists to prevent.

The obvious way to add a host firewall on Debian is to edit /etc/nftables.conf and enable nftables.service. The problem: that file conventionally starts with flush ruleset, and Docker manages its own NAT and FORWARD chains through iptables-nft (the nftables-backed iptables compatibility layer). Every time that ruleset gets loaded, for example a systemctl restart nftables from an Ansible handler after editing the file, it wipes Docker’s NAT rules along with everything else and silently breaks every published container port. AdGuard runs with network_mode: host so it can bind port 53 directly, but the same box runs other containers in bridge mode with published ports. That combination is the actual topology here, not a hypothetical.

The fix: leave /etc/nftables.conf and the stock service alone. Use a separate ruleset file and a separate systemd service:

# /etc/nftables-hostfw.conf
table inet hostfw {
  chain input {
    type filter hook input priority filter; policy drop;
    iif "lo" accept
    ct state established,related accept
    ip protocol icmp accept
    meta l4proto ipv6-icmp accept
    tcp dport 22 accept
    tcp dport 53 accept
    udp dport 53 accept
    tcp dport 3001 accept
    tcp dport { 80, 443 } accept
    udp dport 41641 accept
    ip protocol vrrp accept
  }
}

# /etc/systemd/system/hostfw.service
[Unit]
Description=Host firewall (inet hostfw table, additive - does not touch Docker's tables)
After=network.target docker.service
Wants=docker.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/sbin/nft -f /etc/nftables-hostfw.conf
ExecStop=/usr/sbin/nft delete table inet hostfw

[Install]
WantedBy=multi-user.target

The rules live in a named table (inet hostfw) with policy drop only on that table’s input chain. Loading it adds a table; it never flushes or replaces anyone else’s. After=docker.service and Wants=docker.service keep the boot order predictable: Docker sets up its tables first, then this one loads. Keep in mind how nftables evaluates multiple tables on the same hook: an accept in Docker’s table does not override a drop in this one. Every port the host stack itself serves has to be listed here explicitly.

What this firewall covers: SSH (22), DNS (53; AdGuard runs network_mode: host, so this is host-stack traffic, not Docker-NAT’d), AdGuard’s web UI (3001), the HAProxy VIP (80/443), Tailscale (41641/udp), and Keepalived VRRP.

What it deliberately doesn’t cover: bridge-mode containers like Unbound (5335) and node_exporter (9100). Docker DNATs that traffic in prerouting and it then traverses the FORWARD path, so it never reaches the host’s INPUT hook and this table never sees it. I confirmed that with live testing, not just by reading about how Docker’s iptables integration works. Restricting bridge-mode container ports would need rules in Docker’s own DOCKER-USER chain, with careful IPv4/IPv6 handling to avoid breaking container egress. I deferred this: the MikroTik* already segments these Pis from the wider internet at the network layer, and the risk of getting DOCKER-USER rules wrong on a live DNS server outweighed the marginal benefit of restricting traffic that’s already internal-only.

Validation of the deployment path, not just the live change. I live-tested on the replica Pi first, with a systemd-run rollback timer staged before every individual change (the same dead-man’s-switch pattern as the MikroTik cleanup). Then I re-tested via the real Ansible run, which is a separate code path from the manual test: a playbook can have a templating bug that a manual nft -f would never hit. Then I rebooted, to confirm the systemd service reapplies the ruleset on boot instead of only working because the manual test had left it loaded. Only after the replica was fully green did the same sequence run against the primary DNS node.

Stopping Slow SD-Card Death

Docker’s default json-file log driver has no size limit. On a box with a real disk, that’s eventually a problem. On a Pi whose only storage is an SD card, it’s a slow-motion outage that looks like nothing is wrong until the card is full and everything stops. The fix goes in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Existing container logs were already at 17MB and 2.7MB when I checked. Not catastrophic yet, but headed for “disk full” months out with zero warning. This setting only applies to containers created or recreated after the daemon restart; it doesn’t truncate what’s already there. The existing oversized logs needed a one-time manual cleanup, and the daemon-wide default stops the problem from coming back.

Memory Limits: Catching a Leak Before It Takes the Whole Pi Down

# docker-compose, per service
adguardhome:
  mem_limit: 512m
unbound:
  mem_limit: 256m
promtail:
  mem_limit: 256m
node_exporter:
  mem_limit: 128m
autoheal:
  mem_limit: 64m

These numbers are generous, picked from observed usage with real headroom. The goal isn’t to constrain normal operation. It’s to make sure a memory leak or runaway process in one container hits that container’s cgroup limit and gets OOM-killed before it starves every other process on the Pi, including the resolver everything depends on. I rolled the limits out incrementally on the replica first, checked with docker inspect that they were enforced, and confirmed all containers came back Up after the restart with DNS unaffected throughout. On a box this important, “looks fine” isn’t enough confirmation.

Local Config Backup: The Gap Nobody Noticed

These Pis are physical hardware. Proxmox Backup Server and Velero only cover VMs, LXCs, and the cluster, so neither had ever backed them up. The gap had existed since the Pis were first deployed and never surfaced, because nothing had ever needed a restore.

#!/bin/bash
# /usr/local/bin/backup-rpi-configs.sh
set -euo pipefail
DEST=/opt/backups
STAMP=$(date +%Y%m%d-%H%M%S)
tar czf "${DEST}/configs-${STAMP}.tar.gz" \
  -C / opt/adguardhome/conf opt/unbound 2>/dev/null || true
ls -t "${DEST}"/configs-*.tar.gz 2>/dev/null | tail -n +15 | xargs -r rm --

It runs daily from a systemd timer with a randomized delay (so both Pis don’t hit disk I/O at the same instant) and keeps the 14 most recent snapshots. It is deliberately local-only, with no NFS or git dependency. The NFS server runs as an LXC on the Proxmox host, and a backup that depends on the thing you’re backing up away from defeats the purpose. AdGuard’s config also contains a bcrypt password hash. Pushing that into git history, even on a private remote, is unnecessary exposure for a snapshot whose only job is “let me recover the last known-good config after an accidental change.”

Alerting That Survives the Main Alerting Stack Being Down

This is the piece that mattered in practice. The homelab’s primary alerting path (Prometheus → Alertmanager → Discord) runs on the k3s cluster, which runs on the Proxmox host. On the day I built this, the Proxmox host was down for hardware repair. The entire alerting pipeline was down with it, on exactly the day DNS health mattered most, because DNS was now the only thing still running and nothing was supervising it.

#!/bin/bash
# Independent DNS health check - ZERO dependency on k3s/Prometheus/Alertmanager
WEBHOOK_URL="..."
STATE_FILE="/var/lib/dns-healthcheck.state"
HOSTNAME=$(hostname)

check_dns() {
  dig +short +timeout=3 google.com @127.0.0.1 -p 53 > /dev/null 2>&1 && \
  dig +short +timeout=3 google.com @127.0.0.1 -p 5335 > /dev/null 2>&1
}

PREV_STATE="unknown"
[ -f "$STATE_FILE" ] && PREV_STATE=$(cat "$STATE_FILE")

if check_dns; then CURRENT_STATE="healthy"; else CURRENT_STATE="unhealthy"; fi

if [ "$CURRENT_STATE" != "$PREV_STATE" ]; then
  if [ "$CURRENT_STATE" = "unhealthy" ]; then
    MESSAGE="🔴 **${HOSTNAME}**: DNS resolution failing. This alert is independent of the main monitoring stack."
  else
    MESSAGE="🟢 **${HOSTNAME}**: DNS resolution recovered."
  fi
  curl -s -X POST -H "Content-Type: application/json" \
    -d "{\"content\": \"${MESSAGE}\"}" "${WEBHOOK_URL}" > /dev/null 2>&1 || true
fi

echo "$CURRENT_STATE" > "$STATE_FILE"

A systemd timer runs it every two minutes. Two design choices matter more than the script’s mechanics.

It tests both layers independently: AdGuard on port 53 and Unbound directly on port 5335. AdGuard forwards to Unbound, so testing only the front door can’t tell “AdGuard is up but its upstream resolver died” from “everything is fine.” The && between the two dig calls means both must succeed for the state to be healthy.

It only posts on a state change, not on every run. A health check that posts every two minutes regardless of state either spams a channel until someone mutes it or trains everyone to ignore it. Tracking the previous state in a file means the alert fires exactly twice per incident: once when it breaks, once when it recovers.

The webhook URL reuses the Discord webhook Alertmanager already posts to. While wiring this up I found that URL committed in plaintext in the cluster’s own monitoring config. That needs its own fix, but it was out of scope for this change, so I noted it instead of quietly folding a second, unrelated remediation into the same commit.

What Got Tested, Not Just Written

Every change here got the same validation discipline, because this box matters too much to skip it: replica first, primary only after the replica was fully green; a manual live test and a separate Ansible-driven test, since they’re different code paths; and for anything that should survive a reboot, an actual reboot instead of trusting that a unit file is correct.

The pattern applies well beyond Raspberry Pis. Any unattended edge device (a branch-office router, an IoT gateway, a remote sensor node) has the same problem: no operator watching it, no platform-level recovery, and a failure mode (hard hang) that application-level monitoring can’t see because the monitoring agent is hung too. A hardware watchdog plus an alerting path with zero dependency on the thing being monitored is the minimum bar for “I’ll find out if this breaks,” whatever the device does.