A homelab that needs daily attention is a second job. My goal is a cluster that runs for weeks without intervention, alerts when something breaks, self-heals what it can, and waits for a human only when a human is actually needed.
Update (November 2026): Atlantis’s webhook server no longer drives this flow. Since ADR-021, GitHub Actions workflows on a self-hosted runner pick up the same
atlantis planandatlantis applyPR comments, and the apply job waits for manual approval on a protectedterraform-applyGitHub Environment. The review-then-apply gate described below is unchanged; only the executor moved.
This is the operating model that makes that work. It’s documented from what actually runs today, including the deliberate gaps and the non-goals that keep the system honest.
View the complete homelab infrastructure source on GitHub
What Auto-Updates
Renovate runs every 2 hours via CronJob, tracking container images, Helm charts, and Terraform providers. But not everything gets the same treatment:
Auto-Merge (Patch/Minor/Digest, 3-Day Soak)
Stateless workloads where a bad version costs a git revert, nothing more:
- Dashboard widgets (Homepage, Uptime Kuma display)
- Development tools (pre-commit hooks, linters)
- Non-critical utilities (SearXNG, Mealie, MySpeed)
Renovate holds each update until the release is at least 3 days old (minimumReleaseAge), then opens the PR, CI runs, and the PR auto-merges once it’s green. If the new version breaks, rollback is reverting that commit. ArgoCD’s selfHeal doesn’t roll back a bad release on its own: it keeps the cluster matching git, and after the merge git says “new version”.
PR-Only, Always (Manual Review Required)
Stateful and critical services where a bad upgrade can cause data loss or auth outage:
- Databases: CNPG Postgres, Redis/Valkey
- Auth: Authelia, Vault
- Storage: Garage S3, Velero
- Apps with data migrations: Nextcloud, Paperless, Gitea
Every update (patch, minor, or digest) gets a PR that requires manual review. The failure mode for these is “database schema mismatch” or “OIDC keys rotated”, not a pod restart.
Immich used to be on this list and had to come off it. The phone app updates itself from the app store, and a server that lags behind the app stops accepting mobile uploads. So Immich server and machine learning now auto-merge minor, patch and digest updates, and only major versions wait for review. A rule that looks safe on paper can be the riskier choice when something outside your control moves faster than your review queue.
PR-Only, Always (Major Version Bumps)
Every major version bump, any package, regardless of tier. Major versions break APIs, change configuration formats, and introduce migration steps that no automated tool can reliably handle.
Terraform: Never Auto-Applied
Every Terraform change goes through Atlantis: PR → plan → atlantis apply comment → apply. No exceptions. Terraform changes physical and VM state on a single-point-of-failure host, so a human is always in the loop.
What Alerts (and Where)
Prometheus + Alertmanager route to Discord via webhook. ArgoCD’s notifications-controller sends app-state events through a separate path.
Critical Alerts (Always Discord)
Everything with severity: critical:
- Proxmox host temperature
- k3s control-plane down
- Storage >93% full
- Postgres replication broken
- Certificate expiry soon
- Velero backup failed
Warning Alerts (Selective)
Only explicitly named warnings that matter at steady state:
KubePodCrashLooping: a pod is crash-loopingKubeJobFailed: a CronJob failed (covers Renovate itself)KubeNodeNotReady: a node is unreachableVeleroBackupPartialFailure: backup completed with warnings
Deliberately NOT Alerted
Routine warnings that would make Discord noisy without being actionable:
KubeMemoryQuotaOvercommit: LXC soft ceilings, not real pressureCPUThrottlingHigh: normal behavior for bursty workloadsKubePodNotReadyduring rolling updates: temporary, self-resolving
The principle: an alert nobody acts on trains people to ignore Discord. Every alert must have a corresponding action in the runbook. If there’s no action, there’s no alert.
What Self-Heals
ArgoCD: Automatic Sync + Self-Heal
All 41 live Applications have syncPolicy.automated.selfHeal: true. A merged manifest change deploys itself, and manual kubectl drift on anything ArgoCD tracks gets reverted within minutes.
This is the primary self-healing mechanism. If I patch a Deployment by hand while debugging, ArgoCD reverts it on the next sync cycle. The cluster always converges to the git state.
Docker Containers: restart: unless-stopped
The media stack, Minecraft, AdGuard, and Unbound all run with restart: unless-stopped in Docker Compose, so the process comes back after a crash or host reboot. Data recovery is NFS/ZFS’s job, not Docker’s.
Kyverno: Audit-Only (Deliberately)
Kyverno runs in Audit mode: it reports policy violations in PolicyReports but does not block or auto-remediate. That’s deliberate:
- A bad policy in Enforce mode silently blocks legitimate deploys
- On a single-operator homelab, there’s nobody to notice the block quickly
- Audit mode provides the visibility without the blast radius
The policies exist (require-resource-limits, disallow-latest-tag, disallow-privileged-containers) but they’re watching, not enforcing. When I’m confident they won’t cause false positives, they’ll flip to Enforce.
What Stays Manual (On Purpose)
Terraform Apply and Stateful Service Bumps
Both covered above. Terraform applies only when a human comments atlantis apply, regardless of what changed; it’s the one rule with no exceptions. Authelia, Vault, CNPG, and Garage bumps get reviewed before merge.
Snapshots Before State-Affecting Changes
Proxmox VM/CT snapshot or Velero PV snapshot, taken manually before applying anything that touches running state. The judgment of “is this change state-affecting” doesn’t belong to a script.
Velero Restore Testing
Running, and tested successfully at least once per namespace, but not yet exercised for every service. A backup that’s only partly restore-tested is closer to hope than guarantee.
Full Cluster Rebuild
Documented in DISASTER-RECOVERY.md, not automated. The target is fast, well-documented recovery instead of zero-touch failover, because true HA isn’t achievable on one physical host.
The Non-Goals
These are explicitly not targets:
- Zero-downtime HA. Not achievable with one physical host. Not attempted.
- Fully unattended major-version upgrades. Stateful service upgrades have repeatedly needed human judgment mid-migration (Postgres mount-point gotcha, capability-drop regression). Automating past that trades a 10-minute manual step for a much longer unattended-failure cleanup.
- Alerting on everything. Deliberately tuned to steady-state-relevant signals only.
The Contract
This document is the contract between the infrastructure and the operator. Everything not listed here is assumed to be working. If it’s not working and it’s not in this document, it’s a gap, not an expected manual task.
As the cluster evolves (Cilium CNI, R2 offsite backup, Kyverno enforcement), the categories shift. The principle stays: automate what’s safe, alert what’s important, and leave everything else to a human with context.
This maps directly to enterprise SRE practice: SLO-based alerting replaces threshold alerting, runbooks define the human response for each alert type, and change management gates (like Atlantis’s apply requirement) prevent unreviewed infrastructure changes. The difference is that enterprise environments have teams; a homelab has one person who needs to sleep through the night without Discord notifications.
The Phoenix Project* is the book that most shaped this document. It’s fiction, but the underlying argument (know exactly what’s automated, what’s gated, and why) is the same one this operating model tries to make explicit instead of leaving implicit.