Homelab

Atlantis was running inside k3s, managing the same Proxmox VMs it ran on. When bpg/proxmox issued a qmshutdown for a non-live-update attribute, it killed the Atlantis pod that was executing the apply. Here's the structural hazard and the fix.

My Proxmox host froze under load three times in one week. The root cause wasn't software — it was a marginal PSU triggering TDP throttling, which exposed a ZFS 2.4.1 deadlock bug. Here's the full chain from symptom to fix.

A hands-on comparison of the 5 best mini PCs for homelab use in 2026 — covering Intel N100, Ryzen 5, and Ryzen 7 platforms for Proxmox, K3s, and always-on workloads.

A hands-on comparison of the 5 best NAS devices for home backup in 2026 — covering Synology, QNAP, Terramaster, and UGREEN for personal and homelab use.

A daily offsite backup cron job had a 99% failure rate for over a month. It looked like a credential or network problem. It was neither - Google Drive's API throttles hard on exactly the storage format Proxmox Backup Server uses, and a 24-hour cron window was never going to be enough.

CloudNativePG's auto-generated PodMonitor was missing a single label — Prometheus never scraped it. The same I/O fragility that causes etcd timeouts was triggering leader-election failures, restarting the operator 310 times in 21 days. Here's how I traced both root causes.

Why I replaced a heavyweight monitoring stack for host-level metrics with a single container, how Kyverno caught my first deploy before it hit production, and why internal-only dashboards don't need a public DNS record.

A full tour of the physical rack, compute, and networking gear powering my K3s cluster, Proxmox hosts, and everything I write about here — what I run, why I picked it, and what I'd change.

Adding health probes to 20+ workloads taught me that kubelet sends the Pod IP as the Host header — and apps with host-validation reject it. Here's the full sweep, the gotcha that caught me, and the probe patterns that actually work.

Moving Atlantis from Kubernetes to a dedicated LXC involved repointing the Cloudflare Tunnel. The new tunnel pointed straight at the LXC's IP, bypassing Traefik and Authelia entirely. Atlantis has no auth of its own. For roughly 18 hours, anyone with the URL had full plan/apply access to the infrastructure repo.

Two GiB wasn't enough, so I bumped to 3 GiB. Still OOMKilled. Bumped to 4 GiB. Still OOMKilled. The real fix wasn't memory at all — it was Terraform hash concurrency. Here's how I isolated the actual spike.

The database namespace holding Authelia's session storage had no network restrictions. Any pod in the cluster could reach it. Here's the audit that found it, the traffic pattern that shaped the fix, and the namespace I deliberately left unrestricted.

Discord voice was robotic for people hearing me. Confirmed clean over mobile data — home network path. Zero QoS on the WAN interface meant a 50 Mbit upload ceiling was easy to saturate. Here's the investigation, the fix, and why PCQ per-flow fairness matters.

DISASTER-RECOVERY.md documented the Proxmox Backup Server restore procedure in detail. It had never been executed end-to-end. Running it for the first time - against a scratch VM, alongside the still-running original - found a network conflict the documentation never mentioned.

A gitleaks baseline file is supposed to suppress known-false-positive findings. It turned out to be suppressing the live rpc_secret and admin_token for a production Garage S3 cluster - unrotated since the commit that introduced them to a public repo months earlier.

Deploy a production-grade monitoring stack on bare-metal k3s: Prometheus, Loki with Garage S3 storage, Promtail on edge nodes via Ansible, SNMP monitoring for MikroTik, and Grafana SSO via Authelia OIDC - all GitOps-managed.

Redis running without a PVC still has persistence enabled by default. When it can't write RDB snapshots to a read-only rootfs, it doesn't crash - it silently refuses all writes. Here's how that turned into a full Nextcloud outage and why the logs pointed at the wrong thing first.

A two-node recursive DNS stack with ad filtering, automatic config sync, and transparent failover - fully managed with Ansible.

capabilities.drop: [ALL] and runAsNonRoot: true passed schema validation cleanly. Within minutes of merge, nine containers - including both Postgres instances backing Paperless and Nextcloud - were down. Here's the failure analysis, why a manual kubectl fix got silently undone, and the lesson for any blanket securityContext change.

Migrating Jellyfin off k3s onto a GPU-passthrough LXC meant pointing a Service at an external IP. The EndpointSlice looked completely correct via kubectl - Service existed, endpoint listed right - but Traefik 404'd every request. A second, unrelated gotcha surfaced in the same migration: a PVC silently shared by reference across two unrelated files.

kubectl apply succeeds, the field reverts within seconds, and there's no error anywhere. Two ArgoCD debugging patterns that hit the same homelab three times in one day: repo-server cache staleness reverting live edits, and two Applications silently fighting over the same resource.

A full-history gitleaks scan of a homelab repo that had been running for months turned up 12 distinct plaintext secrets - including an OIDC signing key. Here's the scanning setup, the baseline strategy that doesn't block on pre-existing leaks, and the remediation plan.

Two Raspberry Pis run DNS for an entire network with no one watching them most of the time. A hardware watchdog, fail2ban, an additive nftables host firewall that doesn't conflict with Docker, log size caps, and an alerting path that works even when the rest of the monitoring stack is down.

Replace MinIO with Garage - a single 50MB binary - as the Velero backup target. Full daily cluster backups with Longhorn volume snapshots, deployed via ArgoCD.

Kubernetes Secrets are base64-encoded, not encrypted. Moving secrets out of the cluster into Vault - and syncing them back via External Secrets Operator - gives you rotation, audit logging, and compliance without changing how applications consume secrets.

A single misconfigured resource limit triggered a cascade: OOMKill on the control-plane, load average of 90, 1.2M DNS queries per day, and kubelet reporting the wrong allocatable memory. Here's the full post-mortem.

Most Kubernetes clusters accept any container image, any privilege level, and any resource configuration by default. Kyverno lets you enforce policies at admission time - before anything runs. Here's how to build a supply chain security baseline with Audit-first rollout.

Setting up IPv6 on MikroTik behind a FritzBox with CGN should be straightforward - ULA prefix, NAT66 masquerade, done. Instead, RouterOS 7 started advertising router advertisements on the WAN interface, turning MikroTik into an uninvited IPv6 gateway for FritzBox WiFi clients. Here's the full setup and fix.

Most teams alert when availability drops below a threshold. Burn-rate alerting tells you how fast you're spending your error budget - so you page on trajectory, not just current state. Here's how to implement the Google SRE Workbook approach on a bare-metal k3s cluster.

Deploy Headscale on a bare-metal k3s cluster with Longhorn persistence, Traefik ingress, and Authelia OIDC authentication - fully GitOps-managed via ArgoCD.

How to automate wildcard Let's Encrypt certificates on a bare-metal K3s cluster using cert-manager's DNS-01 challenge with Cloudflare - and why HTTP-01 won't work for internal services.

How to manage an entire Kubernetes homelab - MetalLB, Traefik, Longhorn, Authelia, and more - as a Git repository using ArgoCD's App-of-Apps pattern.

How to get a real external IP on a bare-metal Kubernetes cluster using MetalLB L2 mode, and wire it up with Traefik for automatic HTTPS - fully GitOps-managed with ArgoCD.

How to build a production-grade Kubernetes homelab with K3s, Authelia SSO, Longhorn storage, and ArgoCD - and the five painful mistakes that will cost you hours if you don't know about them.

Take a tour of a fully automated, segmented, and highly available homelab architecture orchestrated entirely via Ansible and GitOps.

A complete guide to automating a local AI stack on Proxmox LXC using Terraform and Ansible, including Open-WebUI and AMD Radeon Vega iGPU workarounds.