Tag
Debugging
All articles tagged with Debugging.
8 articles
- Velero Said Backups Succeeded. The Data Was Never There.09 Aug 2026Velero's daily backup reported 'Completed' for weeks without ever capturing PVC data. The k8s manifests were there, but Postgres, Vaultwarden, and Paperless data was completely missing. Here's how defaultVolumesToFsBackup fixes it and why velero backup describe is the only real verification.
- My Terraform Runner Destroyed Itself Mid-Apply02 Aug 2026Atlantis was running inside k3s, managing the same Proxmox VMs it ran on. When bpg/proxmox issued a qmshutdown for a non-live-update attribute, it killed the Atlantis pod that was executing the apply. Here's the structural hazard and the fix.
- The ZFS ARC Freeze: How a Marginal PSU Killed My Entire Homelab30 Jul 2026My Proxmox host froze under load three times in one week. The root cause wasn't software — it was a marginal PSU triggering TDP throttling, which exposed a ZFS ARC deadlock. Here's the full chain from symptom to fix.
- 310 Restarts in 21 Days: CNPG's Silent PodMonitor Failure and the Leader-Election Trap17 Jul 2026CloudNativePG's auto-generated PodMonitor was missing a single label — Prometheus never scraped it. The same I/O fragility that causes etcd timeouts was triggering leader-election failures, restarting the operator 310 times in 21 days. Here's how I traced both root causes.
- Kubernetes Health Probes: The Host Header Trap That Restarts Healthy Pods15 Jul 2026Adding health probes to 20+ workloads taught me that kubelet sends the Pod IP as the Host header — and apps with host-validation reject it. Here's the full sweep, the gotcha that caught me, and the probe patterns that actually work.
- Renovate OOMKilled Three Times: Why the Fix Wasn't More Memory14 Jul 2026Two GiB wasn't enough, so I bumped to 3 GiB. Still OOMKilled. Bumped to 4 GiB. Still OOMKilled. The real fix wasn't memory at all — it was Terraform hash concurrency. Here's how I isolated the actual spike.
- Discord Voice Choppy? It Was Bufferbloat — Fixed with 51 Lines of Terraform12 Jul 2026Discord voice was robotic for people hearing me. Confirmed clean over mobile data — home network path. Zero QoS on the WAN interface meant a 50 Mbit upload ceiling was easy to saturate. Here's the investigation, the fix, and why PCQ per-flow fairness matters.
- How a 1 GiB Memory Limit Took Down My Entire k3s Cluster18 Jun 2026A single misconfigured resource limit triggered a cascade: OOMKill on the control-plane, load average of 90, 1.2M DNS queries per day, and kubelet reporting the wrong allocatable memory. Here's the full post-mortem.