Debugging

Garage S3's SQLite database corrupted because NFS file-locking semantics don't work with SQLite's WAL mode. The lesson: embedded databases and NFS are incompatible, and the fix is local-path for anything with a .db file.

All VMs and LXCs starting simultaneously spiked the Proxmox host load to 147. The fix was NFS first, k3s nodes 30 seconds apart, and LXCs last — a boot order strategy that costs nothing but prevents boot storms.

VM dedicated memory is a real reservation. LXC dedicated is a soft ceiling. Summing them together overstated pressure by 40GB — but the real risk was underestimating it. Here's the two-tier memory model that prevents a repeat of the ZFS ARC freeze.

paperless-gpt's VISION_LLM_MODEL pointed at a text-only model that fabricated German OCR content for real documents. Combined with an Ollama iGPU that crashed 451 times in one day from an unstable Vulkan/radv fallback, the AI pipeline was hallucinating on hallucinating hardware.

Velero's daily backup reported 'Completed' for weeks without ever capturing PVC data. The k8s manifests were there, but Postgres, Vaultwarden, and Paperless data was completely missing. Here's how defaultVolumesToFsBackup fixes it and why velero backup describe is the only real verification.

Atlantis was running inside k3s, managing the same Proxmox VMs it ran on. When bpg/proxmox issued a qmshutdown for a non-live-update attribute, it killed the Atlantis pod that was executing the apply. Here's the structural hazard and the fix.

My Proxmox host froze under load three times in one week. The root cause wasn't software — it was a marginal PSU triggering TDP throttling, which exposed a ZFS ARC deadlock. Here's the full chain from symptom to fix.

CloudNativePG's auto-generated PodMonitor was missing a single label — Prometheus never scraped it. The same I/O fragility that causes etcd timeouts was triggering leader-election failures, restarting the operator 310 times in 21 days. Here's how I traced both root causes.

Adding health probes to 20+ workloads taught me that kubelet sends the Pod IP as the Host header — and apps with host-validation reject it. Here's the full sweep, the gotcha that caught me, and the probe patterns that actually work.

Two GiB wasn't enough, so I bumped to 3 GiB. Still OOMKilled. Bumped to 4 GiB. Still OOMKilled. The real fix wasn't memory at all — it was Terraform hash concurrency. Here's how I isolated the actual spike.

Discord voice was robotic for people hearing me. Confirmed clean over mobile data — home network path. Zero QoS on the WAN interface meant a 50 Mbit upload ceiling was easy to saturate. Here's the investigation, the fix, and why PCQ per-flow fairness matters.

A single misconfigured resource limit triggered a cascade: OOMKill on the control-plane, load average of 90, 1.2M DNS queries per day, and kubelet reporting the wrong allocatable memory. Here's the full post-mortem.