Kubernetes

Three backup layers, two storage media, one offsite copy — the theory. In practice: Velero's defaultVolumesToFsBackup gap, PBS restore gotchas with IP conflicts, and Google Drive API throttled to 1.6 KiB/s. Here's what actually works.

Garage S3's SQLite database corrupted because NFS file-locking semantics don't work with SQLite's WAL mode. The lesson: embedded databases and NFS are incompatible, and the fix is local-path for anything with a .db file.

A production AKS cluster with private_cluster_enabled, forced-tunneled node egress via a Route Table, and private-endpoint-only ACR/Key Vault — the workload cluster that sits behind a Hub & Spoke and an Azure Firewall.

paperless-gpt's VISION_LLM_MODEL pointed at a text-only model that fabricated German OCR content for real documents. Combined with an Ollama iGPU that crashed 451 times in one day from an unstable Vulkan/radv fallback, the AI pipeline was hallucinating on hallucinating hardware.

HashiCorp Vault OSS doesn't support auto-unseal without a KMS. The workaround: a polling sidecar that reads unseal keys from a Kubernetes Secret and runs vault operator unseal every 5 seconds. The trade-offs, the security boundary, and the seal window.

Running scheduled chaos experiments on a non-production k3s cluster. Weekly pod-kills and latency injection — what breaks, what it teaches, and why chaos testing matters even when nobody's paying for uptime.

Zero inbound firewall rules for external service access. How Cloudflare Tunnel works with split-DNS on AdGuard, the chunked_encoding fix for large Immich uploads, and why routing Atlantis through Traefik matters for Authelia protection.

Velero's daily backup reported 'Completed' for weeks without ever capturing PVC data. The k8s manifests were there, but Postgres, Vaultwarden, and Paperless data was completely missing. Here's how defaultVolumesToFsBackup fixes it and why velero backup describe is the only real verification.

How a single Authelia instance protects Proxmox, PBS, Grafana, ArgoCD, Headscale, and 20+ web apps with OIDC — CNPG-managed Postgres, Redis sessions, hmac_secret in Vault, and a Traefik ForwardAuth middleware that gates every request.

The nfs-provisioner's root-mount volume was backing up the entire shared NFS export as one 33.4GB blob every night — redundant because every app's PVC was already backed up separately. The investigation almost went wrong when the first hypothesis pointed at the wrong disk.

Vault is the trust root every ExternalSecret reads from. Its namespace had zero NetworkPolicies — any pod in the cluster could reach it. The investigation also uncovered a class of ArgoCD drift that silently ignores merged changes.

CloudNativePG's auto-generated PodMonitor was missing a single label — Prometheus never scraped it. The same I/O fragility that causes etcd timeouts was triggering leader-election failures, restarting the operator 310 times in 21 days. Here's how I traced both root causes.

Why I replaced a heavyweight monitoring stack for host-level metrics with a single container, how Kyverno caught my first deploy before it hit production, and why internal-only dashboards don't need a public DNS record.

Adding health probes to 20+ workloads taught me that kubelet sends the Pod IP as the Host header — and apps with host-validation reject it. Here's the full sweep, the gotcha that caught me, and the probe patterns that actually work.

Moving Atlantis from Kubernetes to a dedicated LXC involved repointing the Cloudflare Tunnel. The new tunnel pointed straight at the LXC's IP, bypassing Traefik and Authelia entirely. Atlantis has no auth of its own. For roughly 18 hours, anyone with the URL had full plan/apply access to the infrastructure repo.

Two GiB wasn't enough, so I bumped to 3 GiB. Still OOMKilled. Bumped to 4 GiB. Still OOMKilled. The real fix wasn't memory at all — it was Terraform hash concurrency. Here's how I isolated the actual spike.

The database namespace holding Authelia's session storage had no network restrictions. Any pod in the cluster could reach it. Here's the audit that found it, the traffic pattern that shaped the fix, and the namespace I deliberately left unrestricted.

Deploy a production-grade monitoring stack on bare-metal k3s: Prometheus, Loki with Garage S3 storage, Promtail on edge nodes via Ansible, SNMP monitoring for MikroTik, and Grafana SSO via Authelia OIDC - all GitOps-managed.

Redis running without a PVC still has persistence enabled by default. When it can't write RDB snapshots to a read-only rootfs, it doesn't crash - it silently refuses all writes. Here's how that turned into a full Nextcloud outage and why the logs pointed at the wrong thing first.

capabilities.drop: [ALL] and runAsNonRoot: true passed schema validation cleanly. Within minutes of merge, nine containers - including both Postgres instances backing Paperless and Nextcloud - were down. Here's the failure analysis, why a manual kubectl fix got silently undone, and the lesson for any blanket securityContext change.

Migrating Jellyfin off k3s onto a GPU-passthrough LXC meant pointing a Service at an external IP. The EndpointSlice looked completely correct via kubectl - Service existed, endpoint listed right - but Traefik 404'd every request. A second, unrelated gotcha surfaced in the same migration: a PVC silently shared by reference across two unrelated files.

kubectl apply succeeds, the field reverts within seconds, and there's no error anywhere. Two ArgoCD debugging patterns that hit the same homelab three times in one day: repo-server cache staleness reverting live edits, and two Applications silently fighting over the same resource.

A full-history gitleaks scan of a homelab repo that had been running for months turned up 12 distinct plaintext secrets - including an OIDC signing key. Here's the scanning setup, the baseline strategy that doesn't block on pre-existing leaks, and the remediation plan.

Replace MinIO with Garage - a single 50MB binary - as the Velero backup target. Full daily cluster backups with Longhorn volume snapshots, deployed via ArgoCD.

Kubernetes Secrets are base64-encoded, not encrypted. Moving secrets out of the cluster into Vault - and syncing them back via External Secrets Operator - gives you rotation, audit logging, and compliance without changing how applications consume secrets.

A single misconfigured resource limit triggered a cascade: OOMKill on the control-plane, load average of 90, 1.2M DNS queries per day, and kubelet reporting the wrong allocatable memory. Here's the full post-mortem.

Most Kubernetes clusters accept any container image, any privilege level, and any resource configuration by default. Kyverno lets you enforce policies at admission time - before anything runs. Here's how to build a supply chain security baseline with Audit-first rollout.

Most teams alert when availability drops below a threshold. Burn-rate alerting tells you how fast you're spending your error budget - so you page on trajectory, not just current state. Here's how to implement the Google SRE Workbook approach on a bare-metal k3s cluster.

Deploy Headscale on a bare-metal k3s cluster with Longhorn persistence, Traefik ingress, and Authelia OIDC authentication - fully GitOps-managed via ArgoCD.

HTTP-01 challenges fail behind a default-deny firewall. Switching cert-manager to Cloudflare DNS01 wildcard validation gives every Traefik IngressRoute a *.yourdomain.com certificate without opening port 80 to the internet.

How to manage an entire Kubernetes homelab - MetalLB, Traefik, Longhorn, Authelia, and more - as a Git repository using ArgoCD's App-of-Apps pattern.

How to get a real external IP on a bare-metal Kubernetes cluster using MetalLB L2 mode, and wire it up with Traefik for automatic HTTPS - fully GitOps-managed with ArgoCD.

How to build a production-grade Kubernetes homelab with K3s, Authelia SSO, Longhorn storage, and ArgoCD - and the five painful mistakes that will cost you hours if you don't know about them.