Blog
Notes from production and the homelab
Incidents, fixes and design decisions from Azure work and a single-host Proxmox homelab: what broke, why, and what held.
91 articles · 6 in German
2026
-
Tailscale on k3s: Exit Nodes, Subnet Routes, and ACLs
A Tailscale pod in k3s acts as subnet router and exit node for a Headscale tailnet: scoped routes, tag-gated auto-approval, and an IPv6 exit-node trap.
Networking VPN Tailscale -
Open WebUI Ignores Your Environment Variables: ENABLE_PERSISTENT_CONFIG
ENABLE_LOGIN_FORM=false had no effect on Open WebUI because a database row overrode it. ENABLE_PERSISTENT_CONFIG=false makes the env vars from Git win again.
Kubernetes GitOps AI -
Homelab Gift Guide 2026: Gear I Actually Use
A gift guide for the person who self-hosts: rack hardware, tools, network gear and books I run in my own lab, with honest notes on what to skip.
Hardware Homelab Review -
How to Plan Homelab Hardware in 2026, Role by Role
Plan a homelab by role: compute host, storage and backup, network, edge DNS and power. Short answers, the order to buy in, and links to each detailed guide.
Hardware Homelab Review -
AdGuard Home + Unbound: DNS Filtering Without Upstream Leaks
AdGuard Home sends every DNS query to one public resolver by default. Unbound as a local recursive resolver keeps your full query history off any single provider.
Networking DNS AdGuard -
Immich App Stops Uploading: The Server Must Match the Mobile App Version
The Immich app stopped backing up photos while every server signal was green. The app had auto-updated to 3.3.3, the server was v3.2.2. How I found it.
Homelab Kubernetes Debugging -
Gitea Actions on k3s: Self-Hosted CI Without GitHub
Gitea Actions runs GitHub Actions-style YAML on your own hardware. Gitea and act_runner on k3s keep CI pipelines and runner resources under your control.
Kubernetes Gitea CI/CD -
Raspberry Pi 5 Alternatives Compared 2026
Radxa Rock 5B, Orange Pi 5 Plus and a budget N100 mini PC compared with the Raspberry Pi 5 for homelab edge workloads, and when the Pi is still right.
Hardware Homelab Review -
Unable to mkstemp /tmp/apt.sig: How /tmp 0755 Broke apt
A /tmp at mode 0755 left apt on my Proxmox host unable to verify signatures for two months. A week later, unattended-upgrades skipped 69 Pi updates.
Proxmox Homelab Debugging -
Prometheus Sizing for Homelab: RAM, Retention, and a 2Gi Limit
kube-prometheus-stack on a small k3s cluster: no memory limit took down a 4GB node, then the TSDB filled its NFS volume. The real values that fixed both.
Kubernetes Prometheus Monitoring -
CloudNativePG on k3s: PostgreSQL That Survives Node Failures
StatefulSets on local PersistentVolumes don't survive node failures. CloudNativePG handles PostgreSQL replication, failover and backups without Longhorn.
Kubernetes PostgreSQL CloudNativePG -
Traefik Rate Limiting That Actually Works on k3s
Traefik's RateLimit middleware on k3s and the source criterion that decides whether it works. Why my Immich limit put every visitor into one bucket.
Kubernetes Traefik Security -
Vault Configuration Under Terraform: Staged Migration Without Downtime
Vault OSS on Kubernetes holds secrets for 25+ services. Moving its policies, auth methods and roles to Terraform in 6 reversible stages, with zero downtime.
Kubernetes Vault Terraform -
Velero Backs Up to the Cluster It's Backing Up: Breaking the Backup Circularity
Velero backed up into Garage S3 inside the same cluster, so losing the cluster meant losing the backups. Fixed with Cloudflare R2 offsite and PBS for VMs.
Kubernetes Backup Velero -
The DNS Drift That Wasn't DHCP: How Proxmox Cloud-Init Silently Injects Search Domains
Every DNS lookup in my k3s cluster appended 'home.lan'. I blamed DHCP, but the Terraform fix showed zero diff. Proxmox cloud-init was injecting it.
Kubernetes DNS Proxmox -
k3s vs Nomad vs Docker Swarm: Which Orchestrator to Pick
Pick k3s for Kubernetes workloads. Docker Swarm is in decline since the Docker Engine v29 API break, and Nomad is source-available under BUSL.
Kubernetes Docker DevOps -
Three Fixes Were Merged. None of Them Had Taken Effect.
A drift audit found merged Traefik, Velero and monitoring fixes that never took effect while ArgoCD showed Synced. How the gap opens and how to check it.
Kubernetes GitOps Homelab -
How a YAML List vs Map Difference Crash-Looped My Kyverno Fix
Kyverno's cleanup-controller extraArgs expects a map, not a list, so my leader-election fix crash-looped the pod it was meant to stabilize. How to catch it.
Kubernetes Security Debugging -
Postgres 16→18: Why You Can't Just Swap the Image
Moving two Postgres StatefulSets from 16 to 18 on k3s: a fresh PVC, pg_dump and psql, a new mount layout in the PG18 image, and a role pg_dump never exports.
Kubernetes PostgreSQL Homelab -
Renovate in Kubernetes: OOM, HashiCorp Downloads, and the 6GB Container
Renovate's CronJob failed silently, then OOM'd at every limit up to 4Gi. The cause: hashing 14 platform builds of a Terraform provider, 16 downloads at a time.
Kubernetes GitOps Homelab -
Longhorn vs NFS on a Single-Host k3s Cluster: Why I Switched
On one Proxmox host, all three Longhorn replicas sit on the same NVMe and add no redundancy. A node failure caused Multi-Attach errors, and NFS fixed it.
Kubernetes Storage Homelab -
The Operating Model: What Should Auto-Update vs. What Needs a Human
A written contract for what Renovate auto-merges, what gets a PR, what alerts in Discord, what self-heals and what stays manual, so the homelab runs alone.
Kubernetes Homelab GitOps -
CPU Scheduling for etcd: Why Proxmox cpu.units Matters
Ollama, Minecraft and etcd compete for CPU on one Proxmox host. cpu.units gives the k3s VMs higher weight; without it, leader-election timeouts cascade.
Proxmox Kubernetes Performance -
3-2-1 Backup in Practice: Velero + PBS + Cloudflare R2 Offsite
3-2-1 backup in practice: Velero's defaultVolumesToFsBackup gap, a Google Drive offsite throttled to 1.6 KiB/s, and an R2 budget guard that deleted it.
Kubernetes Backup Homelab -
My Media Stack Lives in Two Containers and a Python CronJob
Jellyfin in a GPU-passthrough LXC, SABnzbd and the *arr apps on a second LXC, and a Python CronJob in k3s. Why the media stack left Kubernetes.
Homelab Proxmox Networking -
SQLite over NFS: Why WAL Corrupted My Garage S3 Metadata
Garage S3's SQLite metadata hit SQLITE_CORRUPT on an NFS volume because WAL mode does not work over network filesystems. The fix: local-path for every .db file.
Kubernetes Storage Homelab -
Private AKS on Azure: No Public API Server, No Public Node IPs, No Default Outbound
Private AKS with private_cluster_enabled, node egress forced through Azure Firewall via a Route Table, and private-endpoint-only ACR and Key Vault.
Azure Terraform Kubernetes -
Staggered VM Boot: How I Prevented a Load Average of 147
All VMs and LXCs starting at once pushed my Proxmox host load to 147. The fix: NFS first, k3s nodes 30 seconds apart, LXCs last. It costs nothing.
Proxmox Homelab Performance -
The Overcommit Guard: How a Python Script Prevents Host Freezes
VM dedicated memory is a reservation, LXC dedicated a soft ceiling. Summing them overstated pressure by 40GB. A two-tier CI guard now prevents ZFS freezes.
Proxmox Homelab Debugging -
GitOps for Firewall Rules: MikroTik + Terraform + Atlantis
Every firewall rule, VLAN, DHCP lease and NAT rule on my MikroTik RB5009 lives in Terraform and ships via Atlantis PRs. The full flow from PR to RouterOS.
Terraform MikroTik GitOps -
paperless-gpt Hallucinated OCR: Wrong VISION_LLM_MODEL
paperless-gpt invented German OCR text because VISION_LLM_MODEL pointed at qwen2.5-coder:7b, a text-only model. The fix, and why Ollama crashed 451 times.
AI Kubernetes Homelab -
Vault Auto-Unseal Without Cloud KMS: The Polling Sidecar Pattern
Vault OSS auto-unseals only through a KMS. Without one: a polling sidecar that unseals from a Kubernetes Secret every 5 seconds, and its security cost.
Kubernetes Security Vault -
Chaos Mesh in a Homelab: The Weekly Pod-Kills That Never Ran
Two weekly Chaos Mesh schedules on a homelab k3s cluster never touched a pod in four months. Why, how it was found, and the guarded game day that replaced them.
Kubernetes Chaos Engineering Homelab -
Cloudflare Tunnel: Outbound-Only Access, No Inbound Ports
cloudflared opens an outbound-only connection, so tunneled services need no inbound firewall rule. Split-DNS on AdGuard and the chunked_encoding fix for Immich.
Kubernetes Networking Cloudflare -
Logic App Standard 403 Forbidden with Private Storage: Fix
Logic App Standard failed every boot with 403 Forbidden. Private Storage needs four Private Endpoints (blob, file, queue, table), not two. Terraform fix inside.
Azure Terraform Logic Apps -
Velero Said Backups Succeeded. The Data Was Never There.
Velero reported 'Completed' for weeks with no PVC data for Postgres, Vaultwarden or Paperless. The defaultVolumesToFsBackup fix and how to verify a backup.
Kubernetes Backup Homelab -
Authelia OIDC on Kubernetes: SSO for Proxmox, Grafana, ArgoCD
One Authelia instance on k3s gives 25+ services a single login via OIDC and Traefik ForwardAuth, backed by CNPG Postgres, Redis sessions and secrets in Vault.
Kubernetes Security Homelab -
My Terraform Runner Destroyed Itself Mid-Apply
Atlantis ran in k3s on the Proxmox VMs it managed. A bpg/proxmox qmshutdown killed the Atlantis pod mid-apply. The structural hazard and the fix.
Terraform GitOps Homelab -
The ZFS ARC Freeze: How a Marginal PSU Killed My Entire Homelab
My Proxmox host froze under load three times in one week. A marginal PSU caused CPU throttling, which exposed a ZFS ARC deadlock. The chain, symptom to fix.
Homelab Proxmox ZFS -
How a Single Volume Was 65% of My Velero Backup and What I Almost Excluded Instead
The nfs-provisioner's root-mount volume backed up the whole NFS export as one 33.4GB blob every night, duplicating every PVC. My first guess was the wrong disk.
Kubernetes Backup NFS -
Best Mini PC for Homelab 2026
A spec-based comparison of the 4 best mini PCs for homelab use in 2026: Intel N100, Ryzen 7 and Intel Core boxes for Proxmox, k3s and always-on workloads.
Hardware Homelab Review -
Best NAS for Home Backup and Homelab 2026: 5 Picks
Synology DS225+ is the best NAS for home backup in 2026. Compared by spec and price with QNAP TS-264, TerraMaster F2-425, Synology DS425+ and UGREEN DXP2800.
Backup Hardware Homelab -
MikroTik vs Ubiquiti UniFi: Which Router for a Homelab?
MikroTik RB5009 wins for a homelab run as code: Terraform support, deeper firewall and VLAN control, lower price. UniFi Dream Router wins on setup and Wi-Fi.
Networking MikroTik Review -
Why a Cloud Backup Sync Was Failing Daily for Six Weeks
A daily offsite backup to Google Drive failed for six weeks. Not credentials, not network: Drive throttles the small-chunk format Proxmox Backup Server uses.
Backup Homelab Ansible -
Zero NetworkPolicies on Vault: How I Found the Biggest Gap in My Cluster and a GitOps Tracking Bug That Hid It
Vault is the trust root for every ExternalSecret, and its namespace had zero NetworkPolicies. The fix also exposed ArgoCD drift that ignores merged changes.
Kubernetes Security GitOps -
310 Restarts in 21 Days: CNPG's Silent PodMonitor Failure and the Leader-Election Trap
CloudNativePG's auto-generated PodMonitor lacked one label, so Prometheus never scraped it, and leader-election timeouts restarted the operator 310 times.
Kubernetes Homelab Debugging -
Beszel: Lightweight Host Monitoring That Doesn't Deserve Its Own Server
Beszel replaced a heavyweight stack for host metrics with one container. Kyverno caught the first deploy, and internal dashboards skip public DNS.
Kubernetes Homelab Monitoring -
Inside My Homelab: The Hardware Behind Every Article on This Blog
A tour of the 10-inch rack, compute and networking gear behind my k3s cluster, Proxmox host and this blog: what I run, why I picked it, and what I'd change.
Homelab Hardware Networking -
Kubernetes Health Probes: The Host Header Trap That Restarts Healthy Pods
Adding probes to 20+ workloads showed kubelet sends the Pod IP as Host header, and apps with host validation reject it. Probe patterns that work.
Kubernetes Homelab Debugging -
Migrating Atlantis to an LXC Accidentally Made It Fully Public
Moving Atlantis from Kubernetes to an LXC repointed the Cloudflare Tunnel past Traefik and Authelia. For about 18 hours the UI was open to anyone.
Security Kubernetes Homelab -
Renovate OOMKilled Three Times: Why the Fix Wasn't More Memory
2 GiB, 3 GiB, 4 GiB: Renovate kept getting OOMKilled. The real fix was capping Terraform provider hash concurrency. How I isolated the spike.
Kubernetes Homelab Debugging -
Zero NetworkPolicies on the Database Namespace: The Gap That Let Any Pod Reach Authelia's Postgres
The database namespace with Authelia's Postgres had no NetworkPolicies, so any pod could reach it. The audit, the fix, and the namespace I left unrestricted.
Kubernetes Homelab Security -
Discord Voice Choppy? It Was Bufferbloat: 51 Lines of Terraform
Discord voice sounded robotic to others but clean over mobile data. With no WAN QoS, a 50 Mbit upload saturated easily. The fix and its FastTrack catch.
Networking Homelab Debugging -
The Disaster Recovery Runbook Nobody Had Actually Run
The Proxmox Backup Server restore in DISASTER-RECOVERY.md had never been run end-to-end. The first run found a network conflict the doc never mentioned.
Backup Proxmox Homelab -
The .gitleaks-baseline.json That Suppressed Live Production Secrets
A gitleaks baseline meant for false positives was suppressing the live rpc_secret and admin_token of a production Garage S3 cluster, public and unrotated.
Security GitOps Homelab -
kube-prometheus-stack and Loki on k3s with Grafana SSO
kube-prometheus-stack and Loki on bare-metal k3s via ArgoCD: logs on Garage S3, Promtail on non-k8s nodes, MikroTik SNMP and Grafana login via Authelia OIDC.
Kubernetes Homelab Monitoring -
Redis Killed Nextcloud and Nobody Noticed for Hours
Redis without a PVC still has RDB persistence on. On a read-only rootfs it doesn't crash, it silently refuses writes. How that took down Nextcloud for hours.
Kubernetes Redis Homelab -
HA DNS for Homelab: Unbound + AdGuard Home + Keepalived on Raspberry Pi
A two-node recursive DNS stack on Raspberry Pi with ad filtering, automatic config sync, and transparent failover, fully managed with Ansible.
Homelab Networking DNS -
I Hardened Pod securityContext and Broke 9 Containers in Production
capabilities.drop: [ALL] and runAsNonRoot: true passed validation. Minutes after merge, nine containers were down, including both Postgres instances.
Kubernetes Security Homelab -
kubectl Said Everything Was Correct. Traefik 404'd Anyway.
A k3s Service pointed at an external Jellyfin LXC: the EndpointSlice looked right in kubectl, but Traefik 404'd every request. Plus a PVC shared by reference.
Kubernetes Networking Homelab -
ArgoCD Gotchas: Cache Staleness and the SharedResourceWarning Nobody Explains
kubectl apply succeeds, then the field reverts within seconds. Two ArgoCD gotchas: repo-server cache staleness and two Applications fighting over a resource.
Kubernetes GitOps Homelab -
I Ran Gitleaks Against My Own Repo and Found 12 Real Secrets
A full-history gitleaks scan of my homelab repo found 12 real plaintext secrets, including an OIDC signing key. The setup, a non-blocking baseline, the fixes.
Security Kubernetes Homelab -
My Firewall Had 77 Rules. Terraform Knew About 22 of Them.
Firewall rebuilds left old RouterOS rules in place. First match wins, so the oldest, broadest rules silently undid a tightening that looked done in Terraform.
MikroTik Terraform Security -
Hardening Unattended Raspberry Pi Edge Nodes: Watchdog, fail2ban, nftables, and the Mistakes That Take Down DNS
Two unattended Raspberry Pis run DNS for a whole network. Hardware watchdog, fail2ban, Docker-safe nftables, log caps and alerts independent of k3s.
Homelab Security Networking -
k3s Backup with Velero and Garage S3 Instead of MinIO
Daily k3s backups with Velero to Garage, a small self-hosted S3 store that replaced MinIO. Deployed via ArgoCD, with the verify and restore steps.
Kubernetes Homelab -
External Secrets Operator + HashiCorp Vault: GitOps Secret Lifecycle in Kubernetes
Kubernetes Secrets are base64, not encryption. Moving them into Vault and syncing them back with External Secrets Operator adds rotation and audit logging.
Kubernetes Security Homelab -
How a 1 GiB Memory Limit Took Down My Entire k3s Cluster
A 1 GiB memory limit set off a cascade: control-plane OOMKill, load average 90, 1.2M DNS queries a day and wrong kubelet allocatable memory. The post-mortem.
Kubernetes Homelab Debugging -
Kyverno: Supply Chain Security as Admission Control on Kubernetes
Kubernetes accepts any image, privilege level and resource config by default. Kyverno enforces policy at admission time. A supply chain baseline, Audit first.
Kubernetes Security Homelab -
IPv6 NAT66 Behind a FritzBox: The RouterOS 7 Bug That Broke WiFi Clients
IPv6 on MikroTik behind a FritzBox with CGN: ULA prefixes, NAT66 masquerade, and the RouterOS 7 RA default that broke FritzBox WiFi clients. Setup and fix.
MikroTik Networking Homelab -
SLO Burn-Rate Alerting with Prometheus: Beyond Threshold Alerts
Burn-rate alerting tracks how fast you spend your error budget, so you page on trajectory, not current state. The Google SRE Workbook approach on k3s.
Kubernetes Monitoring Homelab -
Self-Hosted Tailscale Control Plane: Headscale on k3s with Authelia OIDC
Deploy Headscale on a bare-metal k3s cluster with Longhorn persistence, Traefik ingress, and Authelia OIDC authentication, fully GitOps-managed via ArgoCD.
Kubernetes Homelab Security -
Wildcard TLS on k3s with cert-manager and Cloudflare DNS01
HTTP-01 fails behind a default-deny firewall. cert-manager with Cloudflare DNS01 gives every Traefik route a wildcard certificate without opening port 80.
Kubernetes Homelab Security -
ArgoCD on k3s: GitOps for a Whole Homelab with App-of-Apps
One root Application deploys a whole k3s homelab from Git with ArgoCD App-of-Apps: repo layout, bootstrap order, selfHeal and a rebuild in under 10 minutes.
Kubernetes GitOps Homelab -
MetalLB + Traefik on k3s: Fix LoadBalancer Stuck in Pending
On bare-metal k3s a LoadBalancer Service stays <pending>. Disable ServiceLB, give MetalLB an L2 IP pool and put Traefik behind it, all managed with ArgoCD.
Kubernetes Homelab Networking -
NIS2 Article 21 in Azure: Implementing Network Security Controls with Terraform
How the network security measures of NIS2 Article 21 map to Azure resources, implemented with Terraform. Concrete code, not legal theory.
Azure Terraform Compliance -
Zero-Trust RAG: Defeating the Shared Private Link Deadlock in Azure Terraform
Approve Azure AI Search Shared Private Links programmatically with AzAPI, and why your AI architecture fails an audit without identity chaining.
Azure Terraform AI -
Enterprise Homelab: k3s, Authelia & Longhorn on Proxmox with Terraform
Building a Kubernetes homelab with k3s, Authelia SSO, Longhorn and ArgoCD on Proxmox, plus the five mistakes that cost me hours along the way.
Kubernetes Homelab Security -
Breaking the Loop: Solving Circular Dependencies in Azure Firewall Routing
Azure Firewall forced tunneling in Terraform without cycle errors, and why a plain 0.0.0.0/0 route instantly breaks your Windows VMs.
Azure Terraform Networking -
Architecting an Enterprise-Grade Homelab: My Ansible Master Playbook
How my Ansible master playbook splits a homelab into tiers: HA DNS on Raspberry Pis, app and management nodes, an isolated DMZ, and dedicated AI hosts.
Homelab Ansible -
Automating MikroTik WireGuard VPN with Role-Based Access via Terraform
Deploy a WireGuard VPN on MikroTik with Terraform, with role-based network access that keeps a phone away from what the admin laptop can reach.
MikroTik Terraform Networking -
Automating MikroTik Bridge VLAN Filtering & Proxmox Trunks with Terraform
MikroTik Bridge VLAN Filtering without the lockouts: automate the VLAN matrix, Proxmox trunk ports, and edge access ports with Terraform.
MikroTik Terraform Networking -
Surviving Azure Policies: Zero-Trust Hub & Spoke with Terraform
Build an Azure Hub & Spoke network with Terraform that denies internet ingress from day one and survives DeployIfNotExists policies without breaking CI/CD.
Azure Terraform Networking -
Implementing a Zero-Trust MikroTik Firewall with Terraform
How I run a default-deny MikroTik RouterOS firewall with strict VLAN isolation and FastTrack for established traffic, all managed in Terraform.
MikroTik Terraform Security -
Gemma 4 26B on a Proxmox Mini PC: What Worked and What Didn't
Gemma 4 26B with Ollama in a Proxmox LXC built by Terraform and Ansible. The iGPU workaround never held and a CPU-only run froze the host.
Homelab AI Terraform -
Hardening Azure Acmebot for ISO 27001 & NIS2 Compliance
Network-isolating Azure Acmebot for ISO 27001 and NIS2: Zero-Trust Let's Encrypt automation with Terraform, Azure Private Link, and VNet Integration.
Azure Terraform Security
Auf Deutsch
-
Acmebot härten für ISO 27001 & NIS2: Zero-Trust-Zertifikate in Azure
Der Standard-Deploy von Acmebot hat drei öffentliche Angriffsflächen. So isolierst du ihn mit Private Link, VNet Integration und Private DNS in Terraform.
-
Server für Selbstständige kaufen: Wo Preis-Leistung wirklich stimmt (2026)
Wie viel RAM braucht dein Betrieb, und wann lohnt ein eigener Proxmox-Server statt Cloud? Kaufberatung für Selbstständige mit echten Betriebszahlen.
Hardware Ratgeber Selbstständigkeit -
WireGuard-VPN auf MikroTik automatisiert: Role-based Peers via Terraform
WireGuard-VPN auf MikroTik per Terraform: feste Peer-IPs pro Gerät und rollenbasierter Zugriff, damit das Handy nicht dasselbe erreicht wie der Admin-Laptop.
-
Pod-securityContext gehärtet und 9 Container in Produktion zerlegt
capabilities.drop: [ALL] und runAsNonRoot: true bestanden die Validierung. Kurz nach dem Merge waren neun Container down, darunter beide Postgres-Instanzen.
-
Backup ohne Komplexität: Velero + Garage S3 auf Longhorn im k3s-Cluster
Garage statt MinIO als Backup-Ziel für Velero: ein einzelnes kleines Binary, tägliche Cluster-Backups inklusive Longhorn-Volumes, deployt per ArgoCD.
-
NIS2 für Homelab & gehostete Dienste: Was (d)ein Server wirklich einhalten muss
NIS2 betrifft nicht jeden mit Server daheim. Wann du als Selbstständiger oder Homelab-Betreiber in den Anwendungsbereich fällst und welche Pflichten gelten.
NIS2 Selbstständigkeit Homelab