# woitzik.dev

> Hybrid Cloud Engineer specializing in Azure, Terraform, and Zero-Trust network architecture. Publishes hardened infrastructure templates and deep-dive articles.

## Blog Posts

- [310 Restarts in 21 Days: CNPG's Silent PodMonitor Failure and the Leader-Election Trap](https://woitzik.dev/blog/cnpg-podmonitor-leader-election-restart-loop/): CloudNativePG's auto-generated PodMonitor was missing a single label — Prometheus never scraped it. The same I/O fragility that causes etcd timeouts was triggering leader-election failures, restarting the operator 310 times in 21 days. Here's how I traced both root causes.
- [Beszel: Lightweight Host Monitoring That Doesn't Deserve Its Own Server](https://woitzik.dev/blog/beszel-lightweight-host-monitoring-k3s/): Why I replaced a heavyweight monitoring stack for host-level metrics with a single container, how Kyverno caught my first deploy before it hit production, and why internal-only dashboards don't need a public DNS record.
- [Inside My Homelab: The Hardware Behind Every Article on This Blog](https://woitzik.dev/blog/homelab-hardware-tour-physical-rack/): A full tour of the physical rack, compute, and networking gear powering my K3s cluster, Proxmox hosts, and everything I write about here — what I run, why I picked it, and what I'd change.
- [Kubernetes Health Probes: The Host Header Trap That Restarts Healthy Pods](https://woitzik.dev/blog/kubernetes-health-probes-host-header-trap/): Adding health probes to 20+ workloads taught me that kubelet sends the Pod IP as the Host header — and apps with host-validation reject it. Here's the full sweep, the gotcha that caught me, and the probe patterns that actually work.
- [Migrating Atlantis to an LXC Accidentally Made It Fully Public](https://woitzik.dev/blog/infra-migration-silently-removed-auth-atlantis/): Moving Atlantis from Kubernetes to a dedicated LXC involved repointing the Cloudflare Tunnel. The new tunnel pointed straight at the LXC's IP, bypassing Traefik and Authelia entirely. Atlantis has no auth of its own. For roughly 18 hours, anyone with the URL had full plan/apply access to the infrastructure repo.
- [Renovate OOMKilled Three Times: Why the Fix Wasn't More Memory](https://woitzik.dev/blog/renovate-oomkilled-terraform-hash-concurrency/): Two GiB wasn't enough, so I bumped to 3 GiB. Still OOMKilled. Bumped to 4 GiB. Still OOMKilled. The real fix wasn't memory at all — it was Terraform hash concurrency. Here's how I isolated the actual spike.
- [Zero NetworkPolicies on the Database Namespace: The Gap That Let Any Pod Reach Authelia's Postgres](https://woitzik.dev/blog/kubernetes-database-namespace-network-policies/): The database namespace holding Authelia's session storage had no network restrictions. Any pod in the cluster could reach it. Here's the audit that found it, the traffic pattern that shaped the fix, and the namespace I deliberately left unrestricted.
- [Discord Voice Choppy? It Was Bufferbloat — Fixed with 51 Lines of Terraform](https://woitzik.dev/blog/mikrotik-wan-qos-bufferbloat-discord/): Discord voice was robotic for people hearing me. Confirmed clean over mobile data — home network path. Zero QoS on the WAN interface meant a 50 Mbit upload ceiling was easy to saturate. Here's the investigation, the fix, and why PCQ per-flow fairness matters.
- [The Disaster Recovery Runbook Nobody Had Actually Run](https://woitzik.dev/blog/backup-restore-test-never-verified-network-conflict/): DISASTER-RECOVERY.md documented the Proxmox Backup Server restore procedure in detail. It had never been executed end-to-end. Running it for the first time - against a scratch VM, alongside the still-running original - found a network conflict the documentation never mentioned.
- [The .gitleaks-baseline.json That Suppressed Live Production Secrets](https://woitzik.dev/blog/gitleaks-baseline-suppressing-live-secrets/): A gitleaks baseline file is supposed to suppress known-false-positive findings. It turned out to be suppressing the live rpc_secret and admin_token for a production Garage S3 cluster - unrotated since the commit that introduced them to a public repo months earlier.
- [Full Observability on k3s: kube-prometheus-stack + Loki + Grafana OIDC](https://woitzik.dev/blog/kube-prometheus-loki-grafana-k3s/): Deploy a production-grade monitoring stack on bare-metal k3s: Prometheus, Loki with Garage S3 storage, Promtail on edge nodes via Ansible, SNMP monitoring for MikroTik, and Grafana SSO via Authelia OIDC - all GitOps-managed.
- [Redis Killed Nextcloud and Nobody Noticed for Hours](https://woitzik.dev/blog/redis-rdb-persistence-no-pvc-kubernetes-outage/): Redis running without a PVC still has persistence enabled by default. When it can't write RDB snapshots to a read-only rootfs, it doesn't crash - it silently refuses all writes. Here's how that turned into a full Nextcloud outage and why the logs pointed at the wrong thing first.
- [HA DNS for Homelab: Unbound + AdGuard Home + Keepalived on Raspberry Pi](https://woitzik.dev/blog/unbound-adguard-keepalived-homelab/): A two-node recursive DNS stack with ad filtering, automatic config sync, and transparent failover - fully managed with Ansible.
- [I Hardened Pod securityContext and Broke 9 Containers in Production](https://woitzik.dev/blog/kubernetes-securitycontext-hardening-broke-9-containers/): capabilities.drop: [ALL] and runAsNonRoot: true passed schema validation cleanly. Within minutes of merge, nine containers - including both Postgres instances backing Paperless and Nextcloud - were down. Here's the failure analysis, why a manual kubectl fix got silently undone, and the lesson for any blanket securityContext change.
- [kubectl Said Everything Was Correct. Traefik 404'd Anyway.](https://woitzik.dev/blog/traefik-endpointslice-vs-endpoints-kubernetes/): Migrating Jellyfin off k3s onto a GPU-passthrough LXC meant pointing a Service at an external IP. The EndpointSlice looked completely correct via kubectl - Service existed, endpoint listed right - but Traefik 404'd every request. A second, unrelated gotcha surfaced in the same migration: a PVC silently shared by reference across two unrelated files.
- [ArgoCD Gotchas: Cache Staleness and the SharedResourceWarning Nobody Explains](https://woitzik.dev/blog/argocd-cache-staleness-shared-resource-warning/): kubectl apply succeeds, the field reverts within seconds, and there's no error anywhere. Two ArgoCD debugging patterns that hit the same homelab three times in one day: repo-server cache staleness reverting live edits, and two Applications silently fighting over the same resource.
- [I Ran Gitleaks Against My Own Repo and Found 12 Real Secrets](https://woitzik.dev/blog/gitleaks-secret-scanning-homelab-remediation/): A full-history gitleaks scan of a homelab repo that had been running for months turned up 12 distinct plaintext secrets - including an OIDC signing key. Here's the scanning setup, the baseline strategy that doesn't block on pre-existing leaks, and the remediation plan.
- [My Firewall Had 77 Rules. Terraform Knew About 22 of Them.](https://woitzik.dev/blog/mikrotik-firewall-rule-drift-orphaned-rules/): Multiple rounds of 'reconstruct the firewall' work each added a fresh generation of rules without removing the old one. Because RouterOS evaluates rules in order and stops at the first match, the oldest, broadest generation was silently winning over the newest, narrower one - undoing a security tightening that looked complete in Terraform.
- [Hardening Unattended Raspberry Pi Edge Nodes: Watchdog, fail2ban, nftables, and the Mistakes That Take Down DNS](https://woitzik.dev/blog/raspberry-pi-edge-hardening-watchdog-fail2ban-nftables/): Two Raspberry Pis run DNS for an entire network with no one watching them most of the time. A hardware watchdog, fail2ban, an additive nftables host firewall that doesn't conflict with Docker, log size caps, and an alerting path that works even when the rest of the monitoring stack is down.
- [k3s Backup Without the Complexity: Velero + Garage S3 on Longhorn](https://woitzik.dev/blog/velero-garage-k3s-backup/): Replace MinIO with Garage - a single 50MB binary - as the Velero backup target. Full daily cluster backups with Longhorn volume snapshots, deployed via ArgoCD.
- [External Secrets Operator + HashiCorp Vault: GitOps Secret Lifecycle in Kubernetes](https://woitzik.dev/blog/external-secrets-operator-vault-kubernetes/): Kubernetes Secrets are base64-encoded, not encrypted. Moving secrets out of the cluster into Vault - and syncing them back via External Secrets Operator - gives you rotation, audit logging, and compliance without changing how applications consume secrets.
- [How a 1 GiB Memory Limit Took Down My Entire k3s Cluster](https://woitzik.dev/blog/k3s-cascading-failure-oomkill-dns-storm/): A single misconfigured resource limit triggered a cascade: OOMKill on the control-plane, load average of 90, 1.2M DNS queries per day, and kubelet reporting the wrong allocatable memory. Here's the full post-mortem.
- [Kyverno: Supply Chain Security as Admission Control on Kubernetes](https://woitzik.dev/blog/kyverno-supply-chain-security-kubernetes/): Most Kubernetes clusters accept any container image, any privilege level, and any resource configuration by default. Kyverno lets you enforce policies at admission time - before anything runs. Here's how to build a supply chain security baseline with Audit-first rollout.
- [IPv6 NAT66 Behind a FritzBox: The RouterOS 7 Bug That Broke WiFi Clients](https://woitzik.dev/blog/mikrotik-ipv6-nat66-cgn-routeros7/): Setting up IPv6 on MikroTik behind a FritzBox with CGN should be straightforward - ULA prefix, NAT66 masquerade, done. Instead, RouterOS 7 started advertising router advertisements on the WAN interface, turning MikroTik into an uninvited IPv6 gateway for FritzBox WiFi clients. Here's the full setup and fix.
- [SLO Burn-Rate Alerting with Prometheus: Beyond Threshold Alerts](https://woitzik.dev/blog/slo-burn-rate-alerting-prometheus-k3s/): Most teams alert when availability drops below a threshold. Burn-rate alerting tells you how fast you're spending your error budget - so you page on trajectory, not just current state. Here's how to implement the Google SRE Workbook approach on a bare-metal k3s cluster.
- [Self-Hosted Tailscale Control Plane: Headscale on k3s with Authelia OIDC](https://woitzik.dev/blog/headscale-oidc-k3s-authelia/): Deploy Headscale on a bare-metal k3s cluster with Longhorn persistence, Traefik ingress, and Authelia OIDC authentication - fully GitOps-managed via ArgoCD.
- [Wildcard TLS Certificates on K3s with cert-manager and Cloudflare DNS](https://woitzik.dev/blog/cert-manager-wildcard-k3s-cloudflare/): How to automate wildcard Let's Encrypt certificates on a bare-metal K3s cluster using cert-manager's DNS-01 challenge with Cloudflare - and why HTTP-01 won't work for internal services.
- [GitOps on K3s: Managing a Complete Homelab with ArgoCD](https://woitzik.dev/blog/argocd-gitops-k3s-homelab/): How to manage an entire Kubernetes homelab - MetalLB, Traefik, Longhorn, Authelia, and more - as a Git repository using ArgoCD's App-of-Apps pattern.
- [Bare-Metal LoadBalancer on K3s: MetalLB + Traefik with ArgoCD](https://woitzik.dev/blog/metallb-traefik-k3s-argocd/): How to get a real external IP on a bare-metal Kubernetes cluster using MetalLB L2 mode, and wire it up with Traefik for automatic HTTPS - fully GitOps-managed with ArgoCD.
- [NIS2 Article 21 in Azure: Implementing Network Security Controls with Terraform](https://woitzik.dev/blog/nis2-article-21-azure-terraform/): A technical deep-dive into the network security requirements of NIS2 Article 21 and how to implement them in Azure using Terraform - with concrete code, not legal theory.
- [Zero-Trust RAG: Defeating the Shared Private Link Deadlock in Azure Terraform](https://woitzik.dev/blog/azure-rag-shared-private-link-automation/): How to programmatically approve Azure AI Search Shared Private Links using AzAPI, and why your AI architecture will fail an audit without proper Identity Chaining.
- [Enterprise Homelab: K3s, Authelia & Longhorn on Proxmox with Terraform](https://woitzik.dev/blog/k3s-authelia-proxmox-homelab/): How to build a production-grade Kubernetes homelab with K3s, Authelia SSO, Longhorn storage, and ArgoCD - and the five painful mistakes that will cost you hours if you don't know about them.
- [Breaking the Loop: Solving Circular Dependencies in Azure Firewall Routing](https://woitzik.dev/blog/azure-firewall-cycle-error/): How to implement Azure Firewall Forced Tunneling in Terraform without triggering cycle errors, and why a simple 0.0.0.0/0 route will instantly break your Windows VMs.
- [Architecting an Enterprise-Grade Homelab: My Ansible Master Playbook](https://woitzik.dev/blog/enterprise-homelab-architecture-ansible/): Take a tour of a fully automated, segmented, and highly available homelab architecture orchestrated entirely via Ansible and GitOps.
- [Automating MikroTik WireGuard VPN with Role-Based Access via Terraform](https://woitzik.dev/blog/mikrotik-wireguard-vpn-terraform/): Deploy a WireGuard VPN on MikroTik using Terraform. Learn how to implement role-based network access, isolating mobile devices from full admin laptops.
- [Automating MikroTik Bridge VLAN Filtering & Proxmox Trunks with Terraform](https://woitzik.dev/blog/mikrotik-vlan-filtering-terraform-proxmox/): Master MikroTik's notoriously complex Bridge VLAN Filtering. Learn how to automate dynamic VLAN matrices, Proxmox trunk ports, and edge devices using Terraform.
- [Surviving Azure Policies: Zero-Trust Hub & Spoke with Terraform](https://woitzik.dev/blog/azure-terraform-hub-spoke-zero-trust/): How to build an enterprise-grade Azure network architecture that blocks internet traffic by default and survives aggressive DeployIfNotExists (DINE) policies - without breaking your CI/CD pipeline.
- [Implementing a Zero-Trust MikroTik Firewall with Terraform](https://woitzik.dev/blog/mikrotik-zero-trust-firewall-terraform/): Learn how to enforce strict VLAN isolation, fast-track traffic, and build a default-deny firewall for MikroTik RouterOS using Infrastructure as Code.
- [Deploying Gemma 4 26B on Proxmox: IaC Setup with Terraform, Ansible & AMD iGPU](https://woitzik.dev/blog/deploying-gemma-proxmox-iac/): A complete guide to automating a local AI stack on Proxmox LXC using Terraform and Ansible, including Open-WebUI and AMD Radeon Vega iGPU workarounds.
- [Hardening Azure Acmebot for ISO 27001 & NIS2 Compliance](https://woitzik.dev/blog/hardening-azure-acmebot-iso27001/): A deep dive into architecting a Zero-Trust Let's Encrypt automation using Terraform, Azure Private Link, and VNet Integration.

## Enterprise Modules

- [Azure Acmebot - Enterprise VNet Edition](https://woitzik.dev/templates/): Production-ready Let's Encrypt automation with Private Link isolation
- [Enterprise Hub & Spoke - Zero-Trust Edition](https://woitzik.dev/templates/): Zero-Trust NSGs, centralized Private DNS, DINE policy bypass
- [Azure Firewall - Enterprise Forced Tunneling Edition](https://woitzik.dev/templates/): Cycle-error-free Forced Tunneling with KMS & Azure AD bypasses
- [Enterprise AI RAG - Zero-Trust Networking](https://woitzik.dev/templates/): Automated Shared Private Link approval, VNet injection, Identity Chaining

## Contact

- Website: https://woitzik.dev
- Email: david@woitzik.dev
- GitHub: https://github.com/dwoitzik
- LinkedIn: https://linkedin.com/in/david-woitzik