Two problems. Same database namespace. Discovered one after the other because fixing the first one required understanding the second. All of it running on a BMAX Mini PC* with the MikroTik RB5009* handling the network between the cluster and everything else.
The CNPG (CloudNativePG) operator had restarted 310 times in 21 days. Meanwhile, its PodMonitor, the thing that was supposed to scrape PostgreSQL metrics into Prometheus, had never actually collected a single data point. Both issues were live, both were silent, and both traced back to the same root cause: a mismatch between what the operator assumed about the cluster and what the cluster actually required.
View the complete homelab infrastructure source on GitHub 🐙
The PodMonitor That Wasn’t Scraping
CNPG has a built-in enablePodMonitor: true field in its Cluster CRD. Set it to true, the operator generates a PodMonitor automatically, Prometheus should pick it up. In theory.
In practice, the operator-generated PodMonitor only carries the cnpg.io/cluster label. My kube-prometheus-stack Prometheus instance has a podMonitorSelector that requires release: kube-prometheus-stack on every PodMonitor it watches. Without that label, Prometheus ignores the object entirely.
The PodMonitor showed as Synced and Healthy in ArgoCD. CNPG reported it as created. Everything looked fine. But kubectl get podmonitor -n database showed the operator-generated one had no release label, and kubectl get prometheus -o jsonpath='{.spec.podMonitorSelector}' confirmed the selector required it. The metrics endpoint on :9187 was never scraped.
No CNPG CRD field adds metadata labels to the auto-generated PodMonitor. podMonitorRelabelings only rewrites samples after they’re scraped, it can’t make Prometheus notice the object in the first place. The fix was to disable the operator’s own PodMonitor and replace it with a git-managed one:
# kubernetes/system/postgres/cluster.yml
monitoring:
enablePodMonitor: false
# kubernetes/system/postgres/postgres-monitoring.yml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: postgres-authelia
namespace: database
labels:
cnpg.io/cluster: postgres-authelia
release: kube-prometheus-stack
spec:
namespaceSelector:
matchNames:
- database
selector:
matchLabels:
cnpg.io/cluster: postgres-authelia
podMetricsEndpoints:
- port: metricsBoth labels. The git-managed version is the one Prometheus actually selects. The operator’s own version is disabled. This is the same pattern as every other monitoring resource in this cluster: if Prometheus needs to see it, you manage the label yourself, don’t rely on the operator to guess what your selector requires.
The new file also had to be added to the ArgoCD Application’s directory.include pattern, the postgres Application only syncs explicitly listed YAML files, not everything in the directory. Missing this means the PodMonitor gets created in git but never applied:
include: '{cluster,external-secret,scheduled-backup,backup-config,network-policies,postgres-monitoring}.yml'310 Restarts: Leader-Election Timeout
The PodMonitor fix was the investigation that led me to the operator restart count. While checking the monitoring gap, I noticed kubectl get pods -n cnpg-system showed the operator had restarted hundreds of times.
Every restart’s previous-container log ended the same way:
context deadline exceededOn a Lease PUT, the leader-election renewal. The CNPG operator uses controller-runtime’s standard leader-election mechanism: a single replica holds a Lease object, and when it can’t renew that Lease within the deadline, controller-runtime terminates the process so another instance can take over. With a single replica, there’s no other instance. The process dies, Kubernetes restarts it, it acquires leadership again, and the cycle repeats.
The defaults are --leader-lease-duration=15s and --leader-renew-deadline=10s. On a host with documented I/O fragility (the same one that caused the etcd cascading failure I wrote about earlier), a brief apiserver latency blip pushes the renewal past the 10-second deadline. Controller-runtime kills the process. Kubernetes restarts it. The operator is healthy for hours, then one blip kills it again.
The fix was the same approach I already applied to etcd’s own timeouts in Ansible: scale the defaults by roughly 5x.
# kubernetes/system/cloudnative-pg/application.yml
spec:
source:
helm:
values:
operator:
additionalArgs:
- --leader-lease-duration=60
- --leader-renew-deadline=50I confirmed both flags exist via the live v1.25.0 binary’s --help output before applying. Not guessed, not assumed, the same discipline I apply to every Terraform resource: check what’s actually available, don’t rely on documentation that might be version-skewed.
After applying, restarts dropped to zero. The operator stopped dying on every apiserver latency blip because a 50-second renewal deadline is generous enough to survive the I/O spikes that this host reliably produces.
The Pattern
Both issues share the same root cause: assumptions that don’t match reality.
The PodMonitor assumed the operator would add the right labels. The operator assumed Prometheus would select any PodMonitor with the cluster label. Neither was true, and both failures were silent, the PodMonitor existed, looked healthy, and did nothing.
The leader-election assumed a 10-second renewal deadline was enough. On a host that regularly produces I/O latency spikes, it wasn’t. The operator died, restarted, and the death was invisible unless you specifically checked the restart count.
The fix for both was the same: stop relying on defaults and assumptions, manage the configuration explicitly. Git-managed PodMonitor with the right labels. Explicit leader-election timeouts tuned to the actual hardware’s failure mode.
Configuring operators for real hardware, not the idealized environment the defaults assume, is the same problem in enterprise Azure: AKS node pools have their own I/O profiles, managed PostgreSQL has its own failover timing, and the defaults are tuned for average cases, not your specific failure modes.