Renovate is the dependency update bot that keeps my k3s cluster current, and it’s the “auto-update” half of the operating model that decides what gets merged automatically vs. what waits for a human. It runs as a Kubernetes CronJob every 2 hours, checks for new container images, Helm chart versions, and Terraform provider releases, and opens PRs on GitHub.
For two weeks, it was failing silently on every run. The CronJob reported “Completed.” The GitHub commits showed no new PRs. Nothing looked wrong, because Renovate exits cleanly on a repository config validation error, so every run landed in the CronJob’s history as a success.
View the complete homelab infrastructure source on GitHub
Failure 1: Invalid Preset
The first failure was a config error. The renovate.json file referenced an invalid preset:
{
"extends": ["config:base", ":enableHelpfulPre-commit"]
}
:enableHelpfulPre-commit is not a real Renovate preset. Renovate’s config validation caught the error, skipped the repository, and exited with code 0. A repository config error doesn’t fail the process.
The CronJob’s exit code was 0. Kubernetes considered the job successful. No alert fired. Renovate did nothing on every run.
The fix: remove the invalid preset and replace the deprecated config:base with config:recommended.
{
"extends": ["config:recommended"]
}
Failure 2: OOM at 512Mi
With the preset fixed, Renovate got past config validation and immediately ran out of memory. The container limit was 512Mi, and every run died with JavaScript heap out of memory during the scan. Each failed Job looked like any other entry in the history:
kubectl get jobs -n apps -l app=renovate
# NAME COMPLETIONS DURATION AGE
# renovate-28197 0/1 ... 2m
# renovate-28196 0/1 ... 2h
# renovate-28195 0/1 ... 4h
With failedJobsHistoryLimit: 3, only the last three failures stay visible, and at one run every two hours they rotated out faster than I was checking.
The fix: raise the memory limit to 1Gi and cap the V8 heap below it with NODE_OPTIONS:
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=896"
resources:
limits:
memory: 1Gi
--max-old-space-size caps V8’s own heap. The container’s RSS also includes git clone and worktree data and non-heap buffers, so the heap cap has to stay well under the container limit, not equal to it. Exceed the heap cap and Node throws a heap error you can read in the logs. Exceed the container limit first and the kernel sends SIGKILL, which leaves no log line at all.
Failure 3: 1Gi, 2Gi, 4Gi, All OOMKilled
1Gi held until the repo grew. New Terraform stacks and ExternalSecrets took Renovate from about 190 dependencies to 217, and from 2026-07-08 every run ended in exitCode: 137, reason: OOMKilled for more than 16 hours. The log cut off mid-extraction with no exit message, which is the OOMKill signature. I raised the limit to 2Gi with a 1536Mi heap, keeping about 25% headroom.
On 2026-07-09 it OOM’d again at 2Gi. The kernel’s oom-kill records on the node showed all 20 kills at an anon-rss of about 2054–2064Mi, only 6–16Mi over the 2048Mi cap. That data is censored: the kernel kills the moment the cgroup limit is crossed, so it proves the limit is too small, not what the real peak is. I raised it to 4Gi. That was also OOMKilled, again just over the cap (anon-rss about 4.12–4.15Gi), and the retry loop’s CPU and memory pressure on a node that also runs one of the two etcd members contributed to a cluster-wide apiserver outage that afternoon.
At that point I stopped guessing. I ran Renovate uncapped on a separate non-k3s LXC with 12Gi of headroom, the real GitHub token and the real branches. It wasn’t a leak. The true peak was about 3.99Gi: one transient spike of about 30 seconds, after which GC reclaimed the memory and the run exited 0.
The spike came from a major hashicorp/aws provider bump (5.x to 6.53.0). To keep .terraform.lock.hcl valid, Renovate’s Terraform manager downloads every platform build of the new release from releases.hashicorp.com and SHA256-hashes each one. That release has 14 platform builds. With Renovate’s default concurrentRequestLimit of 16, all 14 zips could be in flight and buffered in memory at once.
The fix had two parts. First, cap concurrency for that host in renovate.json, which flattens the peak at the source:
{
"hostRules": [
{
"matchHost": "releases.hashicorp.com",
"concurrentRequestLimit": 2
}
]
}
Second, size the container from the measured peak instead of a multiplier guess: a 6Gi limit with a 4608Mi heap, about 1.5x the 3.99Gi peak. Terraform provider majors also stopped auto-merging. They now always need a human look, because they’re the bump class that pulls a whole release’s platform builds through that hashing path.
The Silent Failure Pattern
Each failure was quiet in its own way. The invalid preset exited 0 and looked successful. The OOMs produced failed Jobs that nobody was alerted on, and SIGKILL left nothing in the logs. The only consistent symptom was the absence of new PRs on GitHub.
Two PrometheusRule alerts now cover the cases that leave a trace in kube-state-metrics:
- alert: RenovateCronJobStale
expr: |
time() - kube_cronjob_status_last_schedule_time{namespace="apps", cronjob="renovate"} > 21600
for: 5m
labels:
severity: critical
- alert: RenovateJobFailing
expr: |
kube_job_status_failed{namespace="apps", job_name=~"renovate-[0-9]+"} > 0
for: 10m
labels:
severity: warning
RenovateCronJobStale fires when the CronJob hasn’t been scheduled in six hours (three missed runs): suspended, stuck controller, or something cluster-wide blocking scheduling. RenovateJobFailing catches every OOMKill and crash. Neither catches Failure 1: a run that exits 0 without doing anything still passes both rules. That case needs a check on the outcome itself, and I haven’t built it yet.
The Current Configuration
# kubernetes/apps/renovate/renovate.yml (trimmed)
apiVersion: batch/v1
kind: CronJob
metadata:
name: renovate
namespace: apps
spec:
schedule: "0 */2 * * *" # every 2 hours
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
restartPolicy: Never
containers:
- name: renovate
image: ghcr.io/renovatebot/renovate:43.253.2
env:
- name: RENOVATE_TOKEN
valueFrom:
secretKeyRef:
name: renovate-token
key: token
- name: NODE_OPTIONS
value: "--max-old-space-size=4608"
resources:
requests:
cpu: 100m
memory: 1Gi
limits:
cpu: 500m
memory: 6Gi
concurrencyPolicy: Forbid keeps a slow run from overlapping the next schedule tick. restartPolicy: Never means every retry is a fresh pod, so each OOMKill shows up as its own failed pod with its own exit code.
The Lesson
Silent CronJob failures are the same “reports success but did nothing” pattern as ArgoCD showing Synced/Healthy on an Application that never applied. The job “succeeds”, nobody checks, and dependency updates stop. Security patches don’t land, and nobody notices until a vulnerability is disclosed for a package Renovate would have bumped.
More memory alone fixed nothing here. Measuring the real peak found the cause, and alerting on the pipeline itself makes the next failure visible. Every CronJob that does something important needs a check that it did the job, not only that it exited.
The same problem exists in Azure DevOps: a pipeline that succeeds but produces nothing looks the same as one that did its work. Azure DevOps pipeline analytics report pass rate and duration, not whether a run produced meaningful output. The fix is the same in both places: add a step that checks for the expected outcome (PRs opened, artifacts published, deployments completed) and alert when it’s missing.
The Phoenix Project* tells the same story from the human side: the automated system everyone assumes is working, until someone finally checks.