Skip to content
7 min read Updated

Renovate in Kubernetes: OOM, HashiCorp Downloads, and the 6GB Container

Renovate's CronJob failed silently, then OOM'd at every limit up to 4Gi. The cause: hashing 14 platform builds of a Terraform provider, 16 downloads at a time.


Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

Renovate is the dependency update bot that keeps my k3s cluster current, and it’s the “auto-update” half of the operating model that decides what gets merged automatically vs. what waits for a human. It runs as a Kubernetes CronJob every 2 hours, checks for new container images, Helm chart versions, and Terraform provider releases, and opens PRs on GitHub.

For two weeks, it was failing silently on every run. The CronJob reported “Completed.” The GitHub commits showed no new PRs. Nothing looked wrong, because Renovate exits cleanly on a repository config validation error, so every run landed in the CronJob’s history as a success.

View the complete homelab infrastructure source on GitHub

Failure 1: Invalid Preset

The first failure was a config error. The renovate.json file referenced an invalid preset:

{
  "extends": ["config:base", ":enableHelpfulPre-commit"]
}

:enableHelpfulPre-commit is not a real Renovate preset. Renovate’s config validation caught the error, skipped the repository, and exited with code 0. A repository config error doesn’t fail the process.

The CronJob’s exit code was 0. Kubernetes considered the job successful. No alert fired. Renovate did nothing on every run.

The fix: remove the invalid preset and replace the deprecated config:base with config:recommended.

{
  "extends": ["config:recommended"]
}

Failure 2: OOM at 512Mi

With the preset fixed, Renovate got past config validation and immediately ran out of memory. The container limit was 512Mi, and every run died with JavaScript heap out of memory during the scan. Each failed Job looked like any other entry in the history:

kubectl get jobs -n apps -l app=renovate
# NAME              COMPLETIONS   DURATION   AGE
# renovate-28197    0/1           ...        2m
# renovate-28196    0/1           ...        2h
# renovate-28195    0/1           ...        4h

With failedJobsHistoryLimit: 3, only the last three failures stay visible, and at one run every two hours they rotated out faster than I was checking.

The fix: raise the memory limit to 1Gi and cap the V8 heap below it with NODE_OPTIONS:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=896"
resources:
  limits:
    memory: 1Gi

--max-old-space-size caps V8’s own heap. The container’s RSS also includes git clone and worktree data and non-heap buffers, so the heap cap has to stay well under the container limit, not equal to it. Exceed the heap cap and Node throws a heap error you can read in the logs. Exceed the container limit first and the kernel sends SIGKILL, which leaves no log line at all.

Failure 3: 1Gi, 2Gi, 4Gi, All OOMKilled

1Gi held until the repo grew. New Terraform stacks and ExternalSecrets took Renovate from about 190 dependencies to 217, and from 2026-07-08 every run ended in exitCode: 137, reason: OOMKilled for more than 16 hours. The log cut off mid-extraction with no exit message, which is the OOMKill signature. I raised the limit to 2Gi with a 1536Mi heap, keeping about 25% headroom.

On 2026-07-09 it OOM’d again at 2Gi. The kernel’s oom-kill records on the node showed all 20 kills at an anon-rss of about 2054–2064Mi, only 6–16Mi over the 2048Mi cap. That data is censored: the kernel kills the moment the cgroup limit is crossed, so it proves the limit is too small, not what the real peak is. I raised it to 4Gi. That was also OOMKilled, again just over the cap (anon-rss about 4.12–4.15Gi), and the retry loop’s CPU and memory pressure on a node that also runs one of the two etcd members contributed to a cluster-wide apiserver outage that afternoon.

At that point I stopped guessing. I ran Renovate uncapped on a separate non-k3s LXC with 12Gi of headroom, the real GitHub token and the real branches. It wasn’t a leak. The true peak was about 3.99Gi: one transient spike of about 30 seconds, after which GC reclaimed the memory and the run exited 0.

The spike came from a major hashicorp/aws provider bump (5.x to 6.53.0). To keep .terraform.lock.hcl valid, Renovate’s Terraform manager downloads every platform build of the new release from releases.hashicorp.com and SHA256-hashes each one. That release has 14 platform builds. With Renovate’s default concurrentRequestLimit of 16, all 14 zips could be in flight and buffered in memory at once.

The fix had two parts. First, cap concurrency for that host in renovate.json, which flattens the peak at the source:

{
  "hostRules": [
    {
      "matchHost": "releases.hashicorp.com",
      "concurrentRequestLimit": 2
    }
  ]
}

Second, size the container from the measured peak instead of a multiplier guess: a 6Gi limit with a 4608Mi heap, about 1.5x the 3.99Gi peak. Terraform provider majors also stopped auto-merging. They now always need a human look, because they’re the bump class that pulls a whole release’s platform builds through that hashing path.

The Silent Failure Pattern

Each failure was quiet in its own way. The invalid preset exited 0 and looked successful. The OOMs produced failed Jobs that nobody was alerted on, and SIGKILL left nothing in the logs. The only consistent symptom was the absence of new PRs on GitHub.

Two PrometheusRule alerts now cover the cases that leave a trace in kube-state-metrics:

- alert: RenovateCronJobStale
  expr: |
    time() - kube_cronjob_status_last_schedule_time{namespace="apps", cronjob="renovate"} > 21600
  for: 5m
  labels:
    severity: critical

- alert: RenovateJobFailing
  expr: |
    kube_job_status_failed{namespace="apps", job_name=~"renovate-[0-9]+"} > 0
  for: 10m
  labels:
    severity: warning

RenovateCronJobStale fires when the CronJob hasn’t been scheduled in six hours (three missed runs): suspended, stuck controller, or something cluster-wide blocking scheduling. RenovateJobFailing catches every OOMKill and crash. Neither catches Failure 1: a run that exits 0 without doing anything still passes both rules. That case needs a check on the outcome itself, and I haven’t built it yet.

The Current Configuration

# kubernetes/apps/renovate/renovate.yml (trimmed)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: renovate
  namespace: apps
spec:
  schedule: "0 */2 * * *"  # every 2 hours
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: renovate
              image: ghcr.io/renovatebot/renovate:43.253.2
              env:
                - name: RENOVATE_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: renovate-token
                      key: token
                - name: NODE_OPTIONS
                  value: "--max-old-space-size=4608"
              resources:
                requests:
                  cpu: 100m
                  memory: 1Gi
                limits:
                  cpu: 500m
                  memory: 6Gi

concurrencyPolicy: Forbid keeps a slow run from overlapping the next schedule tick. restartPolicy: Never means every retry is a fresh pod, so each OOMKill shows up as its own failed pod with its own exit code.

The Lesson

Silent CronJob failures are the same “reports success but did nothing” pattern as ArgoCD showing Synced/Healthy on an Application that never applied. The job “succeeds”, nobody checks, and dependency updates stop. Security patches don’t land, and nobody notices until a vulnerability is disclosed for a package Renovate would have bumped.

More memory alone fixed nothing here. Measuring the real peak found the cause, and alerting on the pipeline itself makes the next failure visible. Every CronJob that does something important needs a check that it did the job, not only that it exited.


The same problem exists in Azure DevOps: a pipeline that succeeds but produces nothing looks the same as one that did its work. Azure DevOps pipeline analytics report pass rate and duration, not whether a run produced meaningful output. The fix is the same in both places: add a step that checks for the expected outcome (PRs opened, artifacts published, deployments completed) and alert when it’s missing.

The Phoenix Project* tells the same story from the human side: the automated system everyone assumes is working, until someone finally checks.

Was this helpful?
Share
DW

David Woitzik

Hybrid Cloud Engineer

I work mostly with Azure, Terraform, and Zero-Trust network design, and publish the Terraform templates and write-ups I wanted when I was solving the same problems.

Keep reading

More like this in your inbox

New Terraform modules and articles by email. No spam.