Skip to content
6 min read Updated

Velero Said Backups Succeeded. The Data Was Never There.

Velero reported 'Completed' for weeks with no PVC data for Postgres, Vaultwarden or Paperless. The defaultVolumesToFsBackup fix and how to verify a backup.


Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

Velero reported “Completed” on every daily backup. The schedule ran on time, the S3 uploads succeeded, the backup size looked reasonable. I never verified the contents, because “Completed” means completed. Or so I assumed.

On 2026-06-19, I needed to restore a PVC. The backup existed in Garage S3. The restore completed. The PVC was empty.

Postgres data, Vaultwarden passwords, Paperless documents, Nextcloud files: none of it was there. For weeks Velero had backed up Kubernetes manifests and ignored the data in the PersistentVolumeClaims.

View the complete homelab infrastructure source on GitHub

What Velero Actually Backs Up

Velero has two backup modes for PersistentVolumes:

  1. Snapshot-based (default): takes a volume snapshot through a provider plugin or CSI. Works for cloud disks with snapshot APIs (EBS, Azure Disk). Doesn’t work for NFS, local-path, or any storage without a snapshot provider.

  2. Filesystem-based (--default-volumes-to-fs-backup): the Velero node-agent DaemonSet reads the files of each pod volume from the node and uploads them with Kopia (or Restic). Works with almost any storage, including NFS and local-path, as long as the volume is mounted by a running pod.

My storage is NFS (nfs-client from a Proxmox LXC) and local-path. Neither has a snapshot provider. Without filesystem backup, Velero has no way to capture PVC contents. It can only back up the Kubernetes resource definitions: the PV and PVC objects, not the data inside them.

The default is snapshot-based. If you don’t tell Velero to use filesystem backup, it tries snapshots. If no snapshot provider exists, it takes no snapshot and the backup “succeeds” with only manifests.

The Discovery

The backup schedule was:

# kubernetes/system/velero/schedule.yml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
  namespace: velero
spec:
  schedule: "0 5 * * *"
  template:
    includedNamespaces: ["apps", "vault", "database", "argocd"]
    excludedResources: ["events", "events.events.k8s.io"]
    ttl: 720h

No defaultVolumesToFsBackup. No per-backup volume hooks. No snapshot provider configured. Every backup since the schedule was created had captured only Kubernetes manifests.

The backup size was misleading. Manifests for 25+ applications, with their Services, ConfigMaps, Secrets, and ExternalSecrets, add up to a few MB, which looked plausible enough that I never questioned it.

The Fix

# kubernetes/system/velero/schedule.yml
spec:
  template:
    defaultVolumesToFsBackup: true
    includedNamespaces: ["apps", "vault", "database", "argocd"]

One line. defaultVolumesToFsBackup: true tells Velero to use filesystem backup for every pod volume in every included namespace. The node-agent on each node reads the volume data from the pod’s volume directory on the host and Kopia uploads it to the Garage S3 bucket. That requires the node-agent DaemonSet to be deployed (deployNodeAgent: true in the Helm chart).

After adding the flag, the next backup jumped from ~2 MB to ~4 GB: the actual data.

Verification

velero backup describe is the way to verify that PVC data was captured. Without --details it prints a count per phase; with --details it lists every pod and volume instead:

velero backup describe daily-backup-20260620050012 | grep -A3 "Pod Volume Backups"
#   Pod Volume Backups - kopia (specify --details for more information):
#     Completed:  12

If the section reads Pod Volume Backups: <none included> or shows no Completed line, no filesystem backup happened. The backup “completed” but only captured manifests. On the CLI, describe is where this distinction shows up; velero backup get shows Completed either way.

For automated verification, I added a post-backup check:

# After each backup completes: count the PodVolumeBackup objects
# Velero created for this backup that reached phase Completed
COMPLETED=$(kubectl -n velero get podvolumebackups \
  -l velero.io/backup-name="$BACKUP" -o json | \
  jq '[.items[] | select(.status.phase == "Completed")] | length')
if [ "$COMPLETED" -eq 0 ]; then
  echo "WARNING: No pod volume backups captured"
  # Alert via Discord webhook
fi

The Broader Problem: Backup Circularity

The false-positive backup led to a deeper issue, documented in docs/garage-velero-design-2026-07-17.md: Velero backs up to Garage S3, which runs inside the same cluster. If the cluster dies, Velero and Garage die with it. The backup target is inside the thing being backed up.

That’s classic backup circularity:

Velero (k8s) → Garage S3 (k8s) → same cluster

The backup is only useful for recovering individual PVCs or namespaces within a running cluster. Full cluster recovery needs the backup to exist outside the cluster. That’s what the Cloudflare R2 offsite backup scaffolding is for, and it isn’t active yet.

The secondary issue is “torn state”: Velero’s metadata (manifests) and the PVC filesystem backups are stored separately. If one succeeds and the other fails, you can end up with manifests but no data, or data with no manifests to restore it into. defaultVolumesToFsBackup doesn’t solve torn state, but it at least makes sure both parts of the backup are attempted.

What Would Have Caught This

  1. A restore test. Running velero restore into a test namespace monthly would have caught the empty PVCs in the first cycle.

  2. Backup size monitoring. A 2 MB backup for a cluster with 25+ apps and 4 GB of PVC data is obviously wrong. An alert on backup size below a threshold would have caught it.

  3. Post-backup verification. The velero backup describe --details | grep "Pod Volume Backups" check should run after every scheduled backup, not just when you need a restore.


Azure has the same problem. An Azure Backup job reporting “Completed” tells you a recovery point was created, not that a VM restored from it boots and its application data is consistent. Restoring a test VM is the only way to confirm the backup contains what you think it contains. ISO 27001 (Annex A 8.13) expects backups to be tested regularly, and NIS2 lists backup management under business continuity in Article 21(2)(c), for exactly this reason.

Designing Data-Intensive Applications* is the best book I know on why “the operation returned success” and “the data is there” are separate claims that need separate verification.

Was this helpful?
Share
DW

David Woitzik

Hybrid Cloud Engineer

I work mostly with Azure, Terraform, and Zero-Trust network design, and publish the Terraform templates and write-ups I wanted when I was solving the same problems.

Keep reading

More like this in your inbox

New Terraform modules and articles by email. No spam.