Velero’s nightly backup was moving about 51GB, and a single volume took 33.4GB and 207 seconds of that. Backup size alone is unremarkable for a cluster with 15+ PVCs; one volume accounting for 65% of the bytes is not. The nfs-provisioner’s root-mount volume was backing up the entire shared NFS export as a single blob, even though every app’s individual PVC (a subdirectory of the same tree) was already backed up separately.
The fix was a one-line annotation. The investigation almost went wrong, because my first hypothesis pointed at the wrong disk entirely.
View the complete homelab infrastructure source on GitHub
The Symptom
Velero’s backup stats showed a disproportionate volume:
PodVolumeBackup: nfs-provisioner-root
Namespace: nfs-provisioner
Size: 33.4GB
Duration: 207s
Total backup for that run: ~51GB. This single volume was 65% of the total. Every other PVC was 1-4GB.
The Investigation
The nfs-provisioner pod mounts the entire shared NFS export at /persistentvolumes. Every app’s PVC is a subdirectory of this tree. With defaultVolumesToFsBackup: true, Velero backs up every volume a pod mounts, including the root mount that contains all the subdirectories.
Each app’s own pod also mounts its PVC subdirectory separately. Velero backs that up too. So the same data is backed up twice: once as part of the root mount, once as the individual PVC.
The First Hypothesis (Wrong)
My first guess was garage-data/immich-library, Immich’s photo library, which lives on separate USB-attached disks (Seagate Portable 2TB External HDD*). Excluding it would have reduced backup size, but the USB disks aren’t on the contended rpool. The bottleneck was the SSD, not USB, so excluding the Immich volume would have been the wrong fix for the right symptom.
The Right Fix
I verified the target before acting:
# Confirm the root-mount volume targets rpool (the contended disk)
findmnt /persistentvolumes
# /dev/zvol/rpool/nfs ← confirmed rpool
# Confirm the individual PVCs are subdirectories of the same tree
ls /persistentvolumes/
# default-elasticsearch-data-0/
# default-immich-redis-data-0/
# default-minecraft-data/
# ... (each is a separate PVC)
The root mount is the entire NFS export on rpool, and every individual PVC is a subdirectory of it. Backing up the root mount backs up every PVC a second time.
The Exclusion
# kubernetes/system/nfs-provisioner/application.yml
podAnnotations:
backup.velero.io/backup-volumes-excludes: nfs-subdir-external-provisioner-root
One annotation. Velero skips the root-mount volume entirely. Every app’s individual PVC still gets backed up through its own pod’s mount.
Result
Before: 51GB total, 33.4GB from root mount (65%)
After: 17.6GB total, 0GB from root mount
Backup duration dropped from 207s to ~60s. Nothing lost coverage: every PVC is still backed up individually.
The Pattern
This trap applies to any NFS provisioner that mounts the export root in its own pod:
- NFS provisioner pod mounts the entire export root
- App pods mount individual PVC subdirectories
- Velero backs up both, the root mount AND the individual PVCs
- The root mount is redundant because it contains everything the individual PVCs contain
The fix is to exclude the root mount via backup.velero.io/backup-volumes-excludes on the provisioner pod.
How to Find Redundant Backups
# List all PodVolumeBackups and their sizes
kubectl get podvolumebackups -A -o json | \
jq -r '.items[] | "\(.spec.volume) \(.status.progress.totalBytes // 0) \(.metadata.namespace)"' | \
sort -k2 -rn | head -10
If one volume is disproportionately large and its data is already covered by individual PVC backups, it’s redundant.
Velero’s defaultVolumesToFsBackup: true is convenient but dangerous with NFS provisioners. The root mount contains everything, and Velero doesn’t know it’s redundant. Check your PodVolumeBackup stats. If one volume is 65% of your total, you’re probably backing up the same data twice.