PostgreSQL major version upgrades are not swap-the-image operations. When Renovate opened a PR bumping postgres:16.14-alpine to postgres:18.4-alpine for my Nextcloud and Paperless databases, merging it as-is would have crash-looped both: the on-disk format of the data directory changes between major versions, and PG18 refuses to start against a PG16 data directory.
Both are plain StatefulSet Postgres instances (postgres-nextcloud and postgres-paperless in the apps namespace), not operator-managed. My Authelia database runs on a CloudNativePG cluster with no pinned image tag, so Renovate never touched it. The migration took a fresh PVC per app, a dump and restore, a mount-path change the PG18 image enforces, and one missing role that kept Nextcloud from logging in.
View the complete homelab infrastructure source on GitHub
Why You Can’t Swap the Image
PostgreSQL stores data in a format specific to its major version. The PG_VERSION file in the data directory records which version created it:
cat /var/lib/postgresql/data/PG_VERSION
# → 16
Point a PG18 server at that directory and it stops at startup:
FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 18.4.
pg_upgrade is the official tool for in-place major upgrades. It needs the old and the new binaries side by side, and the official postgres:*-alpine images ship exactly one major version. For a database on a 5 GiB volume the simpler path is a logical dump from PG16 and a restore into an empty PG18 instance.
The Migration Procedure
I ran Nextcloud first, then Paperless. Same steps for both.
Step 1: Stop Writes and Dump
# Nextcloud: stop writes during the dump (Paperless: scale its Deployment to 0)
kubectl exec -n apps deploy/nextcloud -- \
su -s /bin/sh www-data -c "php occ maintenance:mode --on"
# Plain SQL dump, streamed straight to the workstation
kubectl exec -n apps postgres-nextcloud-0 -- \
pg_dump -U nextcloud nextcloud > nextcloud-pre18.sql
The dump contains schema and data of that one database. It does not contain roles: roles are cluster-wide objects, and only pg_dumpall --globals-only exports them. That detail comes back below.
As a second recovery path I took a Velero on-demand backup (nextcloud-pg-pre18) before changing anything:
velero backup create nextcloud-pg-pre18 \
--include-namespaces apps \
--selector app=postgres-nextcloud
Step 2: Fresh PVC, New Image
The StatefulSet gets a new, empty PVC instead of the old one. The old PVC stays declared (unused) so ArgoCD’s prune doesn’t delete it, which keeps it as the rollback path:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nextcloud-db-data-pg18
namespace: apps
spec:
accessModes: [ReadWriteOnce]
storageClassName: nfs-client
resources:
requests:
storage: 5Gi
In the StatefulSet, the image moves to postgres:18.4-alpine and the volume to claimName: nextcloud-db-data-pg18. PG18 never sees the PG16 files, not even by accident.
Step 3: The PG18 Mount Layout
The first PG18 pod still refused to start, on an empty volume. The PG16 manifest mounted the PVC at /var/lib/postgresql/data with subPath: data. Since 18, the official image expects a single mount at /var/lib/postgresql and puts the data in a version-specific subdirectory itself (docker-library/postgres#1259). A mount at the old path triggers this from the entrypoint:
Error: in 18+, these Docker images are configured to store database data in a
format which is compatible with "pg_ctlcluster" (specifically, using
major-version-specific directory names).
...
Counter to that, there appears to be PostgreSQL data in:
/var/lib/postgresql/data (unused mount/volume)
The fix is one mount, no subPath:
volumeMounts:
- name: data
mountPath: /var/lib/postgresql
For Paperless I made both changes in one pass instead of hitting the mount error twice.
Step 4: Restore
Once the PG18 pod is Ready with an empty nextcloud database (created from POSTGRES_DB), feed the dump back in:
kubectl exec -i -n apps postgres-nextcloud-0 -- \
psql -U nextcloud -d nextcloud < nextcloud-pre18.sql
No format conversion happens here. A logical dump is SQL, independent of the on-disk format, and the PG18 server writes it in its own format as it loads it.
The Missing Role
The restore printed 230 errors. All of them were ALTER ... OWNER TO oc_dw and GRANT ... TO oc_dw statements failing with:
ERROR: role "oc_dw" does not exist
The data itself was fine: CREATE TABLE and COPY ran as the connecting nextcloud role. But Nextcloud couldn’t authenticate afterwards. Its installer had created its own lower-privilege role, oc_ plus the admin username, and config.php’s dbuser/dbpassword point at that role, not at POSTGRES_USER. The fresh PG18 instance only had the nextcloud role from the env var.
The fix was to create oc_dw with the password from config.php and grant it full privileges on the database:
CREATE ROLE oc_dw WITH LOGIN PASSWORD '<dbpassword from config.php>';
GRANT ALL PRIVILEGES ON DATABASE nextcloud TO oc_dw;
\c nextcloud
GRANT ALL ON SCHEMA public TO oc_dw;
GRANT ALL ON ALL TABLES IN SCHEMA public TO oc_dw;
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO oc_dw;
LOGIN is all it needs. It’s an application role, so no REPLICATION or superuser attributes. Creating it before the restore is the cleaner order: then the ALTER OWNER statements succeed and ownership matches the old cluster. occ config:system:get dbuser tells you the role name up front. Paperless had no such mismatch (PAPERLESS_DBUSER and POSTGRES_USER are both paperless).
After that, occ maintenance:mode --off, then occ status and occ user:list on Nextcloud. On Paperless I checked row counts in django_migrations and documents_document and the celery logs of a fresh pod.
Verify the Dump Before Anything Else
The old PVC is the only copy until the restore is verified, so check the dump before switching. For a plain SQL dump that means looking at it:
# A complete pg_dump ends with this line; a truncated one doesn't
tail -n 3 nextcloud-pre18.sql
# → -- PostgreSQL database dump complete
grep -c '^CREATE TABLE' nextcloud-pre18.sql
For custom-format dumps (pg_dump -Fc), pg_restore --list reads only the table of contents. pg_restore -f /dev/null dump.file reads the whole archive and catches truncation. There is no --dry-run flag.
Rollback and Cleanup
Rollback stayed trivial the whole time: point the StatefulSet back at the old PVC and revert the image tag. Nothing was deleted until both apps had run cleanly on PG18 for more than a day. Then the old PG16 PVCs came out of the manifests, and Velero’s daily backup covers recovery from there.
What I’d Change
- Export globals first.
pg_dumpall --globals-onlynext to the per-database dump would have carriedoc_dwover and avoided both the 230 errors and the failed login. - Read the image changelog for majors. The PG18 mount layout change is documented upstream. I found it from a crash-looping pod instead.
Azure Database for PostgreSQL Flexible Server has the same constraint. Its in-place major version upgrade runs pg_upgrade behind a single API call, and the alternative is a dump-and-restore migration to a new server. The managed service hides the mechanics, but the rule is identical: major versions are not compatible at the storage layer.
The Linux Command Line* is worth having on the shelf for exactly this kind of migration - pg_dump, pg_restore, kubectl cp, and a dozen other shell tools chained together under time pressure go a lot smoother when the shell itself isn’t also something you’re learning in the moment.