Longhorn vs NFS on a Single-Host k3s Cluster: Why I Switched

Longhorn is designed for multi-node Kubernetes clusters. It replicates PersistentVolume data across nodes, so if one node dies, the data survives on the others. On real multi-host hardware it does that job well.

I run three k3s nodes on a single physical host. All three nodes share the same NVMe. When Longhorn replicates a PVC across three nodes, all three replicas live on the same physical disk, so there is no real distribution and no real redundancy. And when a node failed, block-volume failover created a problem I would not have had without it.

View the complete homelab infrastructure source on GitHub

The Multi-Attach Error

Longhorn volumes are ReadWriteOnce (RWO) block devices: Kubernetes attaches each one to exactly one node at a time. When a k3s node becomes unreachable (OOMKill, kubelet crash, network blip), that attachment does not move on its own.

The sequence:

1. k3s-12 becomes unreachable (its kubelet stops renewing its node lease)
2. The node goes NotReady, and the pod on it is stuck in Terminating because nobody can confirm it stopped
3. The controller schedules a replacement pod on k3s-13
4. The volume’s VolumeAttachment still points at k3s-12
5. The attach/detach controller refuses a second attach for an RWO volume and reports a Multi-Attach error on the new pod
6. The new pod sits in ContainerCreating until the old attachment is cleared

The pod that needs the volume (Postgres, for example) can’t start anywhere until the old attachment is gone. Clearing it meant force-deleting the stuck pod and detaching the volume by hand so Longhorn could re-attach it cleanly: a manual step during an outage, exactly when automation should be working.

On a multi-host cluster, this is a real HA scenario: the volume legitimately needs to fail over to a replica on a different physical disk, and the wait protects against two nodes writing at once. On a single-host cluster, the “failover” is to the same NVMe, and the Multi-Attach wait is pure overhead.

The NFS Alternative

NFS (Network File System) doesn’t have a Multi-Attach problem because it isn’t block storage. NFS exports a directory over the network, and any number of clients can mount it at the same time. There’s no attachment state to conflict.

The NFS server runs on a dedicated LXC (ct-srv-nfs-01) with a ZFS-backed dataset:

# terraform/stacks/proxmox/lxc.tf
resource "proxmox_virtual_environment_container" "ct_srv_nfs_01" {
  vm_id = 220

  initialization {
    hostname = "ct-srv-nfs-01"
    # ...
  }

  memory {
    dedicated = 2048
    swap      = 1024
  }

  # host bind mount holding every NFS-exported PVC
  mount_point {
    volume = "/nfs-data"
    path   = "/nfs-data"
  }
  # ...
}

The k3s NFS provisioner creates PVCs on this NFS server:

# kubernetes/system/nfs-provisioner/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      values:
        nfs:
          server: 10.0.20.100
          path: /nfs-data/k8s-general
          mountOptions:
            - nfsvers=4.1
            - hard
        storageClass:
          name: nfs-client
          defaultClass: false
          reclaimPolicy: Retain

When k3s-12 goes down, the pods reschedule to k3s-11 or k3s-13, mount the same NFS export, and continue where they left off. No Multi-Attach error, no manual detachment, no attachment state machine.

The Trade-offs

What NFS Gives Up

NFS gives up replication and node-failure tolerance. If the NFS server dies, every PVC it serves is unavailable. On a single physical host that is the same risk as with Longhorn, because both storage systems live on the same NVMe. Longhorn’s replication doesn’t help when the underlying disk is the single point of failure.

What NFS Gives Back

• No Multi-Attach errors. Pods reschedule freely without volume attachment conflicts.
• Simpler failure modes. NFS is either up or down. No “rebuilding,” “degraded,” or “faulted” volume states.
• ZFS snapshots for backup. PBS backs up the NFS server’s ZFS dataset, giving point-in-time recovery without Longhorn’s snapshot overhead.
• Lower resource usage. Longhorn runs a manager, a CSI plugin, and instance-manager pods on every node. NFS is one server on one LXC.

The Storage Decision Matrix

After migrating from Longhorn to NFS, I defined the rule:

The principle: embedded databases go on local-path (file-locking), everything else goes on NFS (survivability). PostgreSQL also lands on local-path, for a different reason: CNPG handles replication itself, so the storage underneath only needs to be local and fast.

The Migration

Moving PVCs from Longhorn to NFS required:

1. Deploy the NFS server LXC with the correct ZFS dataset
2. Install the nfs-subdir-external-provisioner Helm chart
3. Copy data from Longhorn volumes to NFS (using kubectl cp or a temporary pod)
4. Update each application’s storageClassName from longhorn to nfs-client
5. Delete the old Longhorn PVC (data is already on NFS)
6. Remove Longhorn from the cluster

Step 5 is the nerve-wracking part: deleting a PVC that contains production data. The verification before deletion:

# Verify data exists on NFS
kubectl exec -it temp-pod -- ls -la /mnt/nfs/authelia-data/
# → Confirm database files, config, etc.

# Verify application starts with NFS PVC
kubectl delete pod authelia-xxxxx  # force reschedule
# → Pod starts, mounts NFS, passes health checks

After the migration, Longhorn was uninstalled. The cluster went from three storage replicas on one disk to a single NFS server on the same disk. That is simpler, more predictable, and free of the Multi-Attach stalls.

Distributed storage on a single host is the same anti-pattern as replicating data across “zones” that share one power feed. Azure availability zones exist precisely because each has independent power, cooling, and networking; replication only buys resilience when the failure domains underneath are independent. If the underlying infrastructure isn’t actually distributed, the replication layer adds complexity without adding resilience. Match the storage topology to the real infrastructure topology: single host, single NFS server; multiple hosts, distributed storage.

Designing Data-Intensive Applications* has the clearest explanation I’ve read of why replication only buys you resilience when the replicas fail independently. That is the whole lesson of this migration.