Skip to content
8 min read Updated

Enterprise Homelab: k3s, Authelia & Longhorn on Proxmox with Terraform

Building a Kubernetes homelab with k3s, Authelia SSO, Longhorn and ArgoCD on Proxmox, plus the five mistakes that cost me hours along the way.


Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

Update (2026-09-09): this cluster has since moved persistent storage off Longhorn onto NFS. All three k3s nodes here run as VMs on a single physical host, so Longhorn’s replica-distribution guarantee was never real: every “replica” landed on the same underlying NVMe. When a node dropped out briefly, Longhorn’s RWO failover logic tried to attach the volume on a second node while the first was still recovering. The result was a Multi-Attach error that needed a manual detach to clear, overhead a single-host cluster doesn’t need to carry. NFS has no attachment state to conflict, so that failure mode is gone. Mistake 2 below (Longhorn + iSCSI) and the Longhorn Application manifest stay as they were. The troubleshooting is accurate for anyone running Longhorn on physically separate nodes; it’s just not what this cluster runs anymore.

Most Kubernetes homelab guides stop at kubectl get pods. This one covers the full stack: three k3s nodes provisioned via Terraform on Proxmox VE, running on a BMAX Mini PC* with 64GB RAM, GitOps-managed with ArgoCD, persistent storage via Longhorn, and Authelia as the SSO gateway in front of every service.

It also covers the mistakes I made along the way, because that’s the part most guides leave out.

View the complete homelab infrastructure source on GitHub

The Stack

Proxmox (Bare Metal)
└── Terraform (proxmox provider)
    ├── vm-srv-k3s-11 (Master,  10.0.20.11, VLAN 20)
    ├── vm-srv-k3s-12 (Worker,  10.0.20.12, VLAN 20)
    └── vm-srv-k3s-13 (Worker,  10.0.20.13, VLAN 20)
        │
        └── K3s Cluster
            ├── ArgoCD         (GitOps controller)
            ├── Traefik        (Ingress + TLS termination)
            ├── cert-manager   (Wildcard cert via Let's Encrypt)
            ├── MetalLB        (Bare-metal LoadBalancer)
            ├── Longhorn       (Distributed block storage)
            ├── Authelia       (SSO + 2FA gateway)
            └── Vaultwarden    (Self-hosted Bitwarden)

Everything is managed as code. The VMs are Terraform resources. The cluster applications are ArgoCD Applications pointing at a Git repository. No manual helm install, no imperative kubectl apply in production.

Provisioning the Nodes with Terraform

Each k3s node is a full VM clone from a template (VM ID 9000) on Proxmox, provisioned via the proxmox_virtual_environment_vm provider:

resource "proxmox_virtual_environment_vm" "vm_srv_k3s_11_master" {
  vm_id     = 211
  name      = "vm-srv-k3s-11"
  node_name = local.target_node
  tags      = ["k3s", "master", "kubernetes"]

  clone {
    vm_id = 9000
    full  = true
  }

  cpu    { cores = 4; type = "host" }
  memory { dedicated = 8192 }

  disk {
    datastore_id = local.storage
    interface    = "scsi0"
    size         = 40
    file_format  = "raw"
  }

  network_device {
    bridge  = "vmbr0"
    vlan_id = 20  # Dedicated server VLAN
  }

  initialization {
    ip_config {
      ipv4 { address = "10.0.20.11/24"; gateway = "10.0.20.1" }
    }
    dns { servers = ["10.0.20.5"] }
    user_account {
      username = "dw"
      keys     = ["ssh-ed25519 ..."]
    }
  }
}

cpu.type = "host" passes the host CPU flags through to the guest, so workloads can use AES-NI, AVX and other extensions that the generic kvm64 model hides. Don’t leave it on kvm64 for real workloads.

The two worker definitions follow the same pattern with IPs .12 and .13.

Mistake 1: Docker Hub Rate Limits

The first thing that breaks on a fresh k3s cluster is Docker Hub’s rate limit.

Pods start showing ErrImagePull or ImagePullBackOff. The images exist; Docker Hub has throttled anonymous pulls. In a homelab where you tear down and rebuild constantly, you hit the limit fast.

The fix is to pull the affected images from a different registry:

  • Bitnami images (Postgres, Redis) → public.ecr.aws/bitnami/... (Amazon ECR Public, with much higher anonymous limits)
  • Authelia → ghcr.io/authelia/authelia:latest (GitHub Container Registry)

Every k3s getting-started guide should mention this. Few do.

Mistake 2: Longhorn & iSCSI on WSL

Longhorn requires the iSCSI protocol on the host to mount virtual block devices into containers. On a standard WSL Ubuntu installation, the iSCSI daemon is missing.

Symptom: pods stuck in ContainerCreating forever. Longhorn volumes stay Detached or report volume is not ready for workloads.

Fix: run this on every node (including the WSL host if applicable):

sudo apt-get install -y open-iscsi
sudo systemctl enable iscsid
sudo systemctl start iscsid

Without it, Longhorn cannot attach its virtual disks to the nodes, and the error messages are vague enough that you can spend hours debugging the wrong thing.

The ArgoCD Application for Longhorn itself is straightforward once iSCSI is working:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: longhorn
  namespace: argocd
spec:
  source:
    repoURL: https://charts.longhorn.io
    targetRevision: 1.6.1
    chart: longhorn
    helm:
      values: |
        preUpgradeChecker:
          jobEnabled: false
  destination:
    server: https://kubernetes.default.svc
    namespace: longhorn-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

preUpgradeChecker.jobEnabled: false disables the pre-upgrade check job that fires on every ArgoCD sync and clutters your logs.

Traefik: TLS Termination at the Edge

Traefik runs in kube-system managed by ArgoCD, with HTTP-to-HTTPS redirect enforced at the ingress level and a wildcard certificate from cert-manager as the default TLS store:

source:
  repoURL: https://helm.traefik.io/traefik
  targetRevision: 27.0.2
  chart: traefik
  helm:
    values: |
      ports:
        web:
          redirectTo:
            port: websecure
        websecure:
          tls:
            enabled: true
      ingressRoute:
        dashboard:
          enabled: false
      tlsStore:
        default:
          defaultCertificate:
            secretName: wildcard-woitzik-dev-tls

The Traefik dashboard is disabled because it exposes too much to leave on in a production-adjacent setup. Use kubectl port-forward when you need it.

Mistake 3: Authelia’s Five Failure Modes

Authelia is the most opinionated component in this stack. It fails fast on configuration errors, which is good, but the error messages aren’t always obvious. This section is about getting the first deployment running. For the later setup, where Authelia is the OIDC provider for Proxmox VE, PBS, Grafana and ArgoCD, see Authelia OIDC on Kubernetes: SSO for 25+ services.

The working Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: authelia
  namespace: apps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: authelia
  template:
    metadata:
      labels:
        app: authelia
    spec:
      enableServiceLinks: false  # Critical - see Failure Mode 1
      containers:
      - name: authelia
        image: ghcr.io/authelia/authelia:latest
        ports:
        - containerPort: 9091
        volumeMounts:
        - name: config
          mountPath: /config
        - name: secrets
          mountPath: /config/secrets
          readOnly: true
        env:
        - name: AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE
          value: /config/secrets/storage-password
        - name: AUTHELIA_SESSION_REDIS_PASSWORD_FILE
          value: /config/secrets/redis-password
        - name: AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRET_FILE
          value: /config/secrets/jwt-secret
        - name: AUTHELIA_SESSION_SECRET_FILE
          value: /config/secrets/session-secret
        - name: AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE
          value: /config/secrets/storage-key
      volumes:
      - name: config
        configMap:
          name: authelia-config
      - name: secrets
        secret:
          secretName: authelia-secrets

Failure Mode 1: enableServiceLinks: false

Kubernetes injects environment variables for every Service in the namespace. A Service named authelia produces AUTHELIA_PORT, AUTHELIA_PORT_9091_TCP and similar. Authelia reads every AUTHELIA_-prefixed variable as configuration, so these collide with its own keys and cause a fatal startup error. enableServiceLinks: false turns the injection off.

Failure Mode 2: The Read-Only Filesystem

The filesystem notifier needs a writable path for its notification file (where password reset and verification messages land instead of being emailed). /config is mounted from a ConfigMap, and ConfigMap volumes are read-only in Kubernetes.

Wrong:

notifier:
  filesystem:
    filename: '/config/notification.txt'  # ConfigMap = read-only = crash

Correct:

notifier:
  filesystem:
    filename: '/tmp/notification.txt'  # /tmp is always writable in containers

Failure Mode 3: Backend DNS Names

Authelia connects to Postgres and Redis via Kubernetes internal DNS. Across namespaces, use the full service name:

session:
  redis:
    host: 'redis-authelia.database.svc.cluster.local'
    port: 6379

storage:
  postgres:
    address: 'tcp://postgres-authelia.database.svc.cluster.local:5432'
    database: 'authelia'
    username: 'authelia'

Short names like redis-authelia only resolve within the same namespace. Authelia lives in apps and the databases in database, so it needs at least redis-authelia.database; the fully qualified name avoids any ambiguity.

Failure Mode 4: YAML Corruption via Terminal Paste

Large YAML blocks pasted into a terminal via cat <<EOF can get truncated or mangled without any warning. Authelia then crashes with a fatal parse error. It looks like a config bug, but it’s a paste artifact.

Fix: write the file in an editor (nano is fine) or build the ConfigMap with kubectl create configmap --from-file=.... Don’t trust terminal paste for configs that run to hundreds of lines.

Failure Mode 5: The server.address Key

In current Authelia versions, the server bind address is configured as:

server:
  address: 'tcp://0.0.0.0:9091/'

Older guides set server.host and server.port separately. Those keys are deprecated since Authelia 4.38. Current 4.x releases still map them automatically and log a warning, but they are scheduled for removal, and a major version bump disables the automatic migration. If you copy config from an older guide, check every key name against the current Authelia documentation.

The same release moved session.domain and the top-level default_redirection_url into a session.cookies list, one entry per cookie domain, each with its own authelia_url. The config below uses the new layout. default_redirection_url must not be the same URL as authelia_url.

Secrets are values, not paths. Writing jwt_secret: '/config/secrets/jwt-secret' does not read the file. Authelia takes the string literally, so the path itself becomes the secret. Secrets mounted as files go in through the matching _FILE environment variables (AUTHELIA_SESSION_SECRET_FILE, AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE, AUTHELIA_IDENTITY_VALIDATION_RESET_PASSWORD_JWT_SECRET_FILE), as in the Deployment above. The alternative is the template config filter (X_AUTHELIA_CONFIG_FILTERS=template) with {{ secret "/config/secrets/jwt-secret" }} inside the YAML, which is what my repo uses today.

The Full Authelia Configuration

server:
  address: 'tcp://0.0.0.0:9091/'

log:
  level: 'debug'

authentication_backend:
  file:
    path: '/config/users_database.yml'

session:
  name: 'authelia_session'
  same_site: 'lax'
  expiration: '1h'
  inactivity: '5m'
  remember_me: '1M'
  cookies:
    - domain: 'yourdomain.com'
      authelia_url: 'https://auth.yourdomain.com'
      default_redirection_url: 'https://www.yourdomain.com'
  redis:
    host: 'redis-authelia.database.svc.cluster.local'
    port: 6379
    database_index: 0

storage:
  postgres:
    address: 'tcp://postgres-authelia.database.svc.cluster.local:5432'
    database: 'authelia'
    username: 'authelia'

notifier:
  filesystem:
    filename: '/tmp/notification.txt'

access_control:
  default_policy: 'one_factor'
  rules:
    - domain: 'auth.yourdomain.com'
      policy: 'bypass'

The Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: authelia-ingress
  namespace: apps
spec:
  ingressClassName: traefik
  rules:
  - host: auth.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: authelia
            port:
              number: 9091

The Result

Once Docker Hub rate limits, iSCSI, service link injection, read-only ConfigMaps and deprecated config keys were sorted out, the stack ran cleanly:

  • Every service behind Authelia SSO with Redis-backed sessions
  • Persistent storage via Longhorn across three nodes (since replaced by NFS, see the update at the top)
  • GitOps-managed via ArgoCD: every change is a Git commit
  • Wildcard TLS via cert-manager and Traefik
  • No manual kubectl apply in steady state

For remote access I later added Headscale as a self-hosted Tailscale control plane. It uses the same Authelia instance for OIDC, so VPN access and service SSO share one identity layer.

The whole infrastructure, from bare metal to running pods, is reproducible from a terraform apply and a Git repository. If you work with network isolation and identity in a corporate Azure tenant, the same principles apply there with different primitives.

Was this helpful?
Share
DW

David Woitzik

Hybrid Cloud Engineer

I work mostly with Azure, Terraform, and Zero-Trust network design, and publish the Terraform templates and write-ups I wanted when I was solving the same problems.

Keep reading

More like this in your inbox

New Terraform modules and articles by email. No spam.