Skip to content
8 min read Updated

Authelia OIDC on Kubernetes: SSO for Proxmox, Grafana, ArgoCD

One Authelia instance on k3s gives 25+ services a single login via OIDC and Traefik ForwardAuth, backed by CNPG Postgres, Redis sessions and secrets in Vault.


Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

Every internal service in my homelab goes through the same authentication gate: Authelia. Proxmox, PBS, Grafana, ArgoCD, Headscale, Uptime Kuma, Paperless, Nextcloud: 25+ web services, one login, one session, one set of access rules. The OIDC provider, the Postgres backend, the session store, and the secrets all run inside k3s, backed by CNPG, Redis, and Vault.

Below is the full implementation: how the pieces fit together, why I made certain design decisions, and the specific bugs that bit me along the way. The first Authelia deployment on this cluster, with its five startup failure modes, is covered in the k3s, Authelia and Longhorn on Proxmox build log; this post is the OIDC setup that grew out of it.

View the complete homelab infrastructure source on GitHub

The Architecture

Authelia runs as a Kubernetes Deployment in the apps namespace, protected by the same default-deny NetworkPolicy that applies to everything else. It has three dependencies:

  1. PostgreSQL: CNPG-managed postgres-authelia cluster in the database namespace
  2. Redis: session store, ephemeral (no persistence needed)
  3. Vault: hmac_secret, OIDC private keys, JWT secrets, session secrets

The Traefik ForwardAuth middleware sits in front of every service. When a request hits Traefik, the middleware sends a verification request to Authelia’s /api/verify endpoint. Authelia checks the session cookie against the access rules and returns 200 if the request is allowed. Otherwise it returns a 302 redirect to the login portal for browsers, or a 401 for requests that can’t follow one. (Since Authelia 4.38, /api/verify is the legacy endpoint; new setups should use /api/authz/forward-auth.)

# kubernetes/apps/authelia/middleware.yml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: authelia
  namespace: apps
spec:
  forwardAuth:
    address: http://authelia.apps.svc.cluster.local:9091/api/verify?rd=https://auth.woitzik.dev/
    trustForwardHeader: true
    authResponseHeaders:
      - Remote-User
      - Remote-Groups
      - Remote-Name
      - Remote-Email

Every IngressRoute that needs protection adds middlewares: [{name: authelia}]. Services that need API-level protection (not browser-based) use OIDC client credentials instead.

The OIDC Configuration

Authelia acts as an OIDC provider for services that support it. The ConfigMap defines seven OIDC clients; these are five of them:

# kubernetes/apps/authelia/configmap.yml
identity_providers:
  oidc:
    clients:
      - client_id: 'proxmox'
        client_name: 'Proxmox VE'
        client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'  # hash only
        public: false
        authorization_policy: two_factor
        redirect_uris:
          - 'https://pve.woitzik.dev'
          - 'https://pve.woitzik.dev/api2/extjs/access/oidc/login'
        scopes: [openid, profile, email, groups]
        token_endpoint_auth_method: 'client_secret_basic'
      - client_id: 'pbs'
        client_name: 'Proxmox Backup Server'
        client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
        public: false
        authorization_policy: two_factor
        redirect_uris:
          - 'https://backup.woitzik.dev'
          - 'https://backup.woitzik.dev/api2/extjs/access/oidc/login'
        scopes: [openid, profile, email, groups]
        token_endpoint_auth_method: 'client_secret_basic'
      - client_id: 'argocd'
        client_name: 'Argo CD'
        client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
        public: false
        authorization_policy: two_factor
        redirect_uris:
          - 'https://argo.woitzik.dev/auth/callback'
        scopes: [openid, profile, email, groups]
        token_endpoint_auth_method: 'client_secret_basic'
      - client_id: 'grafana'
        client_name: 'Grafana'
        client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
        public: false
        authorization_policy: two_factor
        redirect_uris:
          - 'https://monitoring.woitzik.dev/login/generic_oauth'
        scopes: [openid, profile, email, groups]
        token_endpoint_auth_method: 'client_secret_basic'
      - client_id: 'headscale'
        client_name: 'Headscale'
        client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
        public: false
        authorization_policy: two_factor
        redirect_uris:
          - 'https://headscale.woitzik.dev/oidc/callback'
        scopes: [openid, profile, email]
        token_endpoint_auth_method: 'client_secret_basic'

The redirect URIs have to match what each service actually sends, character for character. Proxmox VE and PBS redirect back to their web UI origin (https://pve.woitzik.dev, https://backup.woitzik.dev), not to a dedicated callback path. Headscale’s callback path is /oidc/callback, and Grafana’s generic OAuth integration uses /login/generic_oauth. The client_secret values are argon2id hashes: Authelia only needs the hash to verify the secret, and the plaintext lives in each service’s own OIDC settings, so the hash can sit in git.

Key detail: every client uses client_secret_basic, including ArgoCD. I originally had ArgoCD on client_secret_post, assuming it sent the secret in the POST body. It doesn’t. ArgoCD’s built-in OIDC client (not Dex) sends client credentials via HTTP Basic auth, and Authelia rejects a client that authenticates with a method other than the configured one with invalid_client. Setting token_endpoint_auth_method: client_secret_basic on the ArgoCD client fixed the login.

Secrets in Vault

The OIDC hmac_secret, signing keys, and session secrets were originally committed as plain Kubernetes Secrets. This is fine for a homelab, but it means anyone with kubectl get secret can read them.

The migration to Vault:

# kubernetes/apps/authelia/external-secret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: authelia-secrets
  namespace: apps
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: authelia-secrets
    creationPolicy: Merge
  data:
    - secretKey: hmac-secret
      remoteRef:
        key: secret/authelia
        property: hmac-secret
    - secretKey: oidc-issuer-private-key
      remoteRef:
        key: secret/authelia
        property: oidc-issuer-private-key
    - secretKey: jwt-secret
      remoteRef:
        key: secret/authelia
        property: jwt-secret
    - secretKey: session-secret
      remoteRef:
        key: secret/authelia
        property: session-secret
    - secretKey: redis-password
      remoteRef:
        key: secret/authelia
        property: redis-password

creationPolicy: Merge means the External Secrets Operator doesn’t own the Kubernetes Secret. It writes its keys into an existing Secret on each refresh and leaves any other keys alone; it does not create the Secret if it’s missing. That made it the safe migration path: the old Secret stayed in place and ESO took over its keys one by one. If Vault is sealed or unreachable, the existing Secret persists, and Authelia keeps running on the last synced values.

The users database (username, argon2id hash, group memberships) was also migrated from a plain Secret to an ExternalSecret sourced from Vault. It uses the same creationPolicy: Merge approach. Like the other secrets, which are mounted under /config/secrets and pulled into the config with Authelia’s secret template filter, it is mounted as a file (under /config/users), and the Authelia config references it by path.

The Postgres Backend

CNPG manages the postgres-authelia single-instance cluster in the database namespace. WAL archiving goes to Garage S3 for point-in-time recovery:

# kubernetes/system/postgres/cluster.yml (excerpt)
postgresql:
  parameters:
    max_connections: "100"
    shared_buffers: "64MB"
backup:
  retentionPolicy: "7d"
  barmanObjectStore:
    destinationPath: s3://cnpg-backups/authelia
    endpointURL: http://garage.apps.svc.cluster.local:3900
    wal:
      compression: gzip
    data:
      compression: gzip

A daily ScheduledBackup creates a full base backup. WAL archiving plus daily base backups lets me restore to any point in time within the retention window.

The Authelia Schema Bug

One gotcha: Authelia’s Postgres schema version can get ahead of the running image. Once a newer Authelia image has migrated the database, an older image running against it hits a schema mismatch, for example when a Renovate bump is reverted or a tag is pinned back.

The fix was to bump the image version to match the schema:

image: ghcr.io/authelia/authelia:4.39.20  # matches DB schema v24

But the symptom was confusing: Authelia reported healthy in ArgoCD, the pods were running, but login attempts returned 500 errors. The health check endpoint (/api/health) only reports that Authelia’s HTTP server is answering. It doesn’t check database schema compatibility.

Access Control Rules

The access_control section defines who can access what:

access_control:
  rules:
    - domain: auth.woitzik.dev
      policy: bypass
    - domain: "*.woitzik.dev"
      policy: two_factor

Critical detail: auth.woitzik.dev must be the first rule. Authelia matches rules top-down and stops at the first match. If the wildcard *.woitzik.dev → two_factor rule comes before the auth.woitzik.dev → bypass rule, Authelia redirects to itself in an infinite loop.

I hit this bug during initial setup. The logs showed repeated 302 redirects from auth.woitzik.dev back to itself. That looks like a broken redirect URL or cookie domain, but it was the rule order.

The Two-Replica Setup

Authelia runs at two replicas with a PodDisruptionBudget (minAvailable: 1). Redis handles session state, so either replica can serve any session. Both replicas must share the same hmac_secret and JWT signing keys, and they do, since both read from the same Vault-backed ExternalSecret.

# kubernetes/apps/authelia/authelia.yml
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: authelia
          livenessProbe:
            httpGet:
              path: /api/health
              port: 9091
          readinessProbe:
            httpGet:
              path: /api/health
              port: 9091

Both probes hit /api/health, which, as the schema bug showed, only confirms that the process is serving. Blackbox Exporter probes the same endpoint from outside, through Traefik, so it also catches failures on the path users take (ingress, TLS, routing) that an in-pod probe never sees.

What I’d Change

Two things, if starting over:

  1. Start with Vault from day one. Migrating from plain Kubernetes Secrets to Vault-backed ExternalSecrets after the fact required the creationPolicy: Merge dance, which wouldn’t have been necessary if the secrets had never been in git.

  2. Check each client’s token endpoint auth method against its source, not its docs page. The ArgoCD invalid_client failure cost more time than any other client, and the fix was one line once I knew what ArgoCD actually sends.


SSO across 25+ services is the same problem in enterprise Azure: Entra ID is the OIDC provider, app registrations replace the Authelia client definitions, and Conditional Access policies replace the access_control rules. The scale is different (Entra ID handles millions of identities), but the pattern is identical: centralize authentication, put every service behind one gate, and keep the access rules in one place.

Most of the debugging above happened at a shell prompt, grepping logs and testing OIDC flows with curl, not in a dashboard. The Linux Command Line* is still the book I’d hand someone who wants to get faster at that part.

Was this helpful?
Share
DW

David Woitzik

Hybrid Cloud Engineer

I work mostly with Azure, Terraform, and Zero-Trust network design, and publish the Terraform templates and write-ups I wanted when I was solving the same problems.

Keep reading

More like this in your inbox

New Terraform modules and articles by email. No spam.