Every internal service in my homelab goes through the same authentication gate: Authelia. Proxmox, PBS, Grafana, ArgoCD, Headscale, Uptime Kuma, Paperless, Nextcloud: 25+ web services, one login, one session, one set of access rules. The OIDC provider, the Postgres backend, the session store, and the secrets all run inside k3s, backed by CNPG, Redis, and Vault.
Below is the full implementation: how the pieces fit together, why I made certain design decisions, and the specific bugs that bit me along the way. The first Authelia deployment on this cluster, with its five startup failure modes, is covered in the k3s, Authelia and Longhorn on Proxmox build log; this post is the OIDC setup that grew out of it.
View the complete homelab infrastructure source on GitHub
The Architecture
Authelia runs as a Kubernetes Deployment in the apps namespace, protected by the same default-deny NetworkPolicy that applies to everything else. It has three dependencies:
- PostgreSQL: CNPG-managed
postgres-autheliacluster in thedatabasenamespace - Redis: session store, ephemeral (no persistence needed)
- Vault: hmac_secret, OIDC private keys, JWT secrets, session secrets
The Traefik ForwardAuth middleware sits in front of every service. When a request hits Traefik, the middleware sends a verification request to Authelia’s /api/verify endpoint. Authelia checks the session cookie against the access rules and returns 200 if the request is allowed. Otherwise it returns a 302 redirect to the login portal for browsers, or a 401 for requests that can’t follow one. (Since Authelia 4.38, /api/verify is the legacy endpoint; new setups should use /api/authz/forward-auth.)
# kubernetes/apps/authelia/middleware.yml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: authelia
namespace: apps
spec:
forwardAuth:
address: http://authelia.apps.svc.cluster.local:9091/api/verify?rd=https://auth.woitzik.dev/
trustForwardHeader: true
authResponseHeaders:
- Remote-User
- Remote-Groups
- Remote-Name
- Remote-Email
Every IngressRoute that needs protection adds middlewares: [{name: authelia}]. Services that need API-level protection (not browser-based) use OIDC client credentials instead.
The OIDC Configuration
Authelia acts as an OIDC provider for services that support it. The ConfigMap defines seven OIDC clients; these are five of them:
# kubernetes/apps/authelia/configmap.yml
identity_providers:
oidc:
clients:
- client_id: 'proxmox'
client_name: 'Proxmox VE'
client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...' # hash only
public: false
authorization_policy: two_factor
redirect_uris:
- 'https://pve.woitzik.dev'
- 'https://pve.woitzik.dev/api2/extjs/access/oidc/login'
scopes: [openid, profile, email, groups]
token_endpoint_auth_method: 'client_secret_basic'
- client_id: 'pbs'
client_name: 'Proxmox Backup Server'
client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
public: false
authorization_policy: two_factor
redirect_uris:
- 'https://backup.woitzik.dev'
- 'https://backup.woitzik.dev/api2/extjs/access/oidc/login'
scopes: [openid, profile, email, groups]
token_endpoint_auth_method: 'client_secret_basic'
- client_id: 'argocd'
client_name: 'Argo CD'
client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
public: false
authorization_policy: two_factor
redirect_uris:
- 'https://argo.woitzik.dev/auth/callback'
scopes: [openid, profile, email, groups]
token_endpoint_auth_method: 'client_secret_basic'
- client_id: 'grafana'
client_name: 'Grafana'
client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
public: false
authorization_policy: two_factor
redirect_uris:
- 'https://monitoring.woitzik.dev/login/generic_oauth'
scopes: [openid, profile, email, groups]
token_endpoint_auth_method: 'client_secret_basic'
- client_id: 'headscale'
client_name: 'Headscale'
client_secret: '$argon2id$v=19$m=65536,t=3,p=4$...'
public: false
authorization_policy: two_factor
redirect_uris:
- 'https://headscale.woitzik.dev/oidc/callback'
scopes: [openid, profile, email]
token_endpoint_auth_method: 'client_secret_basic'
The redirect URIs have to match what each service actually sends, character for character. Proxmox VE and PBS redirect back to their web UI origin (https://pve.woitzik.dev, https://backup.woitzik.dev), not to a dedicated callback path. Headscale’s callback path is /oidc/callback, and Grafana’s generic OAuth integration uses /login/generic_oauth. The client_secret values are argon2id hashes: Authelia only needs the hash to verify the secret, and the plaintext lives in each service’s own OIDC settings, so the hash can sit in git.
Key detail: every client uses client_secret_basic, including ArgoCD. I originally had ArgoCD on client_secret_post, assuming it sent the secret in the POST body. It doesn’t. ArgoCD’s built-in OIDC client (not Dex) sends client credentials via HTTP Basic auth, and Authelia rejects a client that authenticates with a method other than the configured one with invalid_client. Setting token_endpoint_auth_method: client_secret_basic on the ArgoCD client fixed the login.
Secrets in Vault
The OIDC hmac_secret, signing keys, and session secrets were originally committed as plain Kubernetes Secrets. This is fine for a homelab, but it means anyone with kubectl get secret can read them.
The migration to Vault:
# kubernetes/apps/authelia/external-secret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: authelia-secrets
namespace: apps
spec:
refreshInterval: 1h
secretStoreRef:
name: vault
kind: ClusterSecretStore
target:
name: authelia-secrets
creationPolicy: Merge
data:
- secretKey: hmac-secret
remoteRef:
key: secret/authelia
property: hmac-secret
- secretKey: oidc-issuer-private-key
remoteRef:
key: secret/authelia
property: oidc-issuer-private-key
- secretKey: jwt-secret
remoteRef:
key: secret/authelia
property: jwt-secret
- secretKey: session-secret
remoteRef:
key: secret/authelia
property: session-secret
- secretKey: redis-password
remoteRef:
key: secret/authelia
property: redis-password
creationPolicy: Merge means the External Secrets Operator doesn’t own the Kubernetes Secret. It writes its keys into an existing Secret on each refresh and leaves any other keys alone; it does not create the Secret if it’s missing. That made it the safe migration path: the old Secret stayed in place and ESO took over its keys one by one. If Vault is sealed or unreachable, the existing Secret persists, and Authelia keeps running on the last synced values.
The users database (username, argon2id hash, group memberships) was also migrated from a plain Secret to an ExternalSecret sourced from Vault. It uses the same creationPolicy: Merge approach. Like the other secrets, which are mounted under /config/secrets and pulled into the config with Authelia’s secret template filter, it is mounted as a file (under /config/users), and the Authelia config references it by path.
The Postgres Backend
CNPG manages the postgres-authelia single-instance cluster in the database namespace. WAL archiving goes to Garage S3 for point-in-time recovery:
# kubernetes/system/postgres/cluster.yml (excerpt)
postgresql:
parameters:
max_connections: "100"
shared_buffers: "64MB"
backup:
retentionPolicy: "7d"
barmanObjectStore:
destinationPath: s3://cnpg-backups/authelia
endpointURL: http://garage.apps.svc.cluster.local:3900
wal:
compression: gzip
data:
compression: gzip
A daily ScheduledBackup creates a full base backup. WAL archiving plus daily base backups lets me restore to any point in time within the retention window.
The Authelia Schema Bug
One gotcha: Authelia’s Postgres schema version can get ahead of the running image. Once a newer Authelia image has migrated the database, an older image running against it hits a schema mismatch, for example when a Renovate bump is reverted or a tag is pinned back.
The fix was to bump the image version to match the schema:
image: ghcr.io/authelia/authelia:4.39.20 # matches DB schema v24
But the symptom was confusing: Authelia reported healthy in ArgoCD, the pods were running, but login attempts returned 500 errors. The health check endpoint (/api/health) only reports that Authelia’s HTTP server is answering. It doesn’t check database schema compatibility.
Access Control Rules
The access_control section defines who can access what:
access_control:
rules:
- domain: auth.woitzik.dev
policy: bypass
- domain: "*.woitzik.dev"
policy: two_factor
Critical detail: auth.woitzik.dev must be the first rule. Authelia matches rules top-down and stops at the first match. If the wildcard *.woitzik.dev → two_factor rule comes before the auth.woitzik.dev → bypass rule, Authelia redirects to itself in an infinite loop.
I hit this bug during initial setup. The logs showed repeated 302 redirects from auth.woitzik.dev back to itself. That looks like a broken redirect URL or cookie domain, but it was the rule order.
The Two-Replica Setup
Authelia runs at two replicas with a PodDisruptionBudget (minAvailable: 1). Redis handles session state, so either replica can serve any session. Both replicas must share the same hmac_secret and JWT signing keys, and they do, since both read from the same Vault-backed ExternalSecret.
# kubernetes/apps/authelia/authelia.yml
spec:
replicas: 2
template:
spec:
containers:
- name: authelia
livenessProbe:
httpGet:
path: /api/health
port: 9091
readinessProbe:
httpGet:
path: /api/health
port: 9091
Both probes hit /api/health, which, as the schema bug showed, only confirms that the process is serving. Blackbox Exporter probes the same endpoint from outside, through Traefik, so it also catches failures on the path users take (ingress, TLS, routing) that an in-pod probe never sees.
What I’d Change
Two things, if starting over:
-
Start with Vault from day one. Migrating from plain Kubernetes Secrets to Vault-backed ExternalSecrets after the fact required the
creationPolicy: Mergedance, which wouldn’t have been necessary if the secrets had never been in git. -
Check each client’s token endpoint auth method against its source, not its docs page. The ArgoCD
invalid_clientfailure cost more time than any other client, and the fix was one line once I knew what ArgoCD actually sends.
SSO across 25+ services is the same problem in enterprise Azure: Entra ID is the OIDC provider, app registrations replace the Authelia client definitions, and Conditional Access policies replace the access_control rules. The scale is different (Entra ID handles millions of identities), but the pattern is identical: centralize authentication, put every service behind one gate, and keep the access rules in one place.
Most of the debugging above happened at a shell prompt, grepping logs and testing OIDC flows with curl, not in a dashboard. The Linux Command Line* is still the book I’d hand someone who wants to get faster at that part.