Skip to content
7 min read Updated

Vault Configuration Under Terraform: Staged Migration Without Downtime

Vault OSS on Kubernetes holds secrets for 25+ services. Moving its policies, auth methods and roles to Terraform in 6 reversible stages, with zero downtime.


Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

Vault is the trust root for every other secret on the cluster. 25+ services depend on External Secrets Operator syncing from Vault’s KV v2 engine, all running on a single BMAX Mini PC*. Authelia, databases, ArgoCD notifications and Velero credentials all pull from secret/<app>/* paths. Unsealing this Vault after a restart is a separate problem, solved by the polling sidecar pattern. This article only covers what’s configured once Vault is up.

Vault’s configuration (policies, the Kubernetes auth method, roles) was managed through vault login and CLI commands. No version control, no change history, no reproducibility. If someone changed a policy by hand, nobody else knew.

The goal: bring Vault’s configuration, not its secret values, under Terraform. Staged, low-risk, each stage independently reversible.

View the complete homelab infrastructure source on GitHub

The Scope Boundary

The hardest part of Terraforming Vault is knowing what NOT to Terraform:

In scope (configuration)Out of scope (stays manual)
Vault policiesActual secret values (secret/<app>/*)
Auth methods (Kubernetes auth)Unseal keys, root tokens
Secrets engine mount configVault server config (Helm, Raft)
Auth roles and bindingsTLS (currently disabled)

Secret values stay in Ansible Vault. Terraform manages configuration, never secret data. The Ansible Vault file (ansible/group_vars/all/vault.yml) is the durable source, and ESO syncs from Vault to Kubernetes Secrets. If Terraform state is lost, you re-import the config. No secrets leak.

That is the core principle: Terraform state describes access rules, not the secrets themselves. Losing state means re-importing policies and roles, not exposing credentials.

The Six Stages

Stage 1: Policies (lowest risk)

Import existing Vault policies into Terraform. A policy only describes access rules, and importing one changes nothing about how Vault operates.

resource "vault_policy" "eso" {
  name   = "eso"
  policy = file("${path.module}/policies/eso.hcl")
}

One .hcl file per policy under policies/, extracted from the live Vault instance (vault policy read <name>) and committed to git.

Risk: Zero for a pure import. No restart, no unseal, no client disruption.

Verification: terraform plan shows only import blocks matching live state. vault policy list is unchanged after apply.

Rollback: terraform state rm vault_policy.<name>. That only makes Terraform forget the resource; the policy stays in Vault.

Stage 2: KV v2 Secrets Engine Mount

Import the secret/ KV v2 mount configuration. This covers the engine’s existence and mount settings (type, KV version, description), NOT the secret values.

resource "vault_mount" "secret" {
  path        = "secret"
  type        = "kv"
  options     = { version = "2" }
  description = "KV v2 secrets engine — application secrets"
}

Risk: Low. If the mount already exists with the same config, terraform plan shows no changes. If there’s a config diff, review it carefully before applying.

Verification: vault read sys/mounts/secret/ output matches the HCL. vault kv list secret/ still works.

Stage 3: Kubernetes Auth Method

Import the Kubernetes auth method, the mechanism by which Vault trusts the Kubernetes API server to authenticate pods.

resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
  path = "kubernetes"
}

resource "vault_kubernetes_auth_backend_config" "config" {
  backend            = vault_auth_backend.kubernetes.path
  kubernetes_host    = "https://kubernetes.default.svc"
  kubernetes_ca_cert = file("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt")
}

Risk: Medium. A misconfigured auth method breaks every ESO sync. Do not apply a diff here blindly. If terraform plan shows changes beyond the imports, stop and investigate.

Verification: Trigger one ESO sync (kubectl annotate externalsecret -n <ns> <name> force-sync=$(date)). Check 3-5 apps that depend on Vault secrets: no restarts, no errors.

Stage 4: Kubernetes Auth Roles

Import the roles that map Kubernetes ServiceAccounts to Vault policies. This is the binding between “which pod” and “what it can access.”

resource "vault_kubernetes_auth_backend_role" "eso" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "eso"
  bound_service_account_names      = ["external-secrets"]
  bound_service_account_namespaces = ["external-secrets"]
  token_policies                   = ["eso"]
  token_ttl                        = 3600
}

Risk: Low for importing existing roles, medium for NEW roles, because those change what pods can access. Least-privilege check: compare each role’s bound_service_account_names and token_policies against the live config.

Stage 5: Additional Auth Methods (deferred)

Userpass, OIDC or other auth methods beyond Kubernetes. Only Kubernetes auth is in use today, since all Vault access goes through ESO. If no other auth methods exist, skip this stage.

Stage 6: App Policies (purely additive)

Import the per-app policies, one per ESO-managed secret path. Each policy controls exactly which secret path an app’s role can read.

Risk: Lowest. A policy grants nothing until a role references it.

Why Not Just vault write Everything?

The alternative is simpler: keep using CLI commands, document them in a runbook, hope nobody drifts. The problems with that:

  1. No change history. Who changed the authelia policy last week, and what did it say before? An enabled audit device logs the request, but it’s a request log, not a reviewable diff with a commit message.
  2. No reproducibility. Recreating a Vault configuration from documentation is error-prone. With Terraform, the HCL in git is the source of truth.
  3. No drift detection. terraform plan shows exactly what changed. Without it, drift accumulates silently until something breaks at 3 AM.
  4. No rollback. With the CLI, undo means remembering what it was before and vault write-ing it back. With Terraform, you revert the commit and apply the last known-good config.

The staged approach means you can stop at any stage. If Stage 3 (auth method) makes you nervous, don’t apply it. Stages 1 and 2 already pay off and carry close to zero risk.

The Maintenance Window

Each stage takes 5-20 minutes and is independently reversible. Total: about an hour.

StageTimeImpact
1. Policies15 minZero: read-only import
2. KV mount5 minZero: config import
3. K8s auth10 minZero if import-only
4. Auth roles15 minZero if import-only
6. App policies20 minZero: additive

If any stage shows unexpected drift: STOP, investigate, do not apply. The diff tells you exactly what’s different. Fix it in git or in Vault, then re-plan.

What Stays Manual Longest

ItemWhyWhen to revisit
Secret valuesAnsible Vault is the durable sourceNever; this is the correct pattern
Vault server configManaged by ArgoCD + HelmWhen Vault moves to a non-ArgoCD install
Unseal keys / root tokensSecurity-sensitive, manual by designNever
Vault upgradesRenovate handles via PROngoing

The point of moving Vault config under Terraform is that the configuration becomes version-controlled, reviewable and reproducible. The CLI isn’t the problem; the missing history is. Staging keeps each step small: if a stage makes you nervous, skip it, and the earlier ones still pay off. The hard part is drawing the scope boundary, not the HCL: secrets stay in Ansible Vault, and Terraform only touches the access rules around them.

Further reading: For the security model behind Vault’s policy system, Zero Trust Networks by Evan Gilman and Doug Barth* covers the “never trust, always verify” principles that Vault’s Kubernetes auth method implements. The import-first, modify-later pattern is a change-management idea at heart, and The Phoenix Project by Gene Kim* covers it in the context of reducing deployment risk.

Was this helpful?
Share
DW

David Woitzik

Hybrid Cloud Engineer

I work mostly with Azure, Terraform, and Zero-Trust network design, and publish the Terraform templates and write-ups I wanted when I was solving the same problems.

Keep reading

More like this in your inbox

New Terraform modules and articles by email. No spam.