Skip to content
6 min read Updated

The DNS Drift That Wasn't DHCP: How Proxmox Cloud-Init Silently Injects Search Domains

Every DNS lookup in my k3s cluster appended 'home.lan'. I blamed DHCP, but the Terraform fix showed zero diff. Proxmox cloud-init was injecting it.


Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

Every DNS lookup from every pod in my k3s cluster was appending home.lan to the query. External lookups like registry-1.docker.io became registry-1.docker.io.home.lan, failed locally, recursed all the way out, and added ~341ms of latency to every image pull, API call and webhook. That traffic should have gone straight to Unbound for recursive resolution instead of round-tripping through a phantom search domain. I assumed DHCP was the source, wrote a Terraform fix, and watched the plan show zero diff against live state.

The real source was Proxmox’s cloud-init generator, three layers away from the DHCP server.

The Symptom

Prometheus showed p99 DNS latency at 341ms for external queries. Local queries (services within the cluster) resolved in under 1ms. The gap was consistent across all three k3s nodes.

# p99 of queries CoreDNS forwards upstream (everything outside cluster.local)
histogram_quantile(0.99,
  sum by (le) (rate(coredns_forward_request_duration_seconds_bucket[5m])))
# → ~341ms

With the default ClusterFirst DNS policy, the kubelet copies the node’s search domains into every pod’s /etc/resolv.conf after the cluster ones. If the node has search home.lan, every pod inherits it. With the default ndots:5, any name with fewer than 5 dots is tried against the search list first, which means almost every name.

Wrong Hypothesis: DHCP

My first guess was the RouterOS DHCP server. The network was configured via Terraform, but I’d never explicitly set the domain field:

# Before fix - no domain attribute at all
resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
}

Adding domain = "" should force Terraform to detect and correct any drift:

resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
  domain     = ""  # explicitly empty - should fix drift
}

The plan showed zero changes. Live state already matched domain = "". DHCP was never the source.

Live Verification

Confirmed via the RouterOS REST API:

# Check all DHCP server config paths
curl -s "https://router.lan/rest/ip/dhcp-server" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dhcp-server/network" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dns" | jq '.domain'
# → null

The RouterOS DHCP server had no domain/search config anywhere. domain = "" matched live state because the live state really was empty. DHCP was never handing out home.lan.

The Real Source: Proxmox Cloud-Init Injects the DNS Search Domain

On the k3s nodes themselves:

networkctl status --json | jq '.DNS'
# → {"SearchDomains": ["home.lan"], "ConfigSource": "static"}

ConfigSource: static, not DHCP. The search domain was baked into the node’s netplan config, generated by Proxmox’s cloud-init writer.

When a VM’s cloud-init config sets neither a search domain nor a DNS server (in Terraform: initialization.dns left unset), Proxmox falls back to the PVE node’s own DNS settings, search domain included:

# On the Proxmox host
pvesh get /nodes/pve-mgmt-01/dns
# → search: home.lan

Proxmox’s cloud-init generator reads the PVE node’s DNS config and writes it into the VM’s /etc/netplan/50-cloud-init.yaml. Every k3s VM got home.lan as its search domain from the hypervisor itself, not from DHCP.

Why This Is a Trap

The trap is the layering:

Proxmox PVE node DNS config
    ↓ (cloud-init fallback)
VM netplan search domain
    ↓ (kubelet DNS policy)
Pod /etc/resolv.conf search domain
    ↓ (ndots:5)
Every DNS lookup appends .home.lan
    ↓
~341ms latency on every external query

And the debugging trap: terraform plan showed zero diff on the DHCP resource because DHCP wasn’t the source. A Terraform-native investigation would have concluded “no drift detected” and moved on. The real fix required:

  1. Clearing the PVE node’s default: pvesh set /nodes/pve-mgmt-01/dns -search ""
  2. Regenerating cloud-init on each k3s VM
  3. Declaring domain = "" in the Proxmox VM’s initialization.dns block

Steps 1 and 2 are live changes, one to the hypervisor and one to the running k3s VMs, and no Terraform resource models them. Step 3 declares intent for future VMs but has no effect on already-provisioned ones (the dns block is in lifecycle.ignore_changes).

The Terraform Fix (Future VMs)

# terraform/stacks/proxmox/vm.tf
initialization {
  dns {
    domain  = ""  # explicitly unset - prevents PVE node fallback
    servers = ["10.0.20.5"]
  }
}

This ensures new VMs don’t inherit the PVE node’s search domain. Existing VMs need the manual cloud-init regen.

Lessons

1. DNS config has multiple sources. DHCP is the obvious one, but cloud-init, node-level config, and kubelet policy all contribute. Check all layers.

2. terraform plan zero-diff doesn’t mean no drift. It means the specific resource you checked matches state. The drift might be in a different layer entirely.

3. Check networkctl status --json. The ConfigSource field tells you whether the search domain came from DHCP, static config, or another source. If it says “static”, stop looking at DHCP.

4. Proxmox cloud-init is a hidden config source. If you leave the VM’s DNS settings unset, Proxmox fills them from the PVE node. Always declare them explicitly. It’s the same category of hidden-layer surprise as Proxmox’s onboot attribute silently not round-tripping through Terraform: the provider abstraction has gaps that only show up once you go looking for drift that isn’t there.

Further reading: DNS debugging keeps coming back to the same fundamentals: search domains, ndots, resolution order. Netzwerke: Verstehen, Einrichten, Administrieren* covers those fundamentals in more depth than this Proxmox-specific drift case gets into.


DNS latency from inherited search domains is invisible in most monitoring setups. You’ll see high p99 latency on external queries but no obvious cause. If your pods have ndots:5 (the default) and your nodes have a search domain you didn’t explicitly set, you’re paying the tax on every lookup.

Was this helpful?
Share
DW

David Woitzik

Hybrid Cloud Engineer

I work mostly with Azure, Terraform, and Zero-Trust network design, and publish the Terraform templates and write-ups I wanted when I was solving the same problems.

Keep reading

More like this in your inbox

New Terraform modules and articles by email. No spam.