Every DNS lookup from every pod in my k3s cluster was appending home.lan to the query. External lookups like registry-1.docker.io became registry-1.docker.io.home.lan, failed locally, recursed all the way out, and added ~341ms of latency to every image pull, API call and webhook. That traffic should have gone straight to Unbound for recursive resolution instead of round-tripping through a phantom search domain. I assumed DHCP was the source, wrote a Terraform fix, and watched the plan show zero diff against live state.
The real source was Proxmox’s cloud-init generator, three layers away from the DHCP server.
The Symptom
Prometheus showed p99 DNS latency at 341ms for external queries. Local queries (services within the cluster) resolved in under 1ms. The gap was consistent across all three k3s nodes.
# p99 of queries CoreDNS forwards upstream (everything outside cluster.local)
histogram_quantile(0.99,
sum by (le) (rate(coredns_forward_request_duration_seconds_bucket[5m])))
# → ~341ms
With the default ClusterFirst DNS policy, the kubelet copies the node’s search domains into every pod’s /etc/resolv.conf after the cluster ones. If the node has search home.lan, every pod inherits it. With the default ndots:5, any name with fewer than 5 dots is tried against the search list first, which means almost every name.
Wrong Hypothesis: DHCP
My first guess was the RouterOS DHCP server. The network was configured via Terraform, but I’d never explicitly set the domain field:
# Before fix - no domain attribute at all
resource "routeros_ip_dhcp_server_network" "vlan_networks" {
for_each = local.homelab_vlans
address = "10.0.${each.value}.0/24"
gateway = "10.0.${each.value}.1"
dns_server = "10.0.20.5"
}
Adding domain = "" should force Terraform to detect and correct any drift:
resource "routeros_ip_dhcp_server_network" "vlan_networks" {
for_each = local.homelab_vlans
address = "10.0.${each.value}.0/24"
gateway = "10.0.${each.value}.1"
dns_server = "10.0.20.5"
domain = "" # explicitly empty - should fix drift
}
The plan showed zero changes. Live state already matched domain = "". DHCP was never the source.
Live Verification
Confirmed via the RouterOS REST API:
# Check all DHCP server config paths
curl -s "https://router.lan/rest/ip/dhcp-server" | jq '.[].domain'
# → all null
curl -s "https://router.lan/rest/ip/dhcp-server/network" | jq '.[].domain'
# → all null
curl -s "https://router.lan/rest/ip/dns" | jq '.domain'
# → null
The RouterOS DHCP server had no domain/search config anywhere. domain = "" matched live state because the live state really was empty. DHCP was never handing out home.lan.
The Real Source: Proxmox Cloud-Init Injects the DNS Search Domain
On the k3s nodes themselves:
networkctl status --json | jq '.DNS'
# → {"SearchDomains": ["home.lan"], "ConfigSource": "static"}
ConfigSource: static, not DHCP. The search domain was baked into the node’s netplan config, generated by Proxmox’s cloud-init writer.
When a VM’s cloud-init config sets neither a search domain nor a DNS server (in Terraform: initialization.dns left unset), Proxmox falls back to the PVE node’s own DNS settings, search domain included:
# On the Proxmox host
pvesh get /nodes/pve-mgmt-01/dns
# → search: home.lan
Proxmox’s cloud-init generator reads the PVE node’s DNS config and writes it into the VM’s /etc/netplan/50-cloud-init.yaml. Every k3s VM got home.lan as its search domain from the hypervisor itself, not from DHCP.
Why This Is a Trap
The trap is the layering:
Proxmox PVE node DNS config
↓ (cloud-init fallback)
VM netplan search domain
↓ (kubelet DNS policy)
Pod /etc/resolv.conf search domain
↓ (ndots:5)
Every DNS lookup appends .home.lan
↓
~341ms latency on every external query
And the debugging trap: terraform plan showed zero diff on the DHCP resource because DHCP wasn’t the source. A Terraform-native investigation would have concluded “no drift detected” and moved on. The real fix required:
- Clearing the PVE node’s default:
pvesh set /nodes/pve-mgmt-01/dns -search "" - Regenerating cloud-init on each k3s VM
- Declaring
domain = ""in the Proxmox VM’sinitialization.dnsblock
Steps 1 and 2 are live changes, one to the hypervisor and one to the running k3s VMs, and no Terraform resource models them. Step 3 declares intent for future VMs but has no effect on already-provisioned ones (the dns block is in lifecycle.ignore_changes).
The Terraform Fix (Future VMs)
# terraform/stacks/proxmox/vm.tf
initialization {
dns {
domain = "" # explicitly unset - prevents PVE node fallback
servers = ["10.0.20.5"]
}
}
This ensures new VMs don’t inherit the PVE node’s search domain. Existing VMs need the manual cloud-init regen.
Lessons
1. DNS config has multiple sources. DHCP is the obvious one, but cloud-init, node-level config, and kubelet policy all contribute. Check all layers.
2. terraform plan zero-diff doesn’t mean no drift. It means the specific resource you checked matches state. The drift might be in a different layer entirely.
3. Check networkctl status --json. The ConfigSource field tells you whether the search domain came from DHCP, static config, or another source. If it says “static”, stop looking at DHCP.
4. Proxmox cloud-init is a hidden config source. If you leave the VM’s DNS settings unset, Proxmox fills them from the PVE node. Always declare them explicitly. It’s the same category of hidden-layer surprise as Proxmox’s onboot attribute silently not round-tripping through Terraform: the provider abstraction has gaps that only show up once you go looking for drift that isn’t there.
Further reading: DNS debugging keeps coming back to the same fundamentals: search domains, ndots, resolution order. Netzwerke: Verstehen, Einrichten, Administrieren* covers those fundamentals in more depth than this Proxmox-specific drift case gets into.
DNS latency from inherited search domains is invisible in most monitoring setups. You’ll see high p99 latency on external queries but no obvious cause. If your pods have ndots:5 (the default) and your nodes have a search domain you didn’t explicitly set, you’re paying the tax on every lookup.