My Firewall Had 77 Rules. Terraform Knew About 22 of Them.

I wrote an article about building a zero-trust MikroTik firewall with Terraform: default-deny chains, explicit allow rules, place_before for deterministic ordering. It runs on the RB5009*, whose hardware-offloaded switching matters once you’re carrying multiple VLANs across a trunk. The Terraform code was correct. I ran terraform plan regularly and it showed no drift.

The live router had 77 firewall filter rules. The Terraform configuration tracked 22.

This is how that happened, why a clean terraform plan didn’t catch it, and how a security tightening I had made, verified, and considered done had been silently undone for weeks.

View the complete homelab infrastructure source on GitHub 🐙

How You End Up With Four Generations of the Same Firewall

In hindsight the pattern is obvious. Every significant firewall rework, I wrote a fresh, complete set of rules in firewall_deterministic.tf and ran terraform apply. Terraform created the new rules. It never removed the previous generation, because by then those rules were no longer in its state. Removing a resource from the configuration while it’s still in state makes Terraform destroy it on the next apply. A rule that has already dropped out of state, through a reconstruction pass that started over or resources taken out of state rather than destroyed, stays on the router untouched. A couple of rules had a simpler origin: I created them directly via the RouterOS API during a debugging session and never imported them.

Terraform only manages what’s in its state. A rule that exists on the router but isn’t a resource in the current configuration and state is invisible to terraform plan. There’s no diff to show, because there’s nothing to compare it against. “No changes” means the resources Terraform knows about match reality. It says nothing about resources Terraform was never told to track.

The Bug This Caused

This was worse than clutter. It undid a real security fix.

At some point I had tightened a monitoring rule from “Prometheus can reach all internal VLANs” to “Prometheus can reach only port 9100 on the management VLAN”:

# The narrow, intentional version - added to fix an overly broad rule
resource "routeros_ip_firewall_filter" "fwd_04a_srv_monitoring" {
  action       = "accept"
  chain        = "forward"
  src_address  = "10.0.20.0/24"   # monitoring VLAN
  dst_address  = "10.0.10.0/24"   # management VLAN
  dst_port     = "9100"
  protocol     = "tcp"
  place_before = routeros_ip_firewall_filter.fwd_08_allow_dns.id
  comment      = "04a: SRV - Prometheus scrape to MGMT node_exporter (port 9100)"
}

This rule existed in Terraform. terraform plan showed it as applied, no drift. I had every reason to believe the network was scoped exactly this way.

But RouterOS evaluates firewall rules in order and stops at the first match. Earlier in the live ruleset sat a leftover from a previous generation, the old, broad version:

"04a: SRV - Allow monitoring to all internal VLANs"
src=10.0.20.0/24 dst=10.0.10.0/24 action=accept

No port restriction. No protocol restriction. Because RouterOS hit this rule first, matching traffic was accepted before the router ever evaluated the narrower, newer rule. The port-9100-only restriction I had written, tested, and confirmed in Terraform was never enforced on the live device. The older, broader rule won every time.

That’s the sharpest version of the general problem with ordered rule lists: a rule that looks dead because a newer one superseded it isn’t dead until it’s removed. It sits there until the day its broader match fires first.

Auditing that by hand means comparing prefixes rather than interface names, which is why I keep the subnet calculator open next to the rule set. Expanding 10.0.20.0/24 to its real network, broadcast and host range makes an overlapping-but-not-identical rule visible in a way two similar-looking strings never do.

Finding the Real Scope of the Problem

# Pull live rules via the RouterOS REST API
curl -s -k -u admin:$PASS https://10.0.10.1/rest/ip/firewall/filter | jq length
# → 77

# Count Terraform-managed resources
grep -c 'resource "routeros_ip_firewall_filter"' terraform/stacks/network/firewall_deterministic.tf
# → 22

55 rules existed on the router with no corresponding Terraform resource. I diffed the live rules against the 22 known-good ones by exact field match (action, chain, src/dst address, port, protocol; not comment text, since comments had drifted across generations too). That split the 55 into two groups:

• 36 rules were exact or near-exact duplicates of a currently tracked rule: stale leftover generations of the same intent.
• 19 rules were legitimate, distinct, and still in use: VPN access tiers, Atlantis/MikroDash API access, WireGuard, a Minecraft server port-forward, OIDC redirect routes. I had created them manually at some point and never added them to Terraform. That’s not drift in the dangerous sense, just infrastructure that never made it under IaC.

Deleting Firewall Rules Without Locking Yourself Out

The router is the highest-blast-radius device in the network. If the rule you delete by mistake is the one allowing SSH, there’s no getting back in remotely to fix it. Before deleting anything, I staged a full per-rule restore behind a RouterOS scheduler entry, a dead man’s switch:

/system scheduler add name="restore-firewall-failsafe" \
  start-time=startup interval=5m \
  on-event="/system script run restore-firewall-rules"

The restore script re-creates every rule about to be deleted. Once enabled, the scheduler runs it every five minutes (and at boot, via start-time=startup) until I remove the entry. The procedure:

1. Stage the restore script and the scheduler entry, disabled (disabled=yes).
2. Enable the scheduler.
3. Delete the 36 orphaned rules via direct REST API calls.
4. Immediately verify DNS, SSH, and WAN connectivity from a separate, already-open session.
5. Only if everything checks out: disable and remove the scheduler entry.

If step 4 had failed, the scheduler would have restored the deleted rules within five minutes without needing any further access to the router. The pattern fits any change where the failure mode is “I can no longer reach the device to fix my mistake”: stage the rollback on a timer, and cancel the timer only after confirming success through a separate channel.

What’s Left

41 rules remain: the 22 Terraform-managed ones plus the 19 legitimate manual rules, now tracked as a known gap in docs/OPERATIONS.md instead of invisible clutter. Bringing those 19 under Terraform with import blocks is the obvious next step, but it isn’t urgent: they work, they’re intentional, and they’re documented. The 36 that mattered, because they were undermining a security control, are gone.

The General Lesson

terraform plan showing no drift is a narrower claim than “the live device matches my intent.” It only means the resources Terraform is tracking match their last-applied state. Anything outside that set, whether left behind by an earlier version of the config that dropped out of state or created via direct API/CLI access during a debugging session, stays invisible to the diff until someone looks at the live device.

On an ordered rule list (firewalls, or any first-match system), an orphaned broad rule isn’t neutral clutter. It can take precedence over a narrower rule you believe supersedes it. Periodically diffing live state against Terraform by querying the device directly, instead of trusting plan, is the only way to catch this class of bug.

The same risk exists in Azure NSGs and Azure Firewall Policy: priority-ordered rules where an old, broad rule with a lower priority number can win over a newer, narrower one if nobody cleaned it up after a security tightening. If you manage NSG rule sets at scale, periodically pulling live rule state with az network nsg rule list and diffing it against your Terraform state catches exactly this drift before it becomes a finding in someone else’s audit.