Telling Your Plan's Noise From Your Change

I replaced one alert rule last week. Deleted a CPU-usage rule, added a CPU-throttling rule in its place. One rule, one file, a diff you could review in twenty seconds.

The plan came back with one to add, six to change, and one to destroy.

I hadn’t destroyed anything. I hadn’t touched six other rules. And this is the moment where infrastructure-as-code gets genuinely dangerous, because I had two plausible stories and no way to pick between them:

  1. My change has side effects I don’t understand.
  2. The repository was already out of sync with reality and my plan is just the first one to surface it.

Both are common. They call for opposite responses — the first means stop and read your own diff again, the second means proceed and file a separate cleanup. Guessing wrong in either direction is bad: you either ship a change with unexamined blast radius, or you spend an afternoon auditing a diff that was never the problem.

The technique

Stash your diff. Plan clean main. Diff the two plans.

git stash
terraform plan   # baseline: what does main alone want to do?
git stash pop
terraform plan   # yours: baseline + your change

Anything present in both plans is not yours. It’s drift that was sitting in the repository before you arrived, waiting for the next person to run a plan in that directory.

In my case, clean main planned zero to add, six to change, zero to destroy — the exact same six. So the six were pre-existing drift, fully accounted for, and the real footprint of my change was one add and one destroy. That reframed the review completely: instead of “why does a one-rule swap touch seven resources,” the question became “why is this rule being destroyed rather than updated,” which is a much better question and has an actual answer.

It takes two minutes and it converts an ambient worry into a fact you can put in the pull request. I now do it reflexively any time a plan is larger than my mental model of the diff.

The destroy was real, and it taught me something

The destroy wasn’t drift. My change genuinely destroyed and recreated a resource, and the reason is a class of gotcha worth internalising.

The alert rules were grouped, and the group’s name was part of its resource identity in the provider’s addressing scheme. I renamed the group — from a usage-based name to a throttling-based one, because the rule inside it had changed meaning. To the provider, that’s not a rename. There’s no such operation. It’s a different resource with a different identity, so the old one is destroyed and a new one created.

For alert rules this is mostly harmless: a brief window with no rule, then a new rule. For plenty of other resource types the same mechanic is not harmless at all — anything with attached storage, a stable endpoint, an allocated address, or downstream references.

The general shape: any attribute the provider uses to construct a resource’s identity is effectively immutable, and editing it means replacement rather than update. Names are the common case, because a name feels like metadata. It reads like a label. And for a large class of resources it’s a primary key.

Two habits that follow:

Why the drift was there at all

Worth naming the root cause, because the technique above is a workaround for a process problem.

Six resources had drifted because the plan was only ever run in directories somebody was actively changing. Nobody runs a plan across the whole repository on a schedule, so drift accumulates silently and is discovered by whoever next touches an unrelated resource in the same state. They inherit it. The polite thing is to file a separate cleanup PR; the tempting thing is to fold it into your change and let the plan look scary. I’ve done both and the first is much better — a plan that mixes your change with somebody else’s drift is a plan nobody can review.

The real fix is a scheduled plan-only run that reports drift as its own signal, decoupled from whoever happens to be editing. I’ve wanted that for a while. The two-minute stash trick is what you do until you have it.

Takeaways

The underlying principle generalises past Terraform to anything with a reconcile loop: a diff between desired and actual state does not distinguish your intent from accumulated divergence. If you can’t separate them, you can’t review either one.


← all writing