Telling Your Plan's Noise From Your Change
I replaced one alert rule last week. Deleted a CPU-usage rule, added a CPU-throttling rule in its place. One rule, one file, a diff you could review in twenty seconds.
The plan came back with one to add, six to change, and one to destroy.
I hadn’t destroyed anything. I hadn’t touched six other rules. And this is the moment where infrastructure-as-code gets genuinely dangerous, because I had two plausible stories and no way to pick between them:
- My change has side effects I don’t understand.
- The repository was already out of sync with reality and my plan is just the first one to surface it.
Both are common. They call for opposite responses — the first means stop and read your own diff again, the second means proceed and file a separate cleanup. Guessing wrong in either direction is bad: you either ship a change with unexamined blast radius, or you spend an afternoon auditing a diff that was never the problem.
The technique
Stash your diff. Plan clean main. Diff the two plans.
git stash
terraform plan # baseline: what does main alone want to do?
git stash pop
terraform plan # yours: baseline + your change
Anything present in both plans is not yours. It’s drift that was sitting in the repository before you arrived, waiting for the next person to run a plan in that directory.
In my case, clean main planned zero to add, six to change, zero to destroy — the exact
same six. So the six were pre-existing drift, fully accounted for, and the real footprint of
my change was one add and one destroy. That reframed the review completely: instead of “why
does a one-rule swap touch seven resources,” the question became “why is this rule being
destroyed rather than updated,” which is a much better question and has an actual answer.
It takes two minutes and it converts an ambient worry into a fact you can put in the pull request. I now do it reflexively any time a plan is larger than my mental model of the diff.
The destroy was real, and it taught me something
The destroy wasn’t drift. My change genuinely destroyed and recreated a resource, and the reason is a class of gotcha worth internalising.
The alert rules were grouped, and the group’s name was part of its resource identity in the provider’s addressing scheme. I renamed the group — from a usage-based name to a throttling-based one, because the rule inside it had changed meaning. To the provider, that’s not a rename. There’s no such operation. It’s a different resource with a different identity, so the old one is destroyed and a new one created.
For alert rules this is mostly harmless: a brief window with no rule, then a new rule. For plenty of other resource types the same mechanic is not harmless at all — anything with attached storage, a stable endpoint, an allocated address, or downstream references.
The general shape: any attribute the provider uses to construct a resource’s identity is effectively immutable, and editing it means replacement rather than update. Names are the common case, because a name feels like metadata. It reads like a label. And for a large class of resources it’s a primary key.
Two habits that follow:
- If you’re renaming anything in infrastructure code, check whether the name participates in the resource address before you decide whether it’s a cosmetic change. “Just a rename” is a phrase that has caused real outages.
- If the replacement is unwanted,
terraform state mv(or the equivalent import) lets you carry the existing object to the new address. If it’s harmless, say so explicitly in the PR — a reviewer seeing a destroy deserves a sentence about why it’s fine, not silence.
Why the drift was there at all
Worth naming the root cause, because the technique above is a workaround for a process problem.
Six resources had drifted because the plan was only ever run in directories somebody was actively changing. Nobody runs a plan across the whole repository on a schedule, so drift accumulates silently and is discovered by whoever next touches an unrelated resource in the same state. They inherit it. The polite thing is to file a separate cleanup PR; the tempting thing is to fold it into your change and let the plan look scary. I’ve done both and the first is much better — a plan that mixes your change with somebody else’s drift is a plan nobody can review.
The real fix is a scheduled plan-only run that reports drift as its own signal, decoupled from whoever happens to be editing. I’ve wanted that for a while. The two-minute stash trick is what you do until you have it.
Takeaways
- When a plan is bigger than your diff, stash and plan clean
main. Anything in both plans is not yours. - Put the result in the pull request. “Six changes are pre-existing drift, verified by
planning clean
main; my change is one add, one destroy” is the sentence that lets a reviewer stop worrying and read the actual diff. - Treat any renaming as a possible replacement until you’ve checked the resource address.
- File inherited drift separately. Don’t let it ride along in your change where it becomes indistinguishable from your blast radius.
- If nothing plans on a schedule, drift is discovered by ambush, and the person ambushed is whoever is least expecting it.
The underlying principle generalises past Terraform to anything with a reconcile loop: a diff between desired and actual state does not distinguish your intent from accumulated divergence. If you can’t separate them, you can’t review either one.