One Query Param Deleted Every Alert Rule We Had

The dashboards-as-code migration took about four minutes. Undoing it took two days.

We had a monitoring stack where dashboards and alert rules both lived in the monitoring tool’s own database, edited through its UI, backed up only by the fact that the database happened to be managed Postgres with point-in-time recovery. That’s a normal place to be and a bad place to stay. So when our monitoring platform shipped a Git Sync feature — declare a repository, migrate your resources into it, review dashboard changes as pull requests — it looked like exactly the upgrade we wanted. Everything reviewable, everything diffable, no more “who changed this panel and when.”

What the migrate button actually does

The feature works by introducing a second class of resource. Once you attach a repository, folders and dashboards can be managed (owned by the repo, reconciled from git) or unmanaged (the old kind, owned by whoever last clicked save). The migrate job walks your existing resources, writes them out to the repo, and creates managed copies.

It does not move things. It creates the new and leaves the old.

So after the migration we had two of everything: the original folder tree, plus a managed tree under the repository’s mount point. The UI showed both. The obvious cleanup was to delete the now-redundant legacy folders.

Deleting a folder that still contains resources requires a force flag. Ours took the form of a query parameter on the delete call — the kind of name that reads like “yes, I know the folder isn’t empty, proceed anyway.” Its actual scope was every alert rule in that folder, whether or not it had anything to do with the migration, whether or not it was orphaned.

Seventy-six alert rules. All created and owned through the UI, none of them part of the dashboards being migrated, all gone in one call.

Recovery is two problems, not one

The data came back quickly. The backing database was managed Postgres with continuous backup, so a point-in-time restore to twenty minutes earlier returned all 76 rules and every dashboard. Alerting was blind for under an hour. That was the easy half.

The hard half is that a point-in-time restore doesn’t hand you your cluster back — it creates a new cluster, with a new identifier, and the application has to be pointed at its endpoint by hand. Which meant that from Terraform’s perspective:

• The old, now-idle cluster still existed and still matched state, so terraform plan was clean. Zero drift, nothing to see — while the database actually serving production traffic was invisible to Terraform entirely. This is the worst kind of green.
• The module we used derived the cluster identifier from a naming convention — organization, environment, stage, name — with no input to override it. There was no way to import the restored cluster under the name Terraform expected. Importing it as-is would have baked the restore suffix into the resource name permanently.
• The connection endpoint had been swapped inside a secret that our module deliberately ignores changes to, so the manual edit survived subsequent applies. Extremely convenient during the incident. One more piece of undocumented state afterwards.

The exit was a rename-in-place dance performed entirely outside Terraform: delete the extra instances on the idle cluster, disable its deletion protection, delete it, rename the restored cluster to the name Terraform wanted, rename its instance to match, point the secret back at the original endpoint, then let terraform plan tell us what was still missing (a second instance) and apply that. It worked. It also cost a maintenance window and a written runbook to accomplish what was, on paper, a no-op.

The half-state

We never finished the migration either. Because the pre-restore migrate had already committed dashboards to the repo and deleted them from the monitoring tool, the restore brought them back as unmanaged. Now the repository sync sees two copies and refuses to reconcile — roughly sixty dashboards rejected with, paraphrasing, “resource already exists and is not managed; the repository cannot take it over without an explicit migration.”

That error is correct. It’s also unresolvable without choosing a direction, it doesn’t self-heal, and it generates a warning every single day until somebody does.

While we were in there, the rest of the v1 seams showed themselves:

• Alert rules weren’t supported by the feature at all. We were adopting Git Sync to get review coverage on our monitoring configuration; it covered dashboards only. Alerts were going to keep living in the UI regardless — meaning the thing we deleted was the one thing the migration could never have helped with.
• The resource-type filter on the migrate job was silently ignored. The API accepted a field naming which resource types to migrate and dropped it on the floor. Migration is all-or-nothing, and you find that out by watching it migrate everything.
• Folder titles are unique across the entire instance, not per parent. So the managed tree couldn’t reuse the legacy tree’s folder names, which is why we renamed the originals with a (legacy) suffix, which is the entire reason the “obvious cleanup” existed in the first place. The sharp edge was downstream of a naming constraint.
• The repository’s connection validator kept reporting our branch protection as incompatible even after we granted the sync app both bypass allowances — the pull-request requirement and the required status checks, which are two separate lists. Either it cached its verdict or it never consulted them.

What I’d take from it

A “migrate” verb is not a scoped operation. It’s a bulk write plus a bulk create, and its cleanup path is where the blast radius lives. Read what the cleanup deletes before you run the migration, not after.

Force flags deserve one minute of reading. The parameter answered a question I never asked. If a flag’s name is a verb of last resort, find out which objects it reaches — “force” almost never means “force the thing I’m currently thinking about.”

Restore is not recovery when infrastructure is code. Getting the bytes back took twenty minutes. Getting the restored artifact to fit back inside a module that derives its identity from labels took days. If your DR plan stops at “point-in-time restore,” it is only half a plan — it has to also say how the restored resource re-enters state, and whether your module lets it.

Adopting a v1 migration feature means adopting its irreversibility, not its convenience. The bar isn’t “does it work on the happy path.” It’s “if this half-completes, can I get back?” We couldn’t, fully — and the half-state is still there, generating one sync error a day, waiting for someone to choose between backing all the way out and waiting for the feature to grow up.

We’re waiting for the feature to grow up.