Postgres Won't Auto-Upgrade, and Your GitOps Bot Will Happily Break It Trying

I run a handful of self-hosted apps at home on top of a small Kubernetes cluster, managed the boring, correct way: a dependency bot watches container image tags across everything and opens a pull request whenever a newer version is available. Merge the PR, the GitOps controller rolls it out. This works great for approximately everything, right up until it doesn’t.

Two separate apps in my cluster each run their own Postgres database. The bot opened routine-looking major-version bump PRs for both — the exact same kind of PR it opens weekly for a dozen other images, the kind you approve on autopilot because the last twenty of them were fine. I merged both. Both databases went down. Not “degraded” — down. New pod, crash loop, immediately.

Data intact, service dead — which is almost worse

The failure mode itself was clean, at least: Postgres refuses to start against a data directory that was initialized by an older major version. It doesn’t attempt to migrate anything, doesn’t corrupt anything, doesn’t silently do the wrong thing with your data. It just says, correctly and loudly, “this data directory belongs to an earlier version of me,” and exits. Nothing was lost. But “nothing was lost, the service is just completely down” is a strange kind of relief, and it’s not a great place to be paged from.

The fix in the moment was equally clean: revert the image tag in git, let the GitOps controller sync the rollback, the old server starts against its original data directory, done. A kubectl edit directly against the cluster would not have stuck — the whole point of a GitOps controller is that it reconciles live drift back toward whatever’s declared in git, so the only durable fix has to happen in git.

The trap: “the last few worked” is not a safety signal

Here’s the part that actually matters, and the part that made this an easy mistake to make twice: the automation and the PR look identical regardless of which image is being bumped and regardless of whether that particular engine can survive a major bump unattended. I had, at the same time, a different database on a different engine go through an equally large major-version jump via the exact same automated-PR-and-merge path, with zero issues — that one auto-migrates its data directory on startup as a matter of course. Same bot, same PR shape, same “looks routine” merge decision, wildly different outcome depending entirely on an engine-specific property that the PR itself gives you no way to see.

That’s the actual lesson: “the last several of these bumps went fine” tells you nothing about the next one, if the next one happens to be a database engine with different upgrade semantics than the ones that already went fine. A routine-looking diff is not evidence of a routine-safe change when the safety depends on internal behavior the diff can’t show you.

The fix is a permanent exclusion, not more caution

The right fix isn’t “be more careful reviewing database PRs” — that’s a habit, and habits erode under load exactly when you need them most. The right fix is structural: configure the dependency bot to simply never auto-open (or at minimum never auto-merge) major-version bumps for known stateful database images, full stop. Treat a major version bump for a database as its own deliberately-gated task — one that involves actually checking whether that specific engine needs a planned upgrade path (a native upgrade utility, or a dump/restore) — rather than something that should ever arrive dressed up as a routine dependency update.

The generalizable rule: automated dependency updates are a great default for anything stateless, where “roll back the tag” is always a complete fix. The moment an image represents something stateful, that assumption breaks silently and specifically at major-version boundaries, because whether the engine self-migrates or refuses to start is engine-specific behavior no automation is checking on your behalf. Carve stateful images out of automatic major-bump merges before you find out the hard way which of your databases falls on which side of that line.