The Error Message Named the Wrong Failure Three Times in a Row

I deployed a new service to a cluster last week. It crash-looped with this:

Error: postgres: querying system variables: pq: password authentication failed for user app

That is an unambiguous error message. It names the failure, the user, and the subsystem. It is also, in this case, wrong three times over — and each time I fixed what it said, the symptom stayed identical.

Here’s the chain, because I think the chain is more instructive than any one link.

Failure 1: the password wasn’t wrong, it was absent

I went looking for a mismatch between the password in the database and the password in the secret. There wasn’t one. There was no password in the secret at all.

The secret existed. It had all nine expected keys. Every value was zero bytes.

The deployment had a documented post-apply step — copy the generated database password from parameter store into the application secret — and that step had been skipped. So the application received an empty string, presented it as a password, and the database rejected it. Correctly! From the database’s point of view, authentication genuinely did fail.

“Authentication failed” and “no credential was supplied” produce the same error on the wire. The client library can’t distinguish them and neither can the server log. So the first thing to check on any auth failure isn’t whether the credential is right — it’s whether the credential is there. len(value) == 0 is a two-second check that I now do first, every time, because an empty secret looks exactly like a wrong secret and the debugging paths diverge completely.

Related trap in the same family: a value that’s present but has a trailing newline from however it was piped in. Also authenticates as “wrong password.”

Failure 2: the fix landed but the cluster didn’t hear about it

I populated the secret in the external secret store. Verified it. Restarted nothing, because the secrets operator syncs continuously — that’s the entire point of it.

Same error.

The operator’s refresh interval was one hour. It syncs continuously the way a bus runs continuously: on a schedule, and not because you’d like it to. There was up to an hour of staleness between the source of truth being correct and the in-cluster secret reflecting it, and nothing in the cluster indicated a difference — the resource looked synced, because it was synced, to the value from the last poll.

The fix is to force a resync rather than wait:

kubectl annotate externalsecret my-app force-sync="$(date +%s)" --overwrite

The annotation value doesn’t matter, it just has to change. But the lesson is the one I keep relearning about every reconciler: “declarative and continuously reconciled” describes the steady state, not the propagation delay. When you’re debugging, the refresh interval is the length of time your fix can appear to have failed.

Failure 3: the pod was never going to read it anyway

Secret populated. Resync forced. Confirmed the in-cluster secret now had real values.

Same error.

The container consumed its config via envFrom, and environment variables are resolved at pod creation. The running container holds the values it was handed at birth. A crash-looping container is being restarted, not recreated — same pod, same env block, same empty strings, forever. It will never see the corrected secret no matter how many times it dies.

kubectl rollout restart deploy/my-app

That’s the whole fix, and it’s the step I skipped because the pod was already restarting every thirty seconds. Restarts felt redundant. But a container restart and a pod recreation are different operations with different config semantics, and crash-loop backoff does the one that doesn’t help.

This is also why mounted-file secrets behave better than envFrom for anything that might rotate: a projected volume updates in place, so a process that re-reads the file picks up new values without a rollout. Environment variables are a snapshot with no invalidation story.

Failure 4: and underneath all of it, a placeholder in production

Once the service started, it still wasn’t reachable. The load balancer returned 503.

The manifest that binds the service to its load balancer target group had been merged with a literal placeholder still in it:

targetGroupARN: REPLACE_AFTER_TF_APPLY

The pattern was intentional — the target group is created by infrastructure code, so the value isn’t known at authoring time, and the placeholder was meant to be filled in after the apply. It got merged instead.

The controller rejected the resource as invalid, which meant the binding object never existed, which meant the target group had zero registered targets, which meant the load balancer had nothing to route to. And the GitOps dashboard looked broadly fine, because from its perspective the desired state had been applied — the rejection happened downstream, in a controller’s admission path, not in the sync.

Four distinct failures. One error message, which described none of them accurately.

What I actually changed about how I work

The meta-lesson is about trust. A precise error message feels like evidence, and it earns more credence than it deserves — precision and accuracy are different properties. That message named a user, a subsystem, and a failure mode, and the actual problem was that somebody didn’t paste a string. Read error messages as the innermost layer’s opinion, which is honest, local, and frequently has no idea what actually went wrong.


← all writing