The Error Message Named the Wrong Failure Three Times in a Row
I deployed a new service to a cluster last week. It crash-looped with this:
Error: postgres: querying system variables: pq: password authentication failed for user app
That is an unambiguous error message. It names the failure, the user, and the subsystem. It is also, in this case, wrong three times over — and each time I fixed what it said, the symptom stayed identical.
Here’s the chain, because I think the chain is more instructive than any one link.
Failure 1: the password wasn’t wrong, it was absent
I went looking for a mismatch between the password in the database and the password in the secret. There wasn’t one. There was no password in the secret at all.
The secret existed. It had all nine expected keys. Every value was zero bytes.
The deployment had a documented post-apply step — copy the generated database password from parameter store into the application secret — and that step had been skipped. So the application received an empty string, presented it as a password, and the database rejected it. Correctly! From the database’s point of view, authentication genuinely did fail.
“Authentication failed” and “no credential was supplied” produce the same error on the
wire. The client library can’t distinguish them and neither can the server log. So the
first thing to check on any auth failure isn’t whether the credential is right — it’s
whether the credential is there. len(value) == 0 is a two-second check that I now do
first, every time, because an empty secret looks exactly like a wrong secret and the
debugging paths diverge completely.
Related trap in the same family: a value that’s present but has a trailing newline from however it was piped in. Also authenticates as “wrong password.”
Failure 2: the fix landed but the cluster didn’t hear about it
I populated the secret in the external secret store. Verified it. Restarted nothing, because the secrets operator syncs continuously — that’s the entire point of it.
Same error.
The operator’s refresh interval was one hour. It syncs continuously the way a bus runs continuously: on a schedule, and not because you’d like it to. There was up to an hour of staleness between the source of truth being correct and the in-cluster secret reflecting it, and nothing in the cluster indicated a difference — the resource looked synced, because it was synced, to the value from the last poll.
The fix is to force a resync rather than wait:
kubectl annotate externalsecret my-app force-sync="$(date +%s)" --overwrite
The annotation value doesn’t matter, it just has to change. But the lesson is the one I keep relearning about every reconciler: “declarative and continuously reconciled” describes the steady state, not the propagation delay. When you’re debugging, the refresh interval is the length of time your fix can appear to have failed.
Failure 3: the pod was never going to read it anyway
Secret populated. Resync forced. Confirmed the in-cluster secret now had real values.
Same error.
The container consumed its config via envFrom, and environment variables are resolved
at pod creation. The running container holds the values it was handed at birth. A
crash-looping container is being restarted, not recreated — same pod, same env block, same
empty strings, forever. It will never see the corrected secret no matter how many times it
dies.
kubectl rollout restart deploy/my-app
That’s the whole fix, and it’s the step I skipped because the pod was already restarting every thirty seconds. Restarts felt redundant. But a container restart and a pod recreation are different operations with different config semantics, and crash-loop backoff does the one that doesn’t help.
This is also why mounted-file secrets behave better than envFrom for anything that might
rotate: a projected volume updates in place, so a process that re-reads the file picks up
new values without a rollout. Environment variables are a snapshot with no invalidation
story.
Failure 4: and underneath all of it, a placeholder in production
Once the service started, it still wasn’t reachable. The load balancer returned 503.
The manifest that binds the service to its load balancer target group had been merged with a literal placeholder still in it:
targetGroupARN: REPLACE_AFTER_TF_APPLY
The pattern was intentional — the target group is created by infrastructure code, so the value isn’t known at authoring time, and the placeholder was meant to be filled in after the apply. It got merged instead.
The controller rejected the resource as invalid, which meant the binding object never existed, which meant the target group had zero registered targets, which meant the load balancer had nothing to route to. And the GitOps dashboard looked broadly fine, because from its perspective the desired state had been applied — the rejection happened downstream, in a controller’s admission path, not in the sync.
Four distinct failures. One error message, which described none of them accurately.
What I actually changed about how I work
- On any auth failure, check for empty before checking for wrong. They’re indistinguishable on the wire and the fixes have nothing in common.
- A documented manual post-apply step is a bug with a deadline. Two of the four failures here were “a human was supposed to copy a value.” If a value must move from A to B for the system to work, that’s automation you haven’t written yet, and “it’s in the runbook” is where the outage comes from.
- Never let a placeholder be syntactically valid.
REPLACE_AFTER_TF_APPLYmerged cleanly because it’s a legal string. If the placeholder had been structurally invalid the manifest would have failed validation in CI instead of admission at runtime. Make the unfilled state unmergeable. - When a fix doesn’t take, suspect propagation before suspecting the fix. Refresh intervals, env-var snapshots, and cached renders all present as “my change did nothing.”
The meta-lesson is about trust. A precise error message feels like evidence, and it earns more credence than it deserves — precision and accuracy are different properties. That message named a user, a subsystem, and a failure mode, and the actual problem was that somebody didn’t paste a string. Read error messages as the innermost layer’s opinion, which is honest, local, and frequently has no idea what actually went wrong.