Your Liveness Probe Is Checking the Wrong Thing

The pod was Running. The liveness probe was green. There were 521 items in the queue and every worker was idle. It had been that way for hours.

This is a home media-transcoding box, so the cost of the outage was that some files didn’t get converted. The mechanism is the interesting part, because it’s the same mechanism in anything with a worker pool.

What the probe was checking

The container needs a GPU, and the GPU passthrough had been flaky when I first set it up. So the liveness probe ran a GPU query — nvidia-smi — and treated success as health.

That probe was correct. It never gave a wrong answer. The GPU was visible in the container, continuously, throughout the entire outage. It was answering a question I had stopped caring about roughly a week after I set it up, and it kept answering it for a year.

What actually broke

The app is Python. It has a scheduler thread that pulls from the pending queue and assigns work to the worker pool. That thread crashed:

sqlite3.OperationalError: database is locked

A thread crash is not a process crash. The interpreter kept running. The web UI kept serving. The workers kept existing, doing nothing, because the thing that hands them work was dead. Kubernetes saw a healthy container because the container was healthy by every definition the platform had been given.

Two details made it worse than a plain crash:

The traceback went to stdout, not to the application’s own log file. The app has a log file. I look at the log file. The log file showed a completely normal startup and then nothing unusual, because the fatal error for one thread was written somewhere else entirely. There was no alert to write, because there was no line to match — not in the place anyone would look.

The crash was a boot race. The trigger was three things reaching for the same SQLite file within a second of startup: the scheduler coming up, an on-start full library scan, and a scheduled-cleanup thread. Reproduced on two consecutive restarts. So it wasn’t a freak event — it was a coin flip on every restart, and the restart was also the cure, which is a fun way to build something that self-heals about half the time.

The rewrite

The new probe doesn’t ask whether the process is alive. It asks whether the system is stuck, using the definition a user would give: there is work waiting and nothing is working on it.

Two signals, ANDed:

• queue depth — read straight out of the SQLite file with sqlite3 -readonly
• worker activity — from the app’s own worker status endpoint

Backlog present AND all workers idle → unhealthy. Everything else → healthy.

The AND is load-bearing. Queue depth alone flags a system that’s busy and behind, which is normal. Idle workers alone flag a system with nothing to do, which is also normal. Only the combination is impossible in a working system.

Then three deliberate softenings:

• Dwell time. failureThreshold: 2 at periodSeconds: 300 means the backlogged-and-idle condition has to hold for about ten minutes before anything restarts. A brief gap between tasks can’t trip it.
• No pending work is healthy. An idle system with an empty queue is a correct system, not a broken one.
• An unreachable status API is healthy. If the probe can’t determine the answer, it returns success. A probe that restarts on ambiguity turns a two-second blip into a crashloop, and a crashloop is a much worse outage than the one you were guarding against.

That last one is a rule I’d state generally: liveness probes should fail closed on evidence and open on uncertainty. They’re wired to a destructive action. Make them prove their case.

What the restart was destroying

Fixing the probe means restarts now happen automatically, so I went looking at what a restart actually does — and found a setting called “clear pending tasks on restart” that was enabled and had been silently discarding the entire real backlog every single time.

So the pre-existing behaviour was: scheduler dies, I eventually notice, I restart the pod, the queue is wiped, the system looks fine because there’s nothing left to be behind on. The cure was erasing the evidence of the disease. That’s why this had happened more than once without me understanding it.

I disabled that, and disabled the on-start full scan — one of the three threads racing for the lock at boot. But those two together meant a restart no longer rebuilt the work queue on its own, so I added a postStart hook that polls a cheap version endpoint until the app responds, then triggers a rescan. Deliberately after the startup window rather than during it, so the fix doesn’t reconstruct the exact race it exists to avoid.

That sequence — fix, discover the fix removed something load-bearing, add the thing back in a shape that doesn’t reintroduce the original bug — is most of the actual work in any of these.

The general shape

Process liveness is a proxy for usefulness, and it’s a proxy that only holds for single-threaded programs that die loudly when they fail. That describes almost nothing you run in production. Anything with a worker pool, a scheduler thread, a consumer loop, or an async event loop can be one hundred percent alive and zero percent useful, indefinitely, and every platform-level signal will be green the entire time.

There’s a related tell in what a probe tests. Probes that check the platform’s view — the port is open, the process exists, the device is visible, the container responds to /healthz from a framework you didn’t write — are testing your infrastructure. That’s a real thing to test, once, when it’s new. Probes that check whether the workload is producing output are testing the thing you actually deployed. Most probes are the first kind long after the question stopped being interesting.

Takeaways

• Write the probe against the symptom a user would report, not against the process table. “Work is queued and nothing is progressing” beats “the process exists.”
• AND two signals so “quiet” is never confused with “stuck.”
• Give it dwell time, and make the ambiguous case healthy. The probe’s action is a restart; it should need real evidence.
• Check where your app’s crash tracebacks actually go before you assume you can alert on them. Thread-level exceptions often land somewhere other than the app’s own log.
• When you add a restart-based cure, audit what a restart destroys. If it clears state, it may have been hiding the failure from you for a long time.
• Revisit probes when the fear that motivated them has passed. Mine was guarding a hardware problem I’d fixed a year earlier while ignoring the only failure that had ever actually happened.