Do the Arithmetic Before Blaming the Database

Every slow-application investigation has a moment where someone opens the database dashboard, sees a spike, and the room exhales. There’s the culprit. Read IOPS tripled at exactly the wrong time. Queue up the index review, schedule the instance resize, write the retro.

I’ve been that someone. The dashboard is the most seductive artifact in the whole investigation, because a database under load always looks a bit alarming and there is always a spike somewhere in the window.

There’s a piece of arithmetic that settles the question in about two minutes, before any of that starts, and it’s the first thing I reach for now.

The bound

Managed databases expose some form of average active sessions — the mean number of sessions actively doing something (running queries, waiting on I/O, waiting on locks) at any instant. It’s a load metric, and critically it includes I/O waits, so “the disk was slow” is inside it, not outside it.

Multiply it by the window:

AAS × window (seconds) = total session-seconds the database could possibly have consumed

That’s a hard ceiling. If the database averaged 4 active sessions over a 10-minute window, it spent at most 4 × 600 = 2,400 session-seconds doing anything at all. Not 2,401. There is no mechanism by which a database with 4 average active sessions consumes more than that.

Now measure the other side: how much time did the slow requests actually spend waiting? Number of slow calls × their average duration.

If the second number exceeds the first, the waiting did not happen in the database. Not “probably didn’t.” Couldn’t. The time doesn’t fit.

Suppose 20,000 requests each stalled 3 seconds in that same window. That’s 60,000 seconds of stall against a 2,400-second budget — off by a factor of 25. Whatever the IOPS graph is insinuating, it cannot account for the stalls. The time went somewhere else: a connection pool with no free connections, a lock in the application, a saturated thread pool, a DNS timeout, a downstream service, a retry storm feeding on itself.

Why this particular check earns its place

It’s a refutation, not a correlation. Almost everything else in an RCA toolkit produces evidence toward a hypothesis. Graphs that move together, timelines that line up, a deploy at the right minute. All suggestive, all compatible with coincidence. This check can definitively rule the database out, and “definitively” is rare enough in production debugging to be worth reaching for first.

It costs two numbers. Both are on the default dashboard of any managed database. No query analysis, no profiler, no reproduction, no additional instrumentation.

It runs before the expensive work. The point of an early bound is to redirect effort. Tuning a database that wasn’t the problem can absorb days — you will find slow queries, because every database has slow queries, and fixing them will not fix the outage. Then the incident recurs and nobody trusts the RCA.

It survives being wrong in your favour. If the stall time fits comfortably inside the session-seconds budget, you haven’t proved the database is guilty. You’ve only failed to exonerate it, which is a reason to keep investigating there rather than a verdict. That asymmetry is exactly what you want from a cheap first-pass filter.

The generalization

The specific metric is database-flavoured, but the move isn’t:

Bound the suspect’s maximum possible contribution, then compare it to the observed effect.

• A service is blamed for latency, but its own request logs total 200 seconds of processing across a window in which users waited 3 hours. It’s not the source.
• A cache is blamed for a load spike, but its miss count times the cost of a miss is a small fraction of the extra backend work observed. Something else is generating traffic.
• A cron job is blamed for saturating a disk, but its total bytes written can’t account for the IO observed. Look elsewhere.

Every one of these is capacity arithmetic against an observed effect, and every one can be done before opening a profiler.

The habit

When someone says “it’s the database,” ask two questions before agreeing:

1. What was average active sessions over the window?
2. How much total wall-clock time did the affected requests spend waiting?

If the second is bigger than the first times the window length, the database has an alibi, and the investigation belongs in the application tier.

The reason I wrote this down is that the failure mode isn’t ignorance — it’s momentum. The database dashboard is where investigations go because it’s where the scariest-looking graph lives, and a room that has agreed on a suspect stops looking. Two multiplications are enough to break that, and they’re cheapest at the very start, before anyone has invested a day in being right.