Focal Plane

August 31, 2026

Your OOM-Killed Pod's "Peak Memory" Is a Lie

There’s a piece of Kubernetes advice I believe in and repeat constantly: don’t guess resource limits. Ship the behavioural change, watch the real baseline in metrics for a few days, then size the limits from data. Guessed limits are either too tight (throttling, OOM kills) or too loose (wasted headroom on every node in the fleet). The baseline is cheap to measure and expensive to guess.

There’s a trap inside that advice, and it eats an afternoon every time I forget it.

The trap

A controller on our CI cluster was leaking memory and getting OOM-killed a few times a day. Standard procedure: pull 30 days of container_memory_working_set_bytes, find the peak, add headroom, set request and limit to that number.

The peak was almost exactly the limit.

Naturally. A container cannot allocate past its memory limit — it dies first. So the metric climbs toward the ceiling, flat-tops, and the series ends at a restart. Whatever the process actually wanted, the graph can only ever show you the ceiling.

The observed peak of an OOM-killed container is a lower bound on demand, not a peak. The limit you’re trying to size is truncating the only measurement you’d size from.

Censored versus uncensored memory observation Left: memory usage climbs into the limit, flat-tops, and resets at each OOM restart, so the observed peak equals the limit. Right: usage varies with visible headroom below the limit, so the observed peak is a real peak. OOM-killed: the peak is censored Headroom: the peak is real limit limit demand you can never observe headroom restart restart observed peak = limit → a floor observed peak < limit → a measurement
Same metric, same query. Only the right-hand panel is telling you about the workload.

If you don’t notice, the arithmetic is seductive and wrong. Peak was 2 GiB, add 15%, set 2.3 GiB, ship it. The container now flat-tops at 2.3 GiB and keeps dying — and you now have a fresh 30-day window that also says “peak ≈ limit,” which invites the same 15% again. I’ve watched that loop run three times on one workload. Each round looks like careful data-driven sizing. Each round produces a number derived from the previous guess rather than from the workload.

Reading the shape instead of the number

Peak ≈ limit, plus OOM kills, is not a measurement. It’s a censored observation, and the honest summary is: demand is unknown and strictly greater than the current limit.

So the recovery move isn’t a percentage, it’s a step change. Raise request and limit together by something generous — 30% or more, double it if the leak is fast — and re-observe. The point of the raise isn’t to be correct. It’s to buy an uncensored window. Once the workload runs for a few days without touching the ceiling, the peak you measure is a real peak, and only then do the normal rules apply: I aim for 30–50% over a 30-day observed peak, with memory requests equal to limits so the pod lands in the Guaranteed QoS class.

Two corollaries worth keeping.

A metric measured against an active constraint is shaped by that constraint. This generalizes well past memory. CPU usage under a CPU limit tells you about the limit, not the demand. Queue depth under a hard producer cap tells you about the cap. Connection counts against a pool ceiling tell you about the pool size. Any time your observed maximum sits suspiciously flush against a configured maximum, you are measuring your own configuration and calling it telemetry.

A leak doesn’t have a peak at all. If usage grows monotonically from restart to restart, no limit is the right limit — every number you pick only changes how often the process dies. Sizing is the wrong tool, and the honest move is to say so out loud: set a limit generous enough to make the restart frequency tolerable, label it a stopgap in the commit message, and go file the actual bug. On that controller we bumped to 3 GiB explicitly as a holding action, and the real fix was an upstream version bump.

Distinguishing the two takes one graph. A sawtooth that resets on restart and climbs at a steady rate regardless of traffic is a leak. A noisy curve that correlates with load and happens to clip the ceiling at peak hours is a sizing problem.

Leak sawtooth versus load-correlated sizing problem Left: memory climbs at a constant rate from each restart, unrelated to the traffic curve below it — a leak. Right: memory tracks the traffic curve and clips the limit only at peaks — a sizing problem. Leak: slope ignores traffic Sizing: memory tracks traffic limit limit traffic (flat) — no relationship traffic — same shape, clips at peak no limit is the right limit — go fix the bug size it from an uncensored window
Overlay the traffic curve. If the slope doesn't care what traffic did, sizing is the wrong tool.

The version of the rule I use now

Observe before setting limits — but check that the observation isn’t being taken through the very thing you’re setting.

Before I trust a peak, four questions:

  1. Were there OOM kills in the window? If yes, the peak is a floor. Don’t size from it.
  2. Is the peak within a few percent of the limit? Same answer, even with no kills — the workload may be quietly shaping itself to fit, dropping caches or GCing harder than it should.
  3. Is the series flat-topped, or is there room above it? Visible headroom above the maximum is what makes the maximum meaningful.
  4. Is it a leak? Then stop sizing and go fix it.

Only a window with room above the maximum produces a number worth putting in a manifest. Everything else is a measurement of your own limit, handed back to you as though it were the workload’s opinion.

← all writing