preStop Doesn't Save a Client That Never Reconnects
I added a 15-second preStop sleep to every Prometheus in four clusters to stop an autoscaler
from seeing 503s during restarts. It made no difference. A week later the autoscaler got four
more 503s seconds after a Prometheus replica was told to shut down. The fix was a client-side
setting I hadn’t thought about, and the reason it took me two attempts is that nearly every
piece of Kubernetes graceful-shutdown advice makes an assumption it doesn’t state.
The setup
KEDA drives autoscaling for a set of web and worker deployments. Several of their ScaledObjects use the Prometheus scaler: on every polling interval, KEDA’s operator sends a PromQL query to Prometheus and turns the answer into a replica target for the HPA.
Prometheus runs as an HA pair behind a regular ClusterIP Service, managed by prometheus-operator. Two replicas, a PodDisruptionBudget, each scraping the same targets. On paper, losing one replica for a minute should be invisible to anything that queries through the Service.
It wasn’t. During a rolling restart of Prometheus, KEDA logged a 503 Service Unavailable on
its poll for two worker ScaledObjects.
Attempt one: the textbook fix
The textbook explanation for errors during a rolling restart goes like this. When a pod is deleted, two things happen in parallel: the kubelet starts terminating the container, and the endpoint controller removes the pod from the Service’s endpoints, which kube-proxy on every node then picks up. Neither waits for the other. If the container stops serving before every node has stopped routing to it, some requests land on a pod that’s on its way out.
The standard fix is a preStop hook that sleeps, so the container keeps serving for a few
seconds while endpoint removal propagates. I added one to the prometheus container through
the Prometheus CR’s spec.containers override, which the operator merges by container name:
spec:
containers:
- name: prometheus
lifecycle:
preStop:
sleep:
seconds: 15
That’s the native sleep handler, so there’s no dependency on a shell in the image.
prometheus-operator’s default terminationGracePeriodSeconds is 600, far above 15, so the
sleep can’t eat into Prometheus’s own shutdown. I rolled it out one cluster at a time and
moved on.
It changed nothing
About a week later Karpenter consolidated the node under one of the production Prometheus
replicas. Consolidation evicts the pod, the pod gets its SIGTERM, and seconds into that shutdown
KEDA logged four 503 Service Unavailable responses from Prometheus. The preStop hook was
deployed. The other replica was healthy. The Service’s endpoints were doing what they’re
supposed to do.
The same pattern turned up in the logs once I went looking for it. Over 30 days there were 33 separate bursts of scaler 503s across two clusters, each one lined up with a Prometheus restart and each one cleared within about five minutes.
The assumption
The preStop story only works if the client opens a new connection for the request that
matters.
kube-proxy, in its default iptables mode, doesn’t proxy anything. It writes NAT rules. When a client opens a TCP connection to the Service’s virtual IP, the first packet matches a rule that picks a backend pod and rewrites the destination. The kernel’s connection tracking table remembers that choice, and every later packet on that connection goes to the same pod IP without looking at the Service rules again.
Removing an endpoint changes the rules. It doesn’t touch connections that already exist. A new connection after the update can only pick a ready pod. An established connection stays pinned to whichever pod it picked when it was opened, for as long as both ends keep it open.
KEDA’s scalers use Go’s HTTP client with keep-alive on, which is the normal default. The operator polls on an interval (30 seconds unless you change it), and the client keeps the connection open between polls so it can reuse it. From the network’s point of view, KEDA connected to one Prometheus pod at some point in the past and has been talking to that pod ever since.
So when that pod starts shutting down, the endpoint update is irrelevant: KEDA isn’t asking
the Service where to go, it’s writing to a socket it already has. And the preStop sleep only
buys time for clients that will look up the Service again. It holds the old pod in a
serving state while traffic moves away from it, and keep-alive traffic doesn’t move. When the
pod finally stops serving properly, the next poll on that connection gets the error. (I’ll be
honest that the exact timing of the 503s relative to the sleep is something I inferred from
the logs rather than traced packet by packet. The pinning is the part that matters, and it
explains why the sleep did nothing.)
This also explains why the bursts scaled with the number of ScaledObjects that polled during the restart rather than with anything about Prometheus itself.
The fix
Make every poll a new connection. KEDA has a Helm value for exactly this:
http:
keepAlive:
enabled: false
which sets KEDA_HTTP_DISABLE_KEEP_ALIVE=true on the operator. With keep-alive off, every
scaler query opens a fresh TCP connection to the Service’s IP, the kernel consults the current
NAT rules, and the query can only land on a pod that’s still in the endpoints. A terminating
replica stops receiving queries as soon as kube-proxy has caught up, which is exactly the
window the preStop sleep was meant to cover. The sleep finally does its job, because the
client is now the kind of client it was designed for.
I shipped it alongside an unrelated KEDA version upgrade, gated per cluster, staging first.
It’s worth being clear about the stakes. The HPA replica counts didn’t move during that burst. When a
Prometheus scaler query fails, KEDA holds the current replica count (or the ScaledObject’s
fallback), and five minutes of holding steady during a Prometheus restart did no damage.
This was cleanup, not an outage fix. The real risk is the version of this where queries fail
for much longer and autoscaling runs blind without anyone noticing, which is why the
same piece of work added an alert for continuous failure: KEDA’s Prometheus scaler logging
transport errors (5xx, refused or reset connections, timeouts) in every 2-minute window for
10 minutes. Against the last 30 days, a simple count threshold would have fired about 33 times.
The sustained-failure rule would have fired zero times. Burst size is noise; duration is the
signal.
The other ways out
Turning off keep-alive is a blunt fix that happens to be cheap here. The general problem has a few other answers, and which one you pick depends on what you control.
Drain connections on the server. If you own the server, the right behaviour during
shutdown is to stop reusing connections: send Connection: close on responses once
termination starts, or close idle keep-alive connections, so clients are pushed to reconnect
while the pod is still healthy enough to answer. Go’s http.Server.Shutdown closes idle
connections, but only once you call it, which is after the preStop sleep. Useful drain logic
has to start at the beginning of the termination window, not the end. I don’t control
Prometheus’s shutdown path, so this wasn’t an option.
Bound connection lifetime on the client. An idle timeout shorter than the poll interval effectively disables reuse for a poller anyway. A maximum connection age (some clients and proxies offer one) caps how long a pin can last, but it narrows the window rather than closing it: a connection that happens to be young when the pod starts terminating is still pinned.
Balance per request instead of per connection. A service mesh or any L7 proxy in front of Prometheus routes each HTTP request on its own and drops terminating backends from its pool. That solves it properly, and it’s a lot of machinery to add for one poller.
Retry. A client that retries a failed poll on a new connection would hide most of this. That treats the symptom, and it only works if the retry actually gets a new connection rather than another request on a pooled one to the same pod.
Pay the reconnect cost. That’s what disabling keep-alive is. The cost is one TCP handshake per query per scaler. At one query every 30 seconds per ScaledObject, inside the cluster, that’s nothing. Keep-alive earns its place when a client makes many requests a second, or when every new connection pays for a TLS handshake. A poller that sleeps longer than most idle timeouts between requests gets almost nothing from it.
Takeaways
preStopsleeps and endpoint-removal delays protect clients that open new connections. A client holding a keep-alive connection never consults the Service again, so it gets nothing from them.- kube-proxy’s iptables mode routes connections, not requests. Removing an endpoint changes where new connections go; established ones stay pinned to the old pod IP.
- When graceful-shutdown changes “do nothing”, look at the client’s connection reuse before tuning the sleep.
- For a low-frequency poller, disabling keep-alive is a cheap, complete fix. For high-rate clients, prefer server-side draining or per-request load balancing.
- Alert on how long a dependency has been failing, not on how many errors a restart produced. The second one fires on every restart; the first one fires when it matters.