Smaller Nodes, Bigger Bill

The node change did what the model said it would. After we moved a cluster’s Karpenter pools to different instance families, EC2 spend for that cluster fell about 21%. Then the invoice from the observability vendor came in, and the billed host count had gone up by roughly a sixth. Fewer dollars to the cloud provider, more to the vendor, all from one change.

Taken together, the higher vendor bill gave back somewhere between a quarter and a half of the EC2 saving. Nothing was misconfigured. The model just never priced the fact that the cluster now ran more nodes.

The setup

The cluster is a non-production EKS cluster with two Karpenter NodePools: an on-demand default pool and a spot pool for workloads that tolerate interruption. Over a few weeks both pools got a family review. The default pool had only been allowed a couple of families, and Karpenter kept picking r6i.xlarge. Those nodes were CPU-request-bound at about 90% while roughly half their memory sat stranded. Opening the pools to c, m and r families (generation 6 and up) let Karpenter pick shapes that matched the requests, and node cost per hour fell.

Every one of those PRs priced the change the same way: requests against the instance price list. That’s the obvious model, and as far as EC2 goes it was right.

It also pushed the cluster toward xlarge nodes. The spot pool allowed xlarge, 2xlarge and 4xlarge, and when the weekday scale-up came, Karpenter would add a batch of small nodes at once because they were the cheapest fit for each pending pod.

Two costs that scale with node count, not with vCPU

EC2 is priced per vCPU, near enough. Within a family, a 2xlarge costs about twice an xlarge, so on the cloud bill node size barely matters and bin-packing is what you optimize. Two other costs don’t follow that rule.

The per-host agent. Most APM, infrastructure monitoring and security agents run as a DaemonSet and bill per host. A 2xlarge and an xlarge are one host each. Ours is billed on a common model: the vendor counts hosts every hour, drops the top 1% of hours (about seven in a month), and charges the whole month at the highest count that remains. That’s the 99th percentile of hourly host count. Your bill is not your average fleet. It is roughly your busiest normal hour.

That detail matters for autoscaled clusters. A weekday scale-up that lasts a few hours, or a burst of consolidation churn where old and new nodes overlap, sets the price for the month. The quiet weekend nights you spend at minimum size don’t help at all.

DaemonSet overhead. Every node runs the same set of DaemonSets: the CNI, kube-proxy, a log shipper, the observability agent, a security agent. On this cluster their requests came to about 0.41 vCPU and 1.2 GiB per node. That slice is reserved on every node regardless of size. On a 4-vCPU node it’s a tenth of the CPU. On an 8-vCPU node it’s a twentieth. Halve the node size and you pay the overhead twice as often.

Neither of these shows up when you compare instance prices per vCPU. Both get worse as nodes get smaller.

The dig

The vendor’s own usage metric gives the hourly billable host count, and the agent reports one series per host, so it’s easy to split the count by cluster. In the months before the change, billed hosts were steady. Afterwards they rose by about a sixth, and most of the excess traced back to this cluster’s weekday spot scale-ups: each one added a group of xlarge nodes at the same time, and those hours set the percentile.

The spot pool was also memory-full: close to 100% of memory requested and about 70% of CPU, on xlarge nodes. The scheduler was already hitting the wall on memory, one small node after another.

Then the useful check: what does spot 2xlarge cost per vCPU compared with spot xlarge? About the same. Spot pricing per vCPU is fairly flat across sizes within a family, so on the spot pool bigger nodes are a near-free win. For the live requests, the model said nine xlarge nodes would become about four 2xlarge. EC2 moves a little in our favour. Five fewer billed hosts come off the vendor bill, and the scale-up spikes that set the percentile get smaller too. The vendor saving came out at roughly one and a half to two and a half times the EC2 saving. On a non-production cluster the per-host fee was the bigger lever.

So the spot pool went to 2xlarge only. Not 4xlarge: with spot, the node is also the unit of interruption, and at that size a single reclaim would take out about a quarter of the pool. The family list stayed broad, so the pool still had plenty of spot capacity to draw from.

The on-demand default pool came out differently. On-demand xlarge packs tighter against real requests, and the stranded capacity on a half-empty 2xlarge roughly cancels the host fee it saves. The model came out roughly break-even, so that pool stayed as it was. The production cluster wasn’t worth changing either until its family mix changed.

The insight

Instance size is not only an EC2 decision. It is also a pricing decision for every vendor that bills per node, and for every DaemonSet that reserves a slice of each node. The cloud price list is per vCPU. Everything else is per host. A sizing model that only reads the first one will always favour smaller nodes, because it sees the bin-packing win and none of the costs.

The fix is to price a node the way the bill does: what it costs per hour all-in, divided by what you can actually schedule on it.

Effective cost per usable vCPU by node size (illustrative) With a per-host fee and per-node DaemonSet overhead included, a 2-vCPU node costs about 59% more per usable vCPU than the list price, while a 16-vCPU node costs about 6% more. The penalty for small nodes is much larger once per-host costs are counted. 0.05 0.06 0.07 0.08 2 vCPU 4 vCPU 8 vCPU 16 vCPU node size (axis starts at 0.04) $ per usable vCPU-hour list price per vCPU: 0.05 + per-host fee + DaemonSet overhead DaemonSet overhead only
Illustrative numbers, not real prices: $0.05 per vCPU-hour list price, 0.4 vCPU of DaemonSets per node, and a $20/month per-host fee. The real bill is shaped the same way.

The line drawn from the price list is flat at 0.05. The two lines that model a real node both slope down, and the per-host fee makes the slope steep at the small end. Going from 4 to 8 vCPU saves about 11% per usable vCPU in this example. Very little of that would appear on a price list.

The line is not the whole story. Bigger nodes strand more capacity when the last one is half empty, more so in a small pool, and they widen the blast radius of a single interruption or drain. Those costs pull the other way, which is why the on-demand pool broke even. The point is to put both sides into the same calculation, not to always buy bigger.

Takeaways


← all writing