Pull up a chair. This one stings a little, because it's almost always the same tiny mistake.
I've gone through enough EKS bills at this point to spot the pattern from across the room: team adopts Karpenter expecting savings, bill goes up instead, and nobody can explain why.
Spoiler — it's not Karpenter's fault. It's one missing ingredient.
π€ The Setup That Looks Totally Fine
Most teams copy a NodePool example from the docs, swap in their own labels, and ship it. Looks reasonable. Nothing screams "this is about to be expensive."
Then a month later, finance asks why EC2 spend jumped 60% right after the "cost-saving" migration. Awkward.
π§ The Real Problem: Consolidation Is Off (or Half-Asleep)
Karpenter only saves you money if you let it actually clean up after itself. If your disruption block is missing or too lazy, idle nodes just sit there — like leaving the oven on after dinner because nobody walked back into the kitchen to check.
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 30s
π If this block is missing entirely, Karpenter defaults to a much more conservative cleanup schedule. Nodes that should've been turned off an hour ago just keep running. And billing.
π What This Looks Like in Practice
That little fork in the road is the entire difference between "Karpenter saved us a fortune" and "why does our bill look like this."
⚙️ Other Quiet Cost Leaks
- No
limits.cpuon your NodePool — Karpenter scales as far as your quota allows, no ceiling, like a buffet with no plate limit - Over-broad instance type requirements — letting Karpenter pick from huge, pricey instance families when smaller ones would do the job
- Spot disabled "to be safe" — giving up Karpenter's single biggest savings lever out of an old fear
- Bloated pod resource requests — Karpenter sizes nodes to what you ask for, not what you actually use. Ask for a feast, get billed for a feast.
π How to Check Your Own Cluster
kubectl get nodepool -o yaml | grep -A3 disruption
π If you don't see consolidationPolicy anywhere in that output, congratulations, you found your leak.
kubectl get nodes -o custom-columns=NAME:.metadata.name,AGE:.metadata.creationTimestamp,CAPACITY:.status.capacity.cpu
Cross-reference any old, low-pod-count nodes against your AWS Cost Explorer EC2 line item. The pattern usually jumps out at you immediately.
⚠️ Common Mistakes
- Setting
consolidateAfterway too high "to avoid thrashing" — and quietly paying for hours of idle capacity instead - Confusing
WhenEmpty(only removes fully empty nodes) withWhenEmptyOrUnderutilized(also repacks underused ones) — the second one is doing the actual heavy lifting - PodDisruptionBudgets that accidentally block consolidation on certain workloads entirely
π₯ CloudChef Pro Tip
Track average node utilization weekly.
π If it's sitting under 50%, your consolidation settings aren't doing their job — full stop, no exceptions.
π Continue Your CloudChef Journey
π References
π Final Thoughts
Karpenter doesn't save money on autopilot. It saves money when you actually tell it to be aggressive about killing idle capacity.
π Check that one disruption block today. Five minutes of your time versus months of quietly overpaying — easy trade.
No comments:
Post a Comment