Let me paint you a picture.
It's 11pm on a Tuesday. Traffic died down hours ago. Your Karpenter cluster is sitting on 14 half-empty nodes like a hotel with 200 rooms and 40 guests — all refusing to share a floor.
Meanwhile, AWS is charging you for all 14 rooms. Fully. Every hour. All night.
That's not a Karpenter problem. That's a consolidation problem. And the fix is one field in your YAML that most people either skip or misconfigure and never revisit.
π€ The Situation (Real Talk)
Consolidation is Karpenter's way of saying: "Hey, I can fit everything on fewer nodes. Want me to clean this up?"
The answer should always be yes. But by default — or with the wrong policy — Karpenter either waits forever or never asks at all.
I've seen clusters running WhenEmpty for six months. Six months of nodes sitting at 20% utilization because the policy only evicts nodes with zero pods. One DaemonSet pod. That's all it takes to block a node from ever being consolidated. The hotel room stays paid. The guest is a goldfish.
π§ How Karpenter Consolidation Actually Works
Think of Karpenter's consolidation engine like a smart hotel concierge. After the dinner rush, they walk the floors. They see guests scattered across 10 floors and think: "I can move everyone to 3 floors and close the elevators on the rest. Guests won't notice. Owner saves money."
That's exactly what WhenEmptyOrUnderutilized does. It looks at every node, evaluates whether pods can be moved without violating PodDisruptionBudgets or topology constraints, then quietly migrates them and terminates the empty node.
But here's the part most docs gloss over: Karpenter doesn't just evict pods randomly. It simulates the rescheduling first. It asks: "If I remove this node, can every pod on it land somewhere else right now?" Only if the answer is yes does it pull the trigger.
⚙️ The Two Policies — And Why One Is Much Better
WhenEmpty
Removes a node only when it has zero non-DaemonSet pods. Sounds safe. Is. Also nearly useless in any real cluster where DaemonSets run everywhere.
WhenEmptyOrUnderutilized
The one you actually want. Considers a node for removal when its pods can be repacked onto other nodes. Aggressively right-sizes your fleet after traffic drops.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: production-general
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
limits:
cpu: "200"
memory: 400Gi
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1m # how long a node must be underutilized before eviction starts
π That consolidateAfter: 1m is the patience window. Karpenter waits 1 minute of confirmed underutilization before acting. Too short and you risk unnecessary churn. Too long and you're just paying for idle compute.
π‘️ How to Save Money Without Breaking Pods
This is where it gets real. Consolidation evicts pods. Eviction is only safe if your workloads are designed for it.
Step 1 — Set PodDisruptionBudgets on everything that matters
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
namespace: production
spec:
minAvailable: 2 # at least 2 replicas must stay running during disruption
selector:
matchLabels:
app: api-serverKarpenter respects PDBs. If draining a node would violate this budget, it skips that node entirely and moves on. Your pods stay running. The consolidation still happens on other nodes.
Step 2 — Make sure your pods have proper resource requests
Karpenter uses requests — not actual usage — to simulate whether pods fit on existing nodes. Bloated requests mean Karpenter thinks pods are heavier than they are, and consolidation never triggers because "there's no room." Set requests close to real P95 usage.
Step 3 — Use topology spread constraints if you care about AZ distribution
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api-serverπ Without this, consolidation might pack all pods into one AZ. Karpenter is smart, but it won't protect your availability unless you tell it to care.
⚠️ Common Mistakes That Will Ruin Your Day
- Running WhenEmpty and wondering why consolidation never triggers — DaemonSets. Every time.
- Setting consolidateAfter too low (like 10s) — You'll see nodes thrashing. Traffic ticks up for 20 seconds and Karpenter immediately spins up a node. Traffic drops and it tries to remove it. Repeat. That's churn, not efficiency.
- No PDBs on stateful workloads — Karpenter will happily evict your single-replica database if you let it. Don't let it.
- Forgetting that Jobs and batch workloads resist consolidation — Pods in Running state from Jobs block node eviction. Use appropriate completionMode and ttlSecondsAfterFinished to clean them up.
π₯ CloudChef Pro Tips (Production-Grade)
Use multiple NodePools with different consolidation behavior.
π Your batch workload NodePool can be aggressive (consolidateAfter: 30s). Your production API NodePool should be gentler (consolidateAfter: 5m). Don't apply one setting across everything.
Watch the right metrics.
π Track karpenter_nodes_total and karpenter_disruption_disruptions_total in your Prometheus stack. If disruptions are zero but utilization is low, your PDBs might be blocking consolidation silently.
# Check what's blocking consolidation right now
kubectl get nodeclaims -o wide
kubectl describe nodeclaim <name> | grep -A5 Conditionsπ Continue Your CloudChef Journey
π References
π Final Thoughts
Karpenter consolidation is one of those rare infrastructure features where doing it right costs you nothing except five minutes of configuration. Doing it wrong costs you money every single hour — silently, in the background, forever.
π Check your NodePool disruption policy right now. If it says WhenEmpty, you now know exactly what to do. If it's missing entirely, you have your answer for why the AWS bill looks like that.
No comments:
Post a Comment