Karpenter CapacityBuffers Recipe: Pre-Warmed Nodes, Percentage Buffers and a GPU Cost Leash

Picture a pizza shop on a Friday night. The smart owner doesn't wait for the first order to light the oven. The oven is already hot, the dough is already stretched, and the first customer gets a pizza in minutes instead of waiting for everything to warm up.

Karpenter, by default, is the opposite kind of shop. It launches a node when a pod can't schedule. That's efficient and cheap. It's also why your pod sits in Pending for a minute or two while a fresh node boots, and for GPU nodes, a good deal longer.

CapacityBuffers are Karpenter's way of lighting the oven early. They arrived with Karpenter 1.14, they're still alpha, and they replace a hack many of us have been running for years: the low-priority "dummy" Deployment that exists only to keep a node warm.

Let's cook this one properly, including the part where it can surprise you.

What We Are Cooking Today

• Understand what a CapacityBuffer is (and what it deliberately isn't)
• Enable the feature gate and deploy a fixed-size buffer
• Make it track a Deployment's size with a percentage buffer
• Avoid the traps: loose NodePools, GPU cost, and cold image pulls

How CapacityBuffers Work

A CapacityBuffer defines virtual placeholder pods. They exist only inside Karpenter's scheduling simulation, and they never show up in kubectl get pods. Karpenter provisions real, Ready nodes big enough to hold them, re-checks the buffer on every scheduling cycle, and refills it when real workloads eat the spare room.

Compare that with the old dummy-Deployment trick:

	Dummy low-priority Deployment	CapacityBuffer
Placeholder pods	Real pods you must manage	Virtual pods, no objects to preempt
Sizing	Guess instance shapes by hand	Karpenter runs its usual scheduling simulation
Eviction	Relies on preemption when real pods arrive	No preemption overhead
Consolidation	Normal rules	Empty consolidation is blocked for buffer-backed nodes
Lifecycle	Normal	Drift and expiry still apply; the buffer re-provisions to stay satisfied

Fact vs. opinion: the behavior above is from the Karpenter docs and the AWS Karpenter Blueprints guide. My opinion: this is a genuinely better design than the dummy-Deployment hack. But it's alpha, and the blueprint notes it isn't supported on EKS Auto Mode. Evaluate it, don't bet the farm on it yet.

Ingredients

• Karpenter v1.14.0 or newer
• The karpenter-crd chart at v1.14.0 or newer, because the CapacityBuffer CRD ships there, not in the main chart
• A working EC2NodeClass and NodePool
• A staging cluster (alpha, remember)
• kubectl and helm

Kitchen Prep

First, check what API version your cluster actually serves. The Karpenter docs and the blueprint don't use identical labels, and alpha APIs move:

kubectl api-resources | grep -i capacitybuffer

The manifests below use autoscaling.x-k8s.io/v1beta1, which is what the 1.14 upgrade notes and the AWS blueprint use. If your output shows something different, match it.

Recipe Steps

Step 1: Turn on the feature gate

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --namespace kube-system \
  --set "settings.featureGates.capacityBuffer=true" \
  --reuse-values

Adjust the namespace to wherever you installed Karpenter. Then confirm the gate took:

kubectl -n kube-system get deployment karpenter \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="FEATURE_GATES")].value}'

Expected: output containing CapacityBuffer=true. Pitfall: upgrading the controller but not the karpenter-crd chart, which leaves you without the CRD.

Step 2: Create a dedicated, tainted NodePool

This is the design decision that matters most. A buffer's nodes are ordinary nodes, so any pod that fits can land on them. A dedicated taint keeps your spare capacity for the pods you intend:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: capacity-buffer
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        name: default
        kind: EC2NodeClass
      taints:
        - key: intent
          value: capacity-buffer
          effect: NoSchedule
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]

Step 3: Describe one "chunk" with a PodTemplate

A PodTemplate is a standard Kubernetes object, and here it's just a stencil. Applying it creates no pods. It tells Karpenter what shape of capacity to keep warm:

apiVersion: v1
kind: PodTemplate
metadata:
  name: web-buffer-template
template:
  spec:
    containers:
      - name: placeholder
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
    nodeSelector:
      intent: capacity-buffer
    tolerations:
      - key: intent
        value: capacity-buffer
        effect: NoSchedule

Match the requests to your real workload's requests. A buffer sized for a different shape than your pods is just a more expensive way to be disappointed.

Step 4: Create the buffer

apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: web-app-buffer
spec:
  provisioningStrategy: "buffer.x-k8s.io/active-capacity"
  podTemplateRef:
    name: web-buffer-template
  replicas: 3

Apply both manifests, wait about a minute, then look:

kubectl get capacitybuffer web-app-buffer -o wide
kubectl get nodeclaims -l intent=capacity-buffer

Expected: the buffer reports it's provisioning, and Karpenter either reuses existing slack or launches nodes. Small chunks bin-pack, so 3 chunks of 1 CPU may land on a single node. If you want spare capacity spread across nodes, size the template closer to a full node or add pod anti-affinity to it, as you would for real pods.

Step 5: Prove it works

Your real pod count doesn't change, because the buffer's pods are virtual. The real test is a burst. Scale your Deployment (give it the matching toleration and node selector so it can use the buffer's nodes):

kubectl scale deployment web-app --replicas=5
kubectl get pods -l app=web-app -w

Expected: new pods reach Running within seconds, because the capacity was already Ready. You can also see the consolidation protection in action:

kubectl get events --field-selector reason=Unconsolidatable

Look for a message that the node has buffer pods.

Step 6: Make the buffer track a workload

A flat number is wasteful when a service scales from 5 to 50 replicas. Point the buffer at the Deployment and keep a percentage:

apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: api-service-buffer
spec:
  provisioningStrategy: "buffer.x-k8s.io/active-capacity"
  scalableRef:
    apiGroup: apps
    kind: Deployment
    name: api-service
  percentage: 20

With 5 replicas, that's one chunk. At 20 replicas, it recalculates to four, using the docs' example math of rounding up. The blueprint notes the controller polls about every 30 seconds, so there's a short lag. When not to use it: this tracks current size, not a forecast. For predictive scaling, drive replicas directly from whatever component owns the forecast.

Optional: Pre-warm GPUs, with a leash

GPU nodes report Ready well before the driver and device plugin finish registering the GPU, which can add real minutes to a cold start. A GPU buffer front-loads that. But spare GPUs are expensive spare parts, so cap the buffer:

spec:
  provisioningStrategy: "buffer.x-k8s.io/active-capacity"
  podTemplateRef:
    name: gpu-buffer-template
  limits:
    nvidia.com/gpu: "2"

One gotcha from the blueprint: set requests.nvidia.com/gpu explicitly in the template. Virtual pods never pass through API admission, so the usual defaulting from limits doesn't happen, and without an explicit request the buffer sees zero GPU demand.

Chef Tips

• Tighten the NodePool, tighten the taints. The blueprint's own warning: nothing stops an unrelated pod from consuming your warm node first if your constraints are loose.
• Remember what a buffer doesn't warm. It pre-warms node-level things: OS, drivers, device plugins. It doesn't pull your container image or download model weights. For GPU inference, that's often the larger share of cold-start time, so plan image caching separately.
• Cap anything expensive. Use limits so a buffer can't hold unbounded warm, billed capacity.
• Watch NodePool limits. If a NodePool's CPU or memory limit is exhausted, the buffer can't be fulfilled.

Kitchen Disasters

Symptom	Likely cause	Fix
no matches for kind "CapacityBuffer"	CRD chart not at v1.14.0+	Upgrade the karpenter-crd chart
Buffer exists but nothing happens	Feature gate off	Set settings.featureGates.capacityBuffer=true and verify FEATURE_GATES
Warm node taken by an unrelated pod	NodePool requirements and tolerations too loose	Add a dedicated taint and matching toleration
GPU pod still slow to start	Image pull or model download, not the node	Pre-pull images or cache models; the buffer only warms the node
Buffer never fully provisions	NodePool CPU, memory, or GPU limit exhausted	Raise the NodePool limit or shrink the buffer
Node replaced unexpectedly	Drift or expiry still applies	Expected behavior; the buffer re-provisions to stay satisfied

Serving Suggestions

• Start with one latency-sensitive service and measure time-to-Running before and after. Don't invent a speedup, measure yours.
• Pair a buffer with sensible disruption budgets so spare capacity isn't churned unnecessarily, and revisit our Karpenter guide if you need a refresher on the basics.
• Compare against the other capacity patterns: node reserved headroom (spare slices inside busy nodes) and static NodePools (a fixed node count Karpenter never touches). The blueprint suggests CapacityBuffers as the default for latency-sensitive, workload-shaped needs, and static NodePools only for things like purchased reservations.

Cleanup

Buffers are reversible. Delete the object and the spare capacity becomes normal capacity again, subject to consolidation:

kubectl delete capacitybuffer web-app-buffer api-service-buffer
kubectl delete podtemplate web-buffer-template
kubectl delete nodepool capacity-buffer

Frequently Asked Questions

Does a CapacityBuffer cost money even when idle?

Yes. It keeps real nodes running, and you pay for them. That's the trade: spend a little continuously to avoid waiting at the worst moment.

Is this the same as overprovisioning with a pause-pod Deployment?

Same goal, different mechanism. The old approach uses real low-priority pods that get preempted. Buffers use virtual pods and no preemption.

Will consolidation remove my spare node because it looks empty?

Empty consolidation is blocked for buffer-backed nodes. Other disruption, like drift and expiry, still applies.

Is it production-ready?

It's alpha, so the API may change. The AWS blueprint says to treat it as a pattern to evaluate and to validate your alpha-API risk tolerance first.

Final Thoughts

Just-in-time capacity is Karpenter's superpower, and sometimes "just in time" is just too late. CapacityBuffers give you a dial between "pay for nothing" and "wait for everything." Start small, taint your NodePool, cap the expensive stuff, and measure what actually improves. A hot oven is lovely, as long as you remember it's burning gas.

🔥 Related CloudChef Recipes

Loading related posts...

References

• Karpenter docs: CapacityBuffers
• Karpenter Blueprints for Amazon EKS: CapacityBuffers
• Karpenter docs: Upgrade Guide (v1.15)
• Karpenter docs: Settings and feature gates