The Karpenter 1.15 Upgrade Recipe: Node Repair, disruption-cost and Metric Label Changes

Karpenter 1.15 Is Out: Node Auto Repair Grew Up (and Your Dashboards Might Break)

Picture a restaurant kitchen where a cook faints mid-service. The old rule: drag them out, then start hunting for a replacement while orders pile up. The new rule: call the backup cook first, let the sick one finish handing off their pans, and never pull more than a few people off the line at once.

That's the idea behind the biggest change in Karpenter 1.15, which shipped on October 10, 2026. Node auto repair stopped being a blunt instrument and became a polite, budget-aware disruption method.

But here's the catch, and it's why this post exists. 1.15 also renames metric labels, deprecates an annotation many teams use, and bumps a Bottlerocket default. None of it is dramatic. All of it can quietly break a dashboard, an alert, or a GPU node launch if you upgrade on autopilot.

So let's cook this upgrade properly.

What We Are Cooking Today

• Understand what actually changed in Karpenter 1.15 (and what only matters if you opted into alpha features)
• Audit your cluster for the four things most likely to bite
• Upgrade CRDs and controller in the right order
• Verify nodes, metrics, and disruption behavior afterward

What's New in Karpenter 1.15

Everything below comes from the official 1.15 upgrade guide. I've split it into "affects everyone" and "affects you only if you opted in," because that distinction is where most upgrade anxiety lives.

Affects everyone

• Three metric label changes. The reason label on karpenter_nodepool_allowed_disruptions and karpenter_nodepool_nodes_consuming_budgets is now lowercase (underutilized, not Underutilized). The consolidation_type label on karpenter_voluntary_disruption_queue_failures_total now reports empty, single, or multi instead of the decision. And the error label on karpenter_cloudprovider_errors_total uses unknown instead of an empty string. Any dashboard or alert filtering on the old values will go quiet, which is the worst kind of broken.
• A deprecation. Karpenter now reads the karpenter.sh/disruption-cost annotation to decide which pods are cheaper to disrupt during consolidation. Using controller.kubernetes.io/pod-deletion-cost for that purpose is deprecated, though there's still a fallback for now.
• A Bottlerocket default. The default root volume is now 6Gi, up from 4Gi, because Bottlerocket v1.66.0+ aws-k8s-1.37-nvidia AMIs have a 6 GiB root snapshot and smaller volumes fail with InvalidBlockDeviceMapping. This only applies to EC2NodeClasses that don't set blockDeviceMappings.

Affects you only if you opted into alpha features

• Node auto repair is now a voluntary disruption method (alpha, behind the NodeRepair feature gate). Instead of terminating an unhealthy node on the spot, Karpenter pre-spins a replacement, respects your NodePool disruption budgets under a new Unhealthy reason, and drains while respecting PDBs. With the default budget of 10%, it repairs at most 10% of a NodePool's nodes at a time. The guide marks this as breaking, so if you enabled the gate, read it twice.
• Terminate-first disruption. New alpha gates TerminateFirstDrift and TerminateFirstRepair let Karpenter drift or repair nodes in full capacity reservations and static NodePools at their limit by terminating before launching a replacement. Both are off by default.
• Reboot for repair. The alpha AWS gate RebootForRepair reboots a node in place for transient GPU XID faults instead of replacing it. It needs the ec2:RebootInstances permission.
• Pod deletion cost management. The alpha PodDeletionCostManagement gate has Karpenter manage pod-deletion-cost on ReplicaSet-owned pods. It can issue a large number of pod writes and overwrites values you've set yourself, so migrate to karpenter.sh/disruption-cost first.

Fact vs. opinion: the list above is documented behavior. My opinion: leave every alpha gate off in production until you've watched it behave in staging for a few weeks. Alpha means the authors reserve the right to change their minds.

If You're Coming From 1.14 (or Earlier)

Version 1.14, which landed July 11, had its own changes worth knowing about. It added a Balanced consolidation policy (opt-in per NodePool), Dynamic Resource Allocation support, and a CapacityBuffers API at v1beta1. The upgrade guide calls out one that applies to everyone: 1.14 changed the order in which Karpenter considers nodes for consolidation under every policy. If your bin packing looked different after 1.14, you weren't imagining it, and the project asked users to report outcomes in the linked GitHub issue.

Ingredients

• Karpenter installed via Helm (this recipe uses Helm; adapt for your tooling)
• A staging cluster that actually mirrors production's NodePools
• kubectl, helm, and jq
• Access to your Prometheus alert rules and Grafana dashboards (as code, ideally)
• Your current Helm values file

Kitchen Prep

Karpenter isn't tied to a specific Kubernetes version the way Cluster Autoscaler is, so you can upgrade it on its own schedule. That freedom is nice, and it also means nobody else will remind you. Here's the order of operations:

flowchart LR A[Back up NodePools] --> B[Audit the 4 risks] B --> C[Upgrade CRDs] C --> D[Upgrade controller] D --> E[Verify nodes and metrics] E --> F[Fix dashboards]

Set variables so you don't fat-finger a namespace at 11 p.m.:

export KARPENTER_NAMESPACE=kube-system
export KARPENTER_VERSION=1.15.0

Recipe Steps

Step 1: Back up what Karpenter is managing

kubectl get nodepools,ec2nodeclasses -o yaml > karpenter-config-backup.yaml
kubectl get nodeclaims -o wide

Expected: a YAML file you can restore from, and a list of current NodeClaims. Pitfall: backing up to a laptop folder and forgetting it. Commit it somewhere your team can find it.

Step 2: Audit risk #1, pods using the old annotation

kubectl get pods -A -o json | jq -r '
  .items[]
  | select(.metadata.annotations["controller.kubernetes.io/pod-deletion-cost"] != null)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Anything listed is a pod whose owner set pod-deletion-cost, possibly to steer Karpenter. For those, plan to set karpenter.sh/disruption-cost to the same value. Karpenter still falls back to the old annotation for now, but the guide says that fallback is planned for removal.

Step 3: Audit risk #2, dashboards and alerts

Search your monitoring repo for the changed labels:

grep -rnE 'karpenter_nodepool_allowed_disruptions|karpenter_nodepool_nodes_consuming_budgets|karpenter_voluntary_disruption_queue_failures_total|karpenter_cloudprovider_errors_total' .

For each hit, check whether a query filters on reason, consolidation_type, or error. A query like this one needs updating:

# Before (matches nothing after 1.15)
karpenter_nodepool_allowed_disruptions{reason="Underutilized"}

# After
karpenter_nodepool_allowed_disruptions{reason="underutilized"}

Step 4: Audit risk #3, Bottlerocket GPU node classes

Find EC2NodeClasses using Bottlerocket and check whether they set their own block device mappings:

kubectl get ec2nodeclasses -o json | jq -r '
  .items[]
  | select(.spec.amiFamily == "Bottlerocket" or ((.spec.amiSelectorTerms // []) | tostring | test("bottlerocket"; "i")))
  | "\(.metadata.name): blockDeviceMappings=\(.spec.blockDeviceMappings // "default")"'

If you run Bottlerocket NVIDIA AMIs on Kubernetes 1.37+ and set blockDeviceMappings yourself, make sure the /dev/xvda volumeSize is at least 6Gi. If you rely on defaults, 1.15 handles it, and the guide notes existing nodes won't drift because of it.

Step 5: Audit risk #4, which feature gates are on

kubectl -n $KARPENTER_NAMESPACE get deploy karpenter \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | jq .

Look for FEATURE_GATES. If NodeRepair=true appears, you're in the breaking-change group for repair behavior. The guide also mentions a --legacy-node-repair option (Helm: settings.legacyNodeRepair) that keeps the previous forceful behavior. Clusters with 1000+ nodes that enable repair may see disruption slow down because repair now runs in the disruption controller. The project says it plans to deprecate the legacy path, so treat it as a temporary bridge.

Step 6: Upgrade the CRDs first

Helm only installs CRDs on the first install, so subsequent chart upgrades won't update them. The guide recommends the standalone karpenter-crd chart:

helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd \
  --version $KARPENTER_VERSION \
  --namespace $KARPENTER_NAMESPACE --create-namespace

This matters for 1.15 specifically, because the CRD adds Unhealthy to the budget reasons enum. Skip this step and a budget using that reason will be rejected.

Step 6b: Upgrade the controller

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version $KARPENTER_VERSION \
  --namespace $KARPENTER_NAMESPACE \
  -f karpenter-values.yaml \
  --wait

Use your existing values file so you don't silently reset IRSA annotations, cluster name, or interruption queue settings.

Step 7: Verify

kubectl -n $KARPENTER_NAMESPACE rollout status deploy/karpenter
kubectl -n $KARPENTER_NAMESPACE logs deploy/karpenter --tail=50
kubectl get nodeclaims
kubectl get nodepools

Then do the test that matters: deploy a throwaway workload that forces a scale-up, confirm a NodeClaim launches, delete the workload, and watch consolidation reclaim the node. Check your updated dashboards show data for the disruption metrics.

Optional: Try Node Auto Repair in staging

If you want to evaluate the new repair behavior, enable the NodeRepair feature gate in staging only (check the chart's values for the exact key under settings.featureGates) and consider giving repair its own budget so it can't outrun your tolerance:

spec:
  disruption:
    budgets:
      - nodes: "1"
        reasons:
          - Unhealthy
      - nodes: "10%"

This is illustrative, built from the guide's description of the new Unhealthy budget reason. Check the disruption docs for the exact schema in your version before applying it. Per the guide, default repair policies bound the drain at 10 minutes (5 for GPU XID faults), using the shorter of that and the NodeClaim's terminationGracePeriod.

Chef Tips

• Change one thing at a time. Upgrade the controller first. Enable alpha gates in a separate change, on another day. When production surprises you, you want exactly one suspect.
• Pair repair with PodDisruptionBudgets that mean something. A repair that respects PDBs is only as safe as the PDBs you wrote. A PDB with maxUnavailable: 0 on a single-replica workload is a standoff, not a safety net.
• Don't enable PodDeletionCostManagement casually. It writes to a lot of pods and overwrites your values. Migrate to karpenter.sh/disruption-cost first.
• Treat dashboards as code. The metric label changes are painless when your queries live in Git and a PR can find them with one grep.

Kitchen Disasters

Symptom	Likely cause	Fix
Disruption panels suddenly empty	Queries still filter on capitalized reason values	Update filters to lowercase (underutilized, drifted, and so on)
Budget with Unhealthy reason rejected	CRDs not upgraded alongside the controller	Upgrade the karpenter-crd chart to the same version
GPU nodes fail to launch with InvalidBlockDeviceMapping	Custom blockDeviceMappings below 6Gi on Bottlerocket NVIDIA AMIs	Set /dev/xvda volumeSize to at least 6Gi
Consolidation slower on very large clusters	Node repair now runs in the disruption controller (1000+ nodes with repair on)	Use legacyNodeRepair temporarily and report your use case to the project
Helm error about ownership metadata on CRD chart	Older install managed CRDs differently	Follow the project's troubleshooting guide for the karpenter-crd chart

Serving Suggestions

• Pin the Karpenter version in Git and let Renovate or Dependabot open the bump PR, with the upgrade-guide link in the PR template.
• Add a CI check that greps dashboards and alert rules for deprecated label values before merge.
• If you run GPUs or other accelerators, read the new DRA tasks docs, since 1.14 and 1.15 keep investing there.
• Rehearse the rollback: keep your previous Helm values and chart version handy. For a rollback, redeploy the older controller and CRDs together.

Cleanup

Delete the throwaway test workload, and keep karpenter-config-backup.yaml somewhere durable for at least a release cycle. If you enabled any alpha gate in staging for evaluation and decided against it, turn it off again so staging keeps mirroring production.

Frequently Asked Questions

Do I have to upgrade to 1.15 right now?

No. But Karpenter doesn't have a documented long-term support policy, and the endoflife.date tracker notes that only the latest release appears to be supported. Staying within a release or two of latest is the safer habit.

Is Node Auto Repair on by default?

No. It's alpha and sits behind the NodeRepair feature gate. If you never enabled it, the repair behavior change doesn't affect you. The metric label changes still do.

Do I need to upgrade Karpenter when I upgrade EKS to 1.37?

Karpenter isn't tied to a specific Kubernetes version, but always check the project's compatibility table before an EKS upgrade. One related detail from the 1.15 guide: Bottlerocket NVIDIA AMIs for Kubernetes 1.37 are why the default root volume grew.

What replaces pod-deletion-cost?

The karpenter.sh/disruption-cost annotation, for influencing which pods are cheaper to disrupt during consolidation. Set it to the same value you used before.

Final Thoughts

Karpenter 1.15 isn't a scary release. It's a careful one, and careful releases reward careful upgraders. Back up the config, grep your dashboards, upgrade CRDs before the controller, and keep the alpha features in staging until they've earned your trust. The nodes will take care of themselves. The dashboards, as usual, won't unless you check.

🔥 Related CloudChef Recipes

Related posts load automatically on the live site.

References

• Karpenter docs: Upgrade Guide (v1.15)
• Karpenter docs: Disruption concepts (v1.15)
• Karpenter core releases on GitHub
• endoflife.date: Karpenter release tracker