New recipes every week

Turn Complexity Into
Cloud Recipes

Learn Kubernetes, AI, DevOps and DevSecOps the CloudChef way. Practical guides, real-world examples, no fluff.

✓Free forever ✓No paywall ✓Practical guides ✓Real-world examples
50+Guides
WeeklyNew posts
K8s + AITop topics
FreeAlways
CloudChef Kubernetes Saturday, October 31, 2026 ⏱ Calculating...

EKS 1.37 Upgrade Guide: HPA Scale-to-Zero, DRA Taints, and the Gotchas Nobody Mentions

CC
CloudChef
thecloudchef.io
Amazon EKS 1.37 upgrade guide: Metrics API GA, HPA scale-to-zero and DRA taints - CloudChef

Every few months AWS drops a new Kubernetes minor version into EKS, and every platform team has the same thought: "Great. Another upgrade."

Think of it like replacing the oven in a restaurant kitchen. The new oven is better, no doubt. But you don't swap it in the middle of the dinner rush, and you definitely check that your favorite pans still fit first.

AWS has announced that Amazon EKS and EKS Distro now support Kubernetes 1.37 (upstream release name: "Garhwal", shipped August 26, 2026). This post covers what actually changed, what can still bite you on the way there, and a recipe for upgrading without a 2 a.m. page.

What We Are Cooking Today

  • Understand the three headline changes in 1.37 in plain English
  • Run a pre-flight check for removals you might have missed in 1.35 and 1.36
  • Upgrade a cluster from 1.36 to 1.37 and validate it
  • Try the new HPA scale-to-zero behavior safely in staging

What Is Actually New in 1.37

Facts first, opinions after. These three items come from AWS's announcement.

ChangeStatus in 1.37Why you might care
Metrics API (metrics.k8s.io/v1)General availabilityIt feeds the Horizontal Pod Autoscaler and kubectl top with pod and node CPU and memory usage.
DRA device taints and tolerationsGeneral availabilityAdmins and drivers can taint devices like GPUs so the scheduler avoids them unless a workload tolerates the taint.
HPA scale-to-zeroBeta, on by defaultAn HPA with minReplicas: 0 and object or external metrics can idle a workload down to zero pods and bring it back when demand returns.

The Chef's Take (Opinion)

Scale-to-zero is the one I'd test first. It's the restaurant that turns off the stove when nobody's ordering. Great for the electric bill. But the first customer after a quiet hour waits for the stove to heat up, so expect a cold start.

Why only object or external metrics? Because CPU utilization is a percentage per pod, and with zero pods there's nothing to measure. You need a signal that exists even when the kitchen is empty, like queue depth.

The Metrics API going GA is quieter but matters. A stable API is one less thing to worry about at upgrade time. If you have old tooling that pins to a beta version of this API, it's worth a grep.

Ingredients

  • An EKS cluster currently on 1.36 (EKS upgrades one minor version at a time)
  • AWS CLI v2, kubectl within one minor version of the cluster, and optionally eksctl
  • IAM permissions to update the cluster and node groups
  • A staging cluster. Please. Staging is cheaper than apologies.

Kitchen Prep: Check What Earlier Versions Already Removed

If you're catching up from 1.34 or 1.35, the 1.37 features aren't what will hurt. The removals along the way will. These come from the EKS version notes and AWS's upgrade-check guidance:

  • 1.35: cgroup v1 support removed. Nodes need a cgroup v2 setup and containerd 2.0 or later.
  • 1.36: IPVS mode in kube-proxy removed. kube-proxy fails to start in that mode, which breaks Service routing cluster-wide.
  • 1.36: gitRepo volumes removed, and the externalIPs field on Services deprecated (full removal planned for 1.43).

Quick checks you can run today:

# kube-proxy mode (EKS keeps its config in this ConfigMap)
kubectl get cm kube-proxy-config -n kube-system -o yaml | grep -i "mode:"

# Pods still using gitRepo volumes
kubectl get pods -A -o json | jq -r '.items[] | select(any(.spec.volumes[]?; has("gitRepo"))) | "\(.metadata.namespace)/\(.metadata.name)"'

# Services still using externalIPs
kubectl get svc -A -o json | jq -r '.items[] | select(.spec.externalIPs) | "\(.metadata.namespace)/\(.metadata.name)"'

An empty result is the good result. If kube-proxy says ipvs, fix that before anything else.

Recipe Steps

1. Read the EKS Cluster Insights

aws eks list-insights \
  --cluster-name my-cluster \
  --filter categories=UPGRADE_READINESS

Insights flag deprecated API usage and other upgrade blockers. Anything not PASSING deserves your attention before you touch the control plane.

2. Upgrade the control plane

aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.37

# or with eksctl
eksctl upgrade cluster --name my-cluster --version 1.37 --approve

Using Terraform? Bump the version in code and review the plan. Same result, with a paper trail.

3. Watch the update finish

aws eks describe-update --name my-cluster --update-id <update-id>

Wait for Successful. Don't touch nodes while it says InProgress.

4. Update the add-ons

aws eks describe-addon-versions --kubernetes-version 1.37 --addon-name kube-proxy
aws eks describe-addon-versions --kubernetes-version 1.37 --addon-name coredns
aws eks describe-addon-versions --kubernetes-version 1.37 --addon-name vpc-cni

Pick a compatible version for each and update through your normal path (console, CLI or IaC).

5. Roll the nodes

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name general

Managed node groups roll gracefully, but only if your PodDisruptionBudgets are sane. Running Karpenter? Roll nodes by updating the AMI selection and letting drift replace them.

6. Validate

kubectl version
kubectl get nodes
kubectl top nodes
kubectl get --raw /apis/metrics.k8s.io/v1 | head -c 300

If kubectl top returns numbers, your metrics pipeline survived.

7. (Staging only) Try scale-to-zero

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker
  namespace: jobs
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: worker
  minReplicas: 0
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: queue_depth
        target:
          type: AverageValue
          averageValue: "5"

This assumes something in your cluster serves an external metric called queue_depth. The name here is an example, not a built-in. When to use it: bursty, queue-driven workers. When not to: anything latency-sensitive that can't tolerate a cold start.

Chef Tips

  • Upgrade staging first, then wait a few days. Most upgrade surprises show up on day two, not minute two.
  • Audit PodDisruptionBudgets before node rolls. A PDB with zero allowed disruptions will stall a drain forever.
  • Treat scale-to-zero as an opt-in per workload, not a cluster-wide habit.

Kitchen Disasters

SymptomLikely causeFix
Nodes stay NotReady after the rollOld AMI, or add-ons not updatedCheck the node version, update vpc-cni and kube-proxy, then re-roll
kubectl top returns errorsMetrics server unhealthy or still on an old versionCheck its pods and logs, then update it
Services stop routing after upgrading from 1.35kube-proxy still configured for IPVSSwitch to iptables mode and restart kube-proxy
HPA never leaves zeroThe external metric isn't being servedQuery the external metrics API directly and check the adapter
Node drain hangsRestrictive PDBFix the PDB. Don't force-delete pods in production.

Serving Suggestions

  • Security note: the upstream advisory for CVE-2026-2270 lists 1.37.0 as affected and 1.37.1 as fixed. EKS manages the control plane for you, so check AWS's security bulletins for the status on your platform version. I haven't verified AWS's response, so don't take my word for it. The full breakdown is in the CVE post on this site.
  • Plan your next hop now. Upstream 1.34 reached end of life this week according to release trackers, so anyone still on it is on borrowed time.
  • Pair this upgrade with a look at your Karpenter node pools while you're in there.

Cleanup and Rollback

Here's the honest part: you can't downgrade an EKS control plane. Your rollback plan is your backup plan: Git-managed manifests, a restorable state for stateful apps, and ideally a blue/green cluster for big jumps. For nodes, you can point a node group back at an older AMI as long as it stays within version skew. And delete any staging clusters you spun up for testing. They bill by the hour.

FAQ

Can I skip from 1.35 straight to 1.37?

No. EKS upgrades move one minor version at a time.

Does scale-to-zero work with CPU-based HPAs?

Per AWS's announcement, the beta covers autoscalers using object or external metrics. Resource metrics like CPU don't give you a signal when there are zero pods.

Is HPA scale-to-zero production ready?

It's beta, so treat it that way. Start with non-critical workers.

Final Thoughts

EKS 1.37 isn't a scary release. The upgrade risk is rarely the new version itself. It's the stuff that quietly got removed two versions ago while nobody was looking. Run the pre-flight, upgrade staging, and keep scale-to-zero on a short leash until you've seen it in your own kitchen.

References


πŸ”₯ Trending CloudChef Recipes

⭐ Popular CloudChef Recipes

No comments:

Post a Comment

πŸ’‘ Found this useful?

Share it with your Team or DevOps Friends πŸ‘‡