Every few months AWS drops a new Kubernetes minor version into EKS, and every platform team has the same thought: "Great. Another upgrade."
Think of it like replacing the oven in a restaurant kitchen. The new oven is better, no doubt. But you don't swap it in the middle of the dinner rush, and you definitely check that your favorite pans still fit first.
AWS has announced that Amazon EKS and EKS Distro now support Kubernetes 1.37 (upstream release name: "Garhwal", shipped August 26, 2026). This post covers what actually changed, what can still bite you on the way there, and a recipe for upgrading without a 2 a.m. page.
What We Are Cooking Today
- Understand the three headline changes in 1.37 in plain English
- Run a pre-flight check for removals you might have missed in 1.35 and 1.36
- Upgrade a cluster from 1.36 to 1.37 and validate it
- Try the new HPA scale-to-zero behavior safely in staging
What Is Actually New in 1.37
Facts first, opinions after. These three items come from AWS's announcement.
| Change | Status in 1.37 | Why you might care |
|---|---|---|
Metrics API (metrics.k8s.io/v1) | General availability | It feeds the Horizontal Pod Autoscaler and kubectl top with pod and node CPU and memory usage. |
| DRA device taints and tolerations | General availability | Admins and drivers can taint devices like GPUs so the scheduler avoids them unless a workload tolerates the taint. |
| HPA scale-to-zero | Beta, on by default | An HPA with minReplicas: 0 and object or external metrics can idle a workload down to zero pods and bring it back when demand returns. |
The Chef's Take (Opinion)
Scale-to-zero is the one I'd test first. It's the restaurant that turns off the stove when nobody's ordering. Great for the electric bill. But the first customer after a quiet hour waits for the stove to heat up, so expect a cold start.
Why only object or external metrics? Because CPU utilization is a percentage per pod, and with zero pods there's nothing to measure. You need a signal that exists even when the kitchen is empty, like queue depth.
The Metrics API going GA is quieter but matters. A stable API is one less thing to worry about at upgrade time. If you have old tooling that pins to a beta version of this API, it's worth a grep.
Ingredients
- An EKS cluster currently on 1.36 (EKS upgrades one minor version at a time)
- AWS CLI v2,
kubectlwithin one minor version of the cluster, and optionallyeksctl - IAM permissions to update the cluster and node groups
- A staging cluster. Please. Staging is cheaper than apologies.
Kitchen Prep: Check What Earlier Versions Already Removed
If you're catching up from 1.34 or 1.35, the 1.37 features aren't what will hurt. The removals along the way will. These come from the EKS version notes and AWS's upgrade-check guidance:
- 1.35: cgroup v1 support removed. Nodes need a cgroup v2 setup and containerd 2.0 or later.
- 1.36: IPVS mode in kube-proxy removed. kube-proxy fails to start in that mode, which breaks Service routing cluster-wide.
- 1.36:
gitRepovolumes removed, and theexternalIPsfield on Services deprecated (full removal planned for 1.43).
Quick checks you can run today:
# kube-proxy mode (EKS keeps its config in this ConfigMap)
kubectl get cm kube-proxy-config -n kube-system -o yaml | grep -i "mode:"
# Pods still using gitRepo volumes
kubectl get pods -A -o json | jq -r '.items[] | select(any(.spec.volumes[]?; has("gitRepo"))) | "\(.metadata.namespace)/\(.metadata.name)"'
# Services still using externalIPs
kubectl get svc -A -o json | jq -r '.items[] | select(.spec.externalIPs) | "\(.metadata.namespace)/\(.metadata.name)"'
An empty result is the good result. If kube-proxy says ipvs, fix that before anything else.
Recipe Steps
1. Read the EKS Cluster Insights
aws eks list-insights \
--cluster-name my-cluster \
--filter categories=UPGRADE_READINESS
Insights flag deprecated API usage and other upgrade blockers. Anything not PASSING deserves your attention before you touch the control plane.
2. Upgrade the control plane
aws eks update-cluster-version \
--name my-cluster \
--kubernetes-version 1.37
# or with eksctl
eksctl upgrade cluster --name my-cluster --version 1.37 --approve
Using Terraform? Bump the version in code and review the plan. Same result, with a paper trail.
3. Watch the update finish
aws eks describe-update --name my-cluster --update-id <update-id>
Wait for Successful. Don't touch nodes while it says InProgress.
4. Update the add-ons
aws eks describe-addon-versions --kubernetes-version 1.37 --addon-name kube-proxy
aws eks describe-addon-versions --kubernetes-version 1.37 --addon-name coredns
aws eks describe-addon-versions --kubernetes-version 1.37 --addon-name vpc-cni
Pick a compatible version for each and update through your normal path (console, CLI or IaC).
5. Roll the nodes
aws eks update-nodegroup-version \
--cluster-name my-cluster \
--nodegroup-name general
Managed node groups roll gracefully, but only if your PodDisruptionBudgets are sane. Running Karpenter? Roll nodes by updating the AMI selection and letting drift replace them.
6. Validate
kubectl version
kubectl get nodes
kubectl top nodes
kubectl get --raw /apis/metrics.k8s.io/v1 | head -c 300
If kubectl top returns numbers, your metrics pipeline survived.
7. (Staging only) Try scale-to-zero
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: worker
namespace: jobs
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: worker
minReplicas: 0
maxReplicas: 20
metrics:
- type: External
external:
metric:
name: queue_depth
target:
type: AverageValue
averageValue: "5"
This assumes something in your cluster serves an external metric called queue_depth. The name here is an example, not a built-in. When to use it: bursty, queue-driven workers. When not to: anything latency-sensitive that can't tolerate a cold start.
Chef Tips
- Upgrade staging first, then wait a few days. Most upgrade surprises show up on day two, not minute two.
- Audit PodDisruptionBudgets before node rolls. A PDB with zero allowed disruptions will stall a drain forever.
- Treat scale-to-zero as an opt-in per workload, not a cluster-wide habit.
Kitchen Disasters
| Symptom | Likely cause | Fix |
|---|---|---|
Nodes stay NotReady after the roll | Old AMI, or add-ons not updated | Check the node version, update vpc-cni and kube-proxy, then re-roll |
kubectl top returns errors | Metrics server unhealthy or still on an old version | Check its pods and logs, then update it |
| Services stop routing after upgrading from 1.35 | kube-proxy still configured for IPVS | Switch to iptables mode and restart kube-proxy |
| HPA never leaves zero | The external metric isn't being served | Query the external metrics API directly and check the adapter |
| Node drain hangs | Restrictive PDB | Fix the PDB. Don't force-delete pods in production. |
Serving Suggestions
- Security note: the upstream advisory for CVE-2026-2270 lists 1.37.0 as affected and 1.37.1 as fixed. EKS manages the control plane for you, so check AWS's security bulletins for the status on your platform version. I haven't verified AWS's response, so don't take my word for it. The full breakdown is in the CVE post on this site.
- Plan your next hop now. Upstream 1.34 reached end of life this week according to release trackers, so anyone still on it is on borrowed time.
- Pair this upgrade with a look at your Karpenter node pools while you're in there.
Cleanup and Rollback
Here's the honest part: you can't downgrade an EKS control plane. Your rollback plan is your backup plan: Git-managed manifests, a restorable state for stateful apps, and ideally a blue/green cluster for big jumps. For nodes, you can point a node group back at an older AMI as long as it stays within version skew. And delete any staging clusters you spun up for testing. They bill by the hour.
FAQ
Can I skip from 1.35 straight to 1.37?
No. EKS upgrades move one minor version at a time.
Does scale-to-zero work with CPU-based HPAs?
Per AWS's announcement, the beta covers autoscalers using object or external metrics. Resource metrics like CPU don't give you a signal when there are zero pods.
Is HPA scale-to-zero production ready?
It's beta, so treat it that way. Start with non-critical workers.
Final Thoughts
EKS 1.37 isn't a scary release. The upgrade risk is rarely the new version itself. It's the stuff that quietly got removed two versions ago while nobody was looking. Run the pre-flight, upgrade staging, and keep scale-to-zero on a short leash until you've seen it in your own kitchen.
No comments:
Post a Comment