New recipes every week

Turn Complexity Into
Cloud Recipes

Learn Kubernetes, AI, DevOps and DevSecOps the CloudChef way. Practical guides, real-world examples, no fluff.

✓Free forever ✓No paywall ✓Practical guides ✓Real-world examples
50+Guides
WeeklyNew posts
K8s + AITop topics
FreeAlways
Monday, September 21, 2026 ⏱ Calculating...

Your Container's "Root" Was Always a Lie — Here's the Kubernetes Fix

CC
CloudChef
thecloudchef.io
Kubernetes User Namespaces architecture diagram showing host UID space and container UID space separated by an idmap boundary, CloudChef branding

Here's a sentence that should keep any platform engineer up at night: root inside your container is root on your node. Not "root-ish." Not "root, but sandboxed." Actual UID 0, recognized by the host kernel as UID 0. Every container escape CVE you've ever triaged — a bad mount, a leaked file descriptor, a kernel bug — has been scary specifically because of this one fact.

For ten years, Kubernetes's answer to this was "don't run as root" and a pile of SecurityContext flags bolted on top. Good advice, but it never fixed the underlying problem: if a workload needs UID 0 inside the container — and plenty of legacy images do — you were stuck trusting it with host-root-adjacent power.

With Kubernetes v1.36, that changes. User Namespaces graduated to General Availability, which means a pod can now run as "root" from its own point of view while the kernel maps that identity to an unprivileged, throwaway UID range on the host. Same container image, zero code changes, dramatically smaller blast radius. Let's cook.

What We're Cooking Today

  • Understand what User Namespaces actually change at the kernel level (and why it took a decade to ship)
  • Deploy a pod with hostUsers: false and prove the UID remap is real, not marketing
  • Enforce it cluster-wide with a Kyverno policy so "secure by default" isn't optional
  • Know exactly where this feature still has sharp edges in production

Ingredients

  • Kubernetes v1.36 or later (User Namespaces is GA and on by default — no feature gate flag needed)
  • Linux nodes only — this is not available for Windows nodes
  • A container runtime with user namespace support: containerd 2.0+ or CRI-O 1.25+
  • Cgroup v2 on your nodes (check with stat -fc %T /sys/fs/cgroup/ — it should report cgroup2fs)
  • Kernel 6.3+ recommended if you want idmapped-mount performance on overlayfs volumes; older kernels still work but volume ownership handling is slower
  • kubectl and cluster-admin access to install a policy engine (Kyverno used here, OPA/Gatekeeper works the same way)

Kitchen Prep

Before touching YAML, it's worth understanding the mechanism, because it's genuinely elegant. The feature is built on ID-mapped mounts, a Linux kernel capability. Instead of the kubelet recursively chown-ing every file in a volume so a remapped container can read it — an operation that used to tank startup time on large volumes — the kernel now translates UIDs and GIDs at the mount boundary itself. The files on disk never change owner; the container just sees a different view of who owns what. That translation is an O(1) operation, not an O(files) one.

The practical result: a pod with hostUsers: false gets its own private UID/GID range, invisible to and unreachable from the host's real UID 0. Even capabilities like CAP_NET_ADMIN become namespaced — a workload can hold them and manipulate its own network namespace without those same capabilities meaning anything on the host. That's the detail that matters operationally: it opens the door to running things that previously demanded a fully privileged: true pod, without the fully-privileged blast radius.

flowchart LR
    subgraph Host["Node (Host Kernel)"]
        HostRoot["Host UID 0
(real root)"] HostRange["Allocated range
e.g. UID 524288-589823"] end subgraph Pod["Pod (hostUsers: false)"] ContainerRoot["Container sees
UID 0 (fake root)"] end ContainerRoot -- "kernel idmap translation" --> HostRange HostRoot -. "no path in" .-> ContainerRoot style HostRoot fill:#0F172A,color:#fff style ContainerRoot fill:#2563EB,color:#fff style HostRange fill:#16A34A,color:#fff

Recipe Steps

Step 1 — Confirm your cluster is actually GA-ready

kubectl version --short
# Server Version should read v1.36.x or newer

# On each node, confirm the container runtime version
crictl version

Common pitfall: if you're on 1.33–1.35, the feature is beta and already enabled by default, so most of this still works — but stateful workloads and some volume types had rough edges before GA. Don't backport this tutorial to production clusters below 1.33 without reading the alpha/beta blog posts linked at the bottom.

Step 2 — Deploy a pod with a remapped identity

apiVersion: v1
kind: Pod
metadata:
  name: isolated-workload
  namespace: default
spec:
  hostUsers: false
  containers:
  - name: app
    image: fedora:42
    command: ["sleep", "3600"]
    securityContext:
      runAsUser: 0
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
kubectl apply -f isolated-workload.yaml
kubectl wait --for=condition=Ready pod/isolated-workload

Step 3 — Prove the mapping, don't trust it

From inside the container, this workload is root, same as always:

kubectl exec -it isolated-workload -- id
# uid=0(root) gid=0(root) groups=0(root)

Now look at the same process from the node it's scheduled on:

# SSH to the node, then:
ps -eo pid,uid,cmd | grep sleep
# The UID column will show a large, unprivileged number
# (e.g. 524288), never 0

That gap between "0 inside" and "524288 outside" is the entire security model. A kernel-level container escape from this pod lands the attacker as an unprivileged, single-purpose UID on your node — not as root.

Step 4 — Enforce it, don't just recommend it

A tutorial pod proves the mechanism works. A fleet of engineers copy-pasting old manifests proves nothing unless it's enforced. This Kyverno policy blocks any new pod from running with the host's real user namespace unless a namespace is explicitly exempted:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-user-namespaces
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: block-host-users
    match:
      any:
      - resources:
          kinds: ["Pod"]
    exclude:
      any:
      - resources:
          namespaces: ["kube-system", "kube-node-lease"]
    validate:
      message: "hostUsers must be explicitly set to false. See CloudChef's user namespace runbook."
      pattern:
        spec:
          hostUsers: false
kubectl apply -f require-user-namespaces.yaml
kubectl apply -f isolated-workload.yaml   # passes
kubectl apply -f some-legacy-pod.yaml     # fails admission, forces the fix

Chef Tips

  • Stateful workloads benefit the most. Because ID-mapped mounts avoid the old recursive-chown penalty, PVC-backed StatefulSets that used to eat minutes of startup time on ownership fixups now mount almost instantly under user namespaces — worth benchmarking your slowest-starting stateful app against this.
  • Combine it with, don't replace, Pod Security Admission. User namespaces shrink the blast radius of a root process; they don't make running as root a good idea. Keep restricted PSA labels on your namespaces and treat hostUsers: false as an additional layer, not a substitute.
  • This is the same lever that used to require privileged: true. If you've got a CNI plugin, a monitoring agent, or a legacy app that wants CAP_NET_ADMIN or similar, test it under hostUsers: false before reaching for a fully privileged pod out of habit.

Kitchen Disasters

  • Pod stuck in ContainerCreating: almost always an older container runtime. Confirm containerd 2.0+/CRI-O 1.25+ with crictl version on the node — a runtime that predates userns support will silently fail to start the sandbox.
  • Volume permission errors that worked fine before: some CSI drivers and network filesystems don't yet support idmapped mounts cleanly. Check your storage vendor's Kubernetes 1.36 compatibility notes before rolling this out to every StatefulSet at once.
  • Windows nodes silently ignoring the field: hostUsers is Linux-only. If your node pool is mixed OS, make sure scheduling constraints (nodeSelector/taints) keep these pods off Windows nodes — Kubernetes won't stop you from trying.

Serving Suggestions

Once the Kyverno policy is in Enforce mode cluster-wide, the next hardening step is auditing existing workloads: run the policy in Audit mode first, review the violation report, fix the noisiest offenders, then flip to Enforce. Pair this with your image-scanning pipeline — a remapped root is a much smaller problem, but a container that legitimately doesn't need root at all is still the better answer where you can get there.

Cleanup

kubectl delete pod isolated-workload
kubectl delete clusterpolicy require-user-namespaces

Final Thoughts

User Namespaces reaching GA in v1.36 closes a gap that's been open since the first "don't run containers as root" blog post. It doesn't replace least-privilege thinking — it backstops it, for the mountain of workloads that still can't (or won't, this quarter) give up UID 0. If you run a multi-tenant cluster or anything handling untrusted workloads, this is worth scheduling into your next platform sprint, not your someday-backlog.


πŸ”₯ Trending CloudChef Recipes

⭐ Popular CloudChef Recipes

No comments:

Post a Comment

πŸ’‘ Found this useful?

Share it with your Team or DevOps Friends πŸ‘‡