In 2020, attackers modified a legitimate build of SolarWinds Orion during the software build process. The malicious version was signed with SolarWinds' own certificate, shipped to 18,000 customers as an official update, and sat undetected for months.
That wasn't a Kubernetes attack. But the same class of attack — compromising something you trust in your supply chain — happens to container images every day at a much smaller and less reported scale.
Your cluster pulls images. The question is: do you have any idea what's actually inside them?
π€ How Supply Chain Attacks Target Your Containers
Three main attack vectors. All of them have hit real production environments.
- Compromised base images — You pull
ubuntu:latestornode:20. A vulnerability exists in that image you haven't patched. Your app inherits it. An attacker with a known CVE against the container runtime can potentially escape the container entirely. - Typosquatted images on Docker Hub — Someone registers
minngoon Docker Hub and it looks close enough tomongothat an engineer mistyping once pulls it. That image has a cryptominer baked in. - Compromised CI build pipelines — An attacker gets write access to your build system, inserts malicious code before the Docker build step, and your own CI signs and pushes the backdoored image to your private registry.
π§ A Real Fix: Image Signing and Verification
Image signing means your CI creates a cryptographic signature when it builds and pushes an image. Admission verification means your cluster refuses to run any image that doesn't have a valid signature from your CI — regardless of where the image came from or what it claims to be.
Step 1 — Sign your images in CI using Cosign
# In your GitHub Actions workflow
- name: Sign container image
uses: sigstore/cosign-installer@v3
- name: Sign the image
env:
COSIGN_EXPERIMENTAL: 1
run: |
cosign sign --yes \
registry.company.com/my-app:${{ github.sha }}
With keyless signing via Sigstore, the signature is backed by your CI's OIDC identity — GitHub Actions, GitLab, or whatever you use. No private key to manage or leak.
Step 2 — Verify at admission with Kyverno
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-image-signatures
spec:
validationFailureAction: Enforce
rules:
- name: check-signature
match:
resources:
kinds: ["Pod"]
verifyImages:
- imageReferences:
- "registry.company.com/*"
attestors:
- entries:
- keyless:
subject: "https://github.com/your-org/your-repo/.github/workflows/*"
issuer: "https://token.actions.githubusercontent.com"
π Now any pod that tries to run an unsigned image — or an image signed by anything other than your own CI pipeline — gets rejected at the Kubernetes API before it ever schedules.
π SBOMs — Knowing What's Inside the Can
A Software Bill of Materials (SBOM) is the ingredient list for your container image. Every package, library, and dependency it contains, with versions. When a new CVE drops, you query your SBOM to find out which of your images are affected — instead of finding out via a breach.
# Generate SBOM with Syft
syft registry.company.com/my-app:latest -o spdx-json > sbom.json
# Scan it immediately with Grype
grype sbom:sbom.json --fail-on high
Integrate both into your CI pipeline. Block the build if high-severity CVEs are found. Push the SBOM as an attestation attached to the image.
⚠️ Common Mistakes
- Using
:latesttags in production —:latestis a floating reference. What you tested yesterday isn't guaranteed to be what you deploy tomorrow. Always pin to a specific digest:registry.company.com/my-app@sha256:abc123... - Only scanning images at build time and never again — A vulnerability-free image today can have a critical CVE discovered against it next month. Continuously scan running container images, not just new ones.
- Pulling from Docker Hub without a pull-through cache — Docker Hub rate-limits unauthenticated pulls and has had malicious images appear periodically. Use AWS ECR Public Gallery or your own pull-through cache.
π₯ CloudChef Pro Tip
Enforce image registry allowlists at admission.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-registries
spec:
validationFailureAction: Enforce
rules:
- name: allowed-registries
match:
resources:
kinds: ["Pod"]
validate:
message: "Images must come from approved registries."
pattern:
spec:
containers:
- image: "registry.company.com/* | 123456789.dkr.ecr.us-east-1.amazonaws.com/*"
π This single policy prevents anyone from accidentally (or intentionally) pulling a public image that hasn't gone through your scanning and signing pipeline.
π Continue Your CloudChef Journey
π References
π Final Thoughts
Supply chain security feels abstract until the moment it isn't. The tooling is there — Cosign, Syft, Grype, Kyverno — and most of it is free and open source. Signing images and verifying at admission takes a day to implement properly. That's a day well spent compared to explaining a breach that came through a dependency you didn't even know you were running.
π Sign your images. Know what's in them. Trust nothing you didn't build.
No comments:
Post a Comment