Something happened in your cluster last Thursday at 2:17am. A pod was deleted. A secret was read. Someone exec'd into a production container.
Can you tell me who did it?
If the answer is "I'd have to check Slack" or "I'm not sure we log that" — you don't have audit logging. And without it, you're essentially running a production system with no security camera and no door log.
π€ Why Audit Logs Are Always the Last Thing Set Up
Because they don't make your application run faster. They don't reduce costs. They don't make deployments easier. Audit logs are pure security and compliance value — which means they tend to exist in the "we should do that" backlog until a security incident, an audit, or a compliance certification makes them non-optional.
Don't wait for that moment. Set them up now. Future you will be grateful.
π§ How Kubernetes Audit Logging Actually Works
Every API request that hits the Kubernetes API server — whether from kubectl, a controller, a webhook, or your CI pipeline — can be recorded in an audit log. The log captures who made the request, what resource they touched, what action they took, and what the response was.
The key is the audit policy. Without a policy, nothing gets logged. With a badly written policy, you either log everything (gigabytes per hour, unusable) or log almost nothing (useless for investigation).
⚙️ A Production-Grade Audit Policy
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Never log the following — they're noisy and low value
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: ""
resources: ["endpoints", "services", "services/status"]
# Never log health checks from nodes
- level: None
userGroups: ["system:nodes"]
verbs: ["get"]
resources:
- group: ""
resources: ["nodes", "nodes/status"]
# Log secret, configmap, tokenreviews at Metadata level only
# (we want to know WHO accessed them, not the content)
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps"]
- group: authentication.k8s.io
resources: ["tokenreviews"]
# Log exec and port-forward at Request level — these are high risk
- level: Request
verbs: ["create"]
resources:
- group: ""
resources: ["pods/exec", "pods/portforward", "pods/proxy"]
# Log all other resource modifications at RequestResponse level
- level: RequestResponse
verbs: ["create", "update", "patch", "delete", "deletecollection"]
# Everything else at Metadata
- level: Metadata
π The pods/exec rule is the one I care about most. Anyone who exec's into a production pod should be logged at Request level — enough to know exactly who ran the exec and what namespace and pod they targeted.
π The Events You Should Actually Alert On
Raw audit logs are noise. You need to know which events signal a real problem.
| Event | Why It Matters | Level to Log |
|---|---|---|
pods/exec create | Interactive shell access to production pods | Request |
secrets get at scale | Possible credential harvesting | Metadata |
clusterrolebindings create/modify | Privilege escalation attempt | RequestResponse |
daemonsets create in kube-system | Persistence attempt | RequestResponse |
| Anonymous user requests | Unauthenticated API access | RequestResponse |
pods/portforward | Tunneling from/to pod | Request |
⚙️ Shipping Logs to Somewhere Useful
# Send audit logs to CloudWatch Logs (EKS)
# In your EKS cluster config:
cloudWatch:
clusterLogging:
enableTypes:
- "audit"
- "api"
- "controllerManager"
- "scheduler"
- "authenticator"# Query who exec'd into pods in the last 24 hours (CloudWatch Insights)
fields @timestamp, user.username, objectRef.namespace, objectRef.name, responseStatus.code
| filter verb = "create" and objectRef.subresource = "exec"
| sort @timestamp desc
| limit 50π That query alone is worth setting up. Run it weekly as a sanity check. If you see service accounts exec'ing into pods, something's wrong. If you see humans exec'ing into production pods regularly, you need a different deployment process.
⚠️ Common Mistakes
- Logging everything at RequestResponse — You'll generate gigabytes per hour and drown in noise. Use Metadata for read operations, RequestResponse only for writes and high-risk actions.
- Writing audit logs to the API server node disk only — That's fine until the node fails or gets terminated and you lose everything. Ship to CloudWatch, S3, or a SIEM immediately.
- No retention policy — Most compliance frameworks require 90-365 days of audit log retention. Set a lifecycle policy on your S3 bucket or CloudWatch log group before you need to produce those logs for an auditor.
π₯ CloudChef Pro Tip
Set up a weekly audit log review ritual.
π Five minutes, once a week. Run your exec query. Check for unexpected clusterrolebinding changes. Look for anonymous API requests. It sounds tedious. It's actually the cheapest security monitoring practice that exists, and it catches things that automated tools miss.
π Continue Your CloudChef Journey
π References
- Kubernetes Audit Logging Documentation
- EKS Control Plane Logging
- Falco — Runtime Security for Kubernetes
π Final Thoughts
Audit logs are the closest thing Kubernetes has to a security camera. Without them, when something goes wrong — and eventually, something goes wrong — you're investigating an incident with zero evidence. With them, you have timestamps, identities, and actions. The difference between a 4-hour investigation and a 4-day one.
π Enable audit logging, write a reasonable policy, ship the logs somewhere durable, and set up the two or three alerts that matter. That's all. It's one afternoon of work that pays for itself every time you need to answer "who did that?"
No comments:
Post a Comment