Your Database Password Is in a Kubernetes Secret. Here's Why That's a Problem.

Let me save you from a very embarrassing conversation with your security team.

Kubernetes Secrets are not secret. Not by default. Not even a little bit.

They're base64 encoded. Base64 is an encoding format, not an encryption algorithm. Anyone with kubectl access to your namespace can read every "secret" in it. Anyone with etcd access can read every secret in your entire cluster. It takes three seconds:

kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

And there's your plaintext password. Congratulations. That's your "secret."

---

😤 Why This Keeps Happening

Because "use a Kubernetes Secret" appears in every getting-started guide and it sounds like the right thing to do. The docs use the word "secret." It's a built-in object type. Surely this is how you store sensitive data?

Partially. Kubernetes Secrets provide a separation of concerns — your credentials aren't hardcoded in your YAML. But they provide almost no actual confidentiality unless you add several more layers on top. Most teams don't add those layers.

---

🧠 What Kubernetes Secrets Actually Provide vs What They Don't

What you get	What you don't get
Separation from application code	Encryption (without etcd encryption at rest)
Namespace scoping	Access audit trail by default
RBAC-controllable access	Secret rotation
Environment variable injection	Version history
Volume mounting	Break-glass access controls

---

⚙️ The Proper Secrets Stack

Layer 1: Enable etcd encryption at rest

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

👉 Without this, Secrets in etcd are stored as plaintext. Anyone who gets access to your etcd data — backups, snapshots, direct access — gets everything.

Layer 2: Use External Secrets Operator (ESO) with a real secrets manager

# ESO syncs from AWS Secrets Manager into Kubernetes Secrets automatically
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: production/database
        property: password

This pulls the real secret from AWS Secrets Manager and syncs it into a Kubernetes Secret. You get rotation support, version history, access policies, and CloudTrail audit logging — all from the secrets manager side. The Kubernetes Secret becomes a temporary, auto-refreshed local copy.

Layer 3: Lock down who can read Secrets via RBAC

# Don't give wildcard access to secrets
# Bad:
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]

# Better — scope to specific named secrets:
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials"]
    verbs: ["get"]

---

⚠️ Common Mistakes

• Checking Secrets into git — even encrypted with Sealed Secrets — Sealed Secrets is a reasonable solution, but the moment someone compromises your cluster's private key, everything you ever committed is readable. Treat sealed secrets as short-lived, not permanent archives.
• Injecting Secrets as environment variables — Environment variables are visible in kubectl exec sessions, process listings, and debug tooling. Mount them as files via volume instead when possible.
• No rotation policy — Static credentials that never rotate are credentials waiting for their moment. Set up auto-rotation in your secrets manager and use ESO's refreshInterval to pull updates automatically.

---

🔥 CloudChef Pro Tip

Mount secrets as files, not environment variables.

volumes:
  - name: db-creds
    secret:
      secretName: db-credentials
containers:
  - volumeMounts:
      - name: db-creds
        mountPath: /var/secrets
        readOnly: true

👉 Your app reads /var/secrets/password as a file. It never appears in environment variables. It never shows up in kubectl describe pod output. It's a small change with a meaningful security improvement.

🔗 Continue Your CloudChef Journey

🔥 Related CloudChef Recipes

Loading related posts...

---

📚 References

• External Secrets Operator
• Encrypting Secret Data at Rest
• AWS Secrets Manager

---

🚀 Final Thoughts

A Kubernetes Secret alone isn't security — it's a starting point. Add encryption at rest, an external secrets manager for real credentials, tight RBAC on who can read them, and volume mounts instead of environment variables. All four layers together is what "secrets management" actually means.

👉 The good news: External Secrets Operator is free, well-maintained, and the setup takes an afternoon. Do it before your next security audit finds it for you.