Kubernetes Secrets Management

by Bala

Best practices for managing secrets in Kubernetes

Kubernetes Secrets Management

Proper secrets management is crucial for maintaining security in Kubernetes. This guide covers essential practices for handling sensitive information.

Video Tutorial

Prerequisites

  • Basic understanding of Kubernetes
  • Access to a Kubernetes cluster
  • kubectl CLI tool installed
  • Familiarity with security concepts

Project Structure

.
├── secrets/
│   ├── vault/           # HashiCorp Vault configs
│   ├── sealed/         # Sealed Secrets
│   ├── external/       # External Secrets
│   └── encryption/     # Encryption configs
└── monitoring/
    ├── audit/          # Audit logging
    └── alerts/         # Security alerts

Vault Integration

1. Vault Configuration

apiVersion: vault.banzaicloud.com/v1alpha1
kind: Vault
metadata:
  name: vault
spec:
  size: 1
  image: vault:1.12.0
  bankVaultsImage: banzaicloud/bank-vaults:latest
  config:
    storage:
      file:
        path: /vault/file
    listener:
      tcp:
        address: "0.0.0.0:8200"
        tls_disable: true

2. Vault Auth

apiVersion: vault.banzaicloud.com/v1alpha1
kind: VaultSecret
metadata:
  name: vault-auth
spec:
  path: secret/data/myapp
  type: Opaque
  vault:
    role: myapp
    auth:
      kubernetes:
        role: myapp
        serviceAccount: default

Sealed Secrets

1. SealedSecret Definition

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
spec:
  encryptedData:
    username: AgBy8hCK8s...
    password: AgBy8hCK8s...
  template:
    metadata:
      labels:
        app: myapp
    type: Opaque

2. Secret Template

apiVersion: v1
kind: Secret
metadata:
  name: template-secret
type: Opaque
stringData:
  config.yaml: |
    apiKey: ${API_KEY}
    endpoint: ${ENDPOINT}

External Secrets

1. SecretStore Configuration

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-secret-creds
            key: access-key
          secretAccessKeySecretRef:
            name: aws-secret-creds
            key: secret-key

2. External Secret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: aws-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secret-store
    kind: SecretStore
  target:
    name: application-secret
  data:
  - secretKey: username
    remoteRef:
      key: app/production/username
  - secretKey: password
    remoteRef:
      key: app/production/password

Encryption Configuration

1. Encryption at Rest

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
metadata:
  name: encryption-config
spec:
  resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: <base64-encoded-key>
    - identity: {}

2. Key Rotation

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
spec:
  resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key2
          secret: <new-key>
        - name: key1
          secret: <old-key>
    - identity: {}

Best Practices Checklist

  1. ✅ Use encryption at rest
  2. ✅ Implement secret rotation
  3. ✅ Enable audit logging
  4. ✅ Use RBAC controls
  5. ✅ Monitor secret access
  6. ✅ Regular key rotation
  7. ✅ Secure storage
  8. ✅ Access policies
  9. ✅ Documentation
  10. ✅ Backup strategy

Secret Management Patterns

Centralized Secrets

  • Single source of truth
  • Automated rotation
  • Access control
  • Audit logging

Distributed Secrets

  • Per-namespace secrets
  • Local encryption
  • Limited access
  • Namespace isolation

External Secrets

  • Cloud provider integration
  • Automated sync
  • Version control
  • Disaster recovery

Common Pitfalls

  1. ❌ Hardcoded secrets
  2. ❌ Unencrypted storage
  3. ❌ Poor access control
  4. ❌ Missing rotation
  5. ❌ Insufficient monitoring

Monitoring Setup

1. Audit Policy

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
  verbs: ["create", "update", "delete"]

2. Alert Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: secret-alerts
spec:
  groups:
  - name: secrets
    rules:
    - alert: SecretAccessAnomaly
      expr: rate(secret_access_total{status="unauthorized"}[5m]) > 10
      for: 5m
      labels:
        severity: critical

Secret Rotation

1. Automated Rotation

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "app/production/secret1"
        objectType: "secretsmanager"
  secretObjects:
  - secretName: application-secret
    type: Opaque
    data:
    - objectName: secret1
      key: username

2. Manual Rotation

apiVersion: v1
kind: Secret
metadata:
  name: rotating-secret
  annotations:
    secret-rotation: "true"
    rotation-date: "2025-01-20"
type: Opaque
data:
  api-key: <base64-encoded-key>

Access Control

1. RBAC Policy

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
  resourceNames: ["app-secret"]

2. Network Policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: secret-access
spec:
  podSelector:
    matchLabels:
      role: secret-consumer
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: vault

Conclusion

Implementing these secrets management practices ensures secure handling of sensitive information in your Kubernetes clusters. Regular audits and updates are essential for maintaining security.

Additional Resources