Kubernetes Upgrade Strategies

by Bala

Best practices for upgrading Kubernetes clusters and applications

Kubernetes Upgrade Strategies

Upgrading Kubernetes clusters and applications requires careful planning and execution. This guide covers essential upgrade practices.

Video Tutorial

Prerequisites

  • Basic understanding of Kubernetes
  • Access to a Kubernetes cluster
  • kubectl CLI tool installed
  • Familiarity with cluster management

Project Structure

.
├── upgrade/
│   ├── cluster/         # Cluster upgrade configs
│   ├── apps/           # Application upgrade configs
│   ├── rollback/       # Rollback procedures
│   └── tests/          # Upgrade tests
└── monitoring/
    ├── health/         # Health checks
    └── alerts/         # Upgrade alerts

Cluster Upgrade

1. Control Plane Upgrade

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.27.0
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    rotate-certificates: "true"

2. Node Pool Upgrade

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-upgrade
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    spec:
      containers:
      - name: app
        image: app:v2

Application Upgrade

1. Rolling Update

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
      - name: app
        image: app:v2
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10

2. Blue-Green Deployment

apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  selector:
    app: app
    version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: app-preview
spec:
  selector:
    app: app
    version: v2

Health Checks

1. Readiness Probe

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: app:v2
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20

2. Startup Probe

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: app:v2
    startupProbe:
      httpGet:
        path: /startup
        port: 8080
      failureThreshold: 30
      periodSeconds: 10

Monitoring Setup

1. Upgrade Monitoring

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: upgrade-monitor
spec:
  selector:
    matchLabels:
      app: upgrade-status
  endpoints:
  - port: metrics

2. Alert Configuration

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: upgrade-alerts
spec:
  groups:
  - name: upgrade
    rules:
    - alert: UpgradeFailed
      expr: upgrade_status{state="failed"} > 0
      for: 5m
      labels:
        severity: critical

Best Practices Checklist

  1. ✅ Backup before upgrade
  2. ✅ Test upgrade procedure
  3. ✅ Health monitoring
  4. ✅ Rollback plan
  5. ✅ Version compatibility
  6. ✅ Resource requirements
  7. ✅ Communication plan
  8. ✅ Downtime window
  9. ✅ Documentation
  10. ✅ Post-upgrade tests

Upgrade Patterns

Incremental Upgrade

  • Version by version
  • Component testing
  • Gradual rollout
  • Monitoring at each step

Direct Upgrade

  • Skip versions
  • Full testing
  • Quick deployment
  • Comprehensive monitoring

Canary Upgrade

  • Partial rollout
  • User testing
  • Performance monitoring
  • Gradual expansion

Common Pitfalls

  1. ❌ Insufficient testing
  2. ❌ Missing backups
  3. ❌ Poor monitoring
  4. ❌ No rollback plan
  5. ❌ Resource constraints

Rollback Procedures

1. Deployment Rollback

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 3
  template:
    metadata:
      annotations:
        kubernetes.io/change-cause: "Rolling back to v1"
    spec:
      containers:
      - name: app
        image: app:v1

2. StatefulSet Rollback

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  template:
    spec:
      containers:
      - name: database
        image: database:v1

Testing Strategy

1. Pre-upgrade Tests

apiVersion: batch/v1
kind: Job
metadata:
  name: pre-upgrade-test
spec:
  template:
    spec:
      containers:
      - name: test
        image: test:latest
        command: ["./test-suite.sh"]
      restartPolicy: Never

2. Post-upgrade Tests

apiVersion: batch/v1
kind: Job
metadata:
  name: post-upgrade-test
spec:
  template:
    spec:
      containers:
      - name: test
        image: test:latest
        command: ["./validation-suite.sh"]
      restartPolicy: Never

Version Management

1. Version Labels

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    metadata:
      labels:
        app: myapp
        version: v2.0.0
        kubernetes.io/version: v1.27.0

2. Version Annotations

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kubernetes.io/change-cause: "Upgrade to v2.0.0"
    app.kubernetes.io/version: "2.0.0"
    kubernetes.io/target-version: "1.27.0"

Conclusion

Implementing these upgrade strategies ensures smooth and reliable updates for your Kubernetes clusters and applications. Regular testing and proper planning are essential for successful upgrades.

Additional Resources