Kubernetes Jobs and CronJobs

by Bala

Running batch and scheduled workloads to completion - parallelism, backoff, TTL cleanup, and concurrency control

Introduction

A Deployment keeps its pods running forever, restarting them if they exit. A Job is for the opposite case - work that’s meant to finish: a database migration, a batch report, a one-off data backfill. A CronJob layers a schedule on top of that, creating a new Job on a recurring basis.

A Basic Job

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: myregistry/db-migrate:1.4.0
          command: ["./migrate", "up"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: url

restartPolicy: Never (or OnFailure) is required on every Job pod template - Always, the default for other pod-owning resources, is rejected outright since it would make the pod restart forever and the Job would never be considered complete. backoffLimit: 3 caps retries at 3 failed attempts before the Job is marked Failed - each retry creates a new pod rather than restarting the failed one, so a Job that hits its backoff limit can leave several failed pods behind for you to inspect with kubectl logs.

activeDeadlineSeconds is a hard wall-clock timeout for the whole Job (not per-attempt) - once exceeded, Kubernetes terminates any running pods and marks the Job Failed regardless of backoffLimit, a safety net against a job that’s hung rather than genuinely failing and retrying. ttlSecondsAfterFinished (requires the TTL controller, enabled by default since Kubernetes 1.23) automatically deletes the Job and its pods some time after completion - without it, completed Jobs accumulate indefinitely and have to be cleaned up manually.

Parallel Jobs

apiVersion: batch/v1
kind: Job
metadata:
  name: batch-report
spec:
  completions: 10      # total successful pod completions needed
  parallelism: 3        # up to 3 pods running at once
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: myregistry/batch-worker:2.1.0
          env:
            - name: JOB_COMPLETION_INDEX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

completions: 10 with parallelism: 3 runs up to 3 pods at a time until 10 total have completed successfully - a work queue of fixed, known size processed by a bounded pool of workers. completionMode: Indexed additionally assigns each pod a fixed index from 0 to 9 (available via the batch.kubernetes.io/job-completion-index annotation, or the pod’s hostname), letting each worker deterministically claim a specific shard of the work (index 3 processes rows 300000-399999) instead of coordinating over a shared queue.

CronJobs

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  timeZone: "America/New_York"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  startingDeadlineSeconds: 300
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: myregistry/backup-tool:3.0.0
              command: ["./backup.sh"]

batch/v1 is the only supported CronJob API version on any currently maintained Kubernetes release - batch/v1beta1 graduated to v1 in Kubernetes 1.21 and was removed entirely in 1.25, so a manifest still referencing v1beta1 fails outright on a modern cluster.

concurrencyPolicy: Forbid skips starting a new Job if the previous scheduled run is still going - the right choice for anything that shouldn’t run two copies at once (a backup job, a report that writes to a fixed output location). Allow (the default) runs overlapping instances with no coordination; Replace cancels the still-running previous Job and starts the new one instead.

timeZone (stable since Kubernetes 1.27) lets schedule be interpreted in a named zone instead of the controller’s UTC default - critical for anything genuinely tied to a business’s local time (like “run after markets close”), since a bare cron schedule with no timeZone silently means UTC, not the cluster operator’s local time.

startingDeadlineSeconds: 300 bounds how late a missed run can still start - if the CronJob controller itself was down (a control plane outage, for instance) and the scheduled time has passed by more than this many seconds when it comes back, that run is skipped entirely rather than started late, avoiding a pile-up of backlogged runs all firing at once.

Manually Triggering a CronJob’s Job

kubectl create job nightly-backup-manual --from=cronjob/nightly-backup

This creates a one-off Job using the CronJob’s jobTemplate immediately, outside the schedule - the standard way to test a CronJob’s actual behavior without waiting for (or temporarily changing) its schedule.

Best Practices

  1. Always set ttlSecondsAfterFinished - completed Jobs and their pods otherwise accumulate forever, both cluttering kubectl get jobs and consuming etcd storage for objects nobody needs once they’ve completed.
  2. Set concurrencyPolicy deliberately, not by accepting the Allow default - decide explicitly whether overlapping runs are safe for the specific workload.
  3. Set activeDeadlineSeconds on anything that could hang rather than cleanly fail - a stuck Job with no deadline holds its pod (and whatever resources it’s consuming) indefinitely.
  4. Use completionMode: Indexed for genuinely parallel batch work with a known, fixed size - it removes the need for a separate work-queue system to coordinate which worker does what.

Conclusion

Jobs and CronJobs are the right tool whenever “run to completion” is the actual semantics you want, rather than “keep running forever” - trying to model batch work as a Deployment (with a container that exits and gets endlessly restarted) fights the platform instead of using it.

Additional Resources