GitOps Deployment of CloudNativePG with ArgoCD

by Bala

Deploying the CloudNativePG operator, Barman Cloud Plugin, and a production Postgres cluster declaratively - sync waves, webhook-defaulting drift, and custom health checks

Introduction

The previous guide built a production CloudNativePG cluster with daily backups to Cloudflare R2 by applying manifests with kubectl. This one covers the same setup managed declaratively through ArgoCD instead - and two things about CloudNativePG specifically make that non-trivial: installation has a strict ordering dependency (CRDs and the plugin must exist before a Cluster can reference them), and CloudNativePG’s own admission webhook mutates .spec with defaults at apply time, which left unaddressed shows up as a permanently OutOfSync Application in ArgoCD.

Repository Layout and Sync Waves

Three logical groups of resources, each with a real ordering dependency on the one before it:

  1. The CloudNativePG operator and Barman Cloud Plugin (cluster-scoped CRDs and controllers)
  2. The ObjectStore and its credentials Secret (namespaced, but must exist before the Cluster references them)
  3. The Cluster and ScheduledBackup (depend on both of the above)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cnpg-operator
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
spec:
  project: default
  source:
    repoURL: https://github.com/example/platform-gitops.git
    targetRevision: main
    path: cnpg/operator
  destination:
    server: https://kubernetes.default.svc
    namespace: cnpg-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

cnpg/operator in the source repo should hold a vendored, version-pinned copy of the operator manifest (cnpg-1.30.0.yaml) and the plugin manifest, not a kubectl apply -f <upstream URL> reference - ArgoCD needs an actual file in Git to diff against, and pointing at a moving upstream release URL means every future upstream release silently changes what “in sync” means for your Application without a corresponding Git commit. Pin the version, commit the file, and bump it deliberately (the same discipline as pinning imageName on the Cluster itself).

sync-wave: "-2" (and "-1" for cert-manager if it isn’t already cluster infrastructure managed elsewhere) ensures the operator and plugin are fully reconciled before ArgoCD attempts to apply anything that depends on their CRDs - a Cluster or ObjectStore applied before its CRD exists simply fails, and ArgoCD’s own dependency-free default sync order doesn’t guarantee the ordering CloudNativePG needs.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cnpg-prod-postgres
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: default
  source:
    repoURL: https://github.com/example/platform-gitops.git
    targetRevision: main
    path: cnpg/prod-postgres
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The cnpg/prod-postgres path holds the ObjectStore, Cluster, and ScheduledBackup manifests from the previous guide - a single Application is fine here since they’re small and tightly coupled, but per-resource sync-wave annotations within that path (ObjectStore and its Secret at wave 0, Cluster and ScheduledBackup at wave 1) still matter, since ArgoCD applies resources within one Application in wave order too.

Keeping R2 Credentials Out of Git

The r2-backup-creds Secret from the previous guide can’t be committed in plaintext. This isn’t a CloudNativePG-specific problem - it’s the same tradeoff covered in the Kubernetes secrets management guide on this site: either encrypt the Secret at rest in Git with Sealed Secrets (decrypted only by a controller running in-cluster), or keep the real value entirely out of Git with the External Secrets Operator pulling from a real secrets manager (AWS Secrets Manager, Vault, etc.) and materializing a Kubernetes Secret at sync time. Either approach lets the reference to the credential live in Git and go through the same ArgoCD-managed sync as everything else, while the actual R2 access key never does.

Handling CloudNativePG’s Webhook-Defaulting Drift

CloudNativePG’s Cluster resource has a mutating admission webhook that fills in defaults on .spec itself, not just .status - confirmed directly in the operator’s source (Cluster.Default() in api/v1/cluster_defaults.go). Concretely, if left unset in your manifest, the webhook sets:

  • .spec.imageName to the operator’s currently configured default Postgres image
  • .spec.postgresql.parameters to a fully expanded set of sanitized Postgres configuration parameters - not just the ones you wrote, the complete resolved set
  • .spec.affinity.podAntiAffinityType to Preferred, once pod anti-affinity is enabled
  • .spec.replicationSlots.highAvailability to {enabled: true, slotPrefix: "_cnpg_"}, and .spec.replicationSlots.synchronizeReplicas.enabled to true

Because these land on .spec (not .status, which ArgoCD’s default diff mostly ignores for a well-behaved CRD), the live object’s spec ends up with more fields populated than what’s committed in Git - and ArgoCD will report that Application as permanently OutOfSync, since its diff is comparing your committed manifest against a live object the webhook has already expanded.

Two ways to handle it, and the first is the better default:

Commit the resolved values explicitly. Set imageName (as the previous guide already recommends, for its own reasons), and if you rely on any non-default postgresql.parameters, write them out in full in Git rather than leaving the field unset. This keeps Git as the actual source of truth for what’s running, which is the point of GitOps in the first place - it just means being explicit about values the webhook would otherwise silently fill in for you.

Or, tell ArgoCD to ignore specific fields, for values you’re deliberately fine leaving to the operator’s defaults:

spec:
  ignoreDifferences:
    - group: postgresql.cnpg.io
      kind: Cluster
      jsonPointers:
        - /spec/replicationSlots
        - /spec/affinity/podAntiAffinityType

Don’t reach for ignoreDifferences reflexively on every field that shows up in a diff, though - run argocd app diff cnpg-prod-postgres first and confirm each field you’re about to ignore is genuinely webhook-defaulted noise, not a real drift you’d want to know about (someone running kubectl edit directly against the cluster, for instance, which selfHeal: true would otherwise catch and revert).

Custom Health Checks for CloudNativePG Resources

ArgoCD doesn’t know how to assess the health of a Cluster or Backup out of the box - without a health check, a Cluster just shows Synced (the manifest was applied) with no signal about whether Postgres is actually up and replicated. ArgoCD supports Lua-based custom health checks per resource kind, configured in the argocd-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.health.postgresql.cnpg.io_Cluster: |
    hs = {}
    if obj.status ~= nil and obj.status.readyInstances ~= nil then
      if obj.status.readyInstances == obj.spec.instances then
        hs.status = "Healthy"
        hs.message = obj.status.phase or "All instances ready"
        return hs
      end
      hs.status = "Progressing"
      hs.message = string.format("%d/%d instances ready", obj.status.readyInstances, obj.spec.instances)
      return hs
    end
    hs.status = "Progressing"
    hs.message = "Waiting for status"
    return hs

  resource.customizations.health.postgresql.cnpg.io_Backup: |
    hs = {}
    if obj.status ~= nil and obj.status.phase ~= nil then
      if obj.status.phase == "completed" then
        hs.status = "Healthy"
      elseif obj.status.phase == "failed" then
        hs.status = "Degraded"
      else
        hs.status = "Progressing"
      end
      hs.message = obj.status.phase
      return hs
    end
    hs.status = "Progressing"
    hs.message = "Waiting for backup to start"
    return hs

With this in place, a Cluster stuck below its declared instance count or a Backup that failed shows up as Degraded directly in the ArgoCD UI and CLI - the same place you already look for sync status - rather than requiring a separate kubectl cnpg status check to notice.

Alerting on Backup Failures

Combined with the Notifications engine, a custom health check that surfaces Degraded on a failed Backup also means the existing on-health-degraded trigger (from the built-in notifications catalog) fires for backup failures specifically, once subscribed on the Application that owns the Backup resources - no bespoke trigger needed, since the health check above already does the work of classifying “backup failed” as a health state ArgoCD’s notifications system already knows how to watch.

Declarative Role Management

For anything beyond the single application user created by bootstrap.initdb, CloudNativePG’s DatabaseRole custom resource is the officially recommended approach for GitOps workflows specifically - it decouples role lifecycle from the Cluster resource itself, so adding a new read-only reporting role doesn’t require touching the Cluster manifest at all:

apiVersion: postgresql.cnpg.io/v1
kind: DatabaseRole
metadata:
  name: reporting-role
  namespace: production
spec:
  cluster:
    name: prod-postgres
  name: reporting
  login: true
  createdb: false
  databaseRoleReclaimPolicy: retain
  inRoles:
    - pg_monitor
  passwordSecret:
    name: prod-postgres-reporting-user

databaseRoleReclaimPolicy: retain (the default) leaves the Postgres role in place if this manifest is ever deleted from Git - the safer choice for anything beyond a genuinely ephemeral role, mirroring how PersistentVolumeReclaimPolicy protects against accidental data loss on the storage side.

Best Practices

  1. Vendor and pin the operator/plugin manifests in Git rather than pointing an Application’s source at a live upstream release URL - reproducibility and an honest diff both depend on Git being the actual source of truth for the exact version deployed.
  2. Set sync-waves based on real CRD/resource dependencies, not arbitrarily - operator and plugin first, ObjectStore/Secret next, Cluster/ScheduledBackup last, matching the order kubectl apply would need to succeed manually.
  3. Prefer committing explicit values over blanket ignoreDifferences for CloudNativePG’s webhook-defaulted fields - it keeps Git as the real source of truth and avoids accidentally masking genuine drift alongside the expected noise.
  4. Add the custom health checks before relying on ArgoCD’s UI/notifications for cluster or backup status - without them, ArgoCD only ever tells you the manifest was applied, never whether Postgres is actually healthy.
  5. Manage roles beyond the app user with DatabaseRole resources, not ad-hoc psql commands against a running cluster - it’s the difference between role grants being reviewable in a PR and being invisible until someone goes looking.

Conclusion

CloudNativePG under ArgoCD is a real GitOps workflow, not just “the same YAML, applied differently” - the parts that need actual thought are the install-order dependency on CRDs, the webhook-defaulting drift ArgoCD will otherwise report as permanent OutOfSync, and giving ArgoCD a way to know whether the Postgres cluster underneath is actually healthy rather than just “applied.”

Additional Resources