OpenShift (OCP) Upgrade Checklist: 5 Essential Phases

Planning an OpenShift Container Platform (OCP) upgrade requires structured validation across 5 distinct phases. Because OpenShift performs rolling updates of both the control plane and the host OS (RHCOS), executing pre-upgrade checks prevents node-drain locks, degraded operators, or broken API dependencies.

Phase 1: Pre-Upgrade Review & Path Mapping

  1. Verify Your Upgrade Path:
    • OpenShift does not support skipping Y-stream releases (e.g., jumping from 4.14 to 4.16 directly). You must upgrade sequentially (4.14 → 4.15 → 4.16).
    • Check your configured update channel (stable-4.x, fast-4.x, or eus-4.x).For enterprise production environments, stick to stable-4.x or eus-4.x (Extended Update Support).
    • Query the OpenShift Update Service (OSUS) via CLI:
      adm upgrade
  2. Audit Deprecated & Removed Kubernetes APIs:
    • Before a minor version jump (e.g., 4.15 → 4.16), check if your workloads use removed K8s API versions.
    • Review cluster alerts for API deprecations and migrate manifests accordingly. OCP will block minor version upgrades if administrator acknowledgment is missing for API removals.
  3. Check Operator & OLM Compatibility:
    • Non-core platform operators (installed via OperatorHub/OLM) must be upgraded to versions compatible with the target OCP release before updating the cluster core.

Phase 2: Cluster Health & Capacity Assessment

Run a complete health scan. Never trigger an upgrade on a degraded cluster.

  1. Verify ClusterOperator Status:Ensure all core operators report AVAILABLE=True, PROGRESSING=False, and DEGRADED=False:
    oc get clusteroperator
  2. Verify MachineConfigPool (MCP) Health:Ensure all node pools are updated and none are paused or degraded:
    oc get mcp (If an MCP is paused for canary testing, verify it is intentional before proceeding.)
  3. Check Node Capacity & PodDisruptionBudgets (PDBs):
    • Node Capacity: During node-draining, remaining workers must have enough CPU/RAM capacity to host evacuated workloads.
    • PDB Deadlocks: Inspect custom PodDisruptionBudgets (oc get pdb -A). If minAvailable is set equal to the total replica count (or maxUnavailable: 0), node drains will stall, hanging the upgrade loop.
  4. Clear Active Alerts:In the console, go to Observe → Alerting and resolve all critical alerts.

Phase 3: Mandatory Backups (Non-Negotiable)

Because OpenShift updates are fail-forward (rolling back to an older minor version is not supported), taking backups prior to execution is mandatory:

  1. Backup etcd: Take a manual snapshot of the etcd database from a control plane node:
    oc debug node/<master-node-name> -- /usr/local/bin/cluster-backup.sh /host/var/usr/temp/etcd-backup
  2. Export Cluster Custom Resources (CRDs): Export your custom deployment manifests, routes, and secret configurations using a GitOps tool (ArgoCD/ACM) or CLI backups.
  3. Storage Snapshots: Create CSI volume snapshots for critical stateful application data.

Phase 4: Execution & Monitoring

  1. Trigger the Upgrade:
    • Via Web Console:Navigate to Administration → Cluster Settings → Details, select your desired target version, and confirm.
    • Via CLI:
      oc adm upgrade --to=<target_version>
  2. Monitor the Phased Rollout:
    • Control Plane Phase: Watch the CVO reconcile cluster operators:
      oc get clusterversion -w
    • Worker Pool Phase: Watch the MCO cordon, drain, re-image (rpm-ostree), reboot, and uncordon nodes pool-by-pool:
      oc get nodes
    • oc get mcp

Phase 5: Post-Upgrade Validation

Once the CVO reports Cluster version is <target_version>:

  • [ ] Verify all nodes report Ready and run the new RHCOS kernel version.
  • [ ] Confirm all ClusterOperators return to an AVAILABLE=True state.
  • [ ] Verify routes, ingress controllers, and critical business applications are accepting traffic.
  • [ ] Re-enable any paused MachineConfigPools or automated maintenance windows.

Leave a Reply