Upgrading an OpenShift Container Platform (OCP) cluster is a fully automated, highly orchestrated process managed by the Cluster Version Operator (CVO) and the Machine Config Operator (MCO).

Because OpenShift enforces an immutable architecture, an upgrade is not an in-place modification of existing configuration files. Instead, it is an atomic rollout of a verified system payload image that replaces the underlying operating system and operator layers in a rolling, zero-downtime sequence.

1. Pre-Upgrade: Graph Validation & Path Resolution

Before any data moves, OpenShift ensures the requested upgrade path is safe and structurally certified.

 ┌───────────────────────────┐      Queries      ┌──────────────────────────────┐
 │   Your OCP Cluster        │ ────────────────► │ OpenShift Update Service     │
 │ (Currently running v4.15) │ ◄──────────────── │ (Returns Valid Upgrade Graph)│
 └───────────────────────────┘   Valid Paths     └──────────────────────────────┘

1. The Graph Check: When an administrator triggers an upgrade via the Web Console or the CLI (oc adm upgrade), the CVO contacts the OpenShift Update Service (OSUS).
2. Path Verification: The update service returns an architectural directed acyclic graph (DAG). OpenShift will block the upgrade if you attempt to skip a mandatory “Y-stream” minor release (e.g., trying to jump straight from 4.15 to 4.17 without passing through 4.16) or if a known critical bug exists for your hardware type.
3. The Payload Pull: Once approved, the CVO pulls down a single Release Payload Image from a container registry (like Quay.io). This payload is an OCI image containing every single Kubernetes manifest and container image required for the target version.

2. Phase 1: Control Plane Orchestration (The Brains First)

The CVO unpacks the payload and prioritizes the cluster’s management layer. The worker nodes and your applications are completely untouched during this phase.

Step 1: Core Operators & Storage

The CVO reads the payload dependency map. It applies updates to the Cluster etcd Operator and the Kubernetes API Server Operators first.

• Why? The data store and API layers must be capable of understanding the new Kubernetes object definitions before any other component updates.

Step 2: Sequential Core Reconciliation

The CVO updates the remaining secondary platform operators (such as Ingress, OVN-Network, Monitoring, and Authentication) one by one. It waits for each operator to report Available=True and Progressing=False before moving to the next.

3. Phase 2: Node & Operating System Upgrades (The Engine Room)

Once the control plane operators are stable, the CVO updates the Machine Config Operator (MCO) config spec, handing over the second half of the upgrade loop. The MCO applies updates to the physical or virtual host machines via MachineConfigPools (typically starting with the master pool, followed by the worker pool).

To prevent an application outage, the MCO processes the nodes in a strict rolling update sequence (one node at a time per pool):

Step 1: Evacuation (Cordon & Drain)

The MCO selects the first worker node (worker-0) and marks it as unschedulable (Cordoned). It then uses the Kubernetes eviction API to Drain the node, safely migrating all your running application pods to neighboring worker nodes that have remaining capacity.

Step 2: Atomic OS Layering (rpm-ostree)

The Machine-Config-Daemon running as a root pod on worker-0 pulls down the new Red Hat Enterprise Linux CoreOS (RHCOS) binary target image.

• Because RHCOS is immutable, it does not run traditional yum update commands. Instead, it uses rpm-ostree to write a brand-new, ready-to-boot operating system disk image layer directly alongside the old layer.

Step 3: Reboot & Verification

The daemon executes an automated host reboot. The machine boots up directly into the new operating system layer, which automatically contains the updated versions of the Linux kernel, the CRI-O container runtime, and the kubelet cluster binary.

Step 4: Re-entry & Loop Repeat

Once worker-0 boots up, passes internal network health checks, and registers itself back to the API server as Ready, the MCO uncordons it, allowing application pods to be scheduled back onto it. The MCO then repeats this exact loop on worker-1, then worker-2, continuing until the entire fleet is updated.

4. What Happens if Something Breaks? (Self-Healing Boundaries)

OpenShift builds strict circuit breakers into the upgrade lifecycle to prevent a misconfiguration or hardware failure from taking down your entire infrastructure:

• Operator Upstream Block: If an individual control plane operator (e.g., the Ingress operator) fails to stabilize within its designated upgrade window, the CVO halts the entire cluster upgrade instantly. It will not touch your worker nodes or applications while the control plane is in a degraded state.
• MaxUnavailable Safety Caps: Within your MachineConfigPools, the MCO honors a configuration parameter called maxUnavailable (default is 1). If worker-0 fails to come back online or hangs during a reboot due to an underlying cloud provider issue, the upgrade loop freezes. It will never drain worker-1 until worker-0 is completely healthy, isolating the blast radius to a single node.
• Operating System Rollbacks: If an RHCOS node reboots into the new kernel and experiences a catastrophic hardware compatibility crash, the underlying ostree bootloader detects the panic. It will automatically abort the boot sequence and drop back down to the previous, perfectly functioning OS deployment layer, allowing the node to rejoin the cluster under its old version so you can investigate logs safely.