This is a Senior OpenShift Architect interview question. The interviewer wants to determine whether you understand how OpenShift upgrades are orchestrated safely, rather than just knowing the oc adm upgrade command.

A strong answer should explain the roles of the Cluster Version Operator (CVO), Operators, Machine Config Operator (MCO), Machine API, etcd quorum, and workload scheduling.

High-Level Upgrade Flow

Administrator
      │
      ▼
oc adm upgrade
      │
      ▼
Cluster Version Operator (CVO)
      │
      ▼
Downloads Release Image
      │
      ▼
Verifies Cluster Health
      │
      ▼
Upgrades Cluster Operators
      │
      ▼
Machine Config Operator updates OS
      │
      ▼
Control Plane
(one node at a time)
      │
      ▼
Workers
(one node or batch at a time)
      │
      ▼
Cluster Validation

The key principle is that OpenShift never upgrades the entire cluster simultaneously.

Step 1 – Administrator Starts the Upgrade

Example:

oc adm upgrade --to=4.19.x

The request updates the desired version in the ClusterVersion resource.

Example:

oc get clusterversion

Step 2 – CVO Detects the Version Change

The CVO watches the ClusterVersion object.

When it detects a new desired version, it:

• Downloads the release image
• Verifies compatibility
• Builds the list of manifests to apply

A release image contains:

• Kubernetes components
• Operator manifests
• CRDs
• Platform YAML
• Operand images

Think of the release image as the entire platform packaged as a versioned artifact.

Step 3 – Pre-Upgrade Validation

Before making changes, the CVO checks for blocking conditions such as:

• Degraded ClusterOperators
• Unavailable ClusterOperators
• Failing MachineConfigPools
• Unsupported Operators
• API deprecations
• Cluster health issues
• Insufficient disk space

If a critical issue exists, the upgrade pauses until it’s resolved.

Step 4 – Cluster Operators Upgrade

The CVO updates platform Operators first.

Examples include:

• Authentication Operator
• DNS Operator
• Image Registry Operator
• Monitoring Operator
• Ingress Operator
• Network Operator

Each Operator upgrades itself using the Kubernetes reconciliation loop.

The CVO waits for each Operator to report:

Available = True
Progressing = False
Degraded = False

before continuing.

Step 5 – Machine Config Operator (MCO)

The CVO doesn’t update node operating systems directly.

Instead, it updates the Machine Config Operator.

The MCO creates new MachineConfigs containing:

• New RHCOS version
• kubelet changes
• CRI-O updates
• kernel changes
• system configuration

Step 6 – Control Plane Upgrade

This is the most critical phase.

Example:

Master 1
Master 2
Master 3

The MCO upgrades only one control plane node at a time.

Example:

Before
Master1
Master2
Master3
↓
Drain Master1
↓
Apply MachineConfig
↓
Reboot
↓
Wait until Ready
↓
Upgrade Master2
↓
Upgrade Master3

Why?

Because etcd requires quorum.

With three members:

Node1
Node2
Node3
Majority = 2

Only one control plane node can be unavailable without losing quorum.

Step 7 – Node Drain

Before rebooting a node, the MCO drains it.

Drain means:

• Mark node unschedulable (cordon)
• Evict application pods (respecting Pod Disruption Budgets where possible)
• Wait for workloads to relocate

This minimizes disruption to running applications.

Step 8 – Node Reboot

The node:

• Installs the new RHCOS image
• Updates kubelet
• Updates CRI-O
• Applies kernel changes
• Reboots
• Rejoins the cluster

The MCO waits until the node reports:

Ready

before proceeding.

Step 9 – Worker Upgrade

Workers are upgraded after the control plane is healthy.

Typical sequence:

Worker1
↓
Drain
↓
Reboot
↓
Ready
↓
Worker2
↓
Worker3

For larger clusters, the MachineConfigPool’s maxUnavailable setting controls how many workers can be updated simultaneously.

Example:

spec:
  maxUnavailable: 2

This allows two workers to be upgraded in parallel while maintaining capacity.

Step 10 – Workload Rescheduling

When a worker is drained:

Application Pod
↓
Evicted
↓
Scheduler
↓
Another Worker
↓
Running

Because Kubernetes replicas are distributed across multiple nodes, services can continue to serve traffic if enough healthy replicas remain.

Step 11 – OVN Networking

Networking remains available because:

• Other OVN nodes continue forwarding traffic.
• Existing tunnels remain operational.
• New tunnels are established when upgraded nodes return.

The network is not restarted cluster-wide.

Step 12 – Ingress

The Ingress Operator ensures router availability.

Example:

Router1
Router2
Router3

If Router1 restarts:

Traffic automatically shifts to Router2 and Router3.

Clients typically see no interruption.

Step 13 – Monitoring

During the upgrade:

• Prometheus continues collecting metrics.
• Alertmanager remains available.
• ClusterOperators report status.
• CVO tracks upgrade progress.

Useful commands:

oc get clusterversion
oc get co
oc get mcp
oc get nodes

Step 14 – Completion

The upgrade finishes only when:

ClusterVersion
Available=True
Progressing=False
Failing=False

and all ClusterOperators are healthy.

Why It’s Called “Zero Downtime”

Strictly speaking, OpenShift provides near-zero downtime, not an absolute guarantee.

Applications remain available because:

• Control plane nodes are upgraded one at a time.
• Workers are drained before reboot.
• Pods are rescheduled to healthy nodes.
• Services automatically update endpoints.
• Ingress routers continue serving traffic.
• etcd quorum is maintained.
• Operators wait for each component to become healthy before advancing.

However, downtime can still occur if the application itself is not designed for high availability.

Common Causes of Downtime During an Upgrade

• Only one application replica is running.
• Pod Disruption Budgets are too restrictive or misconfigured.
• Readiness/liveness probes are incorrect.
• Applications maintain in-memory session state without replication.
• Persistent storage cannot fail over.
• Long startup times delay replacement pods.
• Custom Operators are incompatible with the target OpenShift version.

Interview Answer (2-Minute Version)

“The Cluster Version Operator orchestrates OpenShift upgrades by managing the desired cluster version and coordinating all platform Operators. It first validates cluster health, then updates Cluster Operators, and hands node updates to the Machine Config Operator. The MCO upgrades control plane nodes one at a time to preserve etcd quorum, draining and rebooting each node before moving to the next. After the control plane is healthy, worker nodes are upgraded in a controlled manner, respecting maxUnavailable settings and Pod Disruption Budgets. During node drains, Kubernetes reschedules workloads to healthy nodes, while Services and Ingress automatically redirect traffic to available pods. The CVO waits for every Operator and MachineConfigPool to report a healthy state before proceeding, which provides near-zero downtime for properly designed, highly available applications.”