Understanding CVO and MCO in OpenShift

techhadoop OCP

In Red Hat OpenShift, the Cluster Version Operator (CVO) and the Machine Config Operator (MCO) are the two most critical automation engines driving Day-2 operations. Together, they eliminate manual cluster maintenance by treating the entire platform—from the high-level Kubernetes APIs down to the bare-metal Linux kernel—as software that can be updated declaratively.

If you think of an OpenShift cluster as a cruise ship, the CVO is the Captain directing the route and orchestrating the crew, while the MCO is the Chief Engineer down in the engine room making sure the structural hardware and operating systems are tuned and running.

1. The Cluster Version Operator (CVO)

The CVO is responsible for the lifecycle of the OpenShift platform layer. It manages all the internal cluster operators (such as Network, Ingress, Monitoring, and Authentication) and orchestrates automated cluster upgrades.

How CVO Works Internally

Instead of downloading individual binaries, the CVO consumes a single artifact called a Release Payload Image (an OCI container image stored in a registry like Quay.io). This payload contains the exact container images and precise Kubernetes manifests for every single operator needed to run that specific version of OpenShift.

The Step-by-Step Upgrade Loop:
  1. The Target: You or a GitOps pipeline change the cluster’s desired version (e.g., using oc adm upgrade --to=4.16.5).
  2. The Graph: The CVO contacts the OpenShift Update Service to validate the upgrade path graph, ensuring the jump is safe and certified.
  3. The Unpacking: It pulls down the release payload image and reads the release-manifests/ directory.
  4. The Dependency Graph Routing: The CVO updates cluster operators in a highly orchestrated sequence. For example, it updates etcd and the kube-apiserver first. It waits until those components report a Healthy status before it tells secondary operators (like Monitoring or Logging) to update their underlying payloads.
  5. Continuous Verification: The CVO continuously tracks the status of all operators, ensuring the cluster achieves the exact desired state declared in the payload image.

2. The Machine Config Operator (MCO)

While the CVO handles the cluster applications and APIs, the MCO manages the host Operating System (RHCOS/FCOS) running on the physical or virtual worker and master nodes. It treats the operating system configuration itself as “just another Kubernetes object.”

If you need to update a file in /etc, inject an enterprise SSH key, modify a sysctl network parameter, or upgrade the Linux kernel version, you do not SSH into nodes. You write a MachineConfig YAML file, and the MCO executes it.

The Sub-Components of the MCO

The MCO orchestrates its node-level automation using three specialized background utilities:

  • Machine-Config-Controller: Sits on the control plane. It watches for changes to your MachineConfig manifests and compiles (“renders”) them into unified Ignition configuration files. It also coordinates with the node daemons to execute updates safely without dropping active application workloads.
  • Machine-Config-Server: An internal utility that listens on port 22623. When a brand-new bare-metal server boots up for the first time, its internal Ignition agent contacts this server to fetch its initial operating system blueprint.
  • Machine-Config-Daemon (MCD): This runs as a DaemonSet (one pod on every single node in your cluster). The MCD has root access to the host file system. It takes the instructions sent by the controller and writes them directly to the node’s disk.
The MCO Update Sequence (The Safety Loop)

When the CVO updates the cluster version, or when you manually submit a new MachineConfig, the MCO triggers an automated rolling update across your MachineConfigPools (Master pool or Worker pool):

  1. The machine-config-controller generates a new target profile.
  2. The machine-config-daemon on Node 1 picks up the change.
  3. The controller safely drains and cordons Node 1, migrating all application pods to neighboring nodes.
  4. The MCD writes the new OS configuration files to disk using rpm-ostree (atomic OS layering).
  5. If the change requires a kernel tweak or driver reload, the MCD reboots the physical node.
  6. Once Node 1 boots up healthy and passes verify checks, the MCO uncordons it and moves on to Node 2.

Architectural Summary: How They Interact During an Upgrade

When you run a standard cluster upgrade, the CVO and MCO work in lockstep to keep the platform online:

Plaintext

 ┌────────────────────────────────────────────────────────┐
 │ 1. Admin triggers upgrade via CVO (oc adm upgrade)     │
 └───────────────────────────┬────────────────────────────┘
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 2. CVO unpacks payload & updates Cluster Operators     │
 └───────────────────────────┬────────────────────────────┘
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 3. CVO triggers the Machine Config Operator (MCO)      │
 └───────────────────────────┬────────────────────────────┘
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 4. MCO commands Machine Config Daemons to update OS    │
 └───────────────────────────┬────────────────────────────┘
                             ▼
 ┌────────────────────────────────────────────────────────┐
 │ 5.Nodes are drained, updated via rpm-ostree, & rebooted│
 └────────────────────────────────────────────────────────┘

By separating responsibilities—with CVO owning the cluster software state and MCO owning the immutable node operating systems—OpenShift eliminates manual cluster maintenance, providing a highly scalable and resilient platform.

Leave a Reply