Understanding OpenShift MCS and MCP: Key Differences

techhadoop OCP

In OpenShift 4, there is a common point of initial confusion between two similarly named but entirely distinct architectural components within the Machine Config Operator (MCO) ecosystem:

The Machine Config Server (MCS) and the Machine Config Pool (MCP).

To clear this up right away, the MCS is a physical containerized service (the background API utility), while the MCP is a logical grouping mechanism (the scheduler policy). Both work together to manage the lifecycle of Red Hat Enterprise Linux CoreOS (RHCOS) or Fedora CoreOS (FCOS) host nodes.

1. The Machine Config Server (MCS)

The Machine Config Server is an internal, lightweight HTTPS daemon whose primary responsibility is to serve Ignition configuration files to newly booting or provisioning nodes.

How the MCS Operates:
  • The Network Port: The MCS listens inside the cluster on port 22623.
  • The Blueprint Provider: When you add a new bare-metal machine, VM, or AWS instance to scale out your cluster, the machine boots up running a minimalist Ignition agent in memory. This agent instantly reaches out to the OpenShift internal API endpoint (api-int.[cluster].com:22623) to fetch its initialization profile.
  • The Delivery: The MCS intercepts this request, reads the requested machine role (master or worker), parses the latest cluster cryptographic secrets, pull secrets, and system configs, and streams a customized JSON Ignition file back down to the machine. The node then reads these instructions to partition its hard drives, create SSH keys, configure network interfaces, and write the foundational systemd files required to boot into a fully functioning OpenShift node.

2. The Machine Config Pool (MCP)

A Machine Config Pool is a logical Kubernetes Custom Resource (CRD) that groups cluster nodes together based on their structural role. It maps a collection of target machines to a specific Rendered MachineConfig Blueprint.

By default, every OpenShift cluster spins up with two standard pools:

  1. master (Managing the control plane nodes)
  2. worker (Managing the standard compute nodes)

Plaintext

       ┌──────────────────────┐
       │  MachineConfigPool   │
       │       (worker)       │
       └──────────┬───────────┘
                  │
                  ▼ Matches via Selector Labels
   ┌──────────────────────────────┐
   │ Rendered MachineConfig Asset │ (Compiled by MCO)
   └──────────────┬───────────────┘
                  │
        ┌─────────┴─────────┐
        ▼                   ▼
 ┌──────────────┐    ┌──────────────┐
 │ Worker Node  │    │ Worker Node  │ (Monitored by individual
 │   (Host 1)   │    │   (Host 2)   │  Machine-Config-Daemons)
 └──────────────┘    └──────────────┘
Why would you create a Custom MCP?

As a Senior Platform Architect, you will regularly need to create custom Machine Config Pools to segment specific hardware roles within your data center.

For instance, if you have a group of nodes carrying physical GPUs for AI workloads or dedicated routing cards for edge telecommunications, you don’t want a system change intended only for the GPU nodes to cause rolling reboots across your entire generic worker fleet.

You solve this by creating an isolated custom pool:

YAML

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra-nodes # Creates a specialized pool for infrastructural loads (routers, logging)
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker, infra]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: "" # Targets physical nodes carrying this custom label

3. High-Impact Troubleshooting Scenarios (Interview Golden Nuggets)

Understanding how the MCS and MCP interact gives you a massive advantage when debugging cluster installations or Day-2 platform drift.

Scenario A: New Worker Nodes Are Stuck in NotReady or Fail to Boot
  • The Symptoms: You are scaling out your on-premises VMware or Bare-Metal cluster. The new servers power on, but they never register with the Kubernetes API, and logs show an internal server error (ignition: GET error).
  • The Root Cause: The new node cannot contact the Machine Config Server (MCS) on port 22623. This is almost always a network architecture or firewall blockade. The bootstrap or control plane nodes must have port 22623 completely unblocked on the load balancers and internal networks for api-int tracking to function.
Scenario B: The MCP Status Reports DEGRADED=True
  • The Symptoms: You applied a custom MachineConfig to update an enterprise NTP server path or write a config file to /etc/containers/registries.conf, but running oc get mcp shows the pool is Degraded.
  • The Root Cause: One of the Machine-Config-Daemons (MCD) running on a node within that pool failed to apply your change. This usually happens if you provide a malformed base64 string inside the file declaration block, or attempt to modify a system directory path that is locked down by the immutable ostree root file system layer. OpenShift automatically halts the rollout across the rest of the pool to protect your workloads from a cascading outage.

Leave a Reply