Understanding Machine Config Operator (MCO) in OpenShift

techhadoop OCP

MCO (Machine Config Operator) in OpenShift

What is MCO?

The Machine Config Operator (MCO) is an OpenShift operator that manages the operating system and runtime configuration of cluster nodes. It treats the underlying RHCOS (Red Hat CoreOS) nodes as immutable infrastructure — all node-level changes go through MCO, never manual SSH edits.

Core idea: Just as Kubernetes manages application workloads declaratively, MCO manages node OS configuration declaratively.

MCO Architecture

┌────────────────────────────────────────────────────────────┐
│                    MCO (Control Plane)                     │
│                                                            │
│  ┌─────────────────────────────────────────────────────┐   │
│  │            Machine Config Operator (MCO)            │   │
│  │         Watches MachineConfig & MachineConfigPool   │   │
│  └──────────────────────┬──────────────────────────────┘   │
│                         │                                  │
│  ┌──────────────────────▼──────────────────────────────┐   │
│  │         Machine Config Controller (MCC)             │   │
│  │   Renders final config, manages pool updates        │   │
│  └──────────────────────┬──────────────────────────────┘   │
│                         │                                  │
│  ┌──────────────────────▼──────────────────────────────┐   │
│  │         Machine Config Server (MCS)                 │   │
│  │   Serves Ignition configs to new nodes on boot      │   │
│  └──────────────────────┬──────────────────────────────┘   │
└─────────────────────────┼──────────────────────────────────┘
                          │ DaemonSet on every node
┌─────────────────────────▼───────────────────────────────────┐
│              Machine Config Daemon (MCD)                    │
│   Applies config changes on each node, triggers reboots     │
└─────────────────────────────────────────────────────────────┘
Component Responsibilities
ComponentRole
MCOTop-level operator — manages all sub-components
MCC (Controller)Renders MachineConfigs, manages rolling updates per pool
MCS (Server)HTTP server — serves Ignition config to bootstrapping nodes
MCD (Daemon)DaemonSet on every node — applies changes, validates, reboots

Core CRDs

1. MachineConfig (MC)

Defines the desired configuration for a node — files, systemd units, kernel arguments, etc.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-custom-config
  labels:
    machineconfiguration.openshift.io/role: worker  # targets worker pool
spec:
  config:
    ignition:
      version: 3.2.0
    # --- Drop files onto nodes ---
    storage:
      files:
        - path: /etc/chrony.conf
          mode: 0644
          overwrite: true
          contents:
            source: data:,server%20time.example.com%20iburst%0A  # URL-encoded
        - path: /etc/sysctl.d/99-custom.conf
          mode: 0644
          contents:
            source: data:,vm.max_map_count%3D262144%0A
    # --- Systemd units ---
    systemd:
      units:
        - name: kubelet.service
          enabled: true
        - name: my-custom-service.service
          enabled: true
          contents: |
            [Unit]
            Description=My Custom Service
            After=network.target
            [Service]
            ExecStart=/usr/local/bin/my-script.sh
            Restart=always
            [Install]
            WantedBy=multi-user.target
  # --- Kernel arguments ---
  kernelArguments:
    - hugepages=1024
    - intel_iommu=on
  # --- Kernel type ---
  kernelType: default           # default | realtime (for low-latency/NFV)
  # --- Extensions (RHCOS layering) ---
  extensions:
    - usbguard
    - kernel-devel
2. MachineConfigPool (MCP)

Defines a group of nodes and which MachineConfigs apply to them. MCO applies configs to all nodes in a pool via rolling update.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker
spec:
  # Which nodes belong to this pool
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values:
          - worker
          - custom-worker
  # Which nodes (by label) are in this pool
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""
  # Pause updates (maintenance window)
  paused: false
  # Max nodes updating simultaneously
  maxUnavailable: 1             # Or percentage: "33%"
Built-in Pools
PoolTargetsNotes
masterControl plane nodesChanges here are critical — be careful
workerAll worker nodesDefault pool for most changes
Custom poolSpecific node subsetFor GPU nodes, infra nodes, etc.
3. Rendered MachineConfig

MCO merges all MachineConfigs targeting a pool into a single rendered-<pool>-<hash> config. This is what actually gets applied to nodes.

# View rendered configs
oc get mc | grep rendered
# Example output:
rendered-master-abc123def456    3.2.0   45m
rendered-worker-xyz789ghi012    3.2.0   45m

How MCO Applies Changes — Rolling Update Flow

New/Updated MachineConfig applied
            │
            ▼
MCC renders new rendered-<pool>-<hash>
            │
            ▼
MCP detects rendered config changed
            │
            ▼
MCD on node: drain → apply → reboot → validate
            │
            ├─ One node at a time (maxUnavailable: 1)
            ├─ Waits for node Ready before next
            └─ Pool goes Updating → Updated
# Watch rolling update progress
oc get mcp
# NAME     CONFIG                   UPDATED   UPDATING   DEGRADED
# master   rendered-master-abc123   True      False      False
# worker   rendered-worker-xyz789   False     True       False  ← updating
# Watch node-by-node
oc get nodes -w

Common MCO Use Cases

1. Add a Custom File to Nodes
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-sysctl
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/sysctl.d/99-worker-performance.conf
          mode: 0644
          overwrite: true
          contents:
            source: data:text/plain;charset=utf-8;base64,dm0ubWF4X21hcF9jb3VudD0yNjIxNDQ=
2. Add SSH Keys to Nodes
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-ssh-key
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
    passwd:
      users:
        - name: core
          sshAuthorizedKeys:
            - ssh-rsa AAAAB3NzaC1yc2E... admin@example.com
3. Configure Container Runtime (CRI-O)
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-crio-config
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/crio/crio.conf.d/99-custom.conf
          mode: 0644
          contents:
            source: data:text/plain,%5Bcrio.runtime%5D%0Alog_level%20%3D%20%22debug%22
4. Real-Time Kernel for Low-Latency Workloads
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-realtime
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  kernelType: realtime
5. Custom MachineConfigPool (Infra Nodes)
# Step 1: Create the pool
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values: [worker, infra]
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: ""
---
# Step 2: Label the nodes
# oc label node infra-node-1 node-role.kubernetes.io/infra=
# Step 3: Create infra-specific MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-infra-config
  labels:
    machineconfiguration.openshift.io/role: infra
spec:
  config:
    ignition:
      version: 3.2.0

Pausing a MachineConfigPool

Use during maintenance windows to prevent automatic node reboots:

# Pause updates to worker pool
oc patch mcp worker --type merge -p '{"spec":{"paused":true}}'
# Apply your MachineConfigs (they queue up, nothing applies yet)
oc apply -f my-machineconfig.yaml
# Resume — all queued changes apply now
oc patch mcp worker --type merge -p '{"spec":{"paused":false}}'

Troubleshooting MCO

Check Pool & Node Status
# Overall pool health
oc get mcp
# Degraded details
oc describe mcp worker | grep -A10 Degraded
# Node MCO status
oc get node -o wide
oc describe node <node> | grep -i machine
Check MCD Logs on a Node
# Find MCD pod on specific node
oc get pod -n openshift-machine-config-operator \
  -l k8s-app=machine-config-daemon \
  -o wide | grep <node-name>
# Get logs
oc logs -n openshift-machine-config-operator \
  machine-config-daemon-<id> -c machine-config-daemon
# Follow live
oc logs -n openshift-machine-config-operator \
  machine-config-daemon-<id> -c machine-config-daemon -f
Common Errors & Fixes
ErrorCauseFix
Pool Degraded: TrueConfig failed to apply on a nodeCheck MCD logs on degraded node
Node stuck in NotReady after updateOS/config error post-rebootSSH to node, check journalctl -u machine-config-daemon
rendered config not foundMC label mismatchVerify machineconfiguration.openshift.io/role label
Pool updating indefinitelymaxUnavailable too low or node stuck drainingCheck pod disruption budgets, force-drain if safe
Wrong config appliedMultiple MCs with same file pathCheck MC priority via name prefix (00–99)
Pool paused, changes not applyingIntentional or forgotten pauseoc patch mcp worker --type merge -p '{"spec":{"paused":false}}'
MCD Node Annotation — Useful for Debugging
# Check what config a node currently has vs desired
oc get node <node> -o yaml | grep -A5 machineconfiguration
# Key annotations:
# machineconfiguration.openshift.io/currentConfig  ← what's applied
# machineconfiguration.openshift.io/desiredConfig  ← what should be applied
# machineconfiguration.openshift.io/state          ← Done / Degraded / Working

MCO Interview Quick-Fire

  • Can you SSH into RHCOS nodes and manually edit files? Technically yes, but changes will be overwritten by MCO on next update. All changes must go through MachineConfig.
  • What is the naming convention for MachineConfigs? Prefix 00–99 controls merge order; lower numbers apply first. e.g. 00-master, 99-worker-custom.
  • What happens if two MCs write to the same file? The one with the higher numeric prefix (applied last) wins.
  • What triggers a node reboot? Any change to files, systemd units, kernel args, or kernel type managed by MCO.
  • Can you apply MCO changes without a reboot? Only for changes that don’t require it (rare). Most OS-level changes require reboot.
  • What is RHCOS and why does MCO require it? Red Hat CoreOS — immutable, container-optimized OS. MCO is the only supported way to configure it.
  • Where are MCO components running? All in openshift-machine-config-operator namespace.

Leave a Reply