OpenShift Worker Node Components

OpenShift worker nodes are the machines that run application workloads.

The control plane decides what should run and where. Worker nodes perform the actual execution.

Control Plane
     │
     ├── API Server
     ├── Scheduler
     ├── Controllers
     └── etcd
            │
            ▼
        Worker Nodes
     ┌────────┼────────┐
     ▼        ▼        ▼
  Worker-1 Worker-2 Worker-3
     │        │        │
   Pods     Pods      Pods

A worker node can be:

• A physical server
• A virtual machine
• A cloud instance
• A bare-metal host

In OpenShift, worker nodes normally run RHCOS, although some supported configurations can use RHEL workers.

Main Components on a Worker Node

A typical OpenShift worker contains:

RHCOS
├── kubelet
├── CRI-O
├── crun or runc
├── OVN-Kubernetes node components
├── Open vSwitch
├── Machine Config Daemon
├── CoreDNS pod
├── Node Exporter
├── CSI node plugins
├── Logging collector
└── Application Pods

The most important components are:

1. kubelet
2. CRI-O
3. OCI runtime
4. OVN-Kubernetes
5. Machine Config Daemon
6. Node-level monitoring
7. Logging collector
8. CSI node plugins

1. RHCOS

Red Hat Enterprise Linux CoreOS is the operating system used by OpenShift nodes.

It provides:

• Linux kernel
• systemd
• SELinux
• cgroups
• namespaces
• filesystems
• network stack
• container storage
• system services

Applications
     │
     ▼
Containers
     │
     ▼
CRI-O
     │
     ▼
Linux kernel
     │
     ▼
RHCOS

RHCOS is largely immutable and should be managed through OpenShift, especially through the Machine Config Operator.

Administrators should avoid manually installing packages or changing Operator-managed files.

2. kubelet

The kubelet is the main Kubernetes agent running on every node.

Its responsibilities include:

• Registering the node with the API server
• Watching for Pods assigned to the node
• Asking CRI-O to start and stop containers
• Mounting volumes
• Running health probes
• Reporting node and Pod status
• Managing static Pods where applicable
• Enforcing Pod resource configuration

API Server
    │
    ▼
kubelet
    │
    ├── Start Pod
    ├── Stop Pod
    ├── Run probes
    ├── Mount volumes
    └── Report status

Check kubelet status:

oc debug node/<worker-node>
chroot /host
systemctl status kubelet

View logs:

journalctl -u kubelet

3. CRI-O

CRI-O is OpenShift’s container runtime.

The kubelet communicates with CRI-O through the Kubernetes Container Runtime Interface.

kubelet
   │
   ▼
CRI-O
   │
   ▼
crun / runc
   │
   ▼
Linux kernel

CRI-O is responsible for:

• Pulling container images
• Creating Pod sandboxes
• Starting containers
• Stopping containers
• Managing image storage
• Applying cgroups
• Applying SELinux labels
• Preparing mounts
• Integrating with networking

Useful commands:

systemctl status crio
journalctl -u crio
crictl ps
crictl pods
crictl images

4. crun or runc

CRI-O does not create containers directly. It uses an OCI runtime, commonly crun or runc.

The OCI runtime creates the container process using Linux kernel technologies such as:

• PID namespaces
• Network namespaces
• Mount namespaces
• cgroups
• seccomp
• Linux capabilities
• SELinux

CRI-O request
      │
      ▼
crun or runc
      │
      ▼
Container process

This component is very low-level and normally not managed directly by administrators.

5. OVN-Kubernetes Node Components

Each worker participates in the OpenShift software-defined network.

Typical components include:

• ovnkube-node
• Open vSwitch
• OVN controller
• CNI integration
• GENEVE tunnel interfaces

Pod
 │
 ▼
Virtual Ethernet Interface
 │
 ▼
Open vSwitch
 │
 ▼
OVN logical network
 │
 ▼
Another Pod or Service

The node networking layer handles:

• Pod IP assignment
• Pod-to-Pod connectivity
• Service traffic
• NetworkPolicies
• East-west routing
• Egress traffic
• GENEVE encapsulation
• Load balancing for Services

Check OVN components:

oc get pods -n openshift-ovn-kubernetes -o wide

A networking failure on one worker can cause Pods on that node to lose connectivity even though the Pods remain running.

6. Machine Config Daemon

The Machine Config Daemon, or MCD, runs as a DaemonSet on each node.

It is part of the Machine Config Operator.

Its responsibilities include:

• Applying RHCOS configuration
• Updating operating-system files
• Applying systemd units
• Updating kubelet and CRI-O configuration
• Performing node OS updates
• Detecting configuration drift
• Draining and rebooting the node when required

MachineConfig
      │
      ▼
Machine Config Operator
      │
      ▼
Machine Config Daemon
      │
      ▼
Worker node

During an update:

Cordon
  │
  ▼
Drain workloads
  │
  ▼
Apply configuration
  │
  ▼
Reboot if required
  │
  ▼
Node Ready

Check it with:

oc get mcp
oc get pods -n openshift-machine-config-operator -o wide

7. Application Pods

The main purpose of worker nodes is to run application Pods.

Examples:

Worker Node
├── payments-api Pod
├── frontend Pod
├── database Pod
├── logging agent Pod
└── monitoring agent Pod

A Pod can contain one or more containers.

The worker provides:

• CPU
• Memory
• Network
• Storage access
• Container runtime
• Security isolation

The scheduler decides the node placement, but the worker executes the workload.

8. Node Exporter

OpenShift deploys a managed Node Exporter DaemonSet.

It collects host-level metrics such as:

• CPU
• Memory
• Filesystem
• Disk I/O
• Network traffic
• Load average
• Kernel metrics

Worker Node
     │
     ▼
Node Exporter
     │
     ▼
Prometheus

Check it:

oc get daemonset node-exporter -n openshift-monitoring

You generally should not install another standalone Node Exporter on OpenShift nodes.

9. kubelet and cAdvisor Metrics

The kubelet also exposes container-related metrics.

These include:

• Container CPU
• Container memory
• Filesystem usage
• Pod resource usage
• Container restarts

Container
    │
    ▼
kubelet / cAdvisor metrics
    │
    ▼
Prometheus

Difference:

10. Logging Collector

When OpenShift Logging is installed, a collector such as Vector runs as a DaemonSet.

Application stdout/stderr
          │
          ▼
Node log files
          │
          ▼
Vector collector
          │
          ├── Loki
          ├── Splunk
          └── Elasticsearch

It can collect:

• Application logs
• Infrastructure logs
• Audit logs

Check:

oc get pods -n openshift-logging -o wide

11. CSI Node Plugins

Storage drivers commonly deploy node-level CSI components as DaemonSets.

They handle:

• Mounting volumes
• Unmounting volumes
• Attaching storage where applicable
• Formatting volumes
• Exposing block devices to Pods

Pod requests PVC
       │
       ▼
CSI controller
       │
       ▼
CSI node plugin
       │
       ▼
Volume mounted on worker

Examples include:

• AWS EBS CSI
• Azure Disk CSI
• VMware vSphere CSI
• Ceph CSI
• Fibre Channel or SAN CSI integrations

12. CoreDNS Pod

OpenShift normally deploys CoreDNS through a DaemonSet, so worker nodes may run a DNS Pod.

CoreDNS handles:

• Service discovery
• Internal DNS
• External DNS forwarding

Application Pod
      │
      ▼
DNS Service IP
      │
      ▼
CoreDNS Pod

Check:

oc get pods -n openshift-dns -o wide

13. Multus

OpenShift uses Multus when Pods or VMs need multiple network interfaces.

Pod
 ├── eth0 → default OVN network
 └── net1 → VLAN or SR-IOV network

Multus is commonly used for:

• Telco workloads
• OpenShift Virtualization
• Storage networks
• High-performance networking
• SR-IOV

Node-level Multus components run on workers.

14. SR-IOV Components

On specialized worker nodes, the SR-IOV Operator may deploy node agents.

These provide:

• Direct virtual functions
• High-throughput networking
• Low latency
• Hardware offload

Pod
 │
 ▼
Virtual Function
 │
 ▼
Physical NIC

These nodes are often labeled and tainted to isolate specialized workloads.

15. Device Plugins

Device plugins expose hardware resources to Kubernetes.

Examples:

• NVIDIA GPUs
• Intel accelerators
• FPGAs
• Network devices

Example GPU resource:

resources:
  limits:
    nvidia.com/gpu: 1

The scheduler sees the available resource, and the node device plugin makes it accessible to the Pod.

Worker Node Startup Flow

A worker node boot sequence looks like this:

Power on
   │
   ▼
RHCOS boots
   │
   ▼
systemd starts
   │
   ├── NetworkManager
   ├── CRI-O
   ├── kubelet
   └── Machine Config Daemon
            │
            ▼
kubelet connects to API server
            │
            ▼
Node registers
            │
            ▼
OVN networking initializes
            │
            ▼
DaemonSet Pods start
            │
            ▼
Node becomes Ready
            │
            ▼
Scheduler assigns application Pods

Pod Startup on a Worker

When a Pod is assigned to a worker:

Scheduler selects worker-2
           │
           ▼
API server updates Pod nodeName
           │
           ▼
kubelet on worker-2 detects Pod
           │
           ▼
CRI-O pulls image
           │
           ▼
Pod sandbox created
           │
           ▼
OVN configures network
           │
           ▼
CSI mounts volumes
           │
           ▼
Container starts
           │
           ▼
Readiness probe succeeds
           │
           ▼
Pod receives traffic

Worker Node Status

Check workers:

oc get nodes

Example:

NAME       STATUS   ROLES
worker-0   Ready    worker
worker-1   Ready    worker
worker-2   Ready    worker

Show worker nodes only:

oc get nodes -l node-role.kubernetes.io/worker

Detailed node information:

oc describe node worker-0

Important sections include:

• Conditions
• Capacity
• Allocatable
• Taints
• Labels
• Allocated resources
• Events

Node Conditions

Common node conditions include:

Healthy example:

Ready=True
MemoryPressure=False
DiskPressure=False
PIDPressure=False
NetworkUnavailable=False

Capacity vs Allocatable

A node may have:

Capacity:
CPU: 32
Memory: 128 GiB

But allocatable resources are lower:

Allocatable:
CPU: 30
Memory: 118 GiB

The difference is reserved for:

• Operating system
• kubelet
• CRI-O
• OpenShift platform agents
• Eviction thresholds

The scheduler uses allocatable, not total physical capacity.

Worker Labels

Labels help place workloads.

Examples:

oc label node worker-1 workload=payments
oc label node worker-2 node-role.kubernetes.io/infra=""
oc label node gpu-1 accelerator=nvidia

Pods can target those labels:

spec:
  nodeSelector:
    workload: payments

Worker Taints

Taints prevent general workloads from being scheduled.

Example:

oc adm taint node gpu-1 dedicated=gpu:NoSchedule

A Pod must have a matching toleration.

tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

This is used for:

• GPU nodes
• Infrastructure nodes
• Storage nodes
• Special security zones
• High-performance workloads

Worker Pools

MachineConfigPools group nodes with the same operating-system configuration.

Typical pools:

master
worker
infra
gpu
storage

Check:

oc get mcp

Example:

NAME     UPDATED   UPDATING   DEGRADED
worker   True      False      False
infra    True      False      False

Custom worker pools can receive separate:

• Kubelet settings
• Kernel arguments
• CRI-O settings
• Systemd units
• OS files

Worker Node Failure

If a worker fails:

Worker node unavailable
        │
        ▼
Node becomes NotReady
        │
        ▼
Controller detects failure
        │
        ▼
Pods are recreated elsewhere
        │
        ▼
Scheduler selects healthy workers

This depends on:

• Multiple replicas
• Storage accessibility
• Pod disruption controls
• Node failure detection
• Sufficient spare capacity

A single-replica application can experience downtime.

Common Worker Problems

Node NotReady

Check:

oc get nodes
oc describe node <worker>

Then:

oc debug node/<worker>
chroot /host
systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio

Possible causes:

• kubelet failure
• CRI-O failure
• Network failure
• Certificate problem
• Disk pressure
• Memory pressure
• Node OS issue
• API server connectivity issue

DiskPressure

Check:

df -h
df -i
du -sh /var/lib/containers/*
du -sh /var/log/*

Possible causes:

• Large container logs
• Image accumulation
• Container storage exhaustion
• Inode exhaustion
• Failed garbage collection

MemoryPressure

Check:

free -h
top
oc adm top node

Possible causes:

• Oversized workloads
• Incorrect requests and limits
• Memory leak
• Too many Pods
• Platform agents consuming resources

Pods stuck in ContainerCreating

Investigate:

oc describe pod <pod> -n <namespace>

Possible causes:

• CRI-O failure
• Image pull problem
• CNI/OVN failure
• CSI mount failure
• SCC or permission issue
• Node disk pressure

ImagePullBackOff

Check:

• Image name
• Registry access
• ImagePullSecret
• DNS
• Proxy configuration
• Registry certificate trust
• CRI-O logs

journalctl -u crio
crictl images

Useful Worker Troubleshooting Commands

oc get nodes
oc describe node <worker>
oc adm top nodes
oc get pods -A -o wide --field-selector spec.nodeName=<worker>

Node-level access:

oc debug node/<worker>
chroot /host

Inside the host:

systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio
df -h
df -i
free -h
iostat -x 1 10
sar -u 1 10
pidstat -d 1 10

Runtime inspection:

crictl ps
crictl pods
crictl images
crictl info

Control Plane vs Worker Components

Control plane says:
"Run this Pod on worker-2"
Worker-2 says:
"I will pull the image, configure networking, mount storage, and start it"

Interview Answer

An OpenShift worker node is responsible for executing application workloads. It normally runs RHCOS, the kubelet, CRI-O, an OCI runtime such as crun, OVN-Kubernetes networking components, the Machine Config Daemon, and node-level DaemonSets such as Node Exporter, CoreDNS, logging collectors, and CSI plugins.

The kubelet registers the worker with the API server and watches for Pods assigned by the scheduler. It asks CRI-O to create the Pod sandbox and start containers, OVN configures networking, and CSI drivers mount persistent storage. The Machine Config Daemon keeps the RHCOS configuration synchronized with the desired MachineConfigPool state.

For troubleshooting, I start with oc get nodes and oc describe node, inspect node conditions such as Ready, MemoryPressure, and DiskPressure, then use oc debug node and chroot /host to check kubelet, CRI-O, storage, memory, network, and system logs.