OpenShift Worker Node Components

OpenShift worker nodes are the machines that run application workloads.

The control plane decides what should run and where. Worker nodes perform the actual execution.

Control Plane
     │
     ├── API Server
     ├── Scheduler
     ├── Controllers
     └── etcd
            │
            ▼
        Worker Nodes
     ┌────────┼────────┐
     ▼        ▼        ▼
  Worker-1 Worker-2 Worker-3
     │        │        │
   Pods     Pods      Pods

A worker node can be:

• A physical server
• A virtual machine
• A cloud instance
• A bare-metal host

In OpenShift, worker nodes normally run RHCOS, although some supported configurations can use RHEL workers.

Main Components on a Worker Node

A typical OpenShift worker contains:

RHCOS
├── kubelet
├── CRI-O
├── crun or runc
├── OVN-Kubernetes node components
├── Open vSwitch
├── Machine Config Daemon
├── CoreDNS pod
├── Node Exporter
├── CSI node plugins
├── Logging collector
└── Application Pods

The most important components are:

1. kubelet
2. CRI-O
3. OCI runtime
4. OVN-Kubernetes
5. Machine Config Daemon
6. Node-level monitoring
7. Logging collector
8. CSI node plugins

1. RHCOS

Red Hat Enterprise Linux CoreOS is the operating system used by OpenShift nodes.

It provides:

• Linux kernel
• systemd
• SELinux
• cgroups
• namespaces
• filesystems
• network stack
• container storage
• system services

Applications
     │
     ▼
Containers
     │
     ▼
CRI-O
     │
     ▼
Linux kernel
     │
     ▼
RHCOS

RHCOS is largely immutable and should be managed through OpenShift, especially through the Machine Config Operator.

Administrators should avoid manually installing packages or changing Operator-managed files.

2. kubelet

The kubelet is the main Kubernetes agent running on every node.

Its responsibilities include:

• Registering the node with the API server
• Watching for Pods assigned to the node
• Asking CRI-O to start and stop containers
• Mounting volumes
• Running health probes
• Reporting node and Pod status
• Managing static Pods where applicable
• Enforcing Pod resource configuration

API Server
    │
    ▼
kubelet
    │
    ├── Start Pod
    ├── Stop Pod
    ├── Run probes
    ├── Mount volumes
    └── Report status

Check kubelet status:

oc debug node/<worker-node>
chroot /host
systemctl status kubelet

View logs:

journalctl -u kubelet

3. CRI-O

CRI-O is OpenShift’s container runtime.

The kubelet communicates with CRI-O through the Kubernetes Container Runtime Interface.

kubelet
   │
   ▼
CRI-O
   │
   ▼
crun / runc
   │
   ▼
Linux kernel

CRI-O is responsible for:

• Pulling container images
• Creating Pod sandboxes
• Starting containers
• Stopping containers
• Managing image storage
• Applying cgroups
• Applying SELinux labels
• Preparing mounts
• Integrating with networking

Useful commands:

systemctl status crio
journalctl -u crio
crictl ps
crictl pods
crictl images

4. crun or runc

CRI-O does not create containers directly. It uses an OCI runtime, commonly crun or runc.

The OCI runtime creates the container process using Linux kernel technologies such as:

• PID namespaces
• Network namespaces
• Mount namespaces
• cgroups
• seccomp
• Linux capabilities
• SELinux

CRI-O request
      │
      ▼
crun or runc
      │
      ▼
Container process

This component is very low-level and normally not managed directly by administrators.

5. OVN-Kubernetes Node Components

Each worker participates in the OpenShift software-defined network.

Typical components include:

• ovnkube-node
• Open vSwitch
• OVN controller
• CNI integration
• GENEVE tunnel interfaces

Pod
 │
 ▼
Virtual Ethernet Interface
 │
 ▼
Open vSwitch
 │
 ▼
OVN logical network
 │
 ▼
Another Pod or Service

The node networking layer handles:

• Pod IP assignment
• Pod-to-Pod connectivity
• Service traffic
• NetworkPolicies
• East-west routing
• Egress traffic
• GENEVE encapsulation
• Load balancing for Services

Check OVN components:

oc get pods -n openshift-ovn-kubernetes -o wide

A networking failure on one worker can cause Pods on that node to lose connectivity even though the Pods remain running.

6. Machine Config Daemon

The Machine Config Daemon, or MCD, runs as a DaemonSet on each node.

It is part of the Machine Config Operator.

Its responsibilities include:

• Applying RHCOS configuration
• Updating operating-system files
• Applying systemd units
• Updating kubelet and CRI-O configuration
• Performing node OS updates
• Detecting configuration drift
• Draining and rebooting the node when required

MachineConfig
      │
      ▼
Machine Config Operator
      │
      ▼
Machine Config Daemon
      │
      ▼
Worker node

During an update:

Cordon
  │
  ▼
Drain workloads
  │
  ▼
Apply configuration
  │
  ▼
Reboot if required
  │
  ▼
Node Ready

Check it with:

oc get mcp
oc get pods -n openshift-machine-config-operator -o wide

7. Application Pods

The main purpose of worker nodes is to run application Pods.

Examples:

Worker Node
├── payments-api Pod
├── frontend Pod
├── database Pod
├── logging agent Pod
└── monitoring agent Pod

A Pod can contain one or more containers.

The worker provides:

• CPU
• Memory
• Network
• Storage access
• Container runtime
• Security isolation

The scheduler decides the node placement, but the worker executes the workload.

8. Node Exporter

OpenShift deploys a managed Node Exporter DaemonSet.

It collects host-level metrics such as:

• CPU
• Memory
• Filesystem
• Disk I/O
• Network traffic
• Load average
• Kernel metrics

Worker Node
     │
     ▼
Node Exporter
     │
     ▼
Prometheus

Check it:

oc get daemonset node-exporter -n openshift-monitoring

You generally should not install another standalone Node Exporter on OpenShift nodes.

9. kubelet and cAdvisor Metrics

The kubelet also exposes container-related metrics.

These include:

• Container CPU
• Container memory
• Filesystem usage
• Pod resource usage
• Container restarts

Container
    │
    ▼
kubelet / cAdvisor metrics
    │
    ▼
Prometheus

Difference:

10. Logging Collector

When OpenShift Logging is installed, a collector such as Vector runs as a DaemonSet.

Application stdout/stderr
          │
          ▼
Node log files
          │
          ▼
Vector collector
          │
          ├── Loki
          ├── Splunk
          └── Elasticsearch

It can collect:

• Application logs
• Infrastructure logs
• Audit logs

Check:

oc get pods -n openshift-logging -o wide

11. CSI Node Plugins

Storage drivers commonly deploy node-level CSI components as DaemonSets.

They handle:

• Mounting volumes
• Unmounting volumes
• Attaching storage where applicable
• Formatting volumes
• Exposing block devices to Pods

Pod requests PVC
       │
       ▼
CSI controller
       │
       ▼
CSI node plugin
       │
       ▼
Volume mounted on worker

Examples include:

• AWS EBS CSI
• Azure Disk CSI
• VMware vSphere CSI
• Ceph CSI
• Fibre Channel or SAN CSI integrations

12. CoreDNS Pod

OpenShift normally deploys CoreDNS through a DaemonSet, so worker nodes may run a DNS Pod.

CoreDNS handles:

• Service discovery
• Internal DNS
• External DNS forwarding

Application Pod
      │
      ▼
DNS Service IP
      │
      ▼
CoreDNS Pod

Check:

oc get pods -n openshift-dns -o wide

13. Multus

OpenShift uses Multus when Pods or VMs need multiple network interfaces.

Pod
 ├── eth0 → default OVN network
 └── net1 → VLAN or SR-IOV network

Multus is commonly used for:

• Telco workloads
• OpenShift Virtualization
• Storage networks
• High-performance networking
• SR-IOV

Node-level Multus components run on workers.

14. SR-IOV Components

On specialized worker nodes, the SR-IOV Operator may deploy node agents.

These provide:

• Direct virtual functions
• High-throughput networking
• Low latency
• Hardware offload

Pod
 │
 ▼
Virtual Function
 │
 ▼
Physical NIC

These nodes are often labeled and tainted to isolate specialized workloads.

15. Device Plugins

Device plugins expose hardware resources to Kubernetes.

Examples:

• NVIDIA GPUs
• Intel accelerators
• FPGAs
• Network devices

Example GPU resource:

resources:
  limits:
    nvidia.com/gpu: 1

The scheduler sees the available resource, and the node device plugin makes it accessible to the Pod.

Worker Node Startup Flow

A worker node boot sequence looks like this:

Power on
   │
   ▼
RHCOS boots
   │
   ▼
systemd starts
   │
   ├── NetworkManager
   ├── CRI-O
   ├── kubelet
   └── Machine Config Daemon
            │
            ▼
kubelet connects to API server
            │
            ▼
Node registers
            │
            ▼
OVN networking initializes
            │
            ▼
DaemonSet Pods start
            │
            ▼
Node becomes Ready
            │
            ▼
Scheduler assigns application Pods

Pod Startup on a Worker

When a Pod is assigned to a worker:

Scheduler selects worker-2
           │
           ▼
API server updates Pod nodeName
           │
           ▼
kubelet on worker-2 detects Pod
           │
           ▼
CRI-O pulls image
           │
           ▼
Pod sandbox created
           │
           ▼
OVN configures network
           │
           ▼
CSI mounts volumes
           │
           ▼
Container starts
           │
           ▼
Readiness probe succeeds
           │
           ▼
Pod receives traffic

Worker Node Status

Check workers:

oc get nodes

Example:

NAME       STATUS   ROLES
worker-0   Ready    worker
worker-1   Ready    worker
worker-2   Ready    worker

Show worker nodes only:

oc get nodes -l node-role.kubernetes.io/worker

Detailed node information:

oc describe node worker-0

Important sections include:

• Conditions
• Capacity
• Allocatable
• Taints
• Labels
• Allocated resources
• Events

Node Conditions

Common node conditions include:

Healthy example:

Ready=True
MemoryPressure=False
DiskPressure=False
PIDPressure=False
NetworkUnavailable=False

Capacity vs Allocatable

A node may have:

Capacity:
CPU: 32
Memory: 128 GiB

But allocatable resources are lower:

Allocatable:
CPU: 30
Memory: 118 GiB

The difference is reserved for:

• Operating system
• kubelet
• CRI-O
• OpenShift platform agents
• Eviction thresholds

The scheduler uses allocatable, not total physical capacity.

Worker Labels

Labels help place workloads.

Examples:

oc label node worker-1 workload=payments
oc label node worker-2 node-role.kubernetes.io/infra=""
oc label node gpu-1 accelerator=nvidia

Pods can target those labels:

spec:
  nodeSelector:
    workload: payments

Worker Taints

Taints prevent general workloads from being scheduled.

Example:

oc adm taint node gpu-1 dedicated=gpu:NoSchedule

A Pod must have a matching toleration.

tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

This is used for:

• GPU nodes
• Infrastructure nodes
• Storage nodes
• Special security zones
• High-performance workloads

Worker Pools

MachineConfigPools group nodes with the same operating-system configuration.

Typical pools:

master
worker
infra
gpu
storage

Check:

oc get mcp

Example:

NAME     UPDATED   UPDATING   DEGRADED
worker   True      False      False
infra    True      False      False

Custom worker pools can receive separate:

• Kubelet settings
• Kernel arguments
• CRI-O settings
• Systemd units
• OS files

Worker Node Failure

If a worker fails:

Worker node unavailable
        │
        ▼
Node becomes NotReady
        │
        ▼
Controller detects failure
        │
        ▼
Pods are recreated elsewhere
        │
        ▼
Scheduler selects healthy workers

This depends on:

• Multiple replicas
• Storage accessibility
• Pod disruption controls
• Node failure detection
• Sufficient spare capacity

A single-replica application can experience downtime.

Common Worker Problems

Node NotReady

Check:

oc get nodes
oc describe node <worker>

Then:

oc debug node/<worker>
chroot /host
systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio

Possible causes:

• kubelet failure
• CRI-O failure
• Network failure
• Certificate problem
• Disk pressure
• Memory pressure
• Node OS issue
• API server connectivity issue

DiskPressure

Check:

df -h
df -i
du -sh /var/lib/containers/*
du -sh /var/log/*

Possible causes:

• Large container logs
• Image accumulation
• Container storage exhaustion
• Inode exhaustion
• Failed garbage collection

MemoryPressure

Check:

free -h
top
oc adm top node

Possible causes:

• Oversized workloads
• Incorrect requests and limits
• Memory leak
• Too many Pods
• Platform agents consuming resources

Pods stuck in ContainerCreating

Investigate:

oc describe pod <pod> -n <namespace>

Possible causes:

• CRI-O failure
• Image pull problem
• CNI/OVN failure
• CSI mount failure
• SCC or permission issue
• Node disk pressure

ImagePullBackOff

Check:

• Image name
• Registry access
• ImagePullSecret
• DNS
• Proxy configuration
• Registry certificate trust
• CRI-O logs

journalctl -u crio
crictl images

Useful Worker Troubleshooting Commands

oc get nodes
oc describe node <worker>
oc adm top nodes
oc get pods -A -o wide --field-selector spec.nodeName=<worker>

Node-level access:

oc debug node/<worker>
chroot /host

Inside the host:

systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio
df -h
df -i
free -h
iostat -x 1 10
sar -u 1 10
pidstat -d 1 10

Runtime inspection:

crictl ps
crictl pods
crictl images
crictl info

Control Plane vs Worker Components

Control plane says:
"Run this Pod on worker-2"
Worker-2 says:
"I will pull the image, configure networking, mount storage, and start it"

Interview Answer

An OpenShift worker node is responsible for executing application workloads. It normally runs RHCOS, the kubelet, CRI-O, an OCI runtime such as crun, OVN-Kubernetes networking components, the Machine Config Daemon, and node-level DaemonSets such as Node Exporter, CoreDNS, logging collectors, and CSI plugins.

The kubelet registers the worker with the API server and watches for Pods assigned by the scheduler. It asks CRI-O to create the Pod sandbox and start containers, OVN configures networking, and CSI drivers mount persistent storage. The Machine Config Daemon keeps the RHCOS configuration synchronized with the desired MachineConfigPool state.

For troubleshooting, I start with oc get nodes and oc describe node, inspect node conditions such as Ready, MemoryPressure, and DiskPressure, then use oc debug node and chroot /host to check kubelet, CRI-O, storage, memory, network, and system logs.

OpenShift Control Plane Components

In OpenShift, the correct term is control plane, not control panel.

The control plane is the “brain” of the OpenShift cluster. It manages:

• Cluster configuration
• Workload scheduling
• API requests
• Cluster state
• Controllers and Operators
• Authentication and authorization
• Node and workload lifecycle

A typical highly available OpenShift cluster has three control-plane nodes.

                    Users and Administrators
                             │
                         oc / Console
                             │
                             ▼
                      API Load Balancer
                             │
          ┌──────────────────┼──────────────────┐
          ▼                  ▼                  ▼
      master-0            master-1           master-2
          │                  │                  │
          ├── kube-apiserver ├── kube-apiserver ├── kube-apiserver
          ├── etcd           ├── etcd           ├── etcd
          ├── scheduler      ├── scheduler      ├── scheduler
          ├── controller     ├── controller     ├── controller
          └── Operators      └── Operators      └── Operators

1. kube-apiserver

The kube-apiserver is the main entry point into the cluster.

All administrative and platform operations go through it.

Examples:

oc get pods
oc apply -f deployment.yaml
oc delete pod mypod

Request flow:

oc client
   │
   ▼
API load balancer
   │
   ▼
kube-apiserver
   │
   ├── Authentication
   ├── Authorization
   ├── Admission controls
   ├── Resource validation
   └── etcd read/write

The API server handles:

• Kubernetes API requests
• Authentication
• RBAC authorization
• Admission webhooks
• Object validation
• Communication with etcd

It listens on:

TCP 6443

Check it with:

oc get pods -n openshift-kube-apiserver -o wide
oc get co kube-apiserver

2. etcd

etcd is the distributed key-value database that stores the authoritative cluster state.

It stores:

• Deployments
• Pods and desired state
• Services
• Secrets
• ConfigMaps
• Routes
• RBAC
• Nodes
• Operators
• CRDs
• MachineConfig objects

API Server
    │
    ▼
   etcd
    │
    ▼
Cluster state

A standard OpenShift cluster normally has three etcd members:

master-0
master-1
master-2

Quorum requirement:

3 members → 2 required for quorum

If one member fails, the cluster can usually continue.

If two members fail, etcd loses quorum and control-plane operations stop.

Check etcd:

oc get co etcd
oc get pods -n openshift-etcd -o wide

3. kube-scheduler

The scheduler decides which node should run a new Pod.

It evaluates:

• CPU requests
• Memory requests
• Node selectors
• Taints and tolerations
• Affinity and anti-affinity
• Topology spread constraints
• Persistent-volume topology
• Host ports
• Node readiness

Pending Pod
     │
     ▼
Scheduler filters nodes
     │
     ▼
Scheduler scores eligible nodes
     │
     ▼
Pod assigned to worker-2
     │
     ▼
kubelet starts Pod

The scheduler does not start the container. It only assigns the Pod to a node.

Check it with:

oc get co kube-scheduler
oc get pods -n openshift-kube-scheduler -o wide

4. kube-controller-manager

The kube-controller-manager runs multiple Kubernetes controllers.

Controllers continuously compare:

Desired state
      vs
Actual state

and take action to correct differences.

Important controllers include:

• Deployment controller
• ReplicaSet controller
• Node controller
• Job controller
• Service account controller
• EndpointSlice controller
• Namespace controller
• Persistent-volume controller

Example:

Deployment requests 3 replicas
          │
          ▼
Controller sees only 2 Pods
          │
          ▼
Creates another Pod

Check it with:

oc get co kube-controller-manager
oc get pods -n openshift-kube-controller-manager -o wide

5. OpenShift Controller Manager

OpenShift also includes OpenShift-specific controllers.

These manage platform-specific resources and behavior beyond standard Kubernetes.

Examples include:

• OpenShift project behavior
• Build-related resources
• Image resources
• OpenShift authorization functions
• Platform-specific reconciliation

It runs separately from the Kubernetes controller manager.

Check:

oc get co openshift-controller-manager
oc get pods -n openshift-controller-manager

6. OpenShift API Server

The OpenShift API Server provides OpenShift-specific APIs that extend Kubernetes.

Examples include APIs related to:

• Projects
• Routes
• Builds
• Images
• OpenShift-specific authorization
• Security extensions

Architecture:

Client
   │
   ▼
Kubernetes API aggregation layer
   │
   ├── Kubernetes APIs
   └── OpenShift APIs

Check:

oc get co openshift-apiserver
oc get pods -n openshift-apiserver

7. Cluster Version Operator

The Cluster Version Operator, or CVO, manages the overall OpenShift release version.

It is responsible for:

• Installing platform components
• Coordinating upgrades
• Applying release manifests
• Monitoring ClusterOperators
• Ensuring components match the desired release

New OpenShift release
         │
         ▼
Cluster Version Operator
         │
         ▼
Platform Operators upgraded
         │
         ▼
Control-plane and worker updates

Check:

oc get clusterversion
oc get co

8. Machine Config Operator

The Machine Config Operator, or MCO, manages the operating-system configuration of RHCOS nodes.

It controls:

• RHCOS updates
• CRI-O configuration
• kubelet configuration
• Kernel arguments
• Systemd units
• CA certificates
• Registry configuration
• Node files

MachineConfig
      │
      ▼
Machine Config Operator
      │
      ▼
Machine Config Daemon
      │
      ▼
Drain → Apply → Reboot → Ready

Check:

oc get mcp
oc get machineconfig
oc get co machine-config

9. Authentication Operator

The Authentication Operator manages OpenShift OAuth and login services.

It handles:

• Identity providers
• OAuth server
• Login flow
• Authentication certificates
• Token configuration

Example:

User
  │
  ▼
Corporate identity provider
  │
  ▼
OpenShift OAuth
  │
  ▼
OpenShift access token

Check:

oc get co authentication
oc get oauth cluster -o yaml

10. Ingress Operator

The Ingress Operator manages the OpenShift router.

It controls:

• Router Pods
• IngressControllers
• Wildcard certificates
• Router replicas
• Publishing strategy
• Public and private ingress

External client
      │
      ▼
Load balancer
      │
      ▼
Router Pods
      │
      ▼
Route
      │
      ▼
Service
      │
      ▼
Application Pods

Check:

oc get co ingress
oc get ingresscontroller -n openshift-ingress-operator
oc get pods -n openshift-ingress

11. DNS Operator

The DNS Operator manages CoreDNS for cluster Service discovery.

It allows Pods to resolve names such as:

payments-api.banking.svc.cluster.local

Architecture:

Application Pod
      │
      ▼
DNS Service IP
      │
      ▼
CoreDNS
      │
      ├── Internal Service names
      └── External upstream DNS

Check:

oc get co dns
oc get pods -n openshift-dns
oc get dns.operator/default -o yaml

12. Network Operator

The Cluster Network Operator manages the OpenShift network plugin, usually OVN-Kubernetes.

It manages:

• Pod networks
• Service networks
• OVN components
• GENEVE tunnels
• MTU
• Egress features
• NetworkPolicies
• Node networking components

Pod A
  │
  ▼
OVN virtual network
  │
  ▼
Pod B

Check:

oc get co network
oc get network.operator cluster -o yaml
oc get pods -n openshift-ovn-kubernetes

13. Cloud Controller Manager

On cloud platforms, the cloud controller integrates OpenShift with AWS, Azure, or GCP.

It manages functions such as:

• Cloud node information
• Load balancers
• Routes
• Instance metadata
• Cloud volumes, depending on the driver architecture

Example:

Service type LoadBalancer
          │
          ▼
Cloud Controller
          │
          ▼
AWS / Azure / GCP load balancer

14. Machine API Operator

The Machine API Operator manages infrastructure machines on supported platforms.

It handles:

• Machine objects
• MachineSets
• MachineHealthChecks
• Worker creation
• Worker replacement
• Autoscaling integration

MachineSet replicas: 5
         │
         ▼
Machine API Operator
         │
         ▼
Create cloud VMs
         │
         ▼
New OpenShift workers join

Check:

oc get machines -A
oc get machinesets -A
oc get machinehealthchecks -A

15. Monitoring Components

The control plane is monitored by the OpenShift monitoring stack.

Main components include:

• Prometheus
• Alertmanager
• kube-state-metrics
• Node Exporter
• Prometheus Operator
• Thanos components

They monitor:

• API latency
• etcd latency
• Scheduler health
• Operator status
• Node health
• Resource utilization

Check:

oc get pods -n openshift-monitoring

Static Pods on Control-Plane Nodes

Several critical control-plane components run as static Pods:

• etcd
• kube-apiserver
• kube-controller-manager
• kube-scheduler

A static Pod is managed directly by the kubelet on the node.

Static Pod manifest
        │
        ▼
kubelet reads manifest
        │
        ▼
Control-plane Pod starts

This allows core components to start even when the scheduler is unavailable.

Control Plane Request Flow

When you run:

oc create deployment nginx --image=nginx

the full sequence is:

1. oc sends request to API load balancer
2. Load balancer selects a kube-apiserver
3. API server authenticates the user
4. RBAC authorizes the request
5. Admission controls validate the object
6. Deployment is stored in etcd
7. Controller Manager creates a ReplicaSet
8. ReplicaSet controller creates a Pod
9. Scheduler selects a worker node
10. kubelet asks CRI-O to start the container
11. OVN configures Pod networking
12. Pod becomes Running

High Availability

A production OpenShift control plane normally uses three control-plane nodes.

master-0
master-1
master-2

Availability is maintained through:

• Three API server instances
• Three etcd members
• Scheduler leader election
• Controller-manager leader election
• API load balancing
• Rolling upgrades
• Static Pods
• Operator reconciliation

Only one scheduler and controller-manager instance is active as leader at a time, while others remain ready to take over.

Control Plane vs Worker Nodes

Control Plane
     │
     ▼
Decides what should run
Worker Node
     │
     ▼
Runs the actual workload

Important Commands

oc get nodes
oc get clusteroperators
oc get clusterversion
oc get pods -A

Control-plane Pods:

oc get pods -n openshift-etcd
oc get pods -n openshift-kube-apiserver
oc get pods -n openshift-kube-controller-manager
oc get pods -n openshift-kube-scheduler

Operator health:

oc get co

Healthy status:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

Troubleshooting Control Plane

Use this sequence:

API unavailable or slow
        │
        ▼
Check API load balancer and DNS
        │
        ▼
Check kube-apiserver
        │
        ▼
Check etcd health and latency
        │
        ▼
Check control-plane nodes
        │
        ▼
Check scheduler and controllers
        │
        ▼
Check ClusterOperators

Useful commands:

oc get co
oc get nodes
oc get --raw='/readyz?verbose'
oc get pods -n openshift-etcd -o wide
oc get pods -n openshift-kube-apiserver -o wide

For node-level investigation:

oc debug node/<control-plane-node>
chroot /host
systemctl status kubelet
journalctl -u kubelet
iostat -x 1 10

Interview Answer

The OpenShift control plane is responsible for managing the cluster’s desired state and making all scheduling, API, and lifecycle decisions. Its key components are the kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and OpenShift-specific API and controller services.

The API server receives all requests, authenticates and authorizes them, and stores the resulting state in etcd. The controller managers continuously reconcile resources, while the scheduler selects suitable worker nodes for new Pods. OpenShift Operators such as the Cluster Version Operator, Machine Config Operator, Ingress Operator, DNS Operator, Network Operator, and Authentication Operator manage the platform components around the core Kubernetes control plane.

In a highly available cluster, these components run across three control-plane nodes. The API servers are load balanced, etcd maintains quorum, and the scheduler and controller managers use leader election. For troubleshooting, I start with oc get co, check API readiness, etcd health, control-plane Pods and nodes, and then review Operator conditions and logs.

WAL fsync in OpenShift

In OpenShift, WAL fsync normally refers to how quickly etcd can safely write changes to its Write-Ahead Log on disk.

A simple definition:

Before etcd confirms an important cluster-state change, it writes the change to its WAL and requests that the operating system physically persist it to storage using fsync or fdatasync.

This protects the cluster state if an etcd process, control-plane node, or operating system suddenly fails.

What is a WAL?

WAL means Write-Ahead Log.

etcd records a change in the log before applying it to its main backend database.

API change
   │
   ▼
etcd receives proposal
   │
   ▼
Write proposal to WAL
   │
   ▼
fsync to persistent storage
   │
   ▼
Replicate through Raft
   │
   ▼
Commit transaction
   │
   ▼
API request succeeds

Examples of changes written through etcd include:

• Creating a Pod
• Updating a Deployment
• Creating or changing a Secret
• Updating a ConfigMap
• Changing node status
• Updating EndpointSlices
• Operator status updates
• Creating or deleting Kubernetes resources

What does fsync do?

When an application writes data, the operating system may initially place it in memory cache.

Application write
       │
       ▼
Operating-system page cache
       │
       ▼
Storage device later

That is fast, but data still in memory can be lost during a sudden power or operating-system failure.

fsync tells the operating system:

Do not acknowledge this operation until the data
has been flushed to persistent storage.

For etcd:

WAL entry
   │
   ▼
fsync
   │
   ▼
Disk confirms persistence
   │
   ▼
etcd continues the commit

Therefore, WAL fsync latency is directly influenced by the storage system.

Why OpenShift depends on it

The Kubernetes API server stores cluster state in etcd.

oc apply
   │
   ▼
kube-apiserver
   │
   ▼
etcd leader
   │
   ▼
WAL fsync and Raft replication
   │
   ▼
Commit
   │
   ▼
API response

If WAL fsync becomes slow, etcd writes become slow. That can make the OpenShift API slow because API write operations depend on etcd committing changes.

Red Hat notes that slow storage or competing disk activity can cause high fsync latency, API slowness, request timeouts, missed heartbeats, and temporary etcd leader loss. (Red Hat Documentation)

Example of a normal write

Suppose you run:

oc scale deployment payments-api --replicas=5

The flow is:

1. oc sends PATCH request
2. API server validates and authorizes it
3. API server sends the change to etcd
4. etcd leader writes the proposal to WAL
5. WAL is synchronized to storage
6. Proposal is replicated to other etcd members
7. A majority acknowledges it
8. The change is committed
9. API server returns success

If step 5 takes 2 ms, the transaction can proceed quickly.

If step 5 takes 200 ms or more, API requests accumulate and controllers begin reconciling more slowly.

WAL vs etcd backend database

etcd uses both a WAL and a backend database.

Incoming change
      │
      ▼
WAL
Durable sequential record
      │
      ▼
Raft commit
      │
      ▼
Backend database
Current key-value state

They have separate performance metrics:

Interpretation:

High WAL fsync latency
→ synchronous WAL storage problem
High backend commit latency
→ backend database storage or database-pressure issue
Both high
→ general disk contention, throttling, or storage degradation

Main Prometheus metric

The key histogram metric is:

etcd_disk_wal_fsync_duration_seconds_bucket

To calculate the p99 latency:

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

This shows the WAL fsync duration below which approximately 99% of observations occurred during the selected period.

Breaking it down by instance helps identify one slow control-plane node:

master-0 → 0.004 seconds
master-1 → 0.120 seconds
master-2 → 0.005 seconds

Here, master-1 is the likely problem.

Red Hat identifies WAL fsync duration, backend commit duration, and leader changes as important metrics for evaluating etcd storage performance. (Red Hat Documentation)

What is an acceptable value?

The exact alert threshold can vary by OpenShift release, workload, and test method.

Current Red Hat guidance for validating etcd storage includes checking the p99 fsync result produced by its supported fio-based performance test. Some current documentation uses a threshold below 10 ms, while other recent release documentation and test output refer to 20 ms. Use the guidance and alerts supplied for your exact OpenShift version rather than applying one universal value. (Red Hat Documentation)

As a practical operational interpretation:

A few milliseconds
→ healthy low-latency storage
Consistent tens of milliseconds
→ investigate
Large or recurring spikes
→ likely to affect etcd and API performance

Focus on:

• p99 behavior
• Sustained duration
• Differences between members
• Correlation with API latency
• Leader changes and timeouts

A single isolated spike is less concerning than repeated or sustained high latency.

Causes of high WAL fsync latency

1. Slow storage

Examples:

• Mechanical disks
• Slow SAN
• High-latency network-backed block storage
• Poorly configured virtual disks
• Underperforming SSDs

2. IOPS or throughput throttling

Cloud disks and virtual machines can have:

• Disk IOPS limits
• Throughput limits
• Burst-credit exhaustion
• Instance-wide storage limits

3. Noisy neighbours

The etcd virtual disk might share physical infrastructure with:

• Other virtual machines
• Databases
• Backup jobs
• Storage replication
• Large image operations

4. Snapshots and backups

Hypervisor snapshots or storage backups can temporarily increase latency.

5. Control-plane processes producing I/O

Examples:

• Logging agents
• Security scanners
• Backup agents
• Excessive journal writes
• Container image operations

6. Device or filesystem errors

Examples:

• Storage path failures
• NVMe timeouts
• SAN multipath issues
• Filesystem problems
• Disk nearly full

Red Hat recommends low-latency block storage for etcd and advises against sharing its underlying I/O infrastructure with competing I/O-intensive workloads. (Red Hat Documentation)

Symptoms in OpenShift

High WAL fsync latency can produce:

• Slow oc commands
• API request timeouts
• Slow application deployments
• Delayed Operator reconciliation
• Pods remaining Pending longer
• ClusterOperators becoming degraded
• etcd slow fdatasync messages
• Missed Raft heartbeats
• Increased leader elections
• Web console delays

Slow disk
   │
   ▼
Slow WAL fsync
   │
   ▼
Slow etcd commits
   │
   ▼
Slow API server
   │
   ▼
Slow controllers and Operators
   │
   ▼
Cluster-wide control-plane impact

Existing application containers might continue processing traffic, but control-plane changes become slow or fail.

Related metrics

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Failed proposals

rate(etcd_server_proposals_failed_total[5m])

Peer network latency

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Interpret the metrics together:

High fsync + normal peer RTT
→ storage issue
Normal fsync + high peer RTT
→ network issue
High fsync + leader changes
→ storage may be destabilizing Raft
High fsync + pending proposals
→ etcd cannot commit writes fast enough

Node-level troubleshooting

Identify the slow member using Prometheus, then debug its control-plane node:

oc debug node/master-1

Enter the host:

chroot /host

Check where etcd data resides:

findmnt /var/lib/etcd
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
df -h
df -i

Check real-time disk performance:

iostat -x 1 10

Important fields:

Look for:

High w_await
Growing aqu-sz
Sustained device pressure
Spikes matching etcd latency

%util alone is not sufficient for modern parallel devices. Latency and queue depth provide better context.

Find competing processes

Use:

pidstat -d 1 10

Possible output:

PID    kB_rd/s    kB_wr/s    COMMAND
2100      0.00      800.00    etcd
7350      0.00     9000.00    backup-agent

This suggests that backup-agent may be competing with etcd.

Historical statistics:

sar -d 1 10
sar -u 1 10
sar -q 1 10

Check kernel storage errors:

journalctl -k --since "1 hour ago" |
  grep -Ei 'I/O error|timeout|reset|nvme|scsi|xfs'

Check etcd logs

List etcd Pods:

oc get pods -n openshift-etcd -o wide

Review one member:

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h

Search for likely symptoms:

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h |
grep -Ei \
  'slow fdatasync|took too long|timeout|heartbeat|leader|election'

Do not confuse WAL fsync with network replication

A write requires both disk durability and Raft consensus.

etcd leader
   │
   ├── WAL fsync on local disk
   ├── Replication to peers
   └── Majority acknowledgement

Therefore, a slow transaction can come from:

• Local disk fsync latency
• Peer disk latency
• Network round-trip latency
• CPU starvation
• etcd overload

Always compare disk and network metrics before concluding that storage is the only problem.

Remediation

Immediate

• Stop or reschedule competing backup or scanning jobs.
• Resolve storage path failures.
• Fix disk or VM throttling.
• Reduce runaway API writes.
• Pause nonessential bulk deployments.
• Remove unrelated I/O-intensive activity from control-plane storage.

Permanent

• Use dedicated low-latency SSD or NVMe-backed storage.
• Use guaranteed IOPS rather than burst-only storage.
• Ensure the VM instance supports sufficient total storage throughput.
• Isolate control-plane nodes from noisy neighbours.
• Avoid putting application workloads on control-plane nodes.
• Monitor p99 WAL fsync continuously.
• Validate storage with the Red Hat-supported fio procedure.

Do not react by restarting every etcd member. That could destroy quorum.

Also, do not use defragmentation as the first solution: it may reduce internal database fragmentation, but it does not repair slow physical storage.

Interview answer

WAL fsync in OpenShift is the time etcd takes to durably persist a Raft proposal into its Write-Ahead Log. When an API operation changes cluster state, the API server sends it to etcd. The etcd leader records the proposal in the WAL, synchronizes it to persistent storage, replicates it to the other members, and commits it after a majority acknowledges the proposal.

Because fsync waits for storage durability, slow storage directly increases etcd transaction latency and therefore OpenShift API latency. Sustained high fsync latency can cause request timeouts, pending proposals, missed Raft heartbeats, and leader changes. I monitor the p99 value of etcd_disk_wal_fsync_duration_seconds_bucket, compare it across members, and correlate it with backend commit latency, peer network RTT, leader changes, and API latency. At the node level, I use oc debug node, iostat, sar, and pidstat to identify disk latency, queueing, throttling, or competing processes. The permanent solution is isolated, low-latency storage with guaranteed IOPS—not repeatedly restarting etcd or treating defragmentation as a disk-performance fix.

Leader Changes and Pending Proposals in OpenShift etcd

Both metrics describe the stability and performance of the etcd cluster, which stores OpenShift control-plane state.

kube-apiserver
      │
      ▼
etcd leader
      │
      ├── Writes WAL locally
      ├── Replicates proposal to followers
      └── Waits for quorum

A healthy etcd cluster should have:

• A stable leader
• Very few unexpected leader changes
• Pending proposals normally close to zero
• Low WAL fsync latency
• Low peer-network latency

1. What is an etcd leader?

etcd uses the Raft consensus algorithm. In a standard three-member OpenShift etcd cluster:

master-0: etcd leader
master-1: etcd follower
master-2: etcd follower

The leader handles cluster-state writes.

For example:

oc scale deployment payments --replicas=5

The write process is:

API server sends update
        │
        ▼
etcd leader creates proposal
        │
        ▼
Leader writes proposal to WAL
        │
        ▼
Proposal replicated to followers
        │
        ▼
Majority acknowledges
        │
        ▼
Proposal committed
        │
        ▼
API request succeeds

The leader keeps followers synchronized and commits a write only after quorum acknowledges it. (Red Hat Documentation)

2. What is a leader change?

A leader change occurs when the current leader stops being leader and another etcd member is elected.

Before:
master-0 = leader
master-1 = follower
master-2 = follower
Leader heartbeat lost
        │
        ▼
Election timeout reached
        │
        ▼
New election
After:
master-0 = follower/unavailable
master-1 = leader
master-2 = follower

Leader election is a normal high-availability mechanism. The problem is not an occasional leader change during planned maintenance; the problem is frequent or unexpected elections.

Why does the leader change?

Common causes include:

Slow WAL fsync

The leader cannot persist its Raft log quickly enough.

Slow disk
   │
   ▼
Slow WAL fsync
   │
   ▼
Heartbeat processing delayed
   │
   ▼
Followers assume leader failed
   │
   ▼
New election

High network latency or packet loss

Followers do not receive heartbeats in time.

Leader heartbeat
       │
       X packet loss or delay
       │
Follower election timeout
       │
       ▼
Leader election

CPU or memory starvation

The etcd process cannot schedule enough CPU time to send or process heartbeats.

Control-plane node restart

A reboot or static-pod restart can trigger a legitimate election.

Storage or node failure

Examples include:

• Cloud-disk throttling
• SAN congestion
• Datastore latency
• Failed storage path
• Hypervisor pause
• Node hardware failure

Slow storage and competing disk activity can cause long fsync times, missed heartbeats, request timeouts, and temporary leader loss. (Red Hat Documentation)

Leader-change metric

Use:

etcd_server_leader_changes_seen_total

This is a cumulative counter. To see recent changes:

increase(etcd_server_leader_changes_seen_total[15m])

Interpretation:

0 during normal operation
→ Stable leader
1 during a planned master reboot
→ Usually expected
Repeated changes in a short period
→ Investigate immediately

Because each member may expose the counter, inspect it by instance:

sum by (instance) (
  increase(etcd_server_leader_changes_seen_total[15m])
)

Impact of frequent leader changes

During an election, etcd briefly has no active leader.

Leader lost
    │
    ▼
Election in progress
    │
    ▼
Writes temporarily pause
    │
    ▼
API write requests wait or time out

Symptoms can include:

• Slow oc apply, oc create, or oc delete
• API request timeouts
• Operators reconciling slowly
• Delayed node status updates
• Pods taking longer to create
• ClusterOperators becoming degraded
• request timed out messages
• Temporary control-plane instability

During leader loss and reelection, Kubernetes API requests that cause state changes can be interrupted or delayed. (Red Hat Documentation)

3. What is an etcd proposal?

A proposal is a requested change to etcd state that must pass through Raft consensus.

Examples include:

Create Deployment
Update Secret
Delete Pod
Modify ConfigMap
Update Node status
Change Route
Update Operator status

Simplified flow:

API write
   │
   ▼
Proposal created
   │
   ▼
WAL persisted
   │
   ▼
Replicated to followers
   │
   ▼
Quorum reached
   │
   ▼
Proposal committed

4. What are pending proposals?

A pending proposal is a proposal that etcd has received but has not yet committed.

The metric is:

etcd_server_proposals_pending

This is a gauge showing the current number of outstanding proposals.

Normally:

Pending proposals ≈ 0

Brief small increases during bursts can be normal.

A sustained or rising value indicates that etcd is receiving changes faster than it can persist, replicate, and commit them.

Incoming proposals
        │
        ▼
etcd processing capacity insufficient
        │
        ▼
Queue grows
        │
        ▼
Pending proposals increase

Why do proposals remain pending?

Slow disk writes

Proposal
   │
   ▼
Waiting for WAL fsync
   │
   ▼
Proposal remains pending

Slow peer replication

Leader sends proposal
       │
       ▼
Follower response delayed
       │
       ▼
Waiting for quorum

Leader election

Proposals can pause while a new leader is elected.

Excessive Kubernetes API writes

A runaway Operator or automation process may generate more writes than etcd can process.

Examples:

• Controller updating status continuously
• CI/CD loop repeatedly creating resources
• Excessive Kubernetes Events
• Large bulk deployments
• Frequent ConfigMap or Secret updates
• Broken automation repeatedly patching objects

CPU pressure

The etcd member cannot process requests promptly.

Oversized API objects

Large objects require more disk, network, and serialization work.

Pending-proposal interpretation

etcd_server_proposals_pending

Example:

0–a few, briefly
→ Usually normal during a write burst
Continuously above zero
→ etcd is falling behind
Steadily increasing
→ Severe processing, disk, or network bottleneck

Also check failed proposals:

rate(etcd_server_proposals_failed_total[5m])

Interpret together:

Pending rising, failures zero
→ Requests are delayed but may eventually commit
Pending rising, failures rising
→ etcd cannot successfully process part of the workload

Relationship between the two metrics

Leader changes and pending proposals often appear together.

Slow disk or network
        │
        ├── WAL commits slow
        │       └── Pending proposals rise
        │
        └── Heartbeats delayed
                └── Leader changes rise

Example:

WAL fsync p99:         180 ms
Pending proposals:     75
Leader changes:        6 in 15 minutes
API requests:          timing out

This strongly suggests etcd instability, often caused by disk or network latency.

Important correlation matrix

Metrics to examine together

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Failed proposals

rate(etcd_server_proposals_failed_total[5m])

WAL fsync p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

Backend commit p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Peer RTT p99

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Red Hat identifies WAL fsync duration, backend commit latency, leader changes, and peer RTT as important etcd performance signals. (Red Hat Documentation)

Troubleshooting procedure

Step 1: Check etcd health

oc get co etcd
oc describe co etcd
oc get pods -n openshift-etcd -o wide

Look for:

• Degraded conditions
• Pod restarts
• One unhealthy member
• Revision rollout problems

Step 2: Identify the current leader

oc get pods -n openshift-etcd -o wide

Enter a healthy etcd Pod and run:

etcdctl endpoint status --cluster -w table

The output shows which member is the leader. Use the supported certificate environment and command procedure for your OCP version.

Step 3: Check etcd logs

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h |
grep -Ei \
  'leader|election|heartbeat|slow fdatasync|timeout|proposal|took too long'

Look for messages indicating:

leader changed
lost leader
elected leader
failed to send heartbeat
slow fdatasync
request timed out
apply request took too long

Step 4: Check disk performance

On the affected control-plane node:

oc debug node/<master-node>
chroot /host

Then run:

iostat -x 1 10
sar -d 1 10
pidstat -d 1 10
df -h
df -i

Check:

• await and w_await
• aqu-sz
• Storage utilization
• Competing backup or logging processes
• Disk capacity and inodes
• Hypervisor or cloud-disk throttling

Step 5: Check network latency

Compare peer round-trip latency between etcd members.

Also investigate:

• Packet loss
• Firewall state
• MTU mismatch
• NIC errors
• Hypervisor networking
• Cross-site or cross-zone latency
• Network jitter

Step 6: Check excessive API writers

Use API metrics:

sum by (verb, resource) (
  rate(apiserver_request_total[5m])
)

Find high-volume clients:

topk(
  20,
  sum by (user_agent, verb) (
    rate(apiserver_request_total[5m])
  )
)

Look for unexpected increases in:

POST
PUT
PATCH
DELETE

Example production incident

02:00 backup starts on control-plane datastore
             │
             ▼
Disk write latency increases
             │
             ▼
WAL fsync rises to 150 ms
             │
             ├── Proposals queue
             │       └── pending proposals = 60
             │
             └── Heartbeats delayed
                     └── 4 leader elections
             │
             ▼
API becomes slow

Resolution:

• Stop or reschedule the competing backup.
• Move control-plane storage to isolated low-latency disks.
• Verify cloud or datastore IOPS and throughput.
• Confirm pending proposals return near zero.
• Confirm no new leader changes.
• Verify API latency and ClusterOperator health.

What not to do

Avoid:

• Restarting all etcd members simultaneously
• Restarting all control-plane nodes together
• Manually deleting etcd data
• Removing members without the supported procedure
• Changing election timers as the first fix
• Defragmenting repeatedly to solve physical disk latency

OpenShift uses validated etcd timer values for each platform. Changing timers can hide symptoms rather than correct the underlying disk or network problem. (Red Hat Documentation)

Interview answer

An etcd leader change occurs when the current Raft leader becomes unavailable or fails to deliver heartbeats within the election timeout, causing another member to be elected. Occasional leader changes during maintenance can be expected, but frequent changes indicate instability caused by disk fsync latency, network delay, packet loss, CPU starvation, or control-plane node failures. I monitor this using increase(etcd_server_leader_changes_seen_total[15m]).

A pending proposal is an etcd write request that has been accepted but has not yet been committed through Raft consensus. The metric etcd_server_proposals_pending should normally stay close to zero. A sustained increase means etcd cannot persist or replicate writes as fast as they arrive, commonly because of slow WAL storage, high peer latency, resource pressure, or excessive API write activity.

I correlate both metrics with WAL fsync latency, backend commit latency, peer RTT, proposal failures, API request rates, and etcd logs. If leader changes and pending proposals increase together, I treat it as a serious control-plane performance problem and investigate the affected member’s storage, network, CPU, and competing processes.

etcd Operator in OpenShift

The etcd Operator manages the lifecycle, configuration, health, certificates, and membership of the etcd cluster that stores OpenShift’s control-plane state.

In a standard highly available OpenShift cluster, etcd runs on the three control-plane nodes:

                  OpenShift API
                       │
                       ▼
                kube-apiserver
                       │
                       ▼
                etcd cluster
          ┌────────────┼────────────┐
          ▼            ▼            ▼
      master-0      master-1      master-2
       etcd-0        etcd-1        etcd-2
          ▲            ▲            ▲
          └────────────┼────────────┘
                       │
                  etcd Operator

The etcd Operator continually observes the cluster, compares the current state with the required state, and corrects differences through the Kubernetes and etcd management APIs. (Red Hat Documentation)

Why etcd Is Critical

etcd is the authoritative database for Kubernetes and OpenShift.

It stores objects such as:

• Deployments
• Pods and their desired state
• Services
• Routes
• Secrets
• ConfigMaps
• RBAC
• Nodes
• CRDs and Custom Resources
• Operator configuration
• MachineConfig objects
• Cluster configuration

The runtime contents of containers and application databases are not stored in etcd.

oc apply -f deployment.yaml
             │
             ▼
       kube-apiserver
             │
             ▼
            etcd
             │
             ▼
 Deployment object stored
             │
             ▼
 Controllers create Pods

If etcd becomes unavailable, existing containers can often continue running temporarily, but:

• New Pods cannot be scheduled.
• Configuration changes cannot be saved.
• Operators cannot reconcile normally.
• oc commands that require the API begin failing.
• Cluster recovery and automation stop functioning correctly.

etcd Operator vs etcd

These are different components:

Cluster Version Operator
          │
          ▼
      etcd Operator
          │
          ▼
      etcd members
          │
          ▼
   Cluster state database

The Operator itself does not store the cluster state. It manages the etcd processes that do.

Location and Resources

The Operator normally runs in:

openshift-etcd-operator

The etcd static Pods run in:

openshift-etcd

Check them:

oc get pods -n openshift-etcd-operator
oc get pods -n openshift-etcd -o wide

Check the ClusterOperator:

oc get clusteroperator etcd

The cluster-scoped configuration resource is:

oc get etcd cluster -o yaml

The etcd cluster Operator provides the cluster-scoped etcds.operator.openshift.io API and is configured through the etcd/cluster object. (Red Hat Documentation)

Main Responsibilities of the etcd Operator

1. Deploying etcd as static Pods

On each control-plane node, etcd runs as a static Pod.

Static Pod manifest
       │
       ▼
kubelet on master node
       │
       ▼
etcd Pod starts

Typical Pods:

oc get pods -n openshift-etcd -o wide

Example:

etcd-master-0
etcd-master-1
etcd-master-2

Static Pods are managed directly by the kubelet, not by a Deployment.

This is important because core control-plane services must be able to start even when normal Kubernetes scheduling is unavailable.

2. Maintaining etcd membership

A three-member etcd cluster normally has:

Member 1: master-0
Member 2: master-1
Member 3: master-2

The Operator monitors whether the expected members match the available control-plane nodes.

When a control-plane node is properly replaced, the Operator can:

• Generate certificates for the new member
• Add the replacement member to etcd
• Remove stale membership
• Reconcile the new topology

Red Hat documents that when a lost control-plane node is replaced, the etcd cluster Operator handles generating new TLS certificates and adding the new node as an etcd member. (Red Hat Documentation)

3. Preserving quorum

etcd uses the Raft consensus algorithm.

For three members:

Members: 3
Required quorum: 2
Maximum simultaneous failures: 1

For five members:

Members: 5
Required quorum: 3
Maximum simultaneous failures: 2

A standard OpenShift control plane normally uses three members.

master-0    master-1    master-2
 Healthy     Healthy     Failed
      \         /
       Quorum remains

If two of three members are lost:

master-0    master-1    master-2
 Healthy     Failed      Failed
      │
      ▼
  No quorum

The Operator cannot simply recreate lost authoritative state when quorum is gone. You must follow the documented disaster-recovery procedure and restore from a valid backup. Red Hat explicitly distinguishes single-member replacement from loss of the majority of control-plane hosts. (Red Hat Documentation)

4. Managing certificates

etcd communication is secured with TLS.

Certificates include:

• Peer certificates for member-to-member communication
• Server certificates
• Client certificates for API server access
• Certificate authority bundles

etcd-0  ←── mutual TLS ──→  etcd-1
   │                           │
   └────── mutual TLS ────────→ etcd-2

The Operator manages the certificate resources and rolls out new static-Pod revisions when certificates rotate.

It also ensures the kube-apiserver has the required trust and client credentials to connect to etcd securely.

5. Managing static-Pod revisions

Configuration changes are rolled out using versioned revisions.

Current revision 20
        │
 Configuration changes
        ▼
New revision 21 generated
        │
        ▼
Install on control-plane nodes
        │
        ▼
Validate member health

You can inspect revision-related resources:

oc get configmaps -n openshift-etcd
oc get secrets -n openshift-etcd

The Operator ensures the expected configuration, certificates, and manifests are synchronized across the control-plane nodes.

6. Monitoring cluster health

The Operator monitors:

• Member availability
• Quorum
• Endpoint health
• Static-Pod revisions
• Certificate status
• Member synchronization
• Leader stability
• Backup and defragmentation-related conditions
• Storage and API-visible health signals

Check its high-level condition:

oc get co etcd

A healthy status is:

AVAILABLE     True
PROGRESSING   False
DEGRADED      False

Detailed conditions:

oc describe co etcd

7. Supporting member recovery

If one etcd member fails but quorum remains, the recovery process depends on the failure type:

• The control-plane machine is stopped.
• The node is NotReady.
• The etcd Pod is crash-looping.
• The underlying machine was permanently lost.
• The certificates are invalid.

The Operator can reconcile the member when a temporarily unavailable node returns. For permanent loss, replacement must follow the supported procedure.

Red Hat recommends taking an etcd backup before replacing an unhealthy member. (Red Hat Documentation)

8. Automating defragmentation and maintenance

etcd receives a high number of small updates and deletions. Deleted data can leave unused space inside the backend database.

Objects created and updated
          │
          ▼
Objects deleted
          │
          ▼
Unused internal database pages
          │
          ▼
Fragmentation

The etcd Operator performs supported maintenance activities, including automatic defragmentation behavior in current OpenShift releases.

However:

Defragmentation is not a fix for slow physical storage.

If WAL fsync latency is high because the disk is saturated, the solution is usually faster or isolated storage, not repeated defragmentation.

Reconciliation Loop

The etcd Operator follows the normal Operator control-loop model:

Observe control-plane nodes and etcd state
                     │
                     ▼
       Read desired configuration
                     │
                     ▼
 Compare desired state with actual state
                     │
             ┌───────┴────────┐
             │                │
          Matches          Difference
             │                │
             ▼                ▼
         No change      Reconcile resources
                              │
               ┌──────────────┼──────────────┐
               ▼              ▼              ▼
          Update static   Rotate certs   Fix membership
          Pod revision
                              │
                              ▼
                       Validate health
                              │
                              ▼
                       Update status

Examples that trigger reconciliation include:

• A control-plane node is replaced.
• A certificate needs rotation.
• A new OpenShift release changes etcd.
• Static-Pod configuration differs.
• An expected member is missing.
• A member returns after temporary failure.

How an API Request Uses etcd

For example:

oc apply -f app.yaml

The flow is:

oc client
    │
    ▼
API load balancer
    │
    ▼
kube-apiserver
    │
    ├── Authentication
    ├── Authorization
    ├── Admission
    └── Validation
           │
           ▼
          etcd
           │
           ▼
  Persist Kubernetes object

The etcd Operator is not in the request data path. It ensures the etcd cluster receiving the request remains healthy and correctly configured.

etcd Leader and Followers

One etcd member acts as the Raft leader.

                etcd leader
                    │
             Replicates writes
          ┌─────────┴─────────┐
          ▼                   ▼
      follower             follower

A write is committed after a majority acknowledges it:

API write
   │
   ▼
Leader writes WAL
   │
   ├── replicate to follower 1
   └── replicate to follower 2
             │
             ▼
      Majority acknowledges
             │
             ▼
         Commit write

This is why etcd requires:

• Low-latency storage
• Reliable networking
• Low latency between control-plane nodes
• Accurate time synchronization
• Stable control-plane resources

etcd continuously persists many small changes, making fast, low-latency I/O especially important. (Red Hat Documentation)

Useful Troubleshooting Commands

Check ClusterOperator status

oc get co etcd
oc describe co etcd

Check etcd Operator

oc get pods -n openshift-etcd-operator

oc logs -n openshift-etcd-operator \
  deployment/etcd-operator \
  --since=1h

Check etcd Pods

oc get pods -n openshift-etcd -o wide

Inspect Pod containers

oc describe pod -n openshift-etcd <etcd-pod>

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h

Check configuration

oc get etcd cluster -o yaml

Check nodes

oc get nodes
oc describe node <control-plane-node>

Checking Endpoint Health

First identify the etcd Pods:

oc get pods -n openshift-etcd --show-labels

Then enter a healthy etcd Pod:

oc rsh -n openshift-etcd <etcd-pod>

Depending on the OpenShift version and container environment, run the provided etcdctl command with the appropriate certificates:

etcdctl endpoint health --cluster
etcdctl endpoint status --cluster -w table

The status output helps identify:

• Member ID
• Endpoint
• etcd version
• Database size
• Leader
• Raft term and index
• Errors

Use the commands and certificate paths documented for the exact OpenShift version rather than inventing or replacing TLS parameters manually.

Important etcd Metrics

WAL fsync latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

High values indicate slow synchronous writes.

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Database size

etcd_mvcc_db_total_size_in_bytes

These help distinguish:

High WAL latency
      → Storage issue
High peer RTT
      → Network issue
Frequent leader changes
      → Storage, network, or resource instability
Increasing pending proposals
      → etcd cannot process writes quickly enough

Common Failure Scenarios

Scenario 1: One member is unavailable

Three members
   │
   ├── Two healthy
   └── One failed

Result:

• Quorum remains.
• API usually continues functioning.
• The Operator reports degradation.
• Investigate and replace or recover the unhealthy member using the supported procedure.

Do not immediately delete etcd data or membership manually.

Scenario 2: Two members are unavailable

Three members
   │
   ├── One healthy
   └── Two failed

Result:

• Quorum is lost.
• Writes stop.
• API availability is severely affected.
• Normal Operator reconciliation cannot restore the authoritative state.
• Perform control-plane disaster recovery from a valid etcd snapshot.

Scenario 3: etcd Pod is CrashLoopBackOff

Check:

oc describe pod -n openshift-etcd <pod>
oc logs -n openshift-etcd <pod> -c etcd --previous

Possible causes:

• Corrupt or unavailable storage
• Certificate failure
• Invalid member state
• Static-Pod revision problem
• Disk full
• File permissions
• Network or peer connectivity
• Node-level failure

Scenario 4: Slow API caused by etcd

Symptoms:

• Slow oc commands
• API timeouts
• Operators become degraded
• Leader changes
• Slow fdatasync messages

Investigate:

oc debug node/<master-node>
chroot /host
iostat -x 1 10
sar -d 1 10
pidstat -d 1 10
df -h
df -i

Correlate node storage metrics with etcd WAL fsync and backend commit latency.

Backups

The Operator manages etcd operation, but the administrator must maintain a tested backup strategy.

A control-plane backup contains:

etcd snapshot
       +
static Kubernetes resources

Run the documented backup script from a healthy control-plane node and copy the resulting files to secure off-cluster storage.

Backups should be:

• Automated
• Encrypted
• Stored off-cluster
• Access controlled
• Tested regularly
• Matched to documented recovery procedures

An etcd snapshot does not replace application database or persistent-volume backups.

What Not to Do

Avoid:

• Deleting /var/lib/etcd
• Manually editing static-Pod manifests
• Manually removing etcd members without the supported procedure
• Restarting all control-plane nodes together
• Restarting all etcd members simultaneously
• Copying a data directory between members
• Restoring a snapshot into a live healthy cluster
• Treating defragmentation as the first fix for disk contention
• Editing Operator-managed resources directly

Unsafe etcd changes can cause permanent loss of cluster state.

Interview Answer

The etcd Operator manages the OpenShift control-plane etcd cluster. etcd itself stores the authoritative Kubernetes state, while the Operator deploys and maintains the etcd static Pods, certificates, configuration revisions, cluster membership and health. It continuously compares the desired state with the actual state and reconciles differences.

In a standard highly available cluster, etcd runs as three members on the control-plane nodes and requires two members for quorum. If one member fails, the cluster can continue operating while the member is recovered or replaced. If the majority is lost, the Operator cannot recreate the missing state, and the cluster must be restored using the documented disaster-recovery process and a valid etcd snapshot.

For troubleshooting, I begin with oc get co etcd, inspect the Operator conditions and logs, check the etcd static Pods and endpoint health, verify control-plane nodes, and examine WAL fsync, backend commit, peer latency, leader changes and pending proposals. I also verify disk latency and capacity because etcd depends on fast, low-latency storage.

Role of the Scheduler in OpenShift

The scheduler decides which node should run a newly created Pod.

It does not start containers itself. Its job is to select the best eligible node and assign the Pod to it.

User creates Pod
      │
      ▼
API Server stores Pod
      │
      ▼
Pod has no nodeName
      │
      ▼
Scheduler evaluates nodes
      │
      ▼
Scheduler selects one node
      │
      ▼
kubelet starts the Pod

Where the Scheduler Runs

In a standard OpenShift cluster, the Kubernetes scheduler runs on the control-plane nodes as a static Pod.

Check it with:

oc get pods -n openshift-kube-scheduler -o wide

The scheduler is managed by the Kubernetes Scheduler Operator.

Check its status:

oc get co kube-scheduler

Healthy status:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

Scheduler vs Scheduler Operator

These are different components.

Scheduler Operator
        │
        ▼
Manages kube-scheduler
        │
        ▼
kube-scheduler assigns Pods

Scheduling Flow

Suppose a Pod is created:

apiVersion: v1
kind: Pod
metadata:
  name: payments-api
spec:
  containers:
    - name: app
      image: registry.example.com/payments-api:1.0

Initially, the Pod has no node assignment:

spec:
  nodeName: ""

The scheduler watches the API server for these unscheduled Pods.

Step 1: Watch for Pending Pods

The scheduler detects:

Pod: payments-api
Status: Pending
Node: none

It adds the Pod to its scheduling queue.

Step 2: Filter Nodes

The scheduler eliminates nodes that cannot run the Pod.

This is called the filtering phase.

Example:

Available nodes:
worker-1
worker-2
worker-3
worker-4

The scheduler checks:

• CPU availability
• Memory availability
• Node readiness
• Taints and tolerations
• Node selectors
• Node affinity
• Pod affinity and anti-affinity
• Persistent volume topology
• Host ports
• Pod count limits
• Resource constraints

After filtering:

worker-1 → eligible
worker-2 → insufficient memory
worker-3 → taint not tolerated
worker-4 → nodeSelector mismatch

Only worker-1 remains.

Step 3: Score Nodes

If several nodes are eligible, the scheduler scores them.

Example:

worker-1 → score 85
worker-2 → score 72
worker-3 → score 91

The scheduler selects:

worker-3

Scoring can consider:

• Resource balance
• Node affinity preferences
• Pod spreading
• Image locality
• Existing workload distribution
• Topology preferences

Step 4: Bind the Pod

The scheduler updates the Pod through the API server:

spec:
  nodeName: worker-3

This is called binding.

Step 5: kubelet Starts the Pod

The kubelet on worker-3 sees the assignment.

kubelet
   │
   ▼
CRI-O
   │
   ▼
Pull image
   │
   ▼
Configure networking
   │
   ▼
Mount volumes
   │
   ▼
Start container

The scheduler is no longer involved after the node assignment unless the Pod is recreated.

Important Point: Scheduler Does Not Move Running Pods

The scheduler normally handles only Pods that do not yet have a node assignment.

It does not automatically move a running Pod from one node to another just because another node becomes less busy.

Running Pod on worker-1
          │
          ▼
Scheduler does not rebalance it automatically

To redistribute workloads, you may use:

• Descheduler
• Node drain
• Pod eviction
• Deployment rollout
• Cluster autoscaling
• Manual rescheduling

Resource Requests

The scheduler uses resource requests, not actual current consumption, when deciding placement.

Example:

resources:
  requests:
    cpu: "2"
    memory: 4Gi
  limits:
    cpu: "4"
    memory: 8Gi

The scheduler reserves:

2 CPU
4 GiB memory

It does not schedule based on the Pod’s current real-time usage.

This is why incorrect requests can cause poor scheduling.

Requests too high

Pod remains Pending
Insufficient CPU or memory

Requests too low

Node becomes overloaded
Pods compete for resources

Node Selectors

A node selector forces a Pod onto nodes with matching labels.

Example:

spec:
  nodeSelector:
    workload-type: payments

Only nodes labeled:

oc label node worker-3 workload-type=payments

are eligible.

Node Affinity

Node affinity provides more expressive placement rules.

Example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - zone-a
                - zone-b

This means the Pod must run in zone-a or zone-b.

Pod Affinity

Pod affinity places Pods near other Pods.

Example use case:

Application Pod
      near
Cache Pod

This can reduce latency but may reduce fault isolation.

Pod Anti-Affinity

Pod anti-affinity spreads Pods apart.

Example:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: payments-api
        topologyKey: kubernetes.io/hostname

This prevents two payments-api replicas from running on the same node.

worker-1 → payments-api-1
worker-2 → payments-api-2
worker-3 → payments-api-3

This improves high availability.

Topology Spread Constraints

Topology spread constraints distribute Pods across zones or nodes.

Example:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payments-api

This helps distribute replicas evenly across availability zones.

Zone A → 2 Pods
Zone B → 2 Pods
Zone C → 2 Pods

Taints and Tolerations

A taint prevents Pods from being scheduled unless they tolerate it.

Example taint:

oc adm taint nodes worker-3 dedicated=payments:NoSchedule

Only Pods with this toleration can run there:

tolerations:
  - key: dedicated
    operator: Equal
    value: payments
    effect: NoSchedule

Typical OpenShift uses include:

• Control-plane protection
• Infrastructure nodes
• GPU nodes
• Storage nodes
• Dedicated application nodes

Control-Plane Nodes

Control-plane nodes normally have taints that prevent ordinary application workloads.

node-role.kubernetes.io/master:NoSchedule

or:

node-role.kubernetes.io/control-plane:NoSchedule

Platform Pods have the required tolerations, but regular application Pods do not.

Storage-Aware Scheduling

For Pods using persistent storage, the scheduler must consider volume topology.

Example:

PVC is available only in zone-a
         │
         ▼
Pod must be scheduled to node in zone-a

The scheduler evaluates:

• StorageClass
• PersistentVolume
• Volume node affinity
• Availability zone
• CSI driver constraints

A Pod may remain Pending if no eligible node can access the volume.

Scheduler Profiles in OpenShift

OpenShift supports scheduler profiles that influence how workloads are placed.

Common profiles include:

• LowNodeUtilization
• HighNodeUtilization
• NoScoring

A simplified interpretation:

Check scheduler configuration:

oc get scheduler cluster -o yaml

Example:

apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  name: cluster
spec:
  profile: LowNodeUtilization

Use supported configuration through the cluster Scheduler resource rather than editing static Pod manifests.

High Availability

The scheduler runs on control-plane nodes, but only one instance is active as leader at a time.

scheduler-master-0 → leader
scheduler-master-1 → standby
scheduler-master-2 → standby

If the leader fails:

Leader unavailable
       │
       ▼
Leader election
       │
       ▼
Another scheduler becomes active

Existing Pods continue running during a scheduler outage, but new Pods cannot be assigned until scheduling resumes.

What Happens if the Scheduler Is Down?

Existing workloads generally continue running.

However:

• New Pods stay Pending.
• Failed Pods cannot be placed on another node.
• Deployments cannot scale successfully.
• New Jobs remain unscheduled.
• Node drain replacements may remain Pending.
• Cluster upgrades can be affected.

Example:

Deployment replicas desired: 5
Running: 3
Pending: 2

Pending Pod Troubleshooting

Start with:

oc get pod <pod-name> -n <namespace>

Then:

oc describe pod <pod-name> -n <namespace>

Look at Events.

Common messages:

0/10 nodes are available:
3 insufficient memory
2 node(s) had untolerated taint
4 node(s) didn't match node selector
1 node(s) had volume node affinity conflict

This message is often the fastest way to identify the scheduling problem.

Common Scheduling Failures

Insufficient CPU

0/5 nodes are available:
5 Insufficient cpu

Check:

oc adm top nodes
oc describe node <node>

Remember that scheduling uses requested CPU, not actual CPU usage.

Insufficient memory

0/5 nodes are available:
5 Insufficient memory

Check allocated requests:

oc describe node <node>

Look under:

Allocated resources

Untolerated taint

node(s) had untolerated taint

Check:

oc describe node <node> | grep -i taint

Then verify the Pod tolerations.

Node selector mismatch

node(s) didn't match Pod's node affinity/selector

Check:

oc get nodes --show-labels

Compare with:

oc get pod <pod> -o yaml

Pod anti-affinity conflict

The placement rules may be too strict for the available number of nodes.

Example:

3 replicas
Only 2 eligible nodes
Required anti-affinity

The third Pod remains Pending.

Volume node affinity conflict

The Pod and volume are tied to different zones.

Check:

oc describe pod <pod>
oc get pv <pv-name> -o yaml
oc get pvc -n <namespace>

Too many Pods on a node

The node may have reached its Pod capacity.

Check:

oc describe node <node>

Look for:

pods: 250

and the number currently allocated.

Scheduler Operator Troubleshooting

Check the ClusterOperator:

oc get co kube-scheduler

Describe it:

oc describe co kube-scheduler

Check scheduler Pods:

oc get pods -n openshift-kube-scheduler -o wide

Check logs:

oc logs -n openshift-kube-scheduler \
  <scheduler-pod> \
  -c kube-scheduler

Check the Operator:

oc get pods -n openshift-kube-scheduler-operator

oc logs -n openshift-kube-scheduler-operator \
  deployment/openshift-kube-scheduler-operator

Check API readiness:

oc get --raw='/readyz?verbose'

The scheduler depends on a healthy API server and etcd.

Scheduling Troubleshooting Flow

Pod Pending
    │
    ▼
oc describe pod
    │
    ▼
Read FailedScheduling event
    │
    ├── Insufficient resources
    ├── Taints/tolerations
    ├── Node selectors
    ├── Affinity rules
    ├── Storage topology
    └── Pod capacity
           │
           ▼
Check scheduler Operator only if many unrelated Pods are affected

A single Pending Pod usually indicates a workload placement issue.

Many unrelated Pending Pods across the cluster may indicate:

• Scheduler failure
• API server problem
• etcd problem
• Cluster-wide capacity shortage
• Broken scheduler configuration

Scheduler vs Autoscaler

The scheduler and autoscaler have different responsibilities.

Flow:

Pod cannot fit
      │
      ▼
Scheduler marks it unschedulable
      │
      ▼
Cluster Autoscaler detects it
      │
      ▼
New node created
      │
      ▼
Scheduler places Pod

Scheduler vs Descheduler

The descheduler does not directly move a Pod. It evicts it, and then the scheduler assigns the replacement.

Interview Answer

The OpenShift scheduler is responsible for assigning unscheduled Pods to suitable nodes. It watches the API server for Pods that do not have a nodeName, places them in a scheduling queue, filters out nodes that cannot satisfy the Pod’s requirements, scores the remaining eligible nodes, and binds the Pod to the best node.

During filtering, it evaluates resource requests, node readiness, taints and tolerations, node selectors, affinity and anti-affinity, topology spread constraints, host ports, and persistent-volume topology. After the scheduler writes the selected node into the Pod specification, the kubelet on that node uses CRI-O to start the containers.

The scheduler does not run containers and normally does not rebalance already running Pods. It is managed by the Kubernetes Scheduler Operator and runs highly available on the control-plane nodes using leader election. For troubleshooting a Pending Pod, I first use oc describe pod and inspect the FailedScheduling event before checking node capacity, labels, taints, affinity rules, storage topology, and finally the scheduler Operator if the problem affects many workloads.

Ingress Operator in OpenShift

The Ingress Operator manages how HTTP and HTTPS traffic enters an OpenShift cluster.

Its primary job is to deploy and maintain the OpenShift router, which receives external requests and forwards them to the correct application Service and Pods.

The simplest traffic flow is:

Client
  │
  ▼
External Load Balancer
  │
  ▼
OpenShift Router
  │
  ▼
Route
  │
  ▼
Service
  │
  ▼
Application Pods

The Ingress Operator runs in:

openshift-ingress-operator

The router Pods normally run in:

openshift-ingress

Ingress Operator vs Router

These are not the same component.

Ingress Operator
       │
       ▼
IngressController CR
       │
       ▼
Router Deployment
       │
       ▼
Router Pods
       │
       ▼
Routes and Services

The Operator manages the routers. The routers process the actual application traffic.

Main Responsibilities

1. Deploying router Pods

The Ingress Operator creates and maintains router workloads.

Check them with:

oc get pods -n openshift-ingress -o wide

Typical router Pods:

router-default-7d8b5c9d4f-abc12
router-default-7d8b5c9d4f-def34

The Operator ensures the desired number of router replicas is running.

Desired replicas: 2
Actual replicas: 1
        │
        ▼
Ingress Operator detects drift
        │
        ▼
Creates replacement router Pod

2. Managing the IngressController resource

The main configuration object is:

oc get ingresscontroller -n openshift-ingress-operator

The default object is usually:

default

Inspect it:

oc get ingresscontroller default \
  -n openshift-ingress-operator \
  -o yaml

A simplified example:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  replicas: 3
  domain: apps.cluster.example.com

The Operator watches this resource and reconciles the router configuration.

3. Managing the wildcard application domain

Applications commonly use names such as:

payments.apps.cluster.example.com
mobile.apps.cluster.example.com
api.apps.cluster.example.com

The wildcard DNS record:

*.apps.cluster.example.com

normally points to the ingress load balancer.

*.apps.cluster.example.com
             │
             ▼
      External Load Balancer
             │
             ▼
         Router Pods

The IngressController defines the domain that its routers serve.

4. Managing Route traffic

An OpenShift Route exposes a Service externally.

Example:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: payments
  namespace: banking
spec:
  host: payments.apps.cluster.example.com
  to:
    kind: Service
    name: payments-service
  port:
    targetPort: https

Traffic flow:

payments.apps.cluster.example.com
              │
              ▼
          Router Pod
              │
              ▼
        Route: payments
              │
              ▼
   Service: payments-service
              │
              ▼
        Ready payment Pods

The router watches Routes and EndpointSlices and updates its proxy configuration dynamically.

TLS Termination

The Ingress Operator manages router configuration for different TLS models.

Edge termination

Client ──HTTPS──> Router ──HTTP──> Application

The router terminates TLS.

Example:

tls:
  termination: edge

Use this when encryption between the router and application is not required or the internal network is trusted.

Re-encrypt termination

Client ──HTTPS──> Router ──HTTPS──> Application

The router terminates the client connection and creates a new TLS connection to the backend.

Example:

tls:
  termination: reencrypt
  destinationCACertificate: |
    ...

This is commonly used for sensitive enterprise applications.

Passthrough termination

Client ──HTTPS──> Router ──HTTPS──> Application

The router does not decrypt the request. TLS terminates inside the application.

Example:

tls:
  termination: passthrough

Because the router cannot inspect HTTP content, passthrough routing relies primarily on TLS SNI.

Default Ingress Certificate

The default router normally presents a wildcard certificate for:

*.apps.cluster.example.com

The Ingress Operator manages the router’s default certificate configuration.

You can configure a custom certificate through a Secret:

oc create secret tls custom-apps-certificate \
  --cert=apps.crt \
  --key=apps.key \
  -n openshift-ingress

Then reference it:

oc patch ingresscontroller default \
  -n openshift-ingress-operator \
  --type=merge \
  -p '{
    "spec": {
      "defaultCertificate": {
        "name": "custom-apps-certificate"
      }
    }
  }'

The Operator rolls out the router configuration with the new certificate.

Ingress Publishing Strategies

The Ingress Operator supports different ways to expose routers depending on the infrastructure.

LoadBalancerService

Common in public clouds:

Cloud Load Balancer
        │
        ▼
Router Service
        │
        ▼
Router Pods

The Operator creates a Service of type LoadBalancer.

HostNetwork

Router Pods bind directly to ports on the node.

Client
  │
  ▼
Node IP:80/443
  │
  ▼
Router Pod using host network

This is often used in bare-metal or controlled on-premises environments.

NodePortService

The router is exposed through NodePorts.

External Load Balancer
        │
        ▼
NodeIP:NodePort
        │
        ▼
Router Service

An external F5, HAProxy, or NetScaler can send traffic to the NodePorts.

Private or internal load balancer

An ingress controller can be exposed only internally.

Example architecture:

Internet users
      │
      ▼
Public LB
      │
      ▼
Public IngressController
Corporate users
      │
      ▼
Internal LB
      │
      ▼
Private IngressController

This is valuable in banking and enterprise environments.

Multiple Ingress Controllers

You are not limited to the default router.

You might create:

default
public
internal
partner
pci

Example separation:

Public applications
       │
       ▼
public.apps.cluster.example.com
       │
       ▼
Public routers
Internal applications
       │
       ▼
internal.apps.cluster.example.com
       │
       ▼
Internal routers

Each ingress controller can have its own:

• Domain
• Certificate
• Node placement
• Replica count
• Publishing strategy
• Route selector
• Namespace selector
• Load balancer

Route Selection

A custom ingress controller can serve only selected Routes.

Example using a route selector:

spec:
  routeSelector:
    matchLabels:
      ingress: internal

Then label a Route:

oc label route payments ingress=internal -n banking

Only the matching ingress controller serves that Route.

Namespace Selection

An ingress controller can also serve only selected namespaces.

Example:

spec:
  namespaceSelector:
    matchLabels:
      exposure: internal

Label the namespace:

oc label namespace banking exposure=internal

This provides stronger organizational separation.

Router Placement on Infrastructure Nodes

In production, router Pods are often placed on dedicated infrastructure nodes.

Application workers
  └── Business workloads
Infrastructure workers
  ├── Router Pods
  ├── Registry
  ├── Monitoring
  └── Logging

Example node placement:

spec:
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/infra: ""
    tolerations:
      - key: node-role.kubernetes.io/infra
        operator: Exists
        effect: NoSchedule

Benefits include:

• Isolating ingress traffic from application workloads
• Predictable capacity
• Easier network segmentation
• Reduced noisy-neighbour risk
• Better operational control

High Availability

A production ingress controller should have multiple router replicas.

External Load Balancer
        │
  ┌─────┼─────┐
  ▼     ▼     ▼
Router1 Router2 Router3

If one router fails:

Router1 readiness fails
        │
        ▼
Endpoint removed
        │
        ▼
Traffic continues through Router2 and Router3

For high availability:

• Use at least two replicas.
• Spread routers across nodes and zones.
• Use anti-affinity or topology controls.
• Ensure the external load balancer removes unhealthy backends.
• Maintain sufficient capacity during upgrades.

Router Load-Balancing Algorithms

The router can distribute traffic to application Pods using algorithms such as:

• roundrobin
• leastconn
• source

Example annotation:

metadata:
  annotations:
    haproxy.router.openshift.io/balance: leastconn

Round robin

Requests rotate across backend Pods.

Least connections

New requests go to the backend with fewer active connections.

Source

Uses the client source to provide a form of persistence.

Applications should ideally be stateless rather than relying heavily on sticky sessions.

Request Processing

A request follows this path:

1. DNS resolves application hostname
2. Client connects to external load balancer
3. Load balancer selects a router endpoint
4. Router evaluates hostname and Route
5. Router performs TLS processing
6. Router selects the target Service
7. Router selects a ready endpoint
8. Request reaches the application Pod

The router generally sends traffic only to endpoints considered ready.

Ingress Operator Reconciliation

The Ingress Operator continuously compares desired and actual state.

IngressController spec
          │
          ▼
Operator observes router state
          │
          ▼
Compare desired and actual state
          │
    ┌─────┴─────┐
    │           │
 Matches      Drift
    │           │
    ▼           ▼
 No action    Create/update resources
                    │
                    ▼
             Check router health
                    │
                    ▼
              Update status

Examples of events that trigger reconciliation:

• Replica count changes
• Certificate changes
• Domain changes
• Node-placement changes
• Router Pod failure
• OpenShift upgrade
• Publishing-strategy changes

Relationship with Other Components

Cluster Version Operator
        │
        ▼
Ingress Operator
        │
        ├── IngressController CR
        ├── Router Deployment
        ├── Router Service
        ├── Certificates
        └── Status

It also depends on:

Useful Commands

Check ClusterOperator status

oc get co ingress

Healthy state:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

More detail:

oc describe co ingress

Check Ingress Operator Pods

oc get pods -n openshift-ingress-operator

View logs:

oc logs -n openshift-ingress-operator \
  deployment/ingress-operator

Check IngressControllers

oc get ingresscontroller \
  -n openshift-ingress-operator

Describe the default controller:

oc describe ingresscontroller default \
  -n openshift-ingress-operator

Check router Pods

oc get pods -n openshift-ingress -o wide

Check router logs:

oc logs -n openshift-ingress <router-pod>

Check router Service

oc get svc -n openshift-ingress

Check Routes

oc get routes -A

Describe one Route:

oc describe route payments -n banking

Check Service endpoints

oc get svc payments-service -n banking
oc get endpointslices -n banking

Troubleshooting a Degraded Ingress Operator

Use this sequence:

ClusterOperator
      ↓
IngressController
      ↓
Operator logs
      ↓
Router Deployment and Pods
      ↓
Service and external load balancer
      ↓
DNS
      ↓
Route
      ↓
Application Service and endpoints

Step 1: Check Operator condition

oc get co ingress
oc describe co ingress

Look for messages about:

• Router deployment unavailable
• Load balancer provisioning failure
• Certificate problem
• DNS failure
• Insufficient replicas
• Node-placement failure

Step 2: Check IngressController status

oc get ingresscontroller default \
  -n openshift-ingress-operator \
  -o yaml

Review conditions such as:

• Available
• Progressing
• Degraded
• LoadBalancerManaged
• Admitted

Step 3: Check router Pods

oc get pods -n openshift-ingress -o wide

For a failing Pod:

oc describe pod <router-pod> -n openshift-ingress
oc logs <router-pod> -n openshift-ingress

Possible causes:

• Image pull failure
• Node selector mismatch
• Missing toleration
• Port conflict
• Certificate failure
• Insufficient CPU or memory
• Readiness probe failure

Step 4: Check DNS

dig payments.apps.cluster.example.com
dig '*.apps.cluster.example.com'

The application hostname should resolve to the correct ingress VIP or load balancer.

Step 5: Check external access

curl -vk https://payments.apps.cluster.example.com

This helps identify:

• DNS errors
• TCP failures
• TLS failures
• Router responses
• HTTP status codes

Step 6: Check the Route

oc get route payments -n banking
oc describe route payments -n banking

Verify:

• Correct hostname
• Correct Service
• Correct target port
• Correct TLS termination
• Route is admitted by the expected router

Step 7: Check Service and endpoints

oc get svc payments-service -n banking
oc get endpointslices -n banking
oc get pods -n banking

A Route with no ready backend endpoints commonly returns:

503 Service Unavailable

Common Errors

503 Service Unavailable

Usually means the router cannot reach a healthy backend.

Check:

• Pod readiness
• Service selector
• EndpointSlices
• Target port
• Application listening port
• NetworkPolicy

Application route does not resolve

Likely causes:

• Missing wildcard DNS
• Wrong domain
• Wrong load-balancer address
• Public/private DNS mismatch

TLS certificate error

Check:

• Certificate validity
• Certificate chain
• Hostname/SAN
• Secret name
• Route-specific certificate
• IngressController default certificate

Router Pods Pending

Check:

• Node selector
• Taints and tolerations
• Resource pressure
• Pod anti-affinity
• SCC or permissions
• Host port availability

Banking Architecture Example

Internet
   │
   ▼
DDoS protection
   │
   ▼
WAF
   │
   ▼
Public Load Balancer
   │
   ▼
Public IngressController
   │
   ▼
Internet-facing applications
Corporate network
   │
   ▼
Internal Load Balancer
   │
   ▼
Private IngressController
   │
   ▼
Internal banking applications

Recommended controls:

• Separate public and internal ingress controllers
• Dedicated infrastructure nodes
• Re-encrypt or passthrough TLS
• WAF in front of public ingress
• Multiple replicas across failure domains
• Central router access logging
• NetworkPolicies behind the router
• Separate certificates and DNS zones
• Route and namespace selectors
• No direct public access to internal routers

Important Interview Distinction

The Ingress Operator does not directly forward application requests. It manages the IngressController and router resources. The router Pods perform the actual Layer 7 traffic routing.

Ingress Operator
      │
      ▼
Manages Router
      │
      ▼
Router processes traffic

Interview Answer

The OpenShift Ingress Operator manages the lifecycle and configuration of the cluster’s ingress controllers. It creates and maintains the router Deployments, Services, certificates, replica count, publishing strategy, and node placement. The default ingress controller serves the wildcard application domain, normally *.apps.<cluster-domain>, and the router Pods receive HTTP or HTTPS traffic from an external load balancer.

When a request arrives, the router matches the hostname to an OpenShift Route, performs edge, re-encrypt, or passthrough TLS handling, and forwards the request to the target Service and ready Pod endpoints. The Operator continuously reconciles the desired IngressController configuration with the running router resources and replaces failed Pods or rolls out configuration changes. For production environments, I normally use multiple router replicas on dedicated infrastructure nodes and separate public and internal ingress controllers. When troubleshooting, I check oc get co ingress, the IngressController conditions, Operator logs, router Pods, external load balancer, wildcard DNS, Route configuration, Service, EndpointSlices, and application readiness.

API Server Operator in OpenShift

The Kubernetes API Server Operator manages the lifecycle and configuration of the Kubernetes API server in OpenShift.

Its main responsibility is to make sure the kube-apiserver is:

• Installed
• Correctly configured
• Highly available
• Using valid certificates
• Running the version required by OpenShift
• Automatically recovered if a component fails
• Safely rolled out during upgrades

The Operator runs in:

openshift-kube-apiserver-operator

It manages API server instances in:

openshift-kube-apiserver

The Operator is installed and updated through the Cluster Version Operator. Its cluster-scoped configuration resource is KubeAPIServer, named cluster. (Red Hat Documentation)

Where It Fits

Users, kubelets, Operators and controllers
                    │
                    ▼
        API load balancer :6443
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
   master-0      master-1      master-2
 kube-apiserver kube-apiserver kube-apiserver
       ▲            ▲            ▲
       └────────────┼────────────┘
                    │
       Kubernetes API Server Operator
                    │
                    ▼
       Configuration, rollout and health
                    │
                    ▼
                   etcd

The API server is the main entry point into the cluster. Commands such as:

oc get pods
oc create deployment
oc apply -f app.yaml

all pass through the Kubernetes API server.

Kubernetes API Server vs API Server Operator

These are different components.

The flow is:

API Server Operator
        │
        ▼
Manages kube-apiserver
        │
        ▼
kube-apiserver handles API requests

Kubernetes API Server vs OpenShift API Server

OpenShift has two related API server layers.

Kubernetes API server

Handles standard Kubernetes resources:

• Pods
• Deployments
• Services
• Secrets
• ConfigMaps
• Nodes
• RBAC
• StatefulSets

OpenShift API server

Handles OpenShift-specific APIs, such as certain:

• Projects
• Routes-related platform integrations
• Security and authorization extensions
• OpenShift-specific resources

Client request
      │
      ▼
Kubernetes API aggregation layer
      │
      ├── Kubernetes APIs
      │
      └── OpenShift-specific APIs

The Kubernetes API Server Operator manages kube-apiserver, while the OpenShift API Server Operator installs and maintains openshift-apiserver. (Red Hat Documentation)

Main Responsibilities

1. Deploying API server static pods

On each control-plane node, the API server runs as a static pod.

/etc/kubernetes/manifests/
        │
        ▼
kubelet detects manifest
        │
        ▼
kube-apiserver pod starts

Typical API server pods can be viewed with:

oc get pods -n openshift-kube-apiserver -o wide

Example:

kube-apiserver-master-0
kube-apiserver-master-1
kube-apiserver-master-2

Static pods are controlled by the kubelet directly rather than by a Deployment.

2. Managing revisions

The Operator creates versioned API server revisions.

Revision 12
     │
     ▼
Configuration changed
     │
     ▼
Revision 13 created
     │
     ▼
Control-plane nodes updated gradually

You can see revision resources and related configuration in the API server namespace:

oc get configmaps -n openshift-kube-apiserver
oc get secrets -n openshift-kube-apiserver

Revision-based management allows the Operator to roll out a consistent configuration and diagnose which revision each node is running.

3. Rolling updates

The Operator avoids replacing every API server simultaneously.

master-0
Update → Ready
          │
          ▼
master-1
Update → Ready
          │
          ▼
master-2
Update → Ready

This preserves API availability as long as:

• The load balancer has healthy backends.
• Enough control-plane nodes remain available.
• etcd retains quorum.
• The new revision becomes healthy.

During an OpenShift upgrade, the CVO delivers the new release state and the API Server Operator reconciles the API server toward that version.

4. Certificate management

The API server requires several certificates for:

• External API access
• Internal API access
• Communication with etcd
• Authentication of clients
• Communication with aggregated APIs
• Service-network endpoints

The Operator helps manage and rotate API server certificates.

Certificate approaches rotation point
               │
               ▼
Operator creates updated certificate resources
               │
               ▼
New static-pod revision
               │
               ▼
API servers roll out incrementally

This reduces the chance of an API outage caused by expired certificates.

5. API server configuration

The cluster-scoped resource is:

oc get kubeapiserver cluster -o yaml

The corresponding API is:

operator.openshift.io/v1
kind: KubeAPIServer
metadata:
  name: cluster

The resource provides configuration for the Operator that manages kube-apiserver. (Red Hat Documentation)

A simplified example:

apiVersion: operator.openshift.io/v1
kind: KubeAPIServer
metadata:
  name: cluster
spec:
  audit:
    profile: Default

Do not add unsupported fields or manually edit generated static-pod manifests.

Use the supported cluster API:

oc edit kubeapiserver cluster

Red Hat identifies oc edit kubeapiserver as the configuration interface for the Kubernetes API Server Operator. (Red Hat Documentation)

6. Audit policy configuration

The Operator applies the configured API audit profile.

Audit records may include:

• User identity
• Service account identity
• API resource
• Operation or verb
• Source IP
• Request result
• Timestamp

Example operations:

create
update
patch
delete
get
list

Audit configuration should be changed through the KubeAPIServer resource, not by directly modifying the API server static-pod command arguments.

7. etcd connectivity

The API server reads and writes cluster state in etcd.

oc request
    │
    ▼
kube-apiserver
    │
    ▼
Authentication and authorization
    │
    ▼
Admission controls
    │
    ▼
etcd read/write
    │
    ▼
Response

The Operator manages the API server configuration required to connect securely to etcd, but the etcd cluster itself is maintained by the etcd Operator.

A slow etcd backend directly affects API performance:

Slow etcd disk
      │
      ▼
Slow etcd transaction
      │
      ▼
Slow API response
      │
      ▼
Slow oc commands and controller reconciliation

8. Health monitoring

The Operator monitors whether the API server is:

• Available
• Progressing
• Degraded
• Running the expected revision
• Responding to health checks

Check the ClusterOperator:

oc get clusteroperator kube-apiserver

Healthy status:

AVAILABLE     True
PROGRESSING   False
DEGRADED      False

Detailed information:

oc describe clusteroperator kube-apiserver

Reconciliation Process

The Operator continuously performs this loop:

Observe desired configuration
             │
             ▼
Inspect current API server revision
             │
             ▼
Compare desired and actual state
             │
       ┌─────┴─────┐
       │           │
   Matches      Difference
       │           │
       ▼           ▼
 No action    Create new revision
                   │
                   ▼
            Roll out static pods
                   │
                   ▼
              Check readiness
                   │
                   ▼
              Update status

Example: the audit profile changes.

Administrator updates KubeAPIServer CR
                    │
                    ▼
Operator notices configuration change
                    │
                    ▼
New revision generated
                    │
                    ▼
API server nodes updated incrementally
                    │
                    ▼
Operator reports Available

Request Processing Through the API Server

A request normally passes through several stages:

oc apply
   │
   ▼
Load balancer
   │
   ▼
kube-apiserver
   │
   ├── TLS validation
   ├── Authentication
   ├── Authorization
   ├── Admission control
   ├── Resource validation
   └── etcd persistence
          │
          ▼
       Response

For example:

oc create deployment nginx --image=nginx

The API server:

1. Authenticates the user.
2. Checks RBAC authorization.
3. Runs admission controls.
4. Validates the Deployment.
5. Stores it in etcd.
6. Returns the result.
7. Controllers later create the ReplicaSet and Pods.

The Operator maintains the API server that performs these steps; it does not process user API requests itself.

Useful Commands

Check ClusterOperator status

oc get co kube-apiserver
oc describe co kube-apiserver

Check Operator pods

oc get pods -n openshift-kube-apiserver-operator

Check Operator logs

oc logs -n openshift-kube-apiserver-operator \
  deployment/kube-apiserver-operator

Check API server pods

oc get pods -n openshift-kube-apiserver -o wide

Check a specific API server pod

oc describe pod -n openshift-kube-apiserver \
  kube-apiserver-master-0

Check API readiness

oc get --raw='/readyz?verbose'

Check API liveness

oc get --raw='/livez?verbose'

Check configuration

oc get kubeapiserver cluster -o yaml

Check recent events

oc get events -n openshift-kube-apiserver \
  --sort-by='.lastTimestamp'

Troubleshooting a Degraded API Server Operator

Use this sequence:

ClusterOperator
      ↓
Operator conditions
      ↓
Operator pod and logs
      ↓
API server static pods
      ↓
Node health
      ↓
etcd health
      ↓
Certificates and load balancer

Step 1: Check status

oc get co kube-apiserver
oc describe co kube-apiserver

Look at the condition messages under:

• Available
• Progressing
• Degraded

The condition message usually points to the affected revision or node.

Step 2: Check the Operator

oc get pods -n openshift-kube-apiserver-operator

oc logs -n openshift-kube-apiserver-operator \
  deployment/kube-apiserver-operator \
  --since=1h

Look for:

• Revision installation failure
• Certificate errors
• Missing ConfigMaps or Secrets
• Static-pod rollout timeout
• Node installer failure

Step 3: Check API server pods

oc get pods -n openshift-kube-apiserver -o wide

Check containers in a failing static pod:

oc describe pod -n openshift-kube-apiserver \
  <kube-apiserver-pod>

oc logs -n openshift-kube-apiserver \
  <kube-apiserver-pod> \
  -c kube-apiserver \
  --since=1h