OpenShift Load Balancers Explained

In OpenShift, “load balancer” can refer to several different layers:

External client traffic
        │
        ▼
External Load Balancer
        │
        ▼
OpenShift Ingress Router
        │
        ▼
Service
        │
        ▼
Application Pods

There are also dedicated load balancers for the OpenShift API and machine configuration endpoints.

Main Load Balancers in OpenShift

A standard OpenShift cluster normally needs these external endpoints:

The most important distinction is:

API load balancer
        ≠
Application ingress load balancer

They serve different traffic and normally use different backend pools.

High-Level Architecture

                          Administrators
                               │
                               ▼
                    api.cluster.example.com
                               │
                     API Load Balancer
                               │
                 ┌─────────────┼─────────────┐
                 ▼             ▼             ▼
             master-0       master-1       master-2
               :6443          :6443          :6443


                         Application Users
                               │
                               ▼
                 app1.apps.cluster.example.com
                               │
                  Ingress Load Balancer
                               │
                 ┌─────────────┼─────────────┐
                 ▼             ▼             ▼
            router pod     router pod     router pod
                 │             │             │
                 └─────────────┼─────────────┘
                               ▼
                         OpenShift Service
                               │
                         Application Pods

1. API Load Balancer

The API load balancer provides highly available access to the OpenShift control plane.

Clients include:

• oc
• OpenShift web console
• kubelets
• Operators
• Controllers
• Automation pipelines
• Monitoring systems

Traffic path:

oc command
    │
    ▼
api.cluster.example.com:6443
    │
    ▼
External load balancer
    │
    ├── master-0:6443
    ├── master-1:6443
    └── master-2:6443

The load balancer distributes requests across all healthy API servers.

Recommended behavior

Use:

• Layer 4 TCP load balancing
• TCP health checks or HTTPS health checks
• No application-level rewriting
• Source connection stability where required
• All control-plane nodes as backends

Example HAProxy configuration:

frontend api-server
    bind *:6443
    mode tcp
    default_backend api-server-backend
backend api-server-backend
    mode tcp
    balance roundrobin
    option tcp-check
    server master0 10.10.10.10:6443 check
    server master1 10.10.10.11:6443 check
    server master2 10.10.10.12:6443 check

2. Internal API Load Balancer

The internal API name is commonly:

api-int.<cluster>.<domain>

It is used by nodes and internal cluster components.

Worker kubelet
      │
      ▼
api-int.cluster.example.com:6443
      │
      ▼
Internal load balancer
      │
      ▼
Control-plane nodes

In some designs, the same load balancer serves both public and internal API names. In more restricted environments, separate internal and external virtual IPs are used.

3. Machine Config Server Load Balancer

The Machine Config Server listens on port 22623.

It provides Ignition and machine configuration during installation and node provisioning.

New OpenShift node
       │
       ▼
api-int.cluster.example.com:22623
       │
       ▼
Load balancer
       │
       ▼
Machine Config Server

Typical backend targets:

master-0:22623
master-1:22623
master-2:22623

Example HAProxy configuration:

frontend machine-config-server
    bind *:22623
    mode tcp
    default_backend machine-config-backend
backend machine-config-backend
    mode tcp
    balance roundrobin
    server master0 10.10.10.10:22623 check
    server master1 10.10.10.11:22623 check
    server master2 10.10.10.12:22623 check

4. Application Ingress Load Balancer

The application load balancer handles traffic for OpenShift Routes.

Example DNS:

payments.apps.cluster.example.com
mobile.apps.cluster.example.com
banking.apps.cluster.example.com

These normally resolve to the ingress load balancer.

Client
   │
   ▼
payments.apps.cluster.example.com
   │
   ▼
External load balancer
   │
   ▼
OpenShift router pods
   │
   ▼
Service
   │
   ▼
Application pods

The router pods are usually HAProxy-based and managed by the Ingress Operator.

OpenShift Router

The OpenShift router is not the same as the external load balancer.

External Load Balancer
        │
        ▼
Ingress Router Pods
        │
        ▼
OpenShift Services
        │
        ▼
Application Pods

The external load balancer only forwards traffic to the router nodes or router pods.

The router then:

• Matches the hostname
• Matches the route
• Handles TLS
• Selects the backend service
• Load-balances traffic to pods

Route Traffic Flow

Suppose the application route is:

https://payments.apps.cluster.example.com

The request path is:

Browser
   │
   ▼
DNS
   │
   ▼
Ingress load balancer
   │
   ▼
Router pod
   │
   ▼
Route object
   │
   ▼
Service
   │
   ▼
Pod endpoint

OpenShift Route Example

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: payments
  namespace: banking
spec:
  host: payments.apps.cluster.example.com
  to:
    kind: Service
    name: payments-service
  port:
    targetPort: https
  tls:
    termination: edge

The router watches Route objects and dynamically updates its routing configuration.

Load Balancing Inside the Cluster

OpenShift Services provide internal load balancing.

Example:

apiVersion: v1
kind: Service
metadata:
  name: payments-service
spec:
  selector:
    app: payments
  ports:
    - port: 443
      targetPort: 8443

Traffic flow:

payments-service
      │
      ├── payments-pod-1
      ├── payments-pod-2
      └── payments-pod-3

The service exposes a stable virtual IP and distributes traffic to healthy pod endpoints.

Service Types

ClusterIP

Default service type.

Available only inside the cluster

spec:
  type: ClusterIP

Use it for internal service-to-service communication.

NodePort

Exposes a port on every node.

Client
  │
  ▼
NodeIP:NodePort
  │
  ▼
Service
  │
  ▼
Pods

spec:
  type: NodePort

NodePort is usually not preferred as the primary application exposure method in enterprise OpenShift. Routes are more common.

LoadBalancer

Requests an external load balancer from the cloud provider or supported infrastructure integration.

spec:
  type: LoadBalancer

Example flow in AWS:

Service type LoadBalancer
        │
        ▼
Cloud Controller Manager
        │
        ▼
AWS NLB or ELB created
        │
        ▼
OpenShift nodes or pods

This is frequently used for non-HTTP protocols or applications that need their own external load balancer.

ExternalName

Maps a service to an external DNS name.

spec:
  type: ExternalName
  externalName: database.example.com

It does not create a real load balancer.

Ingress Controller Placement

By default, ingress router pods are scheduled according to the Ingress Controller configuration.

For enterprise environments, dedicated infrastructure nodes are recommended:

Worker nodes
    ├── Application workloads
    └── Batch workloads
Infrastructure nodes
    ├── Ingress routers
    ├── Registry
    ├── Monitoring
    └── Logging

Example node labels:

oc label node infra-0 node-role.kubernetes.io/infra=""
oc label node infra-1 node-role.kubernetes.io/infra=""
oc label node infra-2 node-role.kubernetes.io/infra=""

The Ingress Controller can then use a node placement policy.

Typical Bare-Metal Architecture

                         Corporate Network
                               │
                     F5 / HAProxy / NetScaler
                     ┌─────────┴──────────┐
                     │                    │
                  API VIP             Apps VIP
                     │                    │
          ┌──────────┼──────────┐    ┌────┼────┐
          ▼          ▼          ▼    ▼    ▼    ▼
       master-0   master-1   master-2  infra nodes
       :6443      :6443      :6443     :80/:443

Example VIPs:

API VIP:  10.10.20.10
Apps VIP: 10.10.20.20

DNS:

api.ocp.example.com              → 10.10.20.10
api-int.ocp.example.com          → 10.10.20.10
*.apps.ocp.example.com           → 10.10.20.20

Typical Cloud Architecture

In AWS, Azure, or GCP, OpenShift can create and manage cloud load balancers.

Internet
   │
   ▼
Cloud Load Balancer
   │
   ▼
Ingress Router Service
   │
   ▼
Router Pods
   │
   ▼
Application Services

Possible cloud load balancers include:

• AWS Network Load Balancer
• Azure Load Balancer
• Google Cloud Load Balancer

The exact implementation depends on:

• Installation method
• Platform integration
• Ingress Controller configuration
• Service annotations
• Internal or external exposure

Internal vs External Ingress

A cluster can have multiple Ingress Controllers.

Example:

Public Ingress Controller
    └── *.apps.ocp.example.com
Internal Ingress Controller
    └── *.internal.apps.ocp.example.com

This is useful for banking environments:

Internet-facing applications
           │
           ▼
WAF → Public LB → Public Router
Internal banking applications
           │
           ▼
Internal LB → Private Router

You can separate them by:

• DNS domain
• Node placement
• Route labels
• Namespace selectors
• Network zones
• Load balancer scope

TLS Termination

OpenShift Routes support several TLS models.

Edge termination

Client ──HTTPS──> Router ──HTTP──> Pod

The router terminates TLS.

Re-encrypt termination

Client ──HTTPS──> Router ──HTTPS──> Pod

The router decrypts and creates a new TLS connection to the backend.

This is commonly preferred for sensitive applications.

Passthrough termination

Client ──HTTPS──> Router ──HTTPS──> Pod

The router does not decrypt the traffic. TLS terminates in the application pod.

Load Balancer Health Checks

API health check

A common check is:

TCP 6443

or an HTTPS readiness endpoint:

https://<master>:6443/readyz

Ingress health check

Common checks include:

TCP 80
TCP 443

or the router health endpoint, depending on platform configuration.

The load balancer should remove unhealthy endpoints automatically.

Session Persistence

Most OpenShift applications should be stateless.

However, if an application requires sticky sessions, Routes support session affinity through cookies.

Example annotation:

metadata:
  annotations:
    haproxy.router.openshift.io/balance: source

Other algorithms include:

roundrobin
leastconn
source

Avoid depending heavily on session persistence when applications can instead store session state externally.

Load Balancing Algorithms

External load balancer

Typical algorithms:

• Round robin
• Least connections
• Source IP
• Weighted round robin

OpenShift router

Common options:

• Round robin
• Least connections
• Source-based persistence

Service load balancing

Kubernetes and OVN-Kubernetes distribute service traffic across available endpoints.

Failure Scenarios

One API server fails

master-1 fails
      │
      ▼
LB health check fails
      │
      ▼
master-1 removed from pool
      │
      ▼
Traffic continues to master-0 and master-2

One router pod fails

router pod fails
      │
      ▼
Load balancer or Service removes endpoint
      │
      ▼
Traffic continues through remaining router pods

One application pod fails

Application pod fails
      │
      ▼
Readiness probe fails
      │
      ▼
Endpoint removed from Service
      │
      ▼
Traffic goes to healthy pods

This shows the three load-balancing layers:

External LB health
       │
Router readiness
       │
Application pod readiness

Troubleshooting API Load Balancer

Test DNS:

dig api.cluster.example.com

Test the API:

curl -k https://api.cluster.example.com:6443/readyz

Test individual control-plane nodes:

curl -k https://master-0.example.com:6443/readyz
curl -k https://master-1.example.com:6443/readyz
curl -k https://master-2.example.com:6443/readyz

Check API pods:

oc get pods -n openshift-kube-apiserver -o wide

Look for:

• Incorrect backend ports
• Failed health checks
• Missing master node
• TLS inspection
• Idle timeout too low
• Load balancer SNAT exhaustion
• DNS pointing to the wrong VIP
• Firewall blocking port 6443

Troubleshooting Application Load Balancer

Check DNS:

dig payments.apps.cluster.example.com

Test the route:

curl -vk https://payments.apps.cluster.example.com

Check route:

oc get route -n banking
oc describe route payments -n banking

Check router pods:

oc get pods -n openshift-ingress -o wide

Check Ingress Controller:

oc get ingresscontroller -n openshift-ingress-operator
oc describe ingresscontroller default \
  -n openshift-ingress-operator

Check the application service and endpoints:

oc get svc -n banking
oc get endpoints -n banking
oc get endpointslices -n banking

Check pod readiness:

oc get pods -n banking
oc describe pod <pod-name> -n banking

The full troubleshooting path is:

DNS
 ↓
External load balancer
 ↓
Router
 ↓
Route
 ↓
Service
 ↓
EndpointSlice
 ↓
Pod readiness
 ↓
Application

Common Problems

Route returns 503

Usually means the router cannot find a healthy backend.

Check:

oc get endpointslices -n <namespace>
oc get pods -n <namespace>
oc describe route <route> -n <namespace>

Common causes:

• No ready pods
• Wrong Service selector
• Wrong target port
• Failed readiness probe
• Application not listening

API intermittently unavailable

Possible causes:

• Load balancer sending traffic to unhealthy master
• Incorrect health check
• Too-short timeout
• Control-plane API latency
• Network packet loss
• etcd latency
• TLS inspection device interference

Route works internally but not externally

Check:

• Wildcard DNS
• Firewall
• External VIP
• Load balancer pool
• Router node placement
• Port 80/443
• Public versus private ingress configuration

Banking Best Practices

For a regulated banking environment, I would use:

API traffic:
Admin network → Private API LB → Control-plane nodes
Public traffic:
Internet → DDoS protection → WAF → Public LB → Public routers
Internal traffic:
Corporate network → Internal LB → Internal routers

Additional controls:

• Separate API and application VIPs
• Private API endpoint
• Dedicated infra nodes for routers
• Multiple router replicas across failure domains
• Re-encrypt or passthrough TLS for sensitive applications
• WAF in front of public ingress
• Mutual TLS for partner applications
• Centralized load balancer and router access logs
• NetworkPolicies behind the router
• Health checks based on readiness
• Avoid TLS interception on OpenShift API traffic
• Monitor connection count, latency and backend health

Important Interview Distinction

An OpenShift architect should distinguish these three layers:

External Load Balancer
        │
        ▼
OpenShift Router
        │
        ▼
Kubernetes Service
        │
        ▼
Pods

Interview Answer

OpenShift uses load balancing at multiple layers. The API load balancer exposes port 6443 and distributes administrative and internal Kubernetes API traffic across the control-plane nodes. Port 22623 is used for the Machine Config Server during node provisioning. Application traffic is normally sent through a separate ingress load balancer on ports 80 and 443 to OpenShift router pods. The routers evaluate Route objects, terminate or pass through TLS, and forward requests to Kubernetes Services, which then distribute traffic to healthy pod endpoints.

On bare metal, the external load balancer might be F5, HAProxy or NetScaler. In cloud environments, OpenShift integrates with the cloud provider’s load-balancing services. For production, I would use separate API and application VIPs, redundant router replicas across failure domains, proper health checks, dedicated infrastructure nodes and separate public and internal Ingress Controllers. When troubleshooting, I follow the traffic path from DNS to the external load balancer, router, Route, Service, EndpointSlice, pod readiness and finally the application.

CRI-O is the default container runtime in OpenShift Container Platform (OCP). It is a lightweight, Kubernetes-native runtime designed specifically to run containers according to the Open Container Initiative (OCI) standards. Unlike Docker, CRI-O is not a full container platform—it only provides the functionality Kubernetes needs to start, stop, and manage containers.

For OpenShift interviews, CRI-O is one of the most important platform components to understand.

Where CRI-O Fits in OpenShift

                    User
                      │
                  oc CLI / API
                      │
               kube-apiserver
                      │
                 Scheduler
                      │
                 kubelet
                      │
                CRI (gRPC API)
                      │
                  CRI-O
          ┌───────────┴────────────┐
          │                        │
     runc / crun             Image Management
          │                        │
          └───────────┬────────────┘
                      │
               Linux Kernel
          (Namespaces + cgroups + SELinux)

Key point: Kubernetes never talks directly to containers—it communicates with CRI-O through the Container Runtime Interface (CRI).

Why Red Hat Uses CRI-O Instead of Docker

Originally Kubernetes supported Docker through Dockershim.

Kubernetes
     │
Dockershim
     │
 Docker

Problems:

• Docker included many features Kubernetes didn’t need.
• Additional translation layer (Dockershim).
• Higher resource consumption.
• Larger attack surface.
• More maintenance.

When Dockershim was removed from Kubernetes, OpenShift adopted CRI-O because it is:

• Kubernetes-native
• OCI-compliant
• Lightweight
• Easier to secure
• Easier to maintain

OpenShift Node Components

Every worker node typically runs:

Worker Node
RHCOS
│
├── kubelet
├── CRI-O
├── Machine Config Daemon
├── Network (OVN)
├── Node Exporter
└── Monitoring Agents

CRI-O is the service responsible for running all containers on the node.

Container Startup Flow

Suppose you create a Deployment:

oc create deployment nginx --image=nginx

The sequence is:

Deployment
↓
ReplicaSet
↓
Pod
↓
Scheduler
↓
Worker Node
↓
kubelet
↓
CRI-O
↓
runc/crun
↓
Linux Kernel
↓
Container Running

What Happens Internally

Imagine the pod is scheduled to Worker-1.

Step 1

Scheduler assigns:

Pod
↓
Worker-1

Step 2

kubelet notices:

Desired Pod
↓
Need container

Step 3

kubelet calls CRI-O:

CreateContainer()
StartContainer()

through the CRI gRPC API.

Step 4

CRI-O:

• pulls the image
• creates filesystem
• prepares networking
• mounts volumes
• creates namespaces
• configures cgroups
• applies SELinux labels

Step 5

CRI-O launches:

runc
or
crun

Step 6

The OCI runtime asks the Linux kernel to create:

• PID namespace
• Network namespace
• Mount namespace
• User namespace (if configured)
• IPC namespace
• cgroups

The application process then starts.

Components of CRI-O

CRI-O
├── CRI Server
├── Image Manager
├── Runtime Manager
├── Storage Manager
├── Networking
├── Logging
└── OCI Runtime

1. CRI Server

Receives requests from kubelet.

Examples:

RunPodSandbox()
CreateContainer()
StartContainer()
StopContainer()
RemoveContainer()

2. Image Manager

Responsible for:

Pull Image
↓
Verify Image
↓
Store Image
↓
Reuse Cached Image

Uses:

• Quay
• Internal registry
• Docker Hub
• Other OCI registries

3. Storage

Uses the Linux OverlayFS storage driver.

Typical storage location:

/var/lib/containers/storage

Example:

Image Layers
↓
OverlayFS
↓
Writable Layer
↓
Container

4. OCI Runtime

CRI-O does not execute containers directly.

It launches:

runc
or
crun

These create the container using Linux kernel primitives.

Why OpenShift Prefers crun

Recent OpenShift versions prefer crun because it offers:

• Faster startup
• Lower memory usage
• Better cgroup v2 support
• Better performance at scale

Networking

CRI-O does not configure networking itself.

It requests networking from Kubernetes.

CRI-O
↓
CNI Plugin
↓
OVN-Kubernetes
↓
Pod IP
↓
Network Ready

Storage

When a pod uses a PVC:

PVC
↓
CSI Driver
↓
StorageClass
↓
Volume
↓
CRI-O Mount
↓
Container

CRI-O mounts the volume before starting the container.

Security

One reason Red Hat chose CRI-O is security.

Every container starts with:

• SELinux labels
• cgroups
• namespaces
• seccomp profiles
• SCC restrictions
• capabilities dropped
• read-only root filesystem (when configured)

Example:

Container
↓
SELinux
↓
cgroups
↓
Namespaces
↓
Kernel

Logging

CRI-O captures stdout/stderr from containers.

Logs are typically stored under:

/var/log/containers/

These are then collected by logging agents such as Vector or Fluentd.

Flow:

Application
↓
stdout
↓
CRI-O
↓
Node Log
↓
Logging Stack

Image Pull

When an image is not cached:

Pod
↓
CRI-O
↓
Registry
↓
Download
↓
Verify
↓
Store
↓
Start Container

If cached:

Pod
↓
CRI-O
↓
Cached Image
↓
Container Starts

CRI-O vs Docker

Useful Commands

Check the CRI-O service:

systemctl status crio

View logs:

journalctl -u crio

Check running containers:

crictl ps

List images:

crictl images

Inspect pods:

crictl pods

Inspect a container:

crictl inspect <container-id>

Runtime information:

crictl info

Troubleshooting CRI-O

Pod stuck in ContainerCreating

Check:

oc describe pod <pod>

Then inspect:

• Image pull errors
• Volume mount failures
• Network setup failures
• CRI-O logs

journalctl -u crio

ImagePullBackOff

Verify:

crictl images

Check:

• Registry availability
• ImagePullSecrets
• DNS resolution
• Authentication

CrashLoopBackOff

Check:

oc logs <pod>
oc describe pod

Then verify:

• Application startup
• Memory limits
• Exit codes
• Liveness/readiness probes

CRI-O service down

systemctl status crio

If stopped:

systemctl restart crio

Then verify:

systemctl is-active crio

CRI-O and OpenShift Operators

The Machine Config Operator (MCO) manages CRI-O configuration across the cluster.

Example workflow:

MachineConfig
↓
Machine Config Operator
↓
Machine Config Daemon
↓
Update /etc/crio/
↓
Restart CRI-O (if required)
↓
Node Ready

This ensures every worker node has a consistent runtime configuration.

CRI-O Integration with Other OpenShift Components

               OpenShift Cluster

                    API Server
                        │
                    Scheduler
                        │
                     kubelet
                        │
                     CRI-O
        ┌───────────────┼────────────────┐
        │               │                │
     Image Pull     Storage Mount     Networking
        │               │                │
     Registry         CSI Driver      OVN-Kubernetes
        │               │                │
        └───────────────┼────────────────┘
                        │
                    OCI Runtime
                  (crun / runc)
                        │
                  Linux Kernel

Interview Answer (2-Minute Version)

CRI-O is OpenShift’s default container runtime and implements the Kubernetes Container Runtime Interface (CRI). Instead of Kubernetes talking directly to Docker, the kubelet communicates with CRI-O over gRPC. CRI-O is responsible for pulling OCI-compliant images, preparing the container filesystem, mounting storage, configuring networking through the CNI plugins, applying security settings such as SELinux labels, cgroups, seccomp profiles, and Security Context Constraints, and finally launching the container using an OCI runtime such as crun or runc. Unlike Docker, CRI-O is purpose-built for Kubernetes, making it smaller, faster, and more secure. In OpenShift, its configuration is managed centrally by the Machine Config Operator, ensuring consistent runtime settings across all worker nodes. For troubleshooting, I typically start with oc describe pod, then check journalctl -u crio, use crictl ps, crictl images, and crictl inspect, and verify image pulls, storage mounts, networking, and runtime health.

What is Ignition in OpenShift?

Ignition is the first-boot provisioning engine used by Red Hat Enterprise Linux CoreOS (RHCOS) to configure OpenShift nodes.

Think of Ignition as the tool that builds the operating system configuration before Kubernetes starts.

A simple way to remember it is:

Ignition provisions a new node only once, during its first boot. After that, the Machine Config Operator (MCO) manages ongoing configuration changes.

Where Ignition Fits in OpenShift

                   New Server / VM
                          │
                          ▼
                 Boot RHCOS ISO/PXE
                          │
                          ▼
                 Download Ignition File
                          │
                          ▼
                    Ignition Runs
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
 Configure disks      Configure users   Configure files
 Configure network    Install SSH keys  Configure systemd
                          │
                          ▼
                  Reboot (if needed)
                          │
                          ▼
                kubelet starts
                          │
                          ▼
              Node joins OpenShift
                          │
                          ▼
         Machine Config Operator takes over

Why Ignition Exists

Before Kubernetes can run, the operating system must be prepared.

A new node initially knows nothing about:

• The cluster
• SSH keys
• Storage layout
• Network configuration
• Certificates
• kubelet
• CRI-O configuration

Ignition performs all of this automatically during the first boot.

Without Ignition, every node would require manual configuration before joining the cluster.

What Ignition Configures

During first boot, Ignition can configure:

1. Disk Partitioning

Example:

Disk
 │
 ├── EFI
 ├── Boot
 ├── Root
 └── Data

It can:

• Partition disks
• Create filesystems
• Format volumes
• Mount storage

2. Users

Creates users such as:

core

Configures:

• SSH authorized keys
• User groups
• Password hash (if specified)

Example:

passwd:
  users:
    - name: core
      sshAuthorizedKeys:
        - ssh-rsa AAAA...

3. Files

Ignition can create files like:

/etc/motd
/etc/sysctl.conf
/etc/containers/registries.conf

Example:

storage:
  files:
    - path: /etc/example.conf
      contents:
        source: data:text/plain;base64,...

4. Directories

Example:

/opt/company/
/etc/custom/

5. systemd Services

Ignition can enable:

systemd

Example:

Enable service
↓
Start at boot

6. Certificates

Can install:

• CA certificates
• Internal PKI
• Registry certificates

7. kubelet Bootstrap

Creates the initial kubelet configuration so that it can join the cluster.

Ignition During Installation

A typical OpenShift installation looks like this:

openshift-install
        │
        ▼
Generate Ignition Files
        │
 ┌──────┼──────────┐
 │      │          │
 ▼      ▼          ▼
bootstrap.ign
master.ign
worker.ign

Each node type receives its own Ignition configuration.

Bootstrap Node

The bootstrap node receives:

bootstrap.ign

Its job is to:

• Start temporary control plane
• Bootstrap etcd
• Start Kubernetes
• Create permanent control-plane nodes

After installation:

Bootstrap node
↓
Removed

Master Nodes

Master nodes receive:

master.ign

This configures:

• kubelet
• CRI-O
• certificates
• etcd membership
• control plane services

Worker Nodes

Workers receive:

worker.ign

This prepares:

• kubelet
• CRI-O
• networking
• certificates

Workers then register with the cluster.

Example Boot Process

Imagine a brand-new worker VM.

Step 1

Power on

↓

Boot RHCOS

Step 2

Downloads:

worker.ign

Step 3

Ignition starts

Step 4

Creates:

Users
Files
Directories
Storage
SSH Keys
systemd

Step 5

Starts kubelet

Step 6

kubelet connects:

api-int.cluster.example.com

Step 7

Node joins cluster

Ignition Runs Only Once

One of the most important interview points.

First Boot
↓
Ignition
↓
Never runs again

After first boot:

Machine Config Operator
↓
Machine Config Daemon
↓
Node Updates

Ignition vs Machine Config Operator

Many interviewers ask this.

Think of it like:

Ignition
↓
Build the house
↓
Machine Config Operator
↓
Maintain the house

Ignition File Format

Ignition files are JSON.

Example:

{
  "ignition": {
    "version": "3.4.0"
  }
}

Generated automatically by:

openshift-install create ignition-configs

Normally administrators do not edit these JSON files manually.

Where Ignition Comes From

During installation:

Install Config
↓
openshift-install
↓
Ignition Generator
↓
bootstrap.ign
master.ign
worker.ign

Relationship with RHCOS

New RHCOS Node
↓
Ignition
↓
Configure OS
↓
Start kubelet
↓
Join OpenShift
↓
Machine Config Operator

Relationship with Machine Config

Later in cluster life:

Administrator
↓
MachineConfig
↓
Machine Config Operator
↓
Machine Config Daemon
↓
Node Updated

Ignition is not involved anymore.

Security

Ignition can securely provision:

• SSH keys
• Certificates
• Registry trust
• Users
• Files
• Kernel arguments

Since it runs before Kubernetes starts, it establishes the initial trusted configuration.

Typical Files Created

Examples include:

/etc/hostname
/etc/containers/
systemd units
authorized_keys
CA certificates
kubelet configuration

Common Troubleshooting

Node never joins cluster

Check:

journalctl -b

Look for:

Ignition failed
network unreachable
cannot download ignition
certificate errors

Verify Ignition

During installation:

openshift-install wait-for bootstrap-complete

If bootstrap never completes:

Often:

• bootstrap.ign incorrect
• network issue
• DNS
• load balancer
• certificates

Machine Config changes not applied

Remember:

Ignition
DOES NOT RUN AGAIN

The issue is almost certainly with:

• Machine Config Operator
• Machine Config Daemon
• MachineConfigPool

Ignition vs Cloud-init

A common interview comparison.

Best Practices

• Never manually edit generated .ign files unless you fully understand the implications.
• Use openshift-install to generate them.
• Use Ignition only for initial provisioning.
• Use MachineConfig for day-2 operating-system changes.
• Avoid manual changes to RHCOS that bypass the Machine Config Operator.

Interview Answer (2 Minutes)

Ignition is the first-boot provisioning engine used by RHCOS in OpenShift. During installation, the openshift-install utility generates three Ignition files—bootstrap.ign, master.ign, and worker.ign—which are consumed when each node boots for the first time. Ignition configures the operating system by creating users, installing SSH keys and certificates, partitioning disks, creating files and directories, configuring systemd services, and preparing the kubelet so the node can join the cluster.

A key point is that Ignition runs only once, before Kubernetes starts. After the node joins the cluster, ongoing operating-system configuration is managed by the Machine Config Operator and the Machine Config Daemon through MachineConfig resources. In interviews, I emphasize that Ignition is for day-0 provisioning, while the Machine Config Operator handles day-2 lifecycle management, updates, and configuration changes across the cluster.

Booting new VMs with Red Hat Enterprise Linux CoreOS (RHCOS) is a critical part of the OpenShift installation process. During installation, the installer provisions the VMs and boots them from an immutable RHCOS image. These nodes then join the cluster and are managed automatically.

OpenShift Boot Process with RHCOS

                User runs openshift-install
                           │
                           ▼
               Ignition configuration generated
                           │
                           ▼
         Infrastructure creates new RHCOS VM
                           │
                           ▼
              VM boots immutable RHCOS image
                           │
                           ▼
             Ignition downloads machine config
                           │
                           ▼
          Configure hostname, networking, SSH
                           │
                           ▼
               kubelet service starts
                           │
                           ▼
        kubelet connects to Kubernetes API
                           │
                           ▼
      CSR generated and approved automatically
                           │
                           ▼
          Node joins the OpenShift cluster
                           │
                           ▼
Machine Config Operator manages the node

Step 1 – Installer Creates Ignition Files

The installer generates three Ignition configurations:

bootstrap.ign
master.ign
worker.ign

Example:

openshift-install create ignition-configs

Each Ignition file tells the node:

• hostname
• SSH keys
• certificates
• kubelet configuration
• pull secret
• networking
• MachineConfig information

Step 2 – VM Boots RHCOS

The VM boots using the Red Hat CoreOS image.

Examples:

• VMware
• KVM
• Hyper-V
• AWS EC2
• Azure VM
• GCP Compute Engine

Unlike traditional Linux:

No kickstart
No cloud-init
No Ansible required

Instead it boots directly into RHCOS.

Step 3 – Ignition Runs (First Boot Only)

During the first boot:

systemd
      │
      ▼
Ignition

Ignition configures:

Filesystem

/var
/etc

Users

core

SSH Keys

Certificates

Networking

Kubelet configuration

CRI-O configuration

Machine Config

This happens only once.

Step 4 – Immutable Operating System

RHCOS is immutable.

/

is read-only.

Applications never modify the OS.

Instead they use

/var

Only Machine Config Operator changes the OS.

Step 5 – kubelet Starts

Systemd starts:

crio

then

kubelet

Example:

systemctl status kubelet

Output:

Active: active (running)

Step 6 – kubelet Contacts API Server

kubelet connects to

https://api.cluster.example.com:6443

It authenticates using bootstrap credentials.

Step 7 – CSR Created

Each node creates a Certificate Signing Request.

View pending requests:

oc get csr

Example:

csr-abc123 Pending
csr-def456 Pending

Normally OpenShift approves them automatically.

Approve manually:

oc adm certificate approve csr-abc123

Step 8 – Node Joins Cluster

Check:

oc get nodes

Example:

master-0 Ready
master-1 Ready
master-2 Ready
worker-0 Ready
worker-1 Ready

Step 9 – Machine Config Operator Takes Over

After joining:

Machine Config Operator

manages:

Kernel arguments

OS updates

Certificates

Kubelet config

CRI-O config

Network files

SSH keys

You never manually patch RHCOS.

What Actually Boots?

RHCOS includes:

Linux Kernel
systemd
CRI-O
kubelet
podman
rpm-ostree
Ignition
NetworkManager
SELinux
OpenShift components

Boot Sequence Inside RHCOS

BIOS / UEFI
      │
      ▼
GRUB
      │
      ▼
Linux Kernel
      │
      ▼
initramfs
      │
      ▼
Ignition
      │
      ▼
systemd
      │
      ▼
NetworkManager
      │
      ▼
CRI-O
      │
      ▼
kubelet
      │
      ▼
API Server
      │
      ▼
Node Ready

Troubleshooting RHCOS Boot

View Ignition logs

journalctl -b -u ignition

Check kubelet

journalctl -u kubelet

Check CRI-O

journalctl -u crio

Verify Machine Config

oc get mcp

Healthy output:

NAME     UPDATED
master   True
worker   True

Check Machine Config Daemon

oc get pods -n openshift-machine-config-operator

Check Ignition file retrieval

On the node:

journalctl -b | grep ignition

Check node status

oc describe node worker-0

Interview Answer (2-minute version)

In OpenShift, new VMs boot from an immutable Red Hat CoreOS image rather than a traditional Linux installation. During installation, the openshift-install utility generates Ignition configuration files for bootstrap, control plane, and worker nodes. On first boot, the Ignition service configures networking, SSH keys, certificates, kubelet, and other system settings. After that, systemd starts CRI-O and the kubelet, which contacts the Kubernetes API server, generates a certificate signing request (CSR), and joins the cluster once approved. From that point onward, the Machine Config Operator (MCO) manages all operating system configuration and updates using rpm-ostree, ensuring every RHCOS node remains consistent, immutable, and centrally managed. This design improves security, simplifies lifecycle management, and reduces configuration drift across the cluster.

For a Senior OpenShift Platform Engineer/Architect interview, using oc debug node is the standard and supported method to troubleshoot node-level issues because SSH access to RHCOS nodes is often restricted.

Scenario

Problem:
Users report:

• oc get pods is taking 10-20 seconds.
• Pods remain in ContainerCreating.
• API server latency is increasing.
• etcd reports slow fsync operations.

You suspect disk contention on one of the control-plane nodes.

Step 1 – Identify the affected node

First determine which master has the issue.

oc get nodes

Example:

NAME        STATUS
master-0    Ready
master-1    Ready
master-2    Ready

From Grafana you notice:

master-1
etcd_disk_wal_fsync_duration_seconds
P99 = 180 ms

That immediately suggests storage latency on master-1.

Step 2 – Debug the node

Instead of SSH:

oc debug node/master-1

Output:

Creating debug namespace...
Starting pod...
To use host binaries, run:
chroot /host

Enter the host OS:

chroot /host

Now you’re inside the RHCOS operating system.

Step 3 – Check disk utilization (iostat)

First verify the disks.

lsblk

Example:

NAME    SIZE
sda     300G
├─sda1
├─sda2
└─sda3

Run:

iostat -x 1 10

Example output:

Device    r/s    w/s   rkB/s  wkB/s  await svctm %util aqu-sz
sda       12    520    300    9500   65.2   0.9 100.0   8.5

How to interpret

await

65 ms

Average I/O latency.

Healthy SSD:

<5 ms

Good:

<10 ms

Bad:

30-100 ms

Very bad:

>100 ms

%util

100%

Means the device is busy almost continuously.

High utilization together with high latency indicates storage saturation.

aqu-sz

8.5

Average queue depth.

Healthy:

0-1

Large values mean requests are waiting.

w/s

520 writes/sec

High write rates combined with elevated latency can affect etcd because it performs synchronous writes to its WAL.

Step 4 – Historical performance (sar)

Current activity is only part of the picture.

Run:

sar -d 1 10

Example:

DEV       tps    rkB/s   wkB/s   await
sda       480      50    9200      72

This confirms sustained storage latency.

Check CPU wait:

sar -u 1 10

Example:

CPU %user %system %iowait
all  15      12      28

%iowait

28%

Healthy:

<5%

High values indicate CPUs are idle waiting for storage.

Check system load:

sar -q 1 10

Example:

runq-sz
3
plist-sz
2100
ldavg-1
18

A high load average together with elevated I/O wait often points to storage rather than CPU as the bottleneck.

Step 5 – Find which process is causing I/O (pidstat)

Run:

pidstat -d 1 10

Example:

UID PID COMMAND     kB_rd/s  kB_wr/s
0  2145 etcd          0      2200
0  8120 backupd       0      6500
0  4211 rsync         0      4500
0  5001 fluentd       0      1100

Immediately you can see:

backupd
6500 KB/sec

That is a strong indication the backup process is competing with etcd for disk I/O.

Another example:

PID COMMAND
crio

Large writes from CRI-O could indicate:

• image extraction
• image garbage collection
• many pod creations
• heavy logging

Step 6 – Check etcd

Verify etcd health:

oc get pods -n openshift-etcd

Then:

oc rsh -n openshift-etcd <etcd-pod>
etcdctl endpoint health --cluster

Review logs:

oc logs -n openshift-etcd <pod> -c etcd

Typical messages:

slow fdatasync
leader changed
request timed out
apply request took too long

These correlate well with disk contention.

Step 7 – Look for filesystem errors

Run:

dmesg | grep -i error

or

journalctl -k

Look for:

I/O errors
NVMe timeout
SCSI timeout
filesystem errors

Step 8 – Check disk space

df -h

Filesystem
/
95%

Nearly full filesystems can increase write latency, especially if log or container storage is consuming most of the space.

Also check inodes:

df -i

Step 9 – Check mounted filesystems

findmnt

Confirm where /var/lib/etcd resides and whether it shares storage with other high-I/O workloads.

Step 10 – Correlate with Prometheus

Compare host observations with metrics such as:

etcd_disk_wal_fsync_duration_seconds

etcd_disk_backend_commit_duration_seconds

node_disk_io_time_seconds_total

node_disk_io_time_weighted_seconds_total

If iostat shows high latency and the etcd WAL fsync metric spikes at the same time, the storage subsystem is the likely bottleneck.

Real Production Example

Imagine every night at 02:00:

API latency increases
↓
etcd WAL latency rises
↓
Users cannot deploy applications
↓
Pods remain Pending

Using pidstat:

backup-agent
12 MB/sec writes

Using iostat:

await = 140 ms
util = 100%

Using Grafana:

etcd WAL fsync
150 ms

The root cause is a scheduled backup saturating the datastore used by the control-plane nodes.

Resolution:

• Move backups to another storage system or reschedule them.
• Isolate etcd on dedicated low-latency SSD/NVMe storage.
• Ensure sufficient IOPS and throughput.
• Validate recovery by confirming WAL fsync latency, API response times, and ClusterOperator health return to normal.

Interview Answer (2 Minutes)

“When troubleshooting suspected storage-related issues in OpenShift, I start with oc debug node because direct SSH access to RHCOS nodes is often restricted. After entering the host with chroot /host, I use iostat -x to examine disk latency, queue depth, and device utilization. I’m particularly interested in await, %util, and aqu-sz, as high values indicate storage contention. Next, I use sar -d and sar -u to determine whether the problem is sustained over time and whether CPU is spending excessive time in I/O wait. Then I run pidstat -d to identify which processes are generating heavy disk activity, such as backup software, rsync, CRI-O image extraction, or logging agents. I correlate these findings with Prometheus metrics like etcd_disk_wal_fsync_duration_seconds and API latency. If they align, I’ve confirmed a storage bottleneck affecting etcd. The remediation is to eliminate competing disk workloads, provide dedicated low-latency storage for etcd, and then verify recovery by checking API latency, etcd health, and cluster operator status.”

What a Complete OCP Backup Actually Contains

The Two Files — Why Both Are Required

Think of it this way:

snapshot.db                    = THE DATA (what the cluster knows)
static_kuberesources.tar.gz    = THE KEYS TO READ IT (how to access/run it)

Neither file alone is sufficient for a restore. You need both.

File 1 — snapshot_<timestamp>.db

What it is

A raw BoltDB binary dump of the entire etcd key-value store — a complete photograph of all cluster state at one point in time.

What’s inside

etcd stores everything under /registry/ prefixes. The snapshot contains every single key:

/registry/pods/
  └── default/my-app-7d9f8b-xkp2q
  └── kube-system/coredns-abc123
  └── production/frontend-pod
/registry/deployments/
  └── production/frontend
  └── production/backend
/registry/services/
  └── default/kubernetes
  └── production/frontend-svc
/registry/secrets/
  └── kube-system/bootstrap-token-xxxxx
  └── production/db-credentials
  └── openshift-ingress/router-certs
/registry/configmaps/
  └── kube-system/kube-proxy
  └── openshift-config/cluster-config-v1
/registry/clusterroles/
/registry/clusterrolebindings/
/registry/rolebindings/
/registry/namespaces/
/registry/nodes/
/registry/persistentvolumes/
/registry/persistentvolumeclaims/
/registry/apiextensions.k8s.io/customresourcedefinitions/
  └── machineconfigpools.machineconfiguration.openshift.io
  └── clusterversions.config.openshift.io
/registry/operators.coreos.com/
/registry/config.openshift.io/
  └── clusterversion/cluster          ← your OCP version + upgrade history
  └── network/cluster                 ← SDN config (OVN/SDN type, CIDRs)
  └── ingress/cluster
  └── authentication/cluster
  └── oauth/cluster

What it does NOT contain

❌ The actual container images (those live in registries)
❌ Data inside PersistentVolumes (databases, file storage)
❌ The OS-level config of nodes
❌ Anything not represented as a Kubernetes object

Encryption caveat

If etcd encryption at rest is enabled:

Values are encrypted in the snapshot  ← secrets, configmaps are ciphertext
Keys (paths) are NOT encrypted        ← /registry/secrets/... names are visible

The decryption keys live inside static_kuberesources.tar.gz — which is exactly why the two files must be stored separately when encryption is on.

File 2 — static_kuberesources_<timestamp>.tar.gz

What it is

A tar archive of the on-disk configuration, manifests, and TLS certificates for the 4 control plane static pods, at the specific revision that was running when the backup was taken.

The 4 components captured

static-pod-resources/
  ├── etcd-pod-<N>/
  ├── kube-apiserver-pod-<N>/
  ├── kube-controller-manager-pod-<N>/
  └── kube-scheduler-pod-<N>/

Drill-down: etcd-pod-<N>/

etcd-pod-3/
  ├── etcd-pod.yaml                      ← static pod manifest (how kubelet runs etcd)
  ├── secrets/
  │   └── etcd-all-certs/
  │       ├── etcd-peer-master-0.crt     ← peer cert (etcd member-to-member)
  │       ├── etcd-peer-master-0.key     ← peer private key
  │       ├── etcd-serving-master-0.crt  ← server cert (clients connect to this)
  │       ├── etcd-serving-master-0.key
  │       ├── etcd-metric-client.crt     ← Prometheus scraping cert
  │       └── etcd-metric-client.key
  └── configmaps/
      ├── etcd-ca/
      │   └── ca-bundle.crt              ← CA that signed all etcd certs
      ├── etcd-metrics-ca/
      └── etcd-scripts/
          ├── etcd.env                   ← env vars: IPs, cert paths, endpoints
          └── etcd-common-tools          ← shared shell functions

Drill-down: kube-apiserver-pod-<N>/

kube-apiserver-pod-7/
  ├── kube-apiserver-pod.yaml
  ├── secrets/
  │   ├── etcd-client/                   ← cert the API server uses to talk to etcd
  │   ├── server-crt/                    ← API server's own TLS cert
  │   ├── bound-service-account-signing-key/
  │   │   └── service-account.key        ← signs ServiceAccount JWT tokens ⚠️
  │   └── node-kubeconfigs/              ← kubeconfig for kubelet auth
  └── configmaps/
      ├── client-ca/                     ← CA for client cert auth
      ├── etcd-serving-ca/               ← CA to verify etcd's cert
      ├── audit-policies/                ← audit log policy
      └── kube-apiserver-server-ca/

The service-account.key is critical. Every pod’s service account token is signed with this key. If this key changes on restore and doesn’t match what’s in etcd, all in-cluster service account authentication breaks.

Drill-down: kube-controller-manager-pod-<N>/

kube-controller-manager-pod-8/
  ├── kube-controller-manager-pod.yaml
  ├── secrets/
  │   ├── etcd-client/                   ← talks to etcd for leader election
  │   ├── kube-controller-manager-client-cert-key/
  │   └── service-account-private-key/   ← issues new service account tokens
  └── configmaps/
      ├── client-ca/
      └── cluster-config-v1/             ← network CIDRs, cloud provider config

Drill-down: kube-scheduler-pod-<N>/

kube-scheduler-pod-6/
  ├── kube-scheduler-pod.yaml
  ├── secrets/
  │   └── kube-scheduler-client-cert-key/
  └── configmaps/
      ├── client-ca/
      └── policy-configmap/              ← scheduling policies (if customized)

How the Two Files Work Together on Restore

┌─────────────────────────────────────────────────────────┐
│              cluster-restore.sh                         │
│                                                         │
│  snapshot.db  ──────────────────────► etcd data store  │
│                                           │             │
│  static_kuberesources.tar.gz              │             │
│    ├── etcd certs ──────────────► etcd can start  ◄────┘
│    ├── API server certs ────────► API server can start  │
│    ├── service-account.key ─────► tokens still valid    │
│    └── pod manifests ───────────► kubelet knows what    │
│                                   to run                │
└─────────────────────────────────────────────────────────┘

Without snapshot.db → you have certificates but no cluster state to restore. Without static_kuberesources.tar.gz → you have data but can’t decrypt it, can’t start etcd, can’t authenticate.

Visual Summary

snapshot.db
┌────────────────────────────────────────────┐
│  /registry/pods/...          ALL objects   │
│  /registry/deployments/...   ALL config    │
│  /registry/secrets/...       ALL secrets   │  ← may be encrypted
│  /registry/nodes/...         ALL state     │
│  /registry/...               Everything    │
│                                            │
│  Size: typically 50MB–500MB                │
│  Format: BoltDB binary                     │
└────────────────────────────────────────────┘
static_kuberesources.tar.gz
┌────────────────────────────────────────────┐
│  etcd-pod-N/                               │
│    ├── manifest (how to run etcd)          │
│    └── TLS certs (peer, server, client)    │
│                                            │
│  kube-apiserver-pod-N/                     │
│    ├── manifest                            │
│    ├── etcd-client cert                    │
│    └── service-account signing key ⚠️      │
│                                            │
│  kube-controller-manager-pod-N/            │
│    └── manifest + certs                    │
│                                            │
│  kube-scheduler-pod-N/                     │
│    └── manifest + certs                    │
│                                            │
│  Size: typically 1MB–5MB                   │
│  Permissions: chmod 600 (root only)        │
└────────────────────────────────────────────┘

The One Thing to Never Forget

snapshot.db captures what the cluster is. static_kuberesources.tar.gz captures how to run the cluster and how to trust everything in it.

A restore without the tar.gz matching the snapshot.db is guaranteed to fail — the revision numbers, certificate serials, and signing keys must all be consistent from the same point in time.

Here’s a deep-dive explanation of every part of the script.

cluster-backup.sh — Complete Internals Explained

The script lives at /usr/local/bin/cluster-backup.sh on every control plane node. It is maintained as a component of the etcd Cluster Operator and is a wrapper around the etcdctl snapshot save command. Here’s every line of it dissected.

Invocation

/usr/local/bin/cluster-backup.sh [--force] <path-to-backup-dir>
# Example
/usr/local/bin/cluster-backup.sh /home/core/backup

It produces two output files:

/home/core/backup/snapshot_2025-03-01_143022.db
/home/core/backup/static_kuberesources_2025-03-01_143022.tar.gz

Section 1 — Shell Safety Flags

set -o errexit   # exit immediately if any command fails
set -o pipefail  # catch failures inside pipes (cmd1 | cmd2)
set -o errtrace  # ERR traps are inherited by functions/subshells

These three together mean: any error anywhere = immediate exit. No silent failures.

Section 2 — Root Check

if [[ $EUID -ne 0 ]]; then
  echo "This script must be run as root"
  exit 1
fi

The script needs access to:

• /etc/kubernetes/ — TLS certificates and manifests (root-only)
• /var/lib/etcd/ — etcd data directory (root-only)
• etcd’s mTLS-protected API endpoint

So it must run as root. On RHCOS you use sudo or oc debug node/....

Section 3 — --force Flag

IS_DIRTY=""
if [ "$1" == "--force" ]; then
  IS_DIRTY="__POSSIBLY_DIRTY__"
  shift
fi

--force skips the operator-health checks (explained below). It marks the output files with __POSSIBLY_DIRTY__ in their filename as a warning:

snapshot__POSSIBLY_DIRTY__.db
static_kuberesources__POSSIBLY_DIRTY__.tar.gz

This is a deliberate signal: “this backup was taken while the cluster was in an uncertain state.” Never use --force backups for production restores unless it’s your only option.

Section 4 — Backup Directory Setup

BACKUP_DIR="$1"
DATESTRING=$(date "+%F_%H%M%S")
BACKUP_TAR_FILE=${BACKUP_DIR}/static_kuberesources_${DATESTRING}${IS_DIRTY}.tar.gz
SNAPSHOT_FILE="${BACKUP_DIR}/snapshot_${DATESTRING}${IS_DIRTY}.db"
trap 'rm -f ${BACKUP_TAR_FILE} ${SNAPSHOT_FILE}' ERR

The trap is critical: if anything fails mid-run, the partial/corrupt output files are automatically deleted. You’ll never be left with a half-written snapshot that looks valid but isn’t.

Section 5 — Load Environment & Certificates

source_required_dependency \
  /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/etcd.env
source_required_dependency \
  /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/etcd-common-tools

source_required_dependency is a small guard function that verifies the file exists before sourcing it:

function source_required_dependency {
  local src_path="$1"
  if [ ! -f "${src_path}" ]; then
    echo "required dependencies not found, please ensure this script is run
          on a node with a functional etcd static pod"
    exit 1
  fi
  source "${src_path}"
}

What does etcd.env contain?

It exports the environment variables etcdctl needs to connect to the local etcd member via mTLS:

# Inside etcd.env (approximate content):
export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt
export ETCDCTL_CERT=/etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-master-0.crt
export ETCDCTL_KEY=/etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-master-0.key
export NODE_NODE_ENVVAR_NAME_IP=192.168.1.10   # this node's IP
export CONFIG_FILE_DIR=/etc/kubernetes

What does etcd-common-tools contain?

Shared shell functions used by both backup and restore scripts, including:

• dl_etcdctl — downloads the correct etcdctl binary
• check_snapshot_status — validates the snapshot after saving

Section 6 — Certificate Path Fallback

if [ ! -f "${ETCDCTL_CACERT}" ]; then
  echo "Certificate ${ETCDCTL_CACERT} is missing. Checking in different directory"
  export ETCDCTL_CACERT=$(echo ${ETCDCTL_CACERT} | sed -e "s|static-pod-certs|static-pod-resources/etcd-certs|")
  export ETCDCTL_CERT=$(echo ${ETCDCTL_CERT}   | sed -e "s|static-pod-certs|static-pod-resources/etcd-certs|")
  export ETCDCTL_KEY=$(echo ${ETCDCTL_KEY}     | sed -e "s|static-pod-certs|static-pod-resources/etcd-certs|")
  ...
fi

Why is this needed?

The default cert paths in etcd.env point to static-pod-certs/ — a symlink that exists inside the running etcd container. When running the script directly on the host (outside the container), those symlinks don’t resolve. This block remaps the paths to the actual host-side directory static-pod-resources/etcd-certs/, where the same certs are stored.

Inside container:  /etc/kubernetes/static-pod-certs/...
On host directly:  /etc/kubernetes/static-pod-resources/etcd-certs/...
                   (same certs, different path)

Section 7 — Back Up Static Pod Resources

backup_latest_kube_static_resources "${BACKUP_TAR_FILE}"

This is the tar.gz part of the backup. The function:

function backup_latest_kube_static_resources {
  local backup_tar_file="$1"
  local backup_resource_list=("kube-apiserver" "kube-controller-manager" "kube-scheduler" "etcd")
  local latest_resource_dirs=()
  for resource in "${backup_resource_list[@]}"; do
    # 1. Verify the static pod manifest exists
    if [ ! -f "/etc/kubernetes/manifests/${resource}-pod.yaml" ]; then
      echo "error finding manifests for the ${resource} pod."
      exit 1
    fi
    # 2. Find which revision is currently active
    local latest_resource
    latest_resource=$(grep -o -m 1 \
      "/etc/kubernetes/static-pod-resources/${resource}-pod-[0-9]*" \
      "/etc/kubernetes/manifests/${resource}-pod.yaml")
    # 3. Check the operator isn't mid-rollout (unless --force)
    if [ "${IS_DIRTY}" == "" ]; then
      check_if_operator_is_progressing "${resource}"
    fi
    latest_resource_dirs+=("${latest_resource#${CONFIG_FILE_DIR}/}")
  done
  # 4. tar all four resource dirs together
  tar -cpzf "$backup_tar_file" -C "${CONFIG_FILE_DIR}" "${latest_resource_dirs[@]}"
  chmod 600 "$backup_tar_file"
}

Step by step:

① Verify manifests exist — if any of the 4 control plane pods don’t have a manifest file at /etc/kubernetes/manifests/, the node’s kubelet isn’t managing them and the script aborts.

② Find the latest revision — OCP uses a revision-based rollout system. Each time a static pod is updated, a new numbered directory is created:

/etc/kubernetes/static-pod-resources/
  etcd-pod-1/          ← old revision
  etcd-pod-2/          ← old revision
  etcd-pod-3/          ← CURRENT (referenced by manifest)
  kube-apiserver-pod-7/
  kube-controller-manager-pod-8/
  kube-scheduler-pod-6/

The script greps the active manifest to find which numbered revision is in use. This ensures you back up the revision that is actually running, not a stale one.

③ Check operator not progressing — calls:

function check_if_operator_is_progressing {
  local operator="$1"
  if [ ! -f "${KUBECONFIG}" ]; then
    echo "Valid kubeconfig is not found. Exiting!"
    exit 1
  fi
  progressing=$(oc get co "${operator}" \
    -o jsonpath='{.status.conditions[?(@.type=="Progressing")].status}')
  if [ "$progressing" != "False" ]; then
    echo "Currently the $operator operator is progressing.
          A reliable backup requires that a rollout is not in progress. Aborting!"
    exit 1
  fi
}

If any of the 4 operators is mid-rollout (Progressing=True), a backup taken now would capture a half-applied state — inconsistent between etcd data and on-disk manifests. The script refuses unless --force is passed.

④ tar with permissions — packages all 4 resource directories together. chmod 600 restricts the tar to root-only, since it may contain TLS private keys and encryption keys (if etcd encryption at rest is enabled).

What exactly is inside the tar?

static-pod-resources/etcd-pod-3/
  ├── etcd-pod.yaml                  ← pod manifest
  ├── secrets/
  │   ├── etcd-all-certs/            ← peer, server, client certs + keys
  │   └── etcd-metric-client/
  └── configmaps/
      ├── etcd-ca/                   ← CA cert bundle
      └── etcd-scripts/             ← etcd.env, common-tools
static-pod-resources/kube-apiserver-pod-7/
  ├── kube-apiserver-pod.yaml
  ├── secrets/                       ← API server TLS certs + service account keys
  └── configmaps/                    ← kubeconfigs, audit policy, etc.
static-pod-resources/kube-controller-manager-pod-8/  ...
static-pod-resources/kube-scheduler-pod-6/  ...

Section 8 — Download etcdctl

dl_etcdctl

This function (from etcd-common-tools) downloads the etcdctl binary that matches the exact etcd version running in the cluster. It doesn’t use a system-installed etcdctl to avoid version mismatch.

The binary is extracted from the running etcd container image itself via crictl or directly from the image layers, ensuring perfect version alignment.

Section 9 — Take the etcd Snapshot

ETCDCTL_ENDPOINTS="https://${NODE_NODE_ENVVAR_NAME_IP}:2379" \
  etcdctl snapshot save "${SNAPSHOT_FILE}"

This is the core operation. It:

1. Connects to the local etcd member only (not the cluster endpoint) — important because it avoids redirects and ensures you’re snapshotting from a member that is up
2. Uses the mTLS certs from etcd.env to authenticate
3. Calls etcd’s built-in snapshot API — which does a consistent read of the entire BoltDB key-value store
4. Writes the raw BoltDB file to snapshot_<timestamp>.db

Why local endpoint (NODE_IP:2379) and not a load-balanced endpoint?

• The snapshot is a point-in-time consistent read — it needs to come from one member
• etcd’s Raft protocol ensures the member has all committed entries before snapshotting
• Using a VIP/LB might land on a different member per retry, causing inconsistency

What’s inside the .db file?

It’s a raw BoltDB database file containing every key-value pair in etcd:

/registry/pods/default/my-app-xyz
/registry/deployments/production/frontend
/registry/secrets/kube-system/bootstrap-token-xyz
/registry/clusterroles/cluster-admin
... (every Kubernetes object ever created)

If etcd encryption is enabled, the snapshot also contains the encryption keys for the etcd snapshot data.

Section 10 — Validate the Snapshot

check_snapshot_status "${SNAPSHOT_FILE}"
snapshot_failed=$?
if [[ $snapshot_failed -eq 1 ]]; then
  echo "snapshot failed with exit code ${snapshot_failed}"
  exit 1
fi

check_snapshot_status (from etcd-common-tools) runs:

etcdctl snapshot status "${SNAPSHOT_FILE}" --write-out=table

Which verifies the snapshot’s internal hash and outputs something like:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 1bf371f1 |   294858 |       5891 |     120 MB |
+----------+----------+------------+------------+

The cluster-backup.sh script confirms the snapshot’s validity. If the hash doesn’t match the data, the file is corrupt and gets deleted by the trap.

Complete Flow Summary

cluster-backup.sh /home/core/backup
        │
        ├─ 1. Root check
        ├─ 2. Parse --force flag
        ├─ 3. Create backup dir
        ├─ 4. source etcd.env  → loads ETCDCTL_CACERT/CERT/KEY + NODE_IP
        ├─ 5. source etcd-common-tools  → loads dl_etcdctl, check_snapshot_status
        ├─ 6. Fix cert paths if running outside container
        │
        ├─ 7. backup_latest_kube_static_resources()
        │       ├─ For each of 4 static pods:
        │       │     ├─ Verify manifest exists
        │       │     ├─ Find current revision number
        │       │     └─ Check operator not Progressing
        │       └─ tar -czf static_kuberesources_<ts>.tar.gz (chmod 600)
        │
        ├─ 8. dl_etcdctl  → downloads matching etcdctl binary
        ├─ 9. etcdctl snapshot save  → writes snapshot_<ts>.db
        └─ 10. check_snapshot_status  → validates hash
                    ├─ OK  → "snapshot db and kube resources are successfully saved"
                    └─ FAIL → trap deletes both files, exits 1

Important Operational Rules

OCP Control Plane Disaster Recovery — Deep Dive

The Big Picture

In OpenShift, the control plane is the brain. If all 3 master nodes are lost simultaneously, the cluster is completely dead — no API, no scheduling, no networking decisions. This recovery strategy is the step-by-step process to resurrect it from a backup.

Step-by-Step Breakdown

1. Disaster

What happened:

• All 3 control plane nodes are gone (hardware failure, accidental termination, storage corruption, datacenter incident)
• etcd quorum is lost — etcd requires 2 of 3 members to be healthy to function
• The Kubernetes API server goes dark
• Workers enter a “disconnected” state — they keep running existing pods but can’t receive new instructions

Impact:

❌ oc commands fail — API unreachable
❌ No new pods can be scheduled
❌ No config changes possible
❌ Ingress/DNS still works (workers still running) but degrades over time

2. Provision New Control Plane VMs

What this means:

• Spin up 3 new VMs that will become the new master nodes
• Must match the original specs (CPU, RAM, disk, network)
• Assign the same hostnames and IPs as the original masters — this is critical

Why same IPs/hostnames?

etcd certificates are bound to specific IPs/hostnames.
Workers have these hardcoded in their kubeconfig.
Changing them = cascading cert failures.

Checklist:

# Original masters typically look like:
master-0.cluster.example.com  → 192.168.1.10
master-1.cluster.example.com  → 192.168.1.11
master-2.cluster.example.com  → 192.168.1.12
# New VMs must reuse these exact identities

3. Install Matching RHCOS Version

What this means:

• Boot new VMs with Red Hat CoreOS (RHCOS) — the only supported OS for OCP control plane nodes
• Version must exactly match the original cluster version (e.g., 4.14.12)

Why it must match:

OCP tightly couples:
  cluster version  ←→  RHCOS version  ←→  etcd version  ←→  kubelet version
Mismatch = incompatible binaries, API schema drift, etcd data format issues

How to get the right version:

# On a surviving worker or from backup:
oc get clusterversion -o jsonpath='{.items[0].status.desired.version}'
# Get RHCOS ISO for that exact version from:
# https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/

RHCOS is immutable — it’s not installed like traditional Linux. It’s written directly to disk via coreos-installer.

4. Restore etcd from Snapshot + Static Pod Resources

This is the most critical and complex step. Let’s break it down.

What is etcd?

etcd = the cluster's database
       stores ALL cluster state:
         - all Kubernetes objects (pods, deployments, services...)
         - RBAC rules, secrets, configmaps
         - custom resources (CRDs)
         - cluster configuration

What is a Snapshot?

A point-in-time binary backup of the entire etcd database, taken via:

# Taken on a healthy master node:
/usr/local/bin/cluster-backup.sh /home/core/backup
# Produces:
# snapshot_<timestamp>.db          ← etcd data
# static_kuberesources_<timestamp>.tar.gz  ← static pod manifests + certs

What are Static Pod Resources?

These are the manifests + certificates for the control plane components that run as static pods (managed by kubelet directly, not by the API server):

/etc/kubernetes/manifests/
  ├── etcd-pod.yaml               ← etcd itself
  ├── kube-apiserver-pod.yaml     ← API server
  ├── kube-controller-manager-pod.yaml
  └── kube-scheduler-pod.yaml
/etc/kubernetes/static-pod-resources/
  └── (TLS certs, kubeconfigs for each component)

The Restore Process:

# 1. Copy snapshot to the recovery master
scp snapshot.db core@master-0:/home/core/
# 2. Run the restore script (on the new master-0 only)
sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup
# What this script does internally:
#   a) Stops existing etcd static pod
#   b) Moves old data dir aside
#   c) Runs: etcdctl snapshot restore snapshot.db
#   d) Restores static pod manifests + certs from tar.gz
#   e── Restarts etcd as a single-member cluster (peer URLs point to self)
#   f) Waits for etcd to become healthy
# 3. Force new etcd member joins from master-1 and master-2
#    (after their RHCOS is installed, they re-join the etcd cluster)

Why single-member first?

Restoring directly to 3 members causes split-brain.
Safe sequence:
  master-0 → restore (becomes sole etcd member)
  master-1 → joins as new peer
  master-2 → joins as new peer
  Result: healthy 3-member etcd cluster

5. API Server Available Again

What happens:

• Once etcd is healthy, the kube-apiserver static pod starts successfully
• It reads its config from etcd and begins serving requests
• The OpenShift API server (openshift-apiserver) also comes up

Verify:

# From recovery host
export KUBECONFIG=/etc/kubernetes/admin.kubeconfig
oc get nodes          # should return (NotReady initially, that's OK)
oc get pods -A        # cluster state re-appears from etcd
oc get etcd -o=jsonpath='{range .items[0].status.conditions[*]}{.type}{" "}{.status}{"\n"}{end}'

What the API server does now:

Reads all objects from restored etcd snapshot
Reconstructs its in-memory watch cache
Begins serving /api and /apis endpoints
Authentication/RBAC enforced from restored data

6. Control Plane Operators Recover

What this means:

• The Cluster Version Operator (CVO) wakes up and starts reconciling
• All Cluster Operators begin self-healing

Key operators that recover:

openshift-kube-apiserver-operator      → manages API server rollout
openshift-kube-controller-manager-operator
openshift-kube-scheduler-operator
openshift-etcd-operator                → adds master-1, master-2 back to etcd
openshift-authentication-operator
openshift-ingress-operator
openshift-dns-operator
openshift-network-operator            → rebuilds SDN/OVN state

Monitor recovery:

# Watch operators come back
watch oc get clusteroperators
# All should eventually show:
# AVAILABLE=True  PROGRESSING=False  DEGRADED=False
# etcd operator re-adds members:
oc get etcd -o=jsonpath='{.items[0].status.conditions}'

This phase can take 15–45 minutes — operators are re-applying their managed configurations across the cluster.

7. Workers Reconnect

What was happening to workers during the outage:

Workers run a kubelet process that constantly tries to reach the API server.
During outage:
  - Existing pods kept running (containers don't need the API to stay alive)
  - No new pods could be scheduled
  - Liveness/readiness probes kept running locally
  - After ~5 min: node status becomes "Unknown" in etcd (nobody updating it)
  - After 40s default: pods on unknown nodes get eviction tolerations triggered

When API returns:

# Kubelet on each worker re-registers with the API
# Node status flips from Unknown → Ready
watch oc get nodes
# Certificate rotation may be needed if outage was long:
oc get csr | grep Pending
oc adm certificate approve <csr-name>
# Or approve all pending CSRs at once:
oc get csr -o name | xargs oc adm certificate approve

Why CSR approval matters: Workers use client certificates to authenticate to the API. If certs expired during outage, they must re-request and get approved before they can reconnect.

8. Applications Recover

What happens automatically:

Scheduler sees nodes are Ready again
  → Evaluates pending pods
  → Reschedules any evicted pods
  → Deployment controllers reconcile replica counts
  → Services re-populate endpoints
  → Ingress/routes become fully functional again

Verify application recovery:

# Check deployments
oc get deployments -A | grep -v "1/1\|2/2\|3/3"   # find not-fully-ready ones
# Check pods
oc get pods -A | grep -v Running | grep -v Completed
# Check routes
oc get routes -A
# Check persistent volumes re-attached
oc get pvc -A | grep -v Bound

What might NOT auto-recover:

⚠️  Jobs that were mid-run — may need manual restart
⚠️  StatefulSets with PVCs — PV re-attachment can lag
⚠️  Custom operators — may need a pod restart if they cached stale state
⚠️  Any object created AFTER the backup snapshot — permanently lost

Recovery Timeline Summary

The Golden Rule

Your recovery is only as good as your last snapshot. Take cluster-backup.sh snapshots regularly (daily minimum), store them off-cluster (S3, NFS, etc.), and test restores in a non-prod environment before you ever need them in production.

What is RHCOS in OpenShift?

RHCOS — Red Hat Enterprise Linux CoreOS — is the operating system used by OpenShift Container Platform nodes.

It is a container-optimized, largely immutable operating system based on Red Hat Enterprise Linux technologies and designed specifically for running Kubernetes and OpenShift components.

OpenShift Container Platform
          │
          ├── Control-plane nodes
          │       └── RHCOS
          │
          └── Worker nodes
                  └── RHCOS

A typical OpenShift node runs:

RHCOS
├── systemd
├── kubelet
├── CRI-O
├── crun / runc
├── OVN-Kubernetes components
├── Machine Config Daemon
├── NetworkManager
├── SELinux
└── Linux kernel

Why OpenShift Uses RHCOS

Traditional Linux servers are often managed by:

• Installing RPM packages manually
• Editing configuration files directly
• Running configuration-management tools
• Applying individual operating-system patches

RHCOS follows a different model:

Desired node configuration
          │
          ▼
MachineConfig
          │
          ▼
Machine Config Operator
          │
          ▼
Machine Config Daemon
          │
          ▼
Node update and reboot

The OpenShift platform controls the operating-system configuration. This gives the cluster:

• Consistent node configuration
• Controlled operating-system updates
• Reduced configuration drift
• Automated upgrades
• Predictable rollback and recovery
• A smaller operational attack surface

RHCOS Is Largely Immutable

“Immutable” does not mean the filesystem can never change.

It means administrators should not manage the host by manually changing files or installing software. The node should be changed through OpenShift-supported mechanisms.

For example, avoid doing this manually:

dnf install package-name
vi /etc/sysctl.conf
systemctl enable custom-service

Instead, use:

• MachineConfig
• Machine Config Operator
• Operators
• DaemonSets for node-level agents
• Supported OpenShift configuration APIs

Manual changes can:

• Be overwritten during an update
• Create configuration drift
• Make nodes inconsistent
• Cause upgrade failures
• Put the environment outside supported practices

RHCOS Boot Process

A simplified boot sequence is:

Power on
   │
   ▼
Firmware / UEFI
   │
   ▼
Bootloader
   │
   ▼
Linux kernel
   │
   ▼
RHCOS userspace
   │
   ▼
systemd
   │
   ├── NetworkManager
   ├── CRI-O
   ├── kubelet
   └── Machine Config Daemon
            │
            ▼
Node joins OpenShift

During initial installation, Ignition applies the first node configuration.

Ignition

Ignition is used during the first boot to configure the node.

It can configure:

• Files
• Disks
• Filesystems
• Users
• SSH keys
• systemd units
• Certificates
• Initial kubelet settings

Ignition configuration
          │
          ▼
First boot
          │
          ├── Configure storage
          ├── Write system files
          ├── Configure networking
          └── Configure systemd

Ignition is primarily a first-boot provisioning mechanism.

After the cluster is installed, ongoing node configuration is primarily handled by the Machine Config Operator.

Machine Config Operator

The Machine Config Operator, or MCO, manages node operating-system configuration.

Its major components include:

Machine Config Operator
├── Machine Config Controller
├── Machine Config Server
├── Machine Config Daemon
└── Machine Config Pools

Machine Config Controller

Combines configuration from multiple MachineConfig resources into a rendered configuration.

Example:

00-worker
01-worker-kubelet
99-worker-custom
        │
        ▼
rendered-worker-abc123

Machine Config Server

Provides Ignition configuration during node provisioning.

Machine Config Daemon

Runs on each node and:

• Detects configuration changes
• Applies files and systemd settings
• Updates the operating-system image
• Drains the node when necessary
• Reboots the node
• Marks the node updated

Machine Config Pool

Groups nodes that receive the same configuration.

Common pools:

oc get mcp

Typical output:

NAME     UPDATED   UPDATING   DEGRADED
master   True      False      False
worker   True      False      False

Example MachineConfig

The following example creates a file on worker nodes:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-worker-example
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.4.0
    storage:
      files:
        - path: /etc/example.conf
          mode: 0644
          overwrite: true
          contents:
            source: data:text/plain;charset=utf-8,enabled=true

When applied:

MachineConfig created
        │
        ▼
MCO renders new configuration
        │
        ▼
Worker node cordoned
        │
        ▼
Workloads drained
        │
        ▼
Configuration applied
        │
        ▼
Node rebooted if required
        │
        ▼
Node returns Ready

MachineConfig changes should always be tested carefully because they can cause node reboots across the MachineConfigPool.

RHCOS and OpenShift Upgrades

During an OpenShift upgrade, the Cluster Version Operator coordinates the release, while the Machine Config Operator updates node operating systems.

New OpenShift release
        │
        ▼
Cluster Version Operator
        │
        ▼
Machine Config Operator
        │
        ▼
MachineConfigPool update
        │
        ▼
Node-by-node drain and reboot

For workers:

Worker 1
Drain → Update → Reboot → Ready
             │
             ▼
Worker 2
Drain → Update → Reboot → Ready

This rolling process helps maintain application availability.

Application availability still depends on:

• Multiple replicas
• PodDisruptionBudgets
• Anti-affinity
• Topology spread constraints
• Sufficient spare capacity

RHCOS and CRI-O

RHCOS is the operating system, while CRI-O is the container runtime running on that operating system.

OpenShift node
     │
     ▼
RHCOS
     │
     ├── kubelet
     ├── CRI-O
     ├── OVN-Kubernetes
     └── Machine Config Daemon

Container flow:

kubelet
   │
   ▼
CRI-O
   │
   ▼
crun / runc
   │
   ▼
Linux kernel in RHCOS

RHCOS provides the underlying:

• Kernel
• cgroups
• Namespaces
• SELinux
• Filesystems
• Networking
• systemd services

CRI-O uses these capabilities to run containers.

Security Features

RHCOS supports OpenShift’s security posture through:

• SELinux enforcing mode
• Minimal package footprint
• Controlled operating-system updates
• Secure Boot where supported
• FIPS mode when enabled during installation
• Kernel security controls
• Read-only or protected operating-system areas
• Centralized configuration through MCO
• Reduced direct administrative access

The security model is:

Application
     │
     ▼
Container security context
     │
     ▼
SCC / seccomp / capabilities
     │
     ▼
CRI-O
     │
     ▼
SELinux and Linux kernel
     │
     ▼
RHCOS

Accessing an RHCOS Node

Direct SSH should be limited and generally reserved for break-glass troubleshooting.

The preferred OpenShift method is:

oc debug node/<node-name>

Then access the host filesystem:

chroot /host

Example:

oc debug node/worker-0.example.com
chroot /host
systemctl status crio
journalctl -u kubelet

This same oc debug node and chroot /host pattern is useful when investigating node-level storage and operating-system issues.

Exit when finished:

exit
exit

Important RHCOS Directories

Do not manually modify these directories unless following an approved troubleshooting or Red Hat support procedure.

RHCOS vs Traditional RHEL

RHCOS vs FCOS

RHCOS is related to Fedora CoreOS but is intended for enterprise OpenShift use.

Fedora CoreOS
      │
      └── Community and upstream innovation
RHEL CoreOS
      │
      └── Enterprise OpenShift operating system

RHCOS includes Red Hat-supported components and follows the OpenShift lifecycle.

Common Troubleshooting Commands

Check node status

oc get nodes
oc describe node <node-name>

Check MachineConfigPools

oc get mcp
oc describe mcp worker

Check MachineConfigDaemon

oc get pods -n openshift-machine-config-operator -o wide

oc logs -n openshift-machine-config-operator \
  <machine-config-daemon-pod> \
  -c machine-config-daemon

Check node services

oc debug node/<node-name>
chroot /host
systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio

Check failed systemd units

systemctl --failed

Check disk space

df -h
df -i
lsblk

Check operating-system details

cat /etc/os-release
rpm-ostree status

Common RHCOS Problems

MachineConfigPool degraded

Possible causes:

• Manual file changes
• Failed systemd unit
• Node cannot reboot
• Invalid MachineConfig
• Disk-space problem
• Node cannot retrieve configuration
• Configuration mismatch

Check:

oc get mcp
oc describe mcp worker

Then identify the degraded node:

oc get nodes
oc get machineconfigpool worker -o yaml

Node stuck updating

Check:

oc describe node <node>
oc get mcp

Inspect the Machine Config Daemon logs:

oc logs -n openshift-machine-config-operator \
  <mcd-pod> -c machine-config-daemon

Look for:

• Drain failures
• PodDisruptionBudget blocking eviction
• File validation mismatch
• Reboot failure
• Disk pressure
• Invalid configuration

Configuration drift

The Machine Config Daemon may report degradation when a managed file no longer matches the desired configuration.

Typical cause:

Administrator manually edits managed file
                │
                ▼
MCD validates file
                │
                ▼
Unexpected content detected
                │
                ▼
Node or MCP becomes degraded

Correct the configuration through MachineConfig rather than continuing to edit the node manually.

Interview Answer

RHCOS, or Red Hat Enterprise Linux CoreOS, is the operating system used by OpenShift control-plane and worker nodes. It is a container-optimized and largely immutable operating system designed to be managed by OpenShift rather than administered manually. During initial boot, Ignition provisions the node, and after installation the Machine Config Operator manages operating-system files, systemd services, kernel arguments, CRI-O configuration and node updates.

RHCOS runs the kubelet, CRI-O, Machine Config Daemon, OVN networking components and other node services. During an OpenShift upgrade, the Machine Config Operator updates nodes through MachineConfigPools using a controlled drain, update, reboot and return-to-service process. Administrators should avoid manually installing packages or changing managed files because that can introduce configuration drift and cause MachineConfigPool degradation. For troubleshooting, I normally use oc debug node, chroot /host, systemctl, journalctl, and check the MachineConfigPool and Machine Config Daemon status.

Troubleshooting Intermittent API Latency Caused by etcd Disk Contention

The troubleshooting path is:

Slow oc/API requests
        ↓
Confirm API latency
        ↓
Correlate with etcd request latency
        ↓
Check WAL fsync and backend commit latency
        ↓
Identify the affected control-plane node
        ↓
Find competing disk activity or storage throttling
        ↓
Remove contention or move etcd to faster storage
        ↓
Validate etcd and API recovery

etcd must durably write consensus proposals to its write-ahead log before acknowledging them. Slow storage or competing processes can increase fsync duration, causing request timeouts, missed heartbeats and potentially temporary leader loss. During leader elections, API operations that change cluster state can stall. (Red Hat Documentation)

1. Confirm the API latency

First determine whether the issue affects:

• All API operations
• Only write operations
• One API server
• One control-plane node
• A particular resource type
• A particular time window

Test API readiness:

oc get --raw='/readyz?verbose'

Test simple reads:

time oc get nodes
time oc get namespaces
time oc get pods -A --request-timeout=10s

Test the API directly:

curl -k -w '\nDNS: %{time_namelookup}
Connect: %{time_connect}
TLS: %{time_appconnect}
TTFB: %{time_starttransfer}
Total: %{time_total}\n' \
  https://api.cluster.example.com:6443/readyz

This helps distinguish:

DNS or load-balancer latency
TLS connection latency
API processing latency
etcd-backed request latency

Check cluster health at the same time:

oc get clusteroperators
oc get nodes
oc get pods -n openshift-etcd -o wide
oc get pods -n openshift-kube-apiserver -o wide

Look for:

etcd Degraded=True
kube-apiserver Degraded=True
control-plane nodes NotReady
frequent etcd pod restarts
API readiness failures

2. Correlate API latency with etcd latency

Open the OpenShift console and inspect:

Observe
  → Dashboards
  → etcd

Also inspect:

Observe
  → Dashboards
  → Kubernetes / API server

The most important correlation is:

API request latency rises
            +
etcd fsync latency rises
            +
node disk latency rises

If all three rise at the same time, disk contention is a strong hypothesis.

3. Check the key etcd disk metrics

WAL fsync latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

This measures how long etcd takes to persist write-ahead-log records.

A sustained increase indicates:

• Slow underlying disks
• Storage throttling
• Storage queue saturation
• Competing writes
• Hypervisor storage contention
• Cloud disk credit exhaustion

The WAL fsync metric is one of the primary etcd metrics affected by storage I/O performance. (Red Hat Documentation)

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

This measures latency committing the etcd backend database.

Interpretation:

High WAL fsync only
→ WAL device or synchronous-write problem
High backend commit only
→ Backend database write or fragmentation pressure
Both high
→ General disk contention or storage degradation

etcd request latency

histogram_quantile(
  0.99,
  sum by (operation, type, le) (
    rate(etcd_request_duration_seconds_bucket[5m])
  )
)

Look for slow:

• PUT
• POST
• Transactions
• Range requests
• Lease operations

Write operations are usually affected most strongly by slow WAL storage.

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Frequent leader changes during disk-latency spikes suggest etcd members are missing heartbeats or taking too long to process consensus traffic.

Proposal failures

rate(etcd_server_proposals_failed_total[5m])

Also inspect pending proposals:

etcd_server_proposals_pending

A rising pending-proposal count means etcd cannot commit work as quickly as it receives it.

Database size and quota

etcd_mvcc_db_total_size_in_bytes

etcd_mvcc_db_total_size_in_use_in_bytes

etcd_server_quota_backend_bytes

Calculate reclaimable fragmented space:

(
  etcd_mvcc_db_total_size_in_bytes
-
  etcd_mvcc_db_total_size_in_use_in_bytes
) / 1024 / 1024

A large difference indicates internal fragmentation, but fragmentation and disk contention are different problems. Defragmentation may reduce database size; it does not fix fundamentally slow or saturated storage. Red Hat notes that defragmentation blocks the member while it runs, so it must be handled carefully and members must be allowed to recover between operations. (Red Hat Documentation)

4. Identify the affected etcd member

Break metrics down by instance.

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

Example:

master-0: normal
master-1: fsync spikes to 150 ms
master-2: normal

This strongly suggests that the storage attached to master-1 is the problem.

Map etcd pods to nodes:

oc get pods -n openshift-etcd \
  -l app=etcd \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase

Check which member is leader:

oc -n openshift-etcd rsh \
  $(oc get pod -n openshift-etcd \
    -l app=etcd \
    -o name | head -1) \
  etcdctl endpoint status --cluster -w table

In some OpenShift releases, the label can be k8s-app=etcd; verify with:

oc get pods -n openshift-etcd --show-labels

5. Inspect etcd health and logs

Check etcd endpoint health:

ETCD_POD=$(oc get pods -n openshift-etcd \
  -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}')
oc rsh -n openshift-etcd "$ETCD_POD" \
  etcdctl endpoint health --cluster

Check endpoint status:

oc rsh -n openshift-etcd "$ETCD_POD" \
  etcdctl endpoint status --cluster -w table

Look for:

• Slow endpoint response
• Unexpected database-size differences
• Raft index divergence
• Missing members
• A single slow member
• Frequent leadership changes

Review logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --since=2h

Search for:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --since=2h |
  grep -Ei \
  'slow fdatasync|slow request|took too long|leader|election|heartbeat|timeout|apply request'

Typical symptoms include messages resembling:

slow fdatasync
request timed out
leader changed
apply request took too long
failed to send out heartbeat

Also check the Operator:

oc logs -n openshift-etcd-operator \
  deployment/etcd-operator \
  --since=2h

6. Inspect host disk performance

Debug the affected control-plane node:

oc debug node/master-1

Enter the host filesystem:

chroot /host

Check block devices and mounts:

lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
findmnt /var/lib/etcd
df -h
df -i

Check disk utilization:

iostat -x 1 10

Important fields:

Warning signs include:

High await or w_await
Persistently high %util
Growing aqu-sz
Latency spikes matching API incidents

Depending on the device and kernel, %util=100 does not always prove saturation, especially for modern parallel storage. Latency and queue depth are more important.

Check historical activity:

sar -d 1 10
sar -q 1 10
sar -u 1 10

Check processes generating I/O:

pidstat -d 1 10

If installed and approved:

iotop -oPa

Do not install arbitrary tools or modify control-plane hosts during an incident unless this follows the bank’s change process and Red Hat support guidance.

7. Find the source of contention

Common causes include:

Shared-disk contention

etcd storage might share an underlying datastore with:

• VM snapshots
• Backup jobs
• Antivirus or filesystem scanning
• Monitoring agents
• Log collectors
• Container image activity
• Hypervisor migration
• Storage replication
• Other high-I/O virtual machines

A periodic latency spike at the same time each day often points to scheduled backup, snapshot, replication or scanning activity.

Cloud disk throttling

Check the cloud provider for:

• IOPS limits
• Throughput limits
• Queue depth
• Burst-credit exhaustion
• Disk latency
• Instance-level I/O limits

Increasing only the disk IOPS may not help when the VM instance itself has a lower aggregate storage throughput limit.

Virtualized storage contention

For virtualized control-plane nodes, investigate:

OpenShift node
    ↓
Virtual disk
    ↓
Datastore
    ↓
Storage controller
    ↓
Physical disks

The guest might show high latency even though its own I/O rate is low because another VM is saturating the datastore.

Correlate OpenShift data with:

• VMware datastore latency
• vSAN congestion
• SAN controller latency
• HBA queue depth
• Storage path failovers
• Multipath errors
• Hypervisor snapshots

Local filesystem pressure

Check:

journalctl -k --since "2 hours ago" |
  grep -Ei 'I/O error|timeout|reset|nvme|scsi|blk|xfs'

Check for filesystem or device errors:

dmesg -T |
  grep -Ei 'I/O error|timeout|reset|nvme|scsi|xfs'

Check system journals consuming excessive I/O:

journalctl --disk-usage

Do not manually remove /var/lib/etcd, WAL files or etcd database files.

8. Rule out network latency

Disk and network issues can produce similar etcd symptoms.

Check peer round-trip latency:

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Interpretation:

High fsync + normal peer RTT
→ Disk or storage issue
Normal fsync + high peer RTT
→ Network issue
Both high
→ Broader node, hypervisor or infrastructure contention

etcd replication performance depends on network latency, and high peer latency can trigger disruptive leader elections. (Red Hat Documentation)

9. Check whether API workload is contributing

Disk contention may be worsened by excessive API writes.

Identify expensive or high-volume clients through API metrics and audit logs.

Useful metrics include:

sum by (verb, resource) (
  rate(apiserver_request_total[5m])
)

topk(
  20,
  sum by (user_agent, verb) (
    rate(apiserver_request_total[5m])
  )
)

Look for:

• Operators stuck in fast reconciliation loops
• Controllers continuously updating status
• Excessive Events
• CI/CD repeatedly creating and deleting objects
• Monitoring systems making expensive list requests
• Large numbers of Secrets or ConfigMaps
• Broken automation generating thousands of API changes

Compare read and write pressure:

sum by (verb) (
  rate(apiserver_request_total[5m])
)

A sharp increase in POST, PUT, PATCH or DELETE operations can increase etcd write pressure.

The permanent solution may require both:

Faster isolated storage
          +
Reduction of unnecessary API writes

10. Remediation

Immediate containment

Depending on the confirmed cause:

• Stop or reschedule a competing backup or scan.
• Remove unrelated high-I/O workloads from the datastore.
• Resolve a failed storage path.
• Restore cloud disk IOPS or throughput capacity.
• Fix a runaway controller or automation loop.
• Pause nonessential bulk deployments.
• Escalate storage degradation to the infrastructure team.

Do not restart all control-plane nodes or etcd members together.

Permanent storage remediation

Use:

• Dedicated low-latency SSD or NVMe storage
• Guaranteed rather than burst-only IOPS
• Adequate throughput
• Dedicated datastore or storage policy
• No noisy-neighbour workloads
• Sufficient host-level I/O capacity
• Redundant, healthy storage paths

Red Hat recommends low-latency block storage for etcd because slow disks and other disk activity can directly increase WAL fsync latency. (Red Hat Documentation)

Database fragmentation

First determine whether automatic defragmentation is operating successfully:

oc logs -n openshift-etcd-operator \
  deployment/etcd-operator |
  grep -i defrag

Modern OpenShift releases automatically perform etcd defragmentation based on fragmentation thresholds. Manual defragmentation should not be the first response to disk contention and should follow the procedure for the exact OpenShift release. (Red Hat Documentation)

If Red Hat support or the documented procedure requires manual defragmentation:

• Take a valid etcd backup first.
• Verify quorum and endpoint health.
• Process one member at a time.
• Defragment the leader last.
• Wait for the member and cluster to recover between operations.

Defragmentation is blocking for the member on which it runs. (Red Hat Documentation)

11. Validate recovery

After remediation, compare the same incident metrics:

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

increase(etcd_server_leader_changes_seen_total[15m])

Then verify:

oc get --raw='/readyz?verbose'
oc get clusteroperators
oc get nodes
oc get pods -n openshift-etcd
oc get pods -n openshift-kube-apiserver

Success criteria:

WAL fsync latency returns to baseline
Backend commit latency returns to baseline
No new leader elections
No pending proposals
All etcd endpoints healthy
API p99 latency returns to SLO
ClusterOperators are healthy

Production Runbook Summary

Interview Answer

“I would first confirm that the latency is inside the API processing path rather than DNS, TLS or the load balancer. Then I would correlate API server latency with etcd’s WAL fsync and backend commit metrics, broken down by member. If one member shows elevated etcd_disk_wal_fsync_duration_seconds while peer network latency remains normal, that points to disk contention on that control-plane node.

I would inspect etcd endpoint health, leadership changes, pending proposals and logs for slow fdatasync, request timeouts or heartbeat failures. On the affected node, I would use oc debug node, iostat, sar and pidstat to examine disk latency, queue depth and competing processes. I would also check the cloud disk, VMware datastore or SAN layer for IOPS throttling, burst-credit exhaustion, snapshots, backup jobs or noisy neighbours.

Immediate remediation would be to stop the competing I/O or reduce excessive API writes while preserving etcd quorum. The permanent correction would be dedicated low-latency SSD or NVMe storage with guaranteed IOPS and sufficient host-level throughput. I would not restart all etcd members or manually defragment them as an initial reaction. After remediation, I would verify WAL fsync latency, backend commit latency, leader stability, endpoint health, API p99 latency and ClusterOperator status.”