DNS Operator in OpenShift

The DNS Operator deploys, configures, and continuously manages CoreDNS, which provides DNS resolution and Kubernetes Service discovery for Pods inside an OpenShift cluster.

A simple way to remember it:

The DNS Operator manages the DNS platform; CoreDNS answers the DNS queries.

DNS Operator
     │
     ▼
CoreDNS DaemonSet
     │
     ▼
DNS Service IP
     │
     ▼
Pods resolve Services and external names

The DNS Operator is installed automatically during OpenShift installation. It runs as a Deployment, while CoreDNS runs as a DaemonSet managed by that Operator. (Red Hat Documentation)

DNS Operator vs CoreDNS

These are separate components:

DNS Operator
      │
      ▼
Creates and manages
      │
      ├── CoreDNS DaemonSet
      ├── DNS Service
      ├── CoreDNS ConfigMap
      └── Node resolver DaemonSet

The Operator implements the cluster-scoped DNS API in the operator.openshift.io API group. (Red Hat Documentation)

Where the Components Run

The DNS Operator normally runs in:

openshift-dns-operator

CoreDNS and node-resolver components normally run in:

openshift-dns

Check the Operator:

oc get deployment -n openshift-dns-operator

Check DNS Pods:

oc get pods -n openshift-dns -o wide

Check DaemonSets:

oc get daemonset -n openshift-dns

Typical output includes:

dns-default
node-resolver

Main DNS Architecture

Application Pod
      │
      │ DNS query
      ▼
Cluster DNS Service IP
      │
      ▼
CoreDNS Pod
      │
      ├── Internal cluster name?
      │       └── Answer from Kubernetes API data
      │
      └── External name?
              └── Forward to upstream DNS resolver

The default internal cluster domain is:

cluster.local

CoreDNS provides resolution for internal names such as:

service.namespace.svc.cluster.local

The default DNS resource is named default. (Red Hat Documentation)

Kubernetes Service Discovery

Suppose you create this Service:

apiVersion: v1
kind: Service
metadata:
  name: payments-api
  namespace: banking
spec:
  selector:
    app: payments-api
  ports:
    - port: 8443

CoreDNS creates DNS-based service discovery for it.

A Pod in the same namespace can use:

payments-api

A Pod in another namespace can use:

payments-api.banking

The complete DNS name is:

payments-api.banking.svc.cluster.local

Flow:

Client Pod
    │
    ▼
payments-api.banking.svc.cluster.local
    │
    ▼
CoreDNS
    │
    ▼
Service ClusterIP
    │
    ▼
Ready application Pods

Pod DNS Configuration

The kubelet configures each normal Pod to use the cluster DNS Service.

Inside a Pod:

cat /etc/resolv.conf

Typical content:

search banking.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.30.0.10
options ndots:5

The exact DNS Service IP depends on the cluster Service network. OpenShift commonly assigns the DNS Service a stable IP from the Service CIDR. (Red Hat Documentation)

You can check it with:

oc get dns.operator/default -o yaml

Look under:

status:
  clusterDomain: cluster.local
  clusterIP: 172.30.0.10

Why CoreDNS Runs as a DaemonSet

CoreDNS normally runs through the dns-default DaemonSet.

Node 1          Node 2          Node 3
  │               │               │
CoreDNS         CoreDNS         CoreDNS

Benefits:

• DNS capacity scales as nodes are added.
• DNS is distributed across the cluster.
• A single DNS Pod failure does not stop name resolution.
• Requests can be handled close to workloads.
• DNS remains available if individual nodes fail.

The DNS Operator creates the CoreDNS DaemonSet and exposes it through a Service with a stable IP. (Red Hat Documentation)

Internal Name Resolution

CoreDNS watches the Kubernetes API for Services, EndpointSlices, namespaces, and other relevant resources.

For a Service query:

Pod asks for:
payments-api.banking.svc.cluster.local
                │
                ▼
CoreDNS queries its Kubernetes data
                │
                ▼
Returns the Service ClusterIP

For a headless Service:

spec:
  clusterIP: None

CoreDNS can return Pod or endpoint IPs instead of one Service ClusterIP.

This is useful for:

• StatefulSets
• Databases
• Cluster members
• Direct endpoint discovery

External DNS Resolution

If a Pod asks for an external name:

api.example.com

CoreDNS normally forwards the query to an upstream resolver.

Application Pod
      │
      ▼
CoreDNS
      │
      ▼
Corporate or cloud DNS
      │
      ▼
External DNS result

Upstream resolvers normally come from node resolver configuration or from explicit DNS Operator settings.

DNS Forwarding

The DNS Operator supports forwarding particular DNS zones to specific resolvers.

Example requirement:

*.corp.bank.local
        │
        ▼
Corporate DNS servers

A simplified configuration is:

apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  servers:
    - name: corporate-dns
      zones:
        - corp.bank.local
      forwardPlugin:
        upstreams:
          - 10.20.30.10
          - 10.20.30.11

Flow:

Query: database.corp.bank.local
              │
              ▼
CoreDNS matches corp.bank.local
              │
              ▼
Forwards to 10.20.30.10 / 10.20.30.11

Queries that do not match a configured zone fall back to the configured upstream resolvers. (Red Hat Documentation)

Edit DNS configuration with:

oc edit dns.operator/default

Do not manually edit the Operator-generated CoreDNS ConfigMap as the primary configuration method.

DNS Caching

CoreDNS caches successful and unsuccessful DNS responses.

This reduces:

• Query latency
• Load on upstream DNS servers
• Repeated external lookups
• Network traffic

OpenShift supports configuring positive and negative cache TTLs through the DNS Operator resource. (Red Hat Documentation)

Example:

apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  cache:
    positiveTTL: 1h
    negativeTTL: 30s

Be careful with TTL tuning:

• Very low TTLs increase DNS load.
• Very high TTLs can retain stale results longer.
• Negative caching can make a recently created record appear unavailable until the negative TTL expires.

Node Resolver

The DNS Operator also manages a node-resolver DaemonSet.

Check it with:

oc get daemonset node-resolver -n openshift-dns

The node-resolver component helps maintain node-level name resolution information, including managed entries in the node’s /etc/hosts where required.

DNS Operator
      │
      ├── dns-default DaemonSet
      │       └── Cluster DNS
      │
      └── node-resolver DaemonSet
              └── Node hostname resolution support

Reconciliation Loop

The DNS Operator continuously compares desired DNS configuration with the actual resources.

DNS resource: default
        │
        ▼
DNS Operator reads desired state
        │
        ▼
Checks DaemonSets, Service and ConfigMap
        │
        ▼
Difference found?
     ┌──┴──┐
     │     │
    No    Yes
     │     │
     ▼     ▼
  Wait   Recreate or update resources
               │
               ▼
         Validate DNS availability
               │
               ▼
        Update Operator status

Examples that trigger reconciliation:

• CoreDNS Pod fails.
• DNS forwarding configuration changes.
• Cache settings change.
• OpenShift is upgraded.
• A managed ConfigMap changes.
• Node placement is modified.
• A DaemonSet does not match the desired configuration.

DNS Operator Health

Check the ClusterOperator:

oc get clusteroperator dns

Healthy output:

NAME   AVAILABLE   PROGRESSING   DEGRADED
dns    True        False         False

Interpretation:

Detailed information:

oc describe clusteroperator dns

The Operator considers DNS available when the DNS Service has a ClusterIP and at least one CoreDNS Pod is available. (Red Hat Documentation)

Useful Commands

Check the Operator

oc get deployment dns-operator \
  -n openshift-dns-operator

oc logs -n openshift-dns-operator \
  deployment/dns-operator

Check DNS configuration

oc get dns.operator/default -o yaml

oc describe dns.operator/default

Check CoreDNS Pods

oc get pods -n openshift-dns -o wide

Check DaemonSets

oc get daemonset -n openshift-dns

Check DNS Service

oc get service dns-default -n openshift-dns

Check generated configuration

oc get configmap dns-default \
  -n openshift-dns \
  -o yaml

Check events

oc get events -n openshift-dns \
  --sort-by='.lastTimestamp'

Testing DNS from a Pod

Create a temporary troubleshooting Pod:

oc run dns-test \
  --image=registry.access.redhat.com/ubi9/ubi-minimal \
  --restart=Never \
  -- sleep 3600

Enter it:

oc rsh dns-test

Check its resolver configuration:

cat /etc/resolv.conf

Test an internal Service:

getent hosts kubernetes.default.svc.cluster.local

Test your application:

getent hosts payments-api.banking.svc.cluster.local

Test an external address:

getent hosts example.com

Delete the Pod afterward:

oc delete pod dns-test

Troubleshooting Flow

Use this sequence:

Application DNS error
        │
        ▼
Check Pod /etc/resolv.conf
        │
        ▼
Test short and full service names
        │
        ▼
Check Service and EndpointSlices
        │
        ▼
Check DNS Service ClusterIP
        │
        ▼
Check CoreDNS Pods
        │
        ▼
Check DNS Operator status
        │
        ▼
Check CoreDNS and Operator logs
        │
        ▼
Check upstream DNS and network policies

Scenario 1: Internal Service Does Not Resolve

Example:

payments-api.banking.svc.cluster.local

Check the Service:

oc get svc payments-api -n banking

Check namespace and spelling:

oc get namespace banking

Test the full name:

getent hosts payments-api.banking.svc.cluster.local

Then check:

oc get pods -n openshift-dns
oc get svc dns-default -n openshift-dns

Remember:

• DNS can resolve a Service even if its backend Pods are unhealthy.
• DNS resolution does not prove the application itself is reachable.
• EndpointSlices affect application traffic, not necessarily the existence of the Service DNS record.

Scenario 2: External Names Fail but Internal Names Work

For example:

payments-api.banking.svc.cluster.local → works
example.com                            → fails

This usually indicates an upstream-forwarding issue.

Check:

• Upstream DNS server availability
• DNS Operator forwarding configuration
• Firewall access to UDP/TCP port 53
• Node /etc/resolv.conf
• Corporate DNS reachability
• Egress restrictions
• Forwarding loops

Review CoreDNS logs:

oc logs -n openshift-dns <dns-default-pod> \
  -c dns

Scenario 3: One Node Has DNS Problems

If Pods on one node fail DNS while other nodes work:

oc get pods -n openshift-dns -o wide

Check whether the affected node has a healthy CoreDNS Pod.

Then examine:

• Node networking
• OVN connectivity
• DNS DaemonSet Pod
• kubelet configuration
• Service routing
• Firewall rules
• MTU problems
• Node resource pressure

Test from Pods scheduled on both a healthy and affected node.

Scenario 4: DNS Query Is Slow

Possible causes:

• Slow upstream resolver
• DNS forwarding loop
• Packet loss
• Excessive query volume
• Too-low cache TTL
• CoreDNS CPU throttling
• Node network problems
• Search-domain expansion caused by ndots
• External queries being tried as multiple internal names first

Measure lookup time:

time getent hosts external.example.com

Compare internal and external queries separately.

Scenario 5: CoreDNS Pods Are Pending

Check:

oc describe pod <dns-pod> -n openshift-dns

Possible causes:

• Node selector mismatch
• Missing toleration
• Insufficient CPU or memory
• Node taints
• Scheduling restrictions
• Image pull problem

CoreDNS and node-resolver placement can be controlled using node selectors and tolerations in the DNS Operator configuration. (Red Hat Documentation)

Common DNS Errors

Could not resolve host

Possible causes:

• CoreDNS unavailable
• Wrong Pod resolver configuration
• Upstream DNS failure
• NetworkPolicy or firewall blocking DNS
• Typographical error

SERVFAIL

Possible causes:

• Upstream resolver failure
• DNS forwarding loop
• Invalid zone configuration
• DNSSEC or upstream issue

NXDOMAIN

Means the requested name does not exist according to the resolver.

Check:

• Service name
• Namespace
• DNS zone
• Negative cache
• External record creation

DNS works but connection fails

DNS only returned an IP address. Check:

• Service port
• EndpointSlices
• Pod readiness
• NetworkPolicy
• Application process
• TLS configuration

DNS Operator vs DNS Operator Configuration

Avoid directly editing:

CoreDNS DaemonSet
dns-default Service
generated dns-default ConfigMap

These are Operator-managed and changes may be reverted.

Configure DNS using:

oc edit dns.operator/default

The Operator then generates the appropriate CoreDNS configuration and performs reconciliation.

Relationship with Other Operators

Cluster Version Operator
        │
        ▼
DNS Operator
        │
        ├── CoreDNS DaemonSet
        ├── Node Resolver DaemonSet
        ├── DNS Service
        └── DNS ConfigMap

Dependencies include:

Interview Answer

The OpenShift DNS Operator deploys and manages CoreDNS to provide internal name resolution and Kubernetes Service discovery. The Operator runs as a Deployment in openshift-dns-operator, while it manages the dns-default CoreDNS DaemonSet, the node-resolver DaemonSet, a DNS Service with a stable ClusterIP, and the generated CoreDNS configuration in openshift-dns.

Pods send DNS queries to the DNS Service IP configured in their /etc/resolv.conf. CoreDNS resolves internal names such as service.namespace.svc.cluster.local using Kubernetes API information and forwards external or configured private-zone queries to upstream resolvers. The Operator continuously reconciles these resources and supports configuration through the cluster-scoped dns.operator/default object, including forwarding, caching, and node placement.

For troubleshooting, I start with oc get co dns, inspect the DNS Operator and CoreDNS Pods, check the dns-default Service and ConfigMap, test DNS from a Pod, and determine whether the failure affects internal names, external names, or only Pods on a particular node. I then check upstream resolvers, OVN connectivity, NetworkPolicies, firewall rules and CoreDNS logs.

kubelet on OCP Master Nodes

Why kubelet Runs on Masters

In OCP, the control plane components are not system services (systemd units). They run as static pods — and kubelet is the only thing that can run static pods.

Without kubelet on masters:
  ❌ etcd cannot start
  ❌ kube-apiserver cannot start
  ❌ kube-controller-manager cannot start
  ❌ kube-scheduler cannot start

kubelet IS the bootstrap mechanism for the entire control plane.

Static Pods — The Core Concept

kubelet watches one directory constantly:

/etc/kubernetes/manifests/
  ├── etcd-pod.yaml
  ├── kube-apiserver-pod.yaml
  ├── kube-controller-manager-pod.yaml
  └── kube-scheduler-pod.yaml

Any .yaml dropped here → kubelet starts it as a pod. Any .yaml removed → kubelet stops and removes it.

No API server needed. kubelet reads these files directly from disk. This is exactly why cluster-restore.sh works by moving manifests in and out of this directory — it’s controlling the control plane via kubelet’s static pod mechanism.

cluster-restore.sh
  └── mv etcd-pod.yaml → /etc/kubernetes/manifests/
              │
              ▼
           kubelet notices file change (inotify watch)
              │
              ▼
           kubelet tells CRI-O to start the etcd containers
              │
              ▼
           etcd is running

What kubelet Does on a Master (vs Worker)

Master Node Taint (workload isolation)

By default, OCP master nodes carry a taint that prevents regular workloads from landing on them:

oc describe node master-0 | grep Taint
# Taints: node-role.kubernetes.io/master:NoSchedule

So while kubelet on masters CAN run any pod, the scheduler won’t place regular workloads there unless you explicitly tolerate the taint. The taint does NOT affect static pods — kubelet runs those directly, bypassing the scheduler entirely.

kubelet Startup Sequence on a Master

This is what happens when a master node boots:

1. RHCOS boots
        │
        ▼
2. systemd starts kubelet.service
        │
        ▼
3. kubelet reads /etc/kubernetes/manifests/
        │
        ├── etcd-pod.yaml          → starts etcd containers
        ├── kube-apiserver-pod.yaml → starts API server containers
        ├── kube-controller-manager-pod.yaml
        └── kube-scheduler-pod.yaml
        │
        ▼
4. etcd becomes healthy (has quorum)
        │
        ▼
5. kube-apiserver connects to etcd, starts serving
        │
        ▼
6. kubelet registers THIS master node with the API server
        │
        ▼
7. Cluster operators come up (CVO, etcd-operator, etc.)

Notice: kubelet starts etcd, and etcd enables the API, and the API is what kubelet later registers with. kubelet bootstraps its own control plane and then registers with it.

Verify kubelet on a Master

# SSH into a master node
ssh core@master-0
# kubelet is a systemd service
systemctl status kubelet
# kubelet logs
journalctl -u kubelet -f
# kubelet process
ps aux | grep kubelet
# kubelet managing static pods
crictl pods | grep -E "etcd|apiserver|scheduler|controller"

What Happens if kubelet Dies on a Master

kubelet dies on master-0
        │
        ├── etcd container keeps running (CRI-O manages it independently)
        ├── But if etcd crashes → kubelet not there to restart it
        ├── kube-apiserver keeps running (same)
        │
        └── After 5 min: node shows NotReady (no heartbeat to API)
                │
                └── etcd operator detects degraded member
                        └── alerts fire, but cluster may still function
                            if other 2 masters are healthy

Key insight: CRI-O keeps containers alive after kubelet dies — but kubelet is the only thing that will restart a crashed container. Without kubelet, a crashed etcd stays dead.

The Dependency Chain in One View

RHCOS
  └── systemd
        └── kubelet.service          ← runs on master as a systemd unit
              └── reads /etc/kubernetes/manifests/
                    ├── etcd-pod.yaml
                    │     └── CRI-O runs etcd containers
                    │           └── etcd cluster (the database)
                    │
                    ├── kube-apiserver-pod.yaml
                    │     └── CRI-O runs API server containers
                    │           └── Kubernetes/OCP API
                    │
                    ├── kube-controller-manager-pod.yaml
                    │     └── CRI-O runs controller-manager
                    │
                    └── kube-scheduler-pod.yaml
                          └── CRI-O runs scheduler

The entire OCP control plane is just kubelet reading files from a directory and telling CRI-O what to run. This simplicity is what makes the restore procedure work — you control the control plane by controlling what kubelet sees in /etc/kubernetes/manifests/.

OpenShift Worker Node Components

OpenShift worker nodes are the machines that run application workloads.

The control plane decides what should run and where. Worker nodes perform the actual execution.

Control Plane
     │
     ├── API Server
     ├── Scheduler
     ├── Controllers
     └── etcd
            │
            ▼
        Worker Nodes
     ┌────────┼────────┐
     ▼        ▼        ▼
  Worker-1 Worker-2 Worker-3
     │        │        │
   Pods     Pods      Pods

A worker node can be:

• A physical server
• A virtual machine
• A cloud instance
• A bare-metal host

In OpenShift, worker nodes normally run RHCOS, although some supported configurations can use RHEL workers.

Main Components on a Worker Node

A typical OpenShift worker contains:

RHCOS
├── kubelet
├── CRI-O
├── crun or runc
├── OVN-Kubernetes node components
├── Open vSwitch
├── Machine Config Daemon
├── CoreDNS pod
├── Node Exporter
├── CSI node plugins
├── Logging collector
└── Application Pods

The most important components are:

1. kubelet
2. CRI-O
3. OCI runtime
4. OVN-Kubernetes
5. Machine Config Daemon
6. Node-level monitoring
7. Logging collector
8. CSI node plugins

1. RHCOS

Red Hat Enterprise Linux CoreOS is the operating system used by OpenShift nodes.

It provides:

• Linux kernel
• systemd
• SELinux
• cgroups
• namespaces
• filesystems
• network stack
• container storage
• system services

Applications
     │
     ▼
Containers
     │
     ▼
CRI-O
     │
     ▼
Linux kernel
     │
     ▼
RHCOS

RHCOS is largely immutable and should be managed through OpenShift, especially through the Machine Config Operator.

Administrators should avoid manually installing packages or changing Operator-managed files.

2. kubelet

The kubelet is the main Kubernetes agent running on every node.

Its responsibilities include:

• Registering the node with the API server
• Watching for Pods assigned to the node
• Asking CRI-O to start and stop containers
• Mounting volumes
• Running health probes
• Reporting node and Pod status
• Managing static Pods where applicable
• Enforcing Pod resource configuration

API Server
    │
    ▼
kubelet
    │
    ├── Start Pod
    ├── Stop Pod
    ├── Run probes
    ├── Mount volumes
    └── Report status

Check kubelet status:

oc debug node/<worker-node>
chroot /host
systemctl status kubelet

View logs:

journalctl -u kubelet

3. CRI-O

CRI-O is OpenShift’s container runtime.

The kubelet communicates with CRI-O through the Kubernetes Container Runtime Interface.

kubelet
   │
   ▼
CRI-O
   │
   ▼
crun / runc
   │
   ▼
Linux kernel

CRI-O is responsible for:

• Pulling container images
• Creating Pod sandboxes
• Starting containers
• Stopping containers
• Managing image storage
• Applying cgroups
• Applying SELinux labels
• Preparing mounts
• Integrating with networking

Useful commands:

systemctl status crio
journalctl -u crio
crictl ps
crictl pods
crictl images

4. crun or runc

CRI-O does not create containers directly. It uses an OCI runtime, commonly crun or runc.

The OCI runtime creates the container process using Linux kernel technologies such as:

• PID namespaces
• Network namespaces
• Mount namespaces
• cgroups
• seccomp
• Linux capabilities
• SELinux

CRI-O request
      │
      ▼
crun or runc
      │
      ▼
Container process

This component is very low-level and normally not managed directly by administrators.

5. OVN-Kubernetes Node Components

Each worker participates in the OpenShift software-defined network.

Typical components include:

• ovnkube-node
• Open vSwitch
• OVN controller
• CNI integration
• GENEVE tunnel interfaces

Pod
 │
 ▼
Virtual Ethernet Interface
 │
 ▼
Open vSwitch
 │
 ▼
OVN logical network
 │
 ▼
Another Pod or Service

The node networking layer handles:

• Pod IP assignment
• Pod-to-Pod connectivity
• Service traffic
• NetworkPolicies
• East-west routing
• Egress traffic
• GENEVE encapsulation
• Load balancing for Services

Check OVN components:

oc get pods -n openshift-ovn-kubernetes -o wide

A networking failure on one worker can cause Pods on that node to lose connectivity even though the Pods remain running.

6. Machine Config Daemon

The Machine Config Daemon, or MCD, runs as a DaemonSet on each node.

It is part of the Machine Config Operator.

Its responsibilities include:

• Applying RHCOS configuration
• Updating operating-system files
• Applying systemd units
• Updating kubelet and CRI-O configuration
• Performing node OS updates
• Detecting configuration drift
• Draining and rebooting the node when required

MachineConfig
      │
      ▼
Machine Config Operator
      │
      ▼
Machine Config Daemon
      │
      ▼
Worker node

During an update:

Cordon
  │
  ▼
Drain workloads
  │
  ▼
Apply configuration
  │
  ▼
Reboot if required
  │
  ▼
Node Ready

Check it with:

oc get mcp
oc get pods -n openshift-machine-config-operator -o wide

7. Application Pods

The main purpose of worker nodes is to run application Pods.

Examples:

Worker Node
├── payments-api Pod
├── frontend Pod
├── database Pod
├── logging agent Pod
└── monitoring agent Pod

A Pod can contain one or more containers.

The worker provides:

• CPU
• Memory
• Network
• Storage access
• Container runtime
• Security isolation

The scheduler decides the node placement, but the worker executes the workload.

8. Node Exporter

OpenShift deploys a managed Node Exporter DaemonSet.

It collects host-level metrics such as:

• CPU
• Memory
• Filesystem
• Disk I/O
• Network traffic
• Load average
• Kernel metrics

Worker Node
     │
     ▼
Node Exporter
     │
     ▼
Prometheus

Check it:

oc get daemonset node-exporter -n openshift-monitoring

You generally should not install another standalone Node Exporter on OpenShift nodes.

9. kubelet and cAdvisor Metrics

The kubelet also exposes container-related metrics.

These include:

• Container CPU
• Container memory
• Filesystem usage
• Pod resource usage
• Container restarts

Container
    │
    ▼
kubelet / cAdvisor metrics
    │
    ▼
Prometheus

Difference:

10. Logging Collector

When OpenShift Logging is installed, a collector such as Vector runs as a DaemonSet.

Application stdout/stderr
          │
          ▼
Node log files
          │
          ▼
Vector collector
          │
          ├── Loki
          ├── Splunk
          └── Elasticsearch

It can collect:

• Application logs
• Infrastructure logs
• Audit logs

Check:

oc get pods -n openshift-logging -o wide

11. CSI Node Plugins

Storage drivers commonly deploy node-level CSI components as DaemonSets.

They handle:

• Mounting volumes
• Unmounting volumes
• Attaching storage where applicable
• Formatting volumes
• Exposing block devices to Pods

Pod requests PVC
       │
       ▼
CSI controller
       │
       ▼
CSI node plugin
       │
       ▼
Volume mounted on worker

Examples include:

• AWS EBS CSI
• Azure Disk CSI
• VMware vSphere CSI
• Ceph CSI
• Fibre Channel or SAN CSI integrations

12. CoreDNS Pod

OpenShift normally deploys CoreDNS through a DaemonSet, so worker nodes may run a DNS Pod.

CoreDNS handles:

• Service discovery
• Internal DNS
• External DNS forwarding

Application Pod
      │
      ▼
DNS Service IP
      │
      ▼
CoreDNS Pod

Check:

oc get pods -n openshift-dns -o wide

13. Multus

OpenShift uses Multus when Pods or VMs need multiple network interfaces.

Pod
 ├── eth0 → default OVN network
 └── net1 → VLAN or SR-IOV network

Multus is commonly used for:

• Telco workloads
• OpenShift Virtualization
• Storage networks
• High-performance networking
• SR-IOV

Node-level Multus components run on workers.

14. SR-IOV Components

On specialized worker nodes, the SR-IOV Operator may deploy node agents.

These provide:

• Direct virtual functions
• High-throughput networking
• Low latency
• Hardware offload

Pod
 │
 ▼
Virtual Function
 │
 ▼
Physical NIC

These nodes are often labeled and tainted to isolate specialized workloads.

15. Device Plugins

Device plugins expose hardware resources to Kubernetes.

Examples:

• NVIDIA GPUs
• Intel accelerators
• FPGAs
• Network devices

Example GPU resource:

resources:
  limits:
    nvidia.com/gpu: 1

The scheduler sees the available resource, and the node device plugin makes it accessible to the Pod.

Worker Node Startup Flow

A worker node boot sequence looks like this:

Power on
   │
   ▼
RHCOS boots
   │
   ▼
systemd starts
   │
   ├── NetworkManager
   ├── CRI-O
   ├── kubelet
   └── Machine Config Daemon
            │
            ▼
kubelet connects to API server
            │
            ▼
Node registers
            │
            ▼
OVN networking initializes
            │
            ▼
DaemonSet Pods start
            │
            ▼
Node becomes Ready
            │
            ▼
Scheduler assigns application Pods

Pod Startup on a Worker

When a Pod is assigned to a worker:

Scheduler selects worker-2
           │
           ▼
API server updates Pod nodeName
           │
           ▼
kubelet on worker-2 detects Pod
           │
           ▼
CRI-O pulls image
           │
           ▼
Pod sandbox created
           │
           ▼
OVN configures network
           │
           ▼
CSI mounts volumes
           │
           ▼
Container starts
           │
           ▼
Readiness probe succeeds
           │
           ▼
Pod receives traffic

Worker Node Status

Check workers:

oc get nodes

Example:

NAME       STATUS   ROLES
worker-0   Ready    worker
worker-1   Ready    worker
worker-2   Ready    worker

Show worker nodes only:

oc get nodes -l node-role.kubernetes.io/worker

Detailed node information:

oc describe node worker-0

Important sections include:

• Conditions
• Capacity
• Allocatable
• Taints
• Labels
• Allocated resources
• Events

Node Conditions

Common node conditions include:

Healthy example:

Ready=True
MemoryPressure=False
DiskPressure=False
PIDPressure=False
NetworkUnavailable=False

Capacity vs Allocatable

A node may have:

Capacity:
CPU: 32
Memory: 128 GiB

But allocatable resources are lower:

Allocatable:
CPU: 30
Memory: 118 GiB

The difference is reserved for:

• Operating system
• kubelet
• CRI-O
• OpenShift platform agents
• Eviction thresholds

The scheduler uses allocatable, not total physical capacity.

Worker Labels

Labels help place workloads.

Examples:

oc label node worker-1 workload=payments
oc label node worker-2 node-role.kubernetes.io/infra=""
oc label node gpu-1 accelerator=nvidia

Pods can target those labels:

spec:
  nodeSelector:
    workload: payments

Worker Taints

Taints prevent general workloads from being scheduled.

Example:

oc adm taint node gpu-1 dedicated=gpu:NoSchedule

A Pod must have a matching toleration.

tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

This is used for:

• GPU nodes
• Infrastructure nodes
• Storage nodes
• Special security zones
• High-performance workloads

Worker Pools

MachineConfigPools group nodes with the same operating-system configuration.

Typical pools:

master
worker
infra
gpu
storage

Check:

oc get mcp

Example:

NAME     UPDATED   UPDATING   DEGRADED
worker   True      False      False
infra    True      False      False

Custom worker pools can receive separate:

• Kubelet settings
• Kernel arguments
• CRI-O settings
• Systemd units
• OS files

Worker Node Failure

If a worker fails:

Worker node unavailable
        │
        ▼
Node becomes NotReady
        │
        ▼
Controller detects failure
        │
        ▼
Pods are recreated elsewhere
        │
        ▼
Scheduler selects healthy workers

This depends on:

• Multiple replicas
• Storage accessibility
• Pod disruption controls
• Node failure detection
• Sufficient spare capacity

A single-replica application can experience downtime.

Common Worker Problems

Node NotReady

Check:

oc get nodes
oc describe node <worker>

Then:

oc debug node/<worker>
chroot /host
systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio

Possible causes:

• kubelet failure
• CRI-O failure
• Network failure
• Certificate problem
• Disk pressure
• Memory pressure
• Node OS issue
• API server connectivity issue

DiskPressure

Check:

df -h
df -i
du -sh /var/lib/containers/*
du -sh /var/log/*

Possible causes:

• Large container logs
• Image accumulation
• Container storage exhaustion
• Inode exhaustion
• Failed garbage collection

MemoryPressure

Check:

free -h
top
oc adm top node

Possible causes:

• Oversized workloads
• Incorrect requests and limits
• Memory leak
• Too many Pods
• Platform agents consuming resources

Pods stuck in ContainerCreating

Investigate:

oc describe pod <pod> -n <namespace>

Possible causes:

• CRI-O failure
• Image pull problem
• CNI/OVN failure
• CSI mount failure
• SCC or permission issue
• Node disk pressure

ImagePullBackOff

Check:

• Image name
• Registry access
• ImagePullSecret
• DNS
• Proxy configuration
• Registry certificate trust
• CRI-O logs

journalctl -u crio
crictl images

Useful Worker Troubleshooting Commands

oc get nodes
oc describe node <worker>
oc adm top nodes
oc get pods -A -o wide --field-selector spec.nodeName=<worker>

Node-level access:

oc debug node/<worker>
chroot /host

Inside the host:

systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio
df -h
df -i
free -h
iostat -x 1 10
sar -u 1 10
pidstat -d 1 10

Runtime inspection:

crictl ps
crictl pods
crictl images
crictl info

Control Plane vs Worker Components

Control plane says:
"Run this Pod on worker-2"
Worker-2 says:
"I will pull the image, configure networking, mount storage, and start it"

Interview Answer

An OpenShift worker node is responsible for executing application workloads. It normally runs RHCOS, the kubelet, CRI-O, an OCI runtime such as crun, OVN-Kubernetes networking components, the Machine Config Daemon, and node-level DaemonSets such as Node Exporter, CoreDNS, logging collectors, and CSI plugins.

The kubelet registers the worker with the API server and watches for Pods assigned by the scheduler. It asks CRI-O to create the Pod sandbox and start containers, OVN configures networking, and CSI drivers mount persistent storage. The Machine Config Daemon keeps the RHCOS configuration synchronized with the desired MachineConfigPool state.

For troubleshooting, I start with oc get nodes and oc describe node, inspect node conditions such as Ready, MemoryPressure, and DiskPressure, then use oc debug node and chroot /host to check kubelet, CRI-O, storage, memory, network, and system logs.

OpenShift Control Plane Components

In OpenShift, the correct term is control plane, not control panel.

The control plane is the “brain” of the OpenShift cluster. It manages:

• Cluster configuration
• Workload scheduling
• API requests
• Cluster state
• Controllers and Operators
• Authentication and authorization
• Node and workload lifecycle

A typical highly available OpenShift cluster has three control-plane nodes.

                    Users and Administrators
                             │
                         oc / Console
                             │
                             ▼
                      API Load Balancer
                             │
          ┌──────────────────┼──────────────────┐
          ▼                  ▼                  ▼
      master-0            master-1           master-2
          │                  │                  │
          ├── kube-apiserver ├── kube-apiserver ├── kube-apiserver
          ├── etcd           ├── etcd           ├── etcd
          ├── scheduler      ├── scheduler      ├── scheduler
          ├── controller     ├── controller     ├── controller
          └── Operators      └── Operators      └── Operators

1. kube-apiserver

The kube-apiserver is the main entry point into the cluster.

All administrative and platform operations go through it.

Examples:

oc get pods
oc apply -f deployment.yaml
oc delete pod mypod

Request flow:

oc client
   │
   ▼
API load balancer
   │
   ▼
kube-apiserver
   │
   ├── Authentication
   ├── Authorization
   ├── Admission controls
   ├── Resource validation
   └── etcd read/write

The API server handles:

• Kubernetes API requests
• Authentication
• RBAC authorization
• Admission webhooks
• Object validation
• Communication with etcd

It listens on:

TCP 6443

Check it with:

oc get pods -n openshift-kube-apiserver -o wide
oc get co kube-apiserver

2. etcd

etcd is the distributed key-value database that stores the authoritative cluster state.

It stores:

• Deployments
• Pods and desired state
• Services
• Secrets
• ConfigMaps
• Routes
• RBAC
• Nodes
• Operators
• CRDs
• MachineConfig objects

API Server
    │
    ▼
   etcd
    │
    ▼
Cluster state

A standard OpenShift cluster normally has three etcd members:

master-0
master-1
master-2

Quorum requirement:

3 members → 2 required for quorum

If one member fails, the cluster can usually continue.

If two members fail, etcd loses quorum and control-plane operations stop.

Check etcd:

oc get co etcd
oc get pods -n openshift-etcd -o wide

3. kube-scheduler

The scheduler decides which node should run a new Pod.

It evaluates:

• CPU requests
• Memory requests
• Node selectors
• Taints and tolerations
• Affinity and anti-affinity
• Topology spread constraints
• Persistent-volume topology
• Host ports
• Node readiness

Pending Pod
     │
     ▼
Scheduler filters nodes
     │
     ▼
Scheduler scores eligible nodes
     │
     ▼
Pod assigned to worker-2
     │
     ▼
kubelet starts Pod

The scheduler does not start the container. It only assigns the Pod to a node.

Check it with:

oc get co kube-scheduler
oc get pods -n openshift-kube-scheduler -o wide

4. kube-controller-manager

The kube-controller-manager runs multiple Kubernetes controllers.

Controllers continuously compare:

Desired state
      vs
Actual state

and take action to correct differences.

Important controllers include:

• Deployment controller
• ReplicaSet controller
• Node controller
• Job controller
• Service account controller
• EndpointSlice controller
• Namespace controller
• Persistent-volume controller

Example:

Deployment requests 3 replicas
          │
          ▼
Controller sees only 2 Pods
          │
          ▼
Creates another Pod

Check it with:

oc get co kube-controller-manager
oc get pods -n openshift-kube-controller-manager -o wide

5. OpenShift Controller Manager

OpenShift also includes OpenShift-specific controllers.

These manage platform-specific resources and behavior beyond standard Kubernetes.

Examples include:

• OpenShift project behavior
• Build-related resources
• Image resources
• OpenShift authorization functions
• Platform-specific reconciliation

It runs separately from the Kubernetes controller manager.

Check:

oc get co openshift-controller-manager
oc get pods -n openshift-controller-manager

6. OpenShift API Server

The OpenShift API Server provides OpenShift-specific APIs that extend Kubernetes.

Examples include APIs related to:

• Projects
• Routes
• Builds
• Images
• OpenShift-specific authorization
• Security extensions

Architecture:

Client
   │
   ▼
Kubernetes API aggregation layer
   │
   ├── Kubernetes APIs
   └── OpenShift APIs

Check:

oc get co openshift-apiserver
oc get pods -n openshift-apiserver

7. Cluster Version Operator

The Cluster Version Operator, or CVO, manages the overall OpenShift release version.

It is responsible for:

• Installing platform components
• Coordinating upgrades
• Applying release manifests
• Monitoring ClusterOperators
• Ensuring components match the desired release

New OpenShift release
         │
         ▼
Cluster Version Operator
         │
         ▼
Platform Operators upgraded
         │
         ▼
Control-plane and worker updates

Check:

oc get clusterversion
oc get co

8. Machine Config Operator

The Machine Config Operator, or MCO, manages the operating-system configuration of RHCOS nodes.

It controls:

• RHCOS updates
• CRI-O configuration
• kubelet configuration
• Kernel arguments
• Systemd units
• CA certificates
• Registry configuration
• Node files

MachineConfig
      │
      ▼
Machine Config Operator
      │
      ▼
Machine Config Daemon
      │
      ▼
Drain → Apply → Reboot → Ready

Check:

oc get mcp
oc get machineconfig
oc get co machine-config

9. Authentication Operator

The Authentication Operator manages OpenShift OAuth and login services.

It handles:

• Identity providers
• OAuth server
• Login flow
• Authentication certificates
• Token configuration

Example:

User
  │
  ▼
Corporate identity provider
  │
  ▼
OpenShift OAuth
  │
  ▼
OpenShift access token

Check:

oc get co authentication
oc get oauth cluster -o yaml

10. Ingress Operator

The Ingress Operator manages the OpenShift router.

It controls:

• Router Pods
• IngressControllers
• Wildcard certificates
• Router replicas
• Publishing strategy
• Public and private ingress

External client
      │
      ▼
Load balancer
      │
      ▼
Router Pods
      │
      ▼
Route
      │
      ▼
Service
      │
      ▼
Application Pods

Check:

oc get co ingress
oc get ingresscontroller -n openshift-ingress-operator
oc get pods -n openshift-ingress

11. DNS Operator

The DNS Operator manages CoreDNS for cluster Service discovery.

It allows Pods to resolve names such as:

payments-api.banking.svc.cluster.local

Architecture:

Application Pod
      │
      ▼
DNS Service IP
      │
      ▼
CoreDNS
      │
      ├── Internal Service names
      └── External upstream DNS

Check:

oc get co dns
oc get pods -n openshift-dns
oc get dns.operator/default -o yaml

12. Network Operator

The Cluster Network Operator manages the OpenShift network plugin, usually OVN-Kubernetes.

It manages:

• Pod networks
• Service networks
• OVN components
• GENEVE tunnels
• MTU
• Egress features
• NetworkPolicies
• Node networking components

Pod A
  │
  ▼
OVN virtual network
  │
  ▼
Pod B

Check:

oc get co network
oc get network.operator cluster -o yaml
oc get pods -n openshift-ovn-kubernetes

13. Cloud Controller Manager

On cloud platforms, the cloud controller integrates OpenShift with AWS, Azure, or GCP.

It manages functions such as:

• Cloud node information
• Load balancers
• Routes
• Instance metadata
• Cloud volumes, depending on the driver architecture

Example:

Service type LoadBalancer
          │
          ▼
Cloud Controller
          │
          ▼
AWS / Azure / GCP load balancer

14. Machine API Operator

The Machine API Operator manages infrastructure machines on supported platforms.

It handles:

• Machine objects
• MachineSets
• MachineHealthChecks
• Worker creation
• Worker replacement
• Autoscaling integration

MachineSet replicas: 5
         │
         ▼
Machine API Operator
         │
         ▼
Create cloud VMs
         │
         ▼
New OpenShift workers join

Check:

oc get machines -A
oc get machinesets -A
oc get machinehealthchecks -A

15. Monitoring Components

The control plane is monitored by the OpenShift monitoring stack.

Main components include:

• Prometheus
• Alertmanager
• kube-state-metrics
• Node Exporter
• Prometheus Operator
• Thanos components

They monitor:

• API latency
• etcd latency
• Scheduler health
• Operator status
• Node health
• Resource utilization

Check:

oc get pods -n openshift-monitoring

Static Pods on Control-Plane Nodes

Several critical control-plane components run as static Pods:

• etcd
• kube-apiserver
• kube-controller-manager
• kube-scheduler

A static Pod is managed directly by the kubelet on the node.

Static Pod manifest
        │
        ▼
kubelet reads manifest
        │
        ▼
Control-plane Pod starts

This allows core components to start even when the scheduler is unavailable.

Control Plane Request Flow

When you run:

oc create deployment nginx --image=nginx

the full sequence is:

1. oc sends request to API load balancer
2. Load balancer selects a kube-apiserver
3. API server authenticates the user
4. RBAC authorizes the request
5. Admission controls validate the object
6. Deployment is stored in etcd
7. Controller Manager creates a ReplicaSet
8. ReplicaSet controller creates a Pod
9. Scheduler selects a worker node
10. kubelet asks CRI-O to start the container
11. OVN configures Pod networking
12. Pod becomes Running

High Availability

A production OpenShift control plane normally uses three control-plane nodes.

master-0
master-1
master-2

Availability is maintained through:

• Three API server instances
• Three etcd members
• Scheduler leader election
• Controller-manager leader election
• API load balancing
• Rolling upgrades
• Static Pods
• Operator reconciliation

Only one scheduler and controller-manager instance is active as leader at a time, while others remain ready to take over.

Control Plane vs Worker Nodes

Control Plane
     │
     ▼
Decides what should run
Worker Node
     │
     ▼
Runs the actual workload

Important Commands

oc get nodes
oc get clusteroperators
oc get clusterversion
oc get pods -A

Control-plane Pods:

oc get pods -n openshift-etcd
oc get pods -n openshift-kube-apiserver
oc get pods -n openshift-kube-controller-manager
oc get pods -n openshift-kube-scheduler

Operator health:

oc get co

Healthy status:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

Troubleshooting Control Plane

Use this sequence:

API unavailable or slow
        │
        ▼
Check API load balancer and DNS
        │
        ▼
Check kube-apiserver
        │
        ▼
Check etcd health and latency
        │
        ▼
Check control-plane nodes
        │
        ▼
Check scheduler and controllers
        │
        ▼
Check ClusterOperators

Useful commands:

oc get co
oc get nodes
oc get --raw='/readyz?verbose'
oc get pods -n openshift-etcd -o wide
oc get pods -n openshift-kube-apiserver -o wide

For node-level investigation:

oc debug node/<control-plane-node>
chroot /host
systemctl status kubelet
journalctl -u kubelet
iostat -x 1 10

Interview Answer

The OpenShift control plane is responsible for managing the cluster’s desired state and making all scheduling, API, and lifecycle decisions. Its key components are the kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and OpenShift-specific API and controller services.

The API server receives all requests, authenticates and authorizes them, and stores the resulting state in etcd. The controller managers continuously reconcile resources, while the scheduler selects suitable worker nodes for new Pods. OpenShift Operators such as the Cluster Version Operator, Machine Config Operator, Ingress Operator, DNS Operator, Network Operator, and Authentication Operator manage the platform components around the core Kubernetes control plane.

In a highly available cluster, these components run across three control-plane nodes. The API servers are load balanced, etcd maintains quorum, and the scheduler and controller managers use leader election. For troubleshooting, I start with oc get co, check API readiness, etcd health, control-plane Pods and nodes, and then review Operator conditions and logs.

WAL fsync in OpenShift

In OpenShift, WAL fsync normally refers to how quickly etcd can safely write changes to its Write-Ahead Log on disk.

A simple definition:

Before etcd confirms an important cluster-state change, it writes the change to its WAL and requests that the operating system physically persist it to storage using fsync or fdatasync.

This protects the cluster state if an etcd process, control-plane node, or operating system suddenly fails.

What is a WAL?

WAL means Write-Ahead Log.

etcd records a change in the log before applying it to its main backend database.

API change
   │
   ▼
etcd receives proposal
   │
   ▼
Write proposal to WAL
   │
   ▼
fsync to persistent storage
   │
   ▼
Replicate through Raft
   │
   ▼
Commit transaction
   │
   ▼
API request succeeds

Examples of changes written through etcd include:

• Creating a Pod
• Updating a Deployment
• Creating or changing a Secret
• Updating a ConfigMap
• Changing node status
• Updating EndpointSlices
• Operator status updates
• Creating or deleting Kubernetes resources

What does fsync do?

When an application writes data, the operating system may initially place it in memory cache.

Application write
       │
       ▼
Operating-system page cache
       │
       ▼
Storage device later

That is fast, but data still in memory can be lost during a sudden power or operating-system failure.

fsync tells the operating system:

Do not acknowledge this operation until the data
has been flushed to persistent storage.

For etcd:

WAL entry
   │
   ▼
fsync
   │
   ▼
Disk confirms persistence
   │
   ▼
etcd continues the commit

Therefore, WAL fsync latency is directly influenced by the storage system.

Why OpenShift depends on it

The Kubernetes API server stores cluster state in etcd.

oc apply
   │
   ▼
kube-apiserver
   │
   ▼
etcd leader
   │
   ▼
WAL fsync and Raft replication
   │
   ▼
Commit
   │
   ▼
API response

If WAL fsync becomes slow, etcd writes become slow. That can make the OpenShift API slow because API write operations depend on etcd committing changes.

Red Hat notes that slow storage or competing disk activity can cause high fsync latency, API slowness, request timeouts, missed heartbeats, and temporary etcd leader loss. (Red Hat Documentation)

Example of a normal write

Suppose you run:

oc scale deployment payments-api --replicas=5

The flow is:

1. oc sends PATCH request
2. API server validates and authorizes it
3. API server sends the change to etcd
4. etcd leader writes the proposal to WAL
5. WAL is synchronized to storage
6. Proposal is replicated to other etcd members
7. A majority acknowledges it
8. The change is committed
9. API server returns success

If step 5 takes 2 ms, the transaction can proceed quickly.

If step 5 takes 200 ms or more, API requests accumulate and controllers begin reconciling more slowly.

WAL vs etcd backend database

etcd uses both a WAL and a backend database.

Incoming change
      │
      ▼
WAL
Durable sequential record
      │
      ▼
Raft commit
      │
      ▼
Backend database
Current key-value state

They have separate performance metrics:

Interpretation:

High WAL fsync latency
→ synchronous WAL storage problem
High backend commit latency
→ backend database storage or database-pressure issue
Both high
→ general disk contention, throttling, or storage degradation

Main Prometheus metric

The key histogram metric is:

etcd_disk_wal_fsync_duration_seconds_bucket

To calculate the p99 latency:

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

This shows the WAL fsync duration below which approximately 99% of observations occurred during the selected period.

Breaking it down by instance helps identify one slow control-plane node:

master-0 → 0.004 seconds
master-1 → 0.120 seconds
master-2 → 0.005 seconds

Here, master-1 is the likely problem.

Red Hat identifies WAL fsync duration, backend commit duration, and leader changes as important metrics for evaluating etcd storage performance. (Red Hat Documentation)

What is an acceptable value?

The exact alert threshold can vary by OpenShift release, workload, and test method.

Current Red Hat guidance for validating etcd storage includes checking the p99 fsync result produced by its supported fio-based performance test. Some current documentation uses a threshold below 10 ms, while other recent release documentation and test output refer to 20 ms. Use the guidance and alerts supplied for your exact OpenShift version rather than applying one universal value. (Red Hat Documentation)

As a practical operational interpretation:

A few milliseconds
→ healthy low-latency storage
Consistent tens of milliseconds
→ investigate
Large or recurring spikes
→ likely to affect etcd and API performance

Focus on:

• p99 behavior
• Sustained duration
• Differences between members
• Correlation with API latency
• Leader changes and timeouts

A single isolated spike is less concerning than repeated or sustained high latency.

Causes of high WAL fsync latency

1. Slow storage

Examples:

• Mechanical disks
• Slow SAN
• High-latency network-backed block storage
• Poorly configured virtual disks
• Underperforming SSDs

2. IOPS or throughput throttling

Cloud disks and virtual machines can have:

• Disk IOPS limits
• Throughput limits
• Burst-credit exhaustion
• Instance-wide storage limits

3. Noisy neighbours

The etcd virtual disk might share physical infrastructure with:

• Other virtual machines
• Databases
• Backup jobs
• Storage replication
• Large image operations

4. Snapshots and backups

Hypervisor snapshots or storage backups can temporarily increase latency.

5. Control-plane processes producing I/O

Examples:

• Logging agents
• Security scanners
• Backup agents
• Excessive journal writes
• Container image operations

6. Device or filesystem errors

Examples:

• Storage path failures
• NVMe timeouts
• SAN multipath issues
• Filesystem problems
• Disk nearly full

Red Hat recommends low-latency block storage for etcd and advises against sharing its underlying I/O infrastructure with competing I/O-intensive workloads. (Red Hat Documentation)

Symptoms in OpenShift

High WAL fsync latency can produce:

• Slow oc commands
• API request timeouts
• Slow application deployments
• Delayed Operator reconciliation
• Pods remaining Pending longer
• ClusterOperators becoming degraded
• etcd slow fdatasync messages
• Missed Raft heartbeats
• Increased leader elections
• Web console delays

Slow disk
   │
   ▼
Slow WAL fsync
   │
   ▼
Slow etcd commits
   │
   ▼
Slow API server
   │
   ▼
Slow controllers and Operators
   │
   ▼
Cluster-wide control-plane impact

Existing application containers might continue processing traffic, but control-plane changes become slow or fail.

Related metrics

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Failed proposals

rate(etcd_server_proposals_failed_total[5m])

Peer network latency

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Interpret the metrics together:

High fsync + normal peer RTT
→ storage issue
Normal fsync + high peer RTT
→ network issue
High fsync + leader changes
→ storage may be destabilizing Raft
High fsync + pending proposals
→ etcd cannot commit writes fast enough

Node-level troubleshooting

Identify the slow member using Prometheus, then debug its control-plane node:

oc debug node/master-1

Enter the host:

chroot /host

Check where etcd data resides:

findmnt /var/lib/etcd
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
df -h
df -i

Check real-time disk performance:

iostat -x 1 10

Important fields:

Look for:

High w_await
Growing aqu-sz
Sustained device pressure
Spikes matching etcd latency

%util alone is not sufficient for modern parallel devices. Latency and queue depth provide better context.

Find competing processes

Use:

pidstat -d 1 10

Possible output:

PID    kB_rd/s    kB_wr/s    COMMAND
2100      0.00      800.00    etcd
7350      0.00     9000.00    backup-agent

This suggests that backup-agent may be competing with etcd.

Historical statistics:

sar -d 1 10
sar -u 1 10
sar -q 1 10

Check kernel storage errors:

journalctl -k --since "1 hour ago" |
  grep -Ei 'I/O error|timeout|reset|nvme|scsi|xfs'

Check etcd logs

List etcd Pods:

oc get pods -n openshift-etcd -o wide

Review one member:

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h

Search for likely symptoms:

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h |
grep -Ei \
  'slow fdatasync|took too long|timeout|heartbeat|leader|election'

Do not confuse WAL fsync with network replication

A write requires both disk durability and Raft consensus.

etcd leader
   │
   ├── WAL fsync on local disk
   ├── Replication to peers
   └── Majority acknowledgement

Therefore, a slow transaction can come from:

• Local disk fsync latency
• Peer disk latency
• Network round-trip latency
• CPU starvation
• etcd overload

Always compare disk and network metrics before concluding that storage is the only problem.

Remediation

Immediate

• Stop or reschedule competing backup or scanning jobs.
• Resolve storage path failures.
• Fix disk or VM throttling.
• Reduce runaway API writes.
• Pause nonessential bulk deployments.
• Remove unrelated I/O-intensive activity from control-plane storage.

Permanent

• Use dedicated low-latency SSD or NVMe-backed storage.
• Use guaranteed IOPS rather than burst-only storage.
• Ensure the VM instance supports sufficient total storage throughput.
• Isolate control-plane nodes from noisy neighbours.
• Avoid putting application workloads on control-plane nodes.
• Monitor p99 WAL fsync continuously.
• Validate storage with the Red Hat-supported fio procedure.

Do not react by restarting every etcd member. That could destroy quorum.

Also, do not use defragmentation as the first solution: it may reduce internal database fragmentation, but it does not repair slow physical storage.

Interview answer

WAL fsync in OpenShift is the time etcd takes to durably persist a Raft proposal into its Write-Ahead Log. When an API operation changes cluster state, the API server sends it to etcd. The etcd leader records the proposal in the WAL, synchronizes it to persistent storage, replicates it to the other members, and commits it after a majority acknowledges the proposal.

Because fsync waits for storage durability, slow storage directly increases etcd transaction latency and therefore OpenShift API latency. Sustained high fsync latency can cause request timeouts, pending proposals, missed Raft heartbeats, and leader changes. I monitor the p99 value of etcd_disk_wal_fsync_duration_seconds_bucket, compare it across members, and correlate it with backend commit latency, peer network RTT, leader changes, and API latency. At the node level, I use oc debug node, iostat, sar, and pidstat to identify disk latency, queueing, throttling, or competing processes. The permanent solution is isolated, low-latency storage with guaranteed IOPS—not repeatedly restarting etcd or treating defragmentation as a disk-performance fix.

Leader Changes and Pending Proposals in OpenShift etcd

Both metrics describe the stability and performance of the etcd cluster, which stores OpenShift control-plane state.

kube-apiserver
      │
      ▼
etcd leader
      │
      ├── Writes WAL locally
      ├── Replicates proposal to followers
      └── Waits for quorum

A healthy etcd cluster should have:

• A stable leader
• Very few unexpected leader changes
• Pending proposals normally close to zero
• Low WAL fsync latency
• Low peer-network latency

1. What is an etcd leader?

etcd uses the Raft consensus algorithm. In a standard three-member OpenShift etcd cluster:

master-0: etcd leader
master-1: etcd follower
master-2: etcd follower

The leader handles cluster-state writes.

For example:

oc scale deployment payments --replicas=5

The write process is:

API server sends update
        │
        ▼
etcd leader creates proposal
        │
        ▼
Leader writes proposal to WAL
        │
        ▼
Proposal replicated to followers
        │
        ▼
Majority acknowledges
        │
        ▼
Proposal committed
        │
        ▼
API request succeeds

The leader keeps followers synchronized and commits a write only after quorum acknowledges it. (Red Hat Documentation)

2. What is a leader change?

A leader change occurs when the current leader stops being leader and another etcd member is elected.

Before:
master-0 = leader
master-1 = follower
master-2 = follower
Leader heartbeat lost
        │
        ▼
Election timeout reached
        │
        ▼
New election
After:
master-0 = follower/unavailable
master-1 = leader
master-2 = follower

Leader election is a normal high-availability mechanism. The problem is not an occasional leader change during planned maintenance; the problem is frequent or unexpected elections.

Why does the leader change?

Common causes include:

Slow WAL fsync

The leader cannot persist its Raft log quickly enough.

Slow disk
   │
   ▼
Slow WAL fsync
   │
   ▼
Heartbeat processing delayed
   │
   ▼
Followers assume leader failed
   │
   ▼
New election

High network latency or packet loss

Followers do not receive heartbeats in time.

Leader heartbeat
       │
       X packet loss or delay
       │
Follower election timeout
       │
       ▼
Leader election

CPU or memory starvation

The etcd process cannot schedule enough CPU time to send or process heartbeats.

Control-plane node restart

A reboot or static-pod restart can trigger a legitimate election.

Storage or node failure

Examples include:

• Cloud-disk throttling
• SAN congestion
• Datastore latency
• Failed storage path
• Hypervisor pause
• Node hardware failure

Slow storage and competing disk activity can cause long fsync times, missed heartbeats, request timeouts, and temporary leader loss. (Red Hat Documentation)

Leader-change metric

Use:

etcd_server_leader_changes_seen_total

This is a cumulative counter. To see recent changes:

increase(etcd_server_leader_changes_seen_total[15m])

Interpretation:

0 during normal operation
→ Stable leader
1 during a planned master reboot
→ Usually expected
Repeated changes in a short period
→ Investigate immediately

Because each member may expose the counter, inspect it by instance:

sum by (instance) (
  increase(etcd_server_leader_changes_seen_total[15m])
)

Impact of frequent leader changes

During an election, etcd briefly has no active leader.

Leader lost
    │
    ▼
Election in progress
    │
    ▼
Writes temporarily pause
    │
    ▼
API write requests wait or time out

Symptoms can include:

• Slow oc apply, oc create, or oc delete
• API request timeouts
• Operators reconciling slowly
• Delayed node status updates
• Pods taking longer to create
• ClusterOperators becoming degraded
• request timed out messages
• Temporary control-plane instability

During leader loss and reelection, Kubernetes API requests that cause state changes can be interrupted or delayed. (Red Hat Documentation)

3. What is an etcd proposal?

A proposal is a requested change to etcd state that must pass through Raft consensus.

Examples include:

Create Deployment
Update Secret
Delete Pod
Modify ConfigMap
Update Node status
Change Route
Update Operator status

Simplified flow:

API write
   │
   ▼
Proposal created
   │
   ▼
WAL persisted
   │
   ▼
Replicated to followers
   │
   ▼
Quorum reached
   │
   ▼
Proposal committed

4. What are pending proposals?

A pending proposal is a proposal that etcd has received but has not yet committed.

The metric is:

etcd_server_proposals_pending

This is a gauge showing the current number of outstanding proposals.

Normally:

Pending proposals ≈ 0

Brief small increases during bursts can be normal.

A sustained or rising value indicates that etcd is receiving changes faster than it can persist, replicate, and commit them.

Incoming proposals
        │
        ▼
etcd processing capacity insufficient
        │
        ▼
Queue grows
        │
        ▼
Pending proposals increase

Why do proposals remain pending?

Slow disk writes

Proposal
   │
   ▼
Waiting for WAL fsync
   │
   ▼
Proposal remains pending

Slow peer replication

Leader sends proposal
       │
       ▼
Follower response delayed
       │
       ▼
Waiting for quorum

Leader election

Proposals can pause while a new leader is elected.

Excessive Kubernetes API writes

A runaway Operator or automation process may generate more writes than etcd can process.

Examples:

• Controller updating status continuously
• CI/CD loop repeatedly creating resources
• Excessive Kubernetes Events
• Large bulk deployments
• Frequent ConfigMap or Secret updates
• Broken automation repeatedly patching objects

CPU pressure

The etcd member cannot process requests promptly.

Oversized API objects

Large objects require more disk, network, and serialization work.

Pending-proposal interpretation

etcd_server_proposals_pending

Example:

0–a few, briefly
→ Usually normal during a write burst
Continuously above zero
→ etcd is falling behind
Steadily increasing
→ Severe processing, disk, or network bottleneck

Also check failed proposals:

rate(etcd_server_proposals_failed_total[5m])

Interpret together:

Pending rising, failures zero
→ Requests are delayed but may eventually commit
Pending rising, failures rising
→ etcd cannot successfully process part of the workload

Relationship between the two metrics

Leader changes and pending proposals often appear together.

Slow disk or network
        │
        ├── WAL commits slow
        │       └── Pending proposals rise
        │
        └── Heartbeats delayed
                └── Leader changes rise

Example:

WAL fsync p99:         180 ms
Pending proposals:     75
Leader changes:        6 in 15 minutes
API requests:          timing out

This strongly suggests etcd instability, often caused by disk or network latency.

Important correlation matrix

Metrics to examine together

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Failed proposals

rate(etcd_server_proposals_failed_total[5m])

WAL fsync p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

Backend commit p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Peer RTT p99

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Red Hat identifies WAL fsync duration, backend commit latency, leader changes, and peer RTT as important etcd performance signals. (Red Hat Documentation)

Troubleshooting procedure

Step 1: Check etcd health

oc get co etcd
oc describe co etcd
oc get pods -n openshift-etcd -o wide

Look for:

• Degraded conditions
• Pod restarts
• One unhealthy member
• Revision rollout problems

Step 2: Identify the current leader

oc get pods -n openshift-etcd -o wide

Enter a healthy etcd Pod and run:

etcdctl endpoint status --cluster -w table

The output shows which member is the leader. Use the supported certificate environment and command procedure for your OCP version.

Step 3: Check etcd logs

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h |
grep -Ei \
  'leader|election|heartbeat|slow fdatasync|timeout|proposal|took too long'

Look for messages indicating:

leader changed
lost leader
elected leader
failed to send heartbeat
slow fdatasync
request timed out
apply request took too long

Step 4: Check disk performance

On the affected control-plane node:

oc debug node/<master-node>
chroot /host

Then run:

iostat -x 1 10
sar -d 1 10
pidstat -d 1 10
df -h
df -i

Check:

• await and w_await
• aqu-sz
• Storage utilization
• Competing backup or logging processes
• Disk capacity and inodes
• Hypervisor or cloud-disk throttling

Step 5: Check network latency

Compare peer round-trip latency between etcd members.

Also investigate:

• Packet loss
• Firewall state
• MTU mismatch
• NIC errors
• Hypervisor networking
• Cross-site or cross-zone latency
• Network jitter

Step 6: Check excessive API writers

Use API metrics:

sum by (verb, resource) (
  rate(apiserver_request_total[5m])
)

Find high-volume clients:

topk(
  20,
  sum by (user_agent, verb) (
    rate(apiserver_request_total[5m])
  )
)

Look for unexpected increases in:

POST
PUT
PATCH
DELETE

Example production incident

02:00 backup starts on control-plane datastore
             │
             ▼
Disk write latency increases
             │
             ▼
WAL fsync rises to 150 ms
             │
             ├── Proposals queue
             │       └── pending proposals = 60
             │
             └── Heartbeats delayed
                     └── 4 leader elections
             │
             ▼
API becomes slow

Resolution:

• Stop or reschedule the competing backup.
• Move control-plane storage to isolated low-latency disks.
• Verify cloud or datastore IOPS and throughput.
• Confirm pending proposals return near zero.
• Confirm no new leader changes.
• Verify API latency and ClusterOperator health.

What not to do

Avoid:

• Restarting all etcd members simultaneously
• Restarting all control-plane nodes together
• Manually deleting etcd data
• Removing members without the supported procedure
• Changing election timers as the first fix
• Defragmenting repeatedly to solve physical disk latency

OpenShift uses validated etcd timer values for each platform. Changing timers can hide symptoms rather than correct the underlying disk or network problem. (Red Hat Documentation)

Interview answer

An etcd leader change occurs when the current Raft leader becomes unavailable or fails to deliver heartbeats within the election timeout, causing another member to be elected. Occasional leader changes during maintenance can be expected, but frequent changes indicate instability caused by disk fsync latency, network delay, packet loss, CPU starvation, or control-plane node failures. I monitor this using increase(etcd_server_leader_changes_seen_total[15m]).

A pending proposal is an etcd write request that has been accepted but has not yet been committed through Raft consensus. The metric etcd_server_proposals_pending should normally stay close to zero. A sustained increase means etcd cannot persist or replicate writes as fast as they arrive, commonly because of slow WAL storage, high peer latency, resource pressure, or excessive API write activity.

I correlate both metrics with WAL fsync latency, backend commit latency, peer RTT, proposal failures, API request rates, and etcd logs. If leader changes and pending proposals increase together, I treat it as a serious control-plane performance problem and investigate the affected member’s storage, network, CPU, and competing processes.

etcd Operator in OpenShift

The etcd Operator manages the lifecycle, configuration, health, certificates, and membership of the etcd cluster that stores OpenShift’s control-plane state.

In a standard highly available OpenShift cluster, etcd runs on the three control-plane nodes:

                  OpenShift API
                       │
                       ▼
                kube-apiserver
                       │
                       ▼
                etcd cluster
          ┌────────────┼────────────┐
          ▼            ▼            ▼
      master-0      master-1      master-2
       etcd-0        etcd-1        etcd-2
          ▲            ▲            ▲
          └────────────┼────────────┘
                       │
                  etcd Operator

The etcd Operator continually observes the cluster, compares the current state with the required state, and corrects differences through the Kubernetes and etcd management APIs. (Red Hat Documentation)

Why etcd Is Critical

etcd is the authoritative database for Kubernetes and OpenShift.

It stores objects such as:

• Deployments
• Pods and their desired state
• Services
• Routes
• Secrets
• ConfigMaps
• RBAC
• Nodes
• CRDs and Custom Resources
• Operator configuration
• MachineConfig objects
• Cluster configuration

The runtime contents of containers and application databases are not stored in etcd.

oc apply -f deployment.yaml
             │
             ▼
       kube-apiserver
             │
             ▼
            etcd
             │
             ▼
 Deployment object stored
             │
             ▼
 Controllers create Pods

If etcd becomes unavailable, existing containers can often continue running temporarily, but:

• New Pods cannot be scheduled.
• Configuration changes cannot be saved.
• Operators cannot reconcile normally.
• oc commands that require the API begin failing.
• Cluster recovery and automation stop functioning correctly.

etcd Operator vs etcd

These are different components:

Cluster Version Operator
          │
          ▼
      etcd Operator
          │
          ▼
      etcd members
          │
          ▼
   Cluster state database

The Operator itself does not store the cluster state. It manages the etcd processes that do.

Location and Resources

The Operator normally runs in:

openshift-etcd-operator

The etcd static Pods run in:

openshift-etcd

Check them:

oc get pods -n openshift-etcd-operator
oc get pods -n openshift-etcd -o wide

Check the ClusterOperator:

oc get clusteroperator etcd

The cluster-scoped configuration resource is:

oc get etcd cluster -o yaml

The etcd cluster Operator provides the cluster-scoped etcds.operator.openshift.io API and is configured through the etcd/cluster object. (Red Hat Documentation)

Main Responsibilities of the etcd Operator

1. Deploying etcd as static Pods

On each control-plane node, etcd runs as a static Pod.

Static Pod manifest
       │
       ▼
kubelet on master node
       │
       ▼
etcd Pod starts

Typical Pods:

oc get pods -n openshift-etcd -o wide

Example:

etcd-master-0
etcd-master-1
etcd-master-2

Static Pods are managed directly by the kubelet, not by a Deployment.

This is important because core control-plane services must be able to start even when normal Kubernetes scheduling is unavailable.

2. Maintaining etcd membership

A three-member etcd cluster normally has:

Member 1: master-0
Member 2: master-1
Member 3: master-2

The Operator monitors whether the expected members match the available control-plane nodes.

When a control-plane node is properly replaced, the Operator can:

• Generate certificates for the new member
• Add the replacement member to etcd
• Remove stale membership
• Reconcile the new topology

Red Hat documents that when a lost control-plane node is replaced, the etcd cluster Operator handles generating new TLS certificates and adding the new node as an etcd member. (Red Hat Documentation)

3. Preserving quorum

etcd uses the Raft consensus algorithm.

For three members:

Members: 3
Required quorum: 2
Maximum simultaneous failures: 1

For five members:

Members: 5
Required quorum: 3
Maximum simultaneous failures: 2

A standard OpenShift control plane normally uses three members.

master-0    master-1    master-2
 Healthy     Healthy     Failed
      \         /
       Quorum remains

If two of three members are lost:

master-0    master-1    master-2
 Healthy     Failed      Failed
      │
      ▼
  No quorum

The Operator cannot simply recreate lost authoritative state when quorum is gone. You must follow the documented disaster-recovery procedure and restore from a valid backup. Red Hat explicitly distinguishes single-member replacement from loss of the majority of control-plane hosts. (Red Hat Documentation)

4. Managing certificates

etcd communication is secured with TLS.

Certificates include:

• Peer certificates for member-to-member communication
• Server certificates
• Client certificates for API server access
• Certificate authority bundles

etcd-0  ←── mutual TLS ──→  etcd-1
   │                           │
   └────── mutual TLS ────────→ etcd-2

The Operator manages the certificate resources and rolls out new static-Pod revisions when certificates rotate.

It also ensures the kube-apiserver has the required trust and client credentials to connect to etcd securely.

5. Managing static-Pod revisions

Configuration changes are rolled out using versioned revisions.

Current revision 20
        │
 Configuration changes
        ▼
New revision 21 generated
        │
        ▼
Install on control-plane nodes
        │
        ▼
Validate member health

You can inspect revision-related resources:

oc get configmaps -n openshift-etcd
oc get secrets -n openshift-etcd

The Operator ensures the expected configuration, certificates, and manifests are synchronized across the control-plane nodes.

6. Monitoring cluster health

The Operator monitors:

• Member availability
• Quorum
• Endpoint health
• Static-Pod revisions
• Certificate status
• Member synchronization
• Leader stability
• Backup and defragmentation-related conditions
• Storage and API-visible health signals

Check its high-level condition:

oc get co etcd

A healthy status is:

AVAILABLE     True
PROGRESSING   False
DEGRADED      False

Detailed conditions:

oc describe co etcd

7. Supporting member recovery

If one etcd member fails but quorum remains, the recovery process depends on the failure type:

• The control-plane machine is stopped.
• The node is NotReady.
• The etcd Pod is crash-looping.
• The underlying machine was permanently lost.
• The certificates are invalid.

The Operator can reconcile the member when a temporarily unavailable node returns. For permanent loss, replacement must follow the supported procedure.

Red Hat recommends taking an etcd backup before replacing an unhealthy member. (Red Hat Documentation)

8. Automating defragmentation and maintenance

etcd receives a high number of small updates and deletions. Deleted data can leave unused space inside the backend database.

Objects created and updated
          │
          ▼
Objects deleted
          │
          ▼
Unused internal database pages
          │
          ▼
Fragmentation

The etcd Operator performs supported maintenance activities, including automatic defragmentation behavior in current OpenShift releases.

However:

Defragmentation is not a fix for slow physical storage.

If WAL fsync latency is high because the disk is saturated, the solution is usually faster or isolated storage, not repeated defragmentation.

Reconciliation Loop

The etcd Operator follows the normal Operator control-loop model:

Observe control-plane nodes and etcd state
                     │
                     ▼
       Read desired configuration
                     │
                     ▼
 Compare desired state with actual state
                     │
             ┌───────┴────────┐
             │                │
          Matches          Difference
             │                │
             ▼                ▼
         No change      Reconcile resources
                              │
               ┌──────────────┼──────────────┐
               ▼              ▼              ▼
          Update static   Rotate certs   Fix membership
          Pod revision
                              │
                              ▼
                       Validate health
                              │
                              ▼
                       Update status

Examples that trigger reconciliation include:

• A control-plane node is replaced.
• A certificate needs rotation.
• A new OpenShift release changes etcd.
• Static-Pod configuration differs.
• An expected member is missing.
• A member returns after temporary failure.

How an API Request Uses etcd

For example:

oc apply -f app.yaml

The flow is:

oc client
    │
    ▼
API load balancer
    │
    ▼
kube-apiserver
    │
    ├── Authentication
    ├── Authorization
    ├── Admission
    └── Validation
           │
           ▼
          etcd
           │
           ▼
  Persist Kubernetes object

The etcd Operator is not in the request data path. It ensures the etcd cluster receiving the request remains healthy and correctly configured.

etcd Leader and Followers

One etcd member acts as the Raft leader.

                etcd leader
                    │
             Replicates writes
          ┌─────────┴─────────┐
          ▼                   ▼
      follower             follower

A write is committed after a majority acknowledges it:

API write
   │
   ▼
Leader writes WAL
   │
   ├── replicate to follower 1
   └── replicate to follower 2
             │
             ▼
      Majority acknowledges
             │
             ▼
         Commit write

This is why etcd requires:

• Low-latency storage
• Reliable networking
• Low latency between control-plane nodes
• Accurate time synchronization
• Stable control-plane resources

etcd continuously persists many small changes, making fast, low-latency I/O especially important. (Red Hat Documentation)

Useful Troubleshooting Commands

Check ClusterOperator status

oc get co etcd
oc describe co etcd

Check etcd Operator

oc get pods -n openshift-etcd-operator

oc logs -n openshift-etcd-operator \
  deployment/etcd-operator \
  --since=1h

Check etcd Pods

oc get pods -n openshift-etcd -o wide

Inspect Pod containers

oc describe pod -n openshift-etcd <etcd-pod>

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h

Check configuration

oc get etcd cluster -o yaml

Check nodes

oc get nodes
oc describe node <control-plane-node>

Checking Endpoint Health

First identify the etcd Pods:

oc get pods -n openshift-etcd --show-labels

Then enter a healthy etcd Pod:

oc rsh -n openshift-etcd <etcd-pod>

Depending on the OpenShift version and container environment, run the provided etcdctl command with the appropriate certificates:

etcdctl endpoint health --cluster
etcdctl endpoint status --cluster -w table

The status output helps identify:

• Member ID
• Endpoint
• etcd version
• Database size
• Leader
• Raft term and index
• Errors

Use the commands and certificate paths documented for the exact OpenShift version rather than inventing or replacing TLS parameters manually.

Important etcd Metrics

WAL fsync latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

High values indicate slow synchronous writes.

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Database size

etcd_mvcc_db_total_size_in_bytes

These help distinguish:

High WAL latency
      → Storage issue
High peer RTT
      → Network issue
Frequent leader changes
      → Storage, network, or resource instability
Increasing pending proposals
      → etcd cannot process writes quickly enough

Common Failure Scenarios

Scenario 1: One member is unavailable

Three members
   │
   ├── Two healthy
   └── One failed

Result:

• Quorum remains.
• API usually continues functioning.
• The Operator reports degradation.
• Investigate and replace or recover the unhealthy member using the supported procedure.

Do not immediately delete etcd data or membership manually.

Scenario 2: Two members are unavailable

Three members
   │
   ├── One healthy
   └── Two failed

Result:

• Quorum is lost.
• Writes stop.
• API availability is severely affected.
• Normal Operator reconciliation cannot restore the authoritative state.
• Perform control-plane disaster recovery from a valid etcd snapshot.

Scenario 3: etcd Pod is CrashLoopBackOff

Check:

oc describe pod -n openshift-etcd <pod>
oc logs -n openshift-etcd <pod> -c etcd --previous

Possible causes:

• Corrupt or unavailable storage
• Certificate failure
• Invalid member state
• Static-Pod revision problem
• Disk full
• File permissions
• Network or peer connectivity
• Node-level failure

Scenario 4: Slow API caused by etcd

Symptoms:

• Slow oc commands
• API timeouts
• Operators become degraded
• Leader changes
• Slow fdatasync messages

Investigate:

oc debug node/<master-node>
chroot /host
iostat -x 1 10
sar -d 1 10
pidstat -d 1 10
df -h
df -i

Correlate node storage metrics with etcd WAL fsync and backend commit latency.

Backups

The Operator manages etcd operation, but the administrator must maintain a tested backup strategy.

A control-plane backup contains:

etcd snapshot
       +
static Kubernetes resources

Run the documented backup script from a healthy control-plane node and copy the resulting files to secure off-cluster storage.

Backups should be:

• Automated
• Encrypted
• Stored off-cluster
• Access controlled
• Tested regularly
• Matched to documented recovery procedures

An etcd snapshot does not replace application database or persistent-volume backups.

What Not to Do

Avoid:

• Deleting /var/lib/etcd
• Manually editing static-Pod manifests
• Manually removing etcd members without the supported procedure
• Restarting all control-plane nodes together
• Restarting all etcd members simultaneously
• Copying a data directory between members
• Restoring a snapshot into a live healthy cluster
• Treating defragmentation as the first fix for disk contention
• Editing Operator-managed resources directly

Unsafe etcd changes can cause permanent loss of cluster state.

Interview Answer

The etcd Operator manages the OpenShift control-plane etcd cluster. etcd itself stores the authoritative Kubernetes state, while the Operator deploys and maintains the etcd static Pods, certificates, configuration revisions, cluster membership and health. It continuously compares the desired state with the actual state and reconciles differences.

In a standard highly available cluster, etcd runs as three members on the control-plane nodes and requires two members for quorum. If one member fails, the cluster can continue operating while the member is recovered or replaced. If the majority is lost, the Operator cannot recreate the missing state, and the cluster must be restored using the documented disaster-recovery process and a valid etcd snapshot.

For troubleshooting, I begin with oc get co etcd, inspect the Operator conditions and logs, check the etcd static Pods and endpoint health, verify control-plane nodes, and examine WAL fsync, backend commit, peer latency, leader changes and pending proposals. I also verify disk latency and capacity because etcd depends on fast, low-latency storage.

Role of the Scheduler in OpenShift

The scheduler decides which node should run a newly created Pod.

It does not start containers itself. Its job is to select the best eligible node and assign the Pod to it.

User creates Pod
      │
      ▼
API Server stores Pod
      │
      ▼
Pod has no nodeName
      │
      ▼
Scheduler evaluates nodes
      │
      ▼
Scheduler selects one node
      │
      ▼
kubelet starts the Pod

Where the Scheduler Runs

In a standard OpenShift cluster, the Kubernetes scheduler runs on the control-plane nodes as a static Pod.

Check it with:

oc get pods -n openshift-kube-scheduler -o wide

The scheduler is managed by the Kubernetes Scheduler Operator.

Check its status:

oc get co kube-scheduler

Healthy status:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

Scheduler vs Scheduler Operator

These are different components.

Scheduler Operator
        │
        ▼
Manages kube-scheduler
        │
        ▼
kube-scheduler assigns Pods

Scheduling Flow

Suppose a Pod is created:

apiVersion: v1
kind: Pod
metadata:
  name: payments-api
spec:
  containers:
    - name: app
      image: registry.example.com/payments-api:1.0

Initially, the Pod has no node assignment:

spec:
  nodeName: ""

The scheduler watches the API server for these unscheduled Pods.

Step 1: Watch for Pending Pods

The scheduler detects:

Pod: payments-api
Status: Pending
Node: none

It adds the Pod to its scheduling queue.

Step 2: Filter Nodes

The scheduler eliminates nodes that cannot run the Pod.

This is called the filtering phase.

Example:

Available nodes:
worker-1
worker-2
worker-3
worker-4

The scheduler checks:

• CPU availability
• Memory availability
• Node readiness
• Taints and tolerations
• Node selectors
• Node affinity
• Pod affinity and anti-affinity
• Persistent volume topology
• Host ports
• Pod count limits
• Resource constraints

After filtering:

worker-1 → eligible
worker-2 → insufficient memory
worker-3 → taint not tolerated
worker-4 → nodeSelector mismatch

Only worker-1 remains.

Step 3: Score Nodes

If several nodes are eligible, the scheduler scores them.

Example:

worker-1 → score 85
worker-2 → score 72
worker-3 → score 91

The scheduler selects:

worker-3

Scoring can consider:

• Resource balance
• Node affinity preferences
• Pod spreading
• Image locality
• Existing workload distribution
• Topology preferences

Step 4: Bind the Pod

The scheduler updates the Pod through the API server:

spec:
  nodeName: worker-3

This is called binding.

Step 5: kubelet Starts the Pod

The kubelet on worker-3 sees the assignment.

kubelet
   │
   ▼
CRI-O
   │
   ▼
Pull image
   │
   ▼
Configure networking
   │
   ▼
Mount volumes
   │
   ▼
Start container

The scheduler is no longer involved after the node assignment unless the Pod is recreated.

Important Point: Scheduler Does Not Move Running Pods

The scheduler normally handles only Pods that do not yet have a node assignment.

It does not automatically move a running Pod from one node to another just because another node becomes less busy.

Running Pod on worker-1
          │
          ▼
Scheduler does not rebalance it automatically

To redistribute workloads, you may use:

• Descheduler
• Node drain
• Pod eviction
• Deployment rollout
• Cluster autoscaling
• Manual rescheduling

Resource Requests

The scheduler uses resource requests, not actual current consumption, when deciding placement.

Example:

resources:
  requests:
    cpu: "2"
    memory: 4Gi
  limits:
    cpu: "4"
    memory: 8Gi

The scheduler reserves:

2 CPU
4 GiB memory

It does not schedule based on the Pod’s current real-time usage.

This is why incorrect requests can cause poor scheduling.

Requests too high

Pod remains Pending
Insufficient CPU or memory

Requests too low

Node becomes overloaded
Pods compete for resources

Node Selectors

A node selector forces a Pod onto nodes with matching labels.

Example:

spec:
  nodeSelector:
    workload-type: payments

Only nodes labeled:

oc label node worker-3 workload-type=payments

are eligible.

Node Affinity

Node affinity provides more expressive placement rules.

Example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - zone-a
                - zone-b

This means the Pod must run in zone-a or zone-b.

Pod Affinity

Pod affinity places Pods near other Pods.

Example use case:

Application Pod
      near
Cache Pod

This can reduce latency but may reduce fault isolation.

Pod Anti-Affinity

Pod anti-affinity spreads Pods apart.

Example:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: payments-api
        topologyKey: kubernetes.io/hostname

This prevents two payments-api replicas from running on the same node.

worker-1 → payments-api-1
worker-2 → payments-api-2
worker-3 → payments-api-3

This improves high availability.

Topology Spread Constraints

Topology spread constraints distribute Pods across zones or nodes.

Example:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payments-api

This helps distribute replicas evenly across availability zones.

Zone A → 2 Pods
Zone B → 2 Pods
Zone C → 2 Pods

Taints and Tolerations

A taint prevents Pods from being scheduled unless they tolerate it.

Example taint:

oc adm taint nodes worker-3 dedicated=payments:NoSchedule

Only Pods with this toleration can run there:

tolerations:
  - key: dedicated
    operator: Equal
    value: payments
    effect: NoSchedule

Typical OpenShift uses include:

• Control-plane protection
• Infrastructure nodes
• GPU nodes
• Storage nodes
• Dedicated application nodes

Control-Plane Nodes

Control-plane nodes normally have taints that prevent ordinary application workloads.

node-role.kubernetes.io/master:NoSchedule

or:

node-role.kubernetes.io/control-plane:NoSchedule

Platform Pods have the required tolerations, but regular application Pods do not.

Storage-Aware Scheduling

For Pods using persistent storage, the scheduler must consider volume topology.

Example:

PVC is available only in zone-a
         │
         ▼
Pod must be scheduled to node in zone-a

The scheduler evaluates:

• StorageClass
• PersistentVolume
• Volume node affinity
• Availability zone
• CSI driver constraints

A Pod may remain Pending if no eligible node can access the volume.

Scheduler Profiles in OpenShift

OpenShift supports scheduler profiles that influence how workloads are placed.

Common profiles include:

• LowNodeUtilization
• HighNodeUtilization
• NoScoring

A simplified interpretation:

Check scheduler configuration:

oc get scheduler cluster -o yaml

Example:

apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  name: cluster
spec:
  profile: LowNodeUtilization

Use supported configuration through the cluster Scheduler resource rather than editing static Pod manifests.

High Availability

The scheduler runs on control-plane nodes, but only one instance is active as leader at a time.

scheduler-master-0 → leader
scheduler-master-1 → standby
scheduler-master-2 → standby

If the leader fails:

Leader unavailable
       │
       ▼
Leader election
       │
       ▼
Another scheduler becomes active

Existing Pods continue running during a scheduler outage, but new Pods cannot be assigned until scheduling resumes.

What Happens if the Scheduler Is Down?

Existing workloads generally continue running.

However:

• New Pods stay Pending.
• Failed Pods cannot be placed on another node.
• Deployments cannot scale successfully.
• New Jobs remain unscheduled.
• Node drain replacements may remain Pending.
• Cluster upgrades can be affected.

Example:

Deployment replicas desired: 5
Running: 3
Pending: 2

Pending Pod Troubleshooting

Start with:

oc get pod <pod-name> -n <namespace>

Then:

oc describe pod <pod-name> -n <namespace>

Look at Events.

Common messages:

0/10 nodes are available:
3 insufficient memory
2 node(s) had untolerated taint
4 node(s) didn't match node selector
1 node(s) had volume node affinity conflict

This message is often the fastest way to identify the scheduling problem.

Common Scheduling Failures

Insufficient CPU

0/5 nodes are available:
5 Insufficient cpu

Check:

oc adm top nodes
oc describe node <node>

Remember that scheduling uses requested CPU, not actual CPU usage.

Insufficient memory

0/5 nodes are available:
5 Insufficient memory

Check allocated requests:

oc describe node <node>

Look under:

Allocated resources

Untolerated taint

node(s) had untolerated taint

Check:

oc describe node <node> | grep -i taint

Then verify the Pod tolerations.

Node selector mismatch

node(s) didn't match Pod's node affinity/selector

Check:

oc get nodes --show-labels

Compare with:

oc get pod <pod> -o yaml

Pod anti-affinity conflict

The placement rules may be too strict for the available number of nodes.

Example:

3 replicas
Only 2 eligible nodes
Required anti-affinity

The third Pod remains Pending.

Volume node affinity conflict

The Pod and volume are tied to different zones.

Check:

oc describe pod <pod>
oc get pv <pv-name> -o yaml
oc get pvc -n <namespace>

Too many Pods on a node

The node may have reached its Pod capacity.

Check:

oc describe node <node>

Look for:

pods: 250

and the number currently allocated.

Scheduler Operator Troubleshooting

Check the ClusterOperator:

oc get co kube-scheduler

Describe it:

oc describe co kube-scheduler

Check scheduler Pods:

oc get pods -n openshift-kube-scheduler -o wide

Check logs:

oc logs -n openshift-kube-scheduler \
  <scheduler-pod> \
  -c kube-scheduler

Check the Operator:

oc get pods -n openshift-kube-scheduler-operator

oc logs -n openshift-kube-scheduler-operator \
  deployment/openshift-kube-scheduler-operator

Check API readiness:

oc get --raw='/readyz?verbose'

The scheduler depends on a healthy API server and etcd.

Scheduling Troubleshooting Flow

Pod Pending
    │
    ▼
oc describe pod
    │
    ▼
Read FailedScheduling event
    │
    ├── Insufficient resources
    ├── Taints/tolerations
    ├── Node selectors
    ├── Affinity rules
    ├── Storage topology
    └── Pod capacity
           │
           ▼
Check scheduler Operator only if many unrelated Pods are affected

A single Pending Pod usually indicates a workload placement issue.

Many unrelated Pending Pods across the cluster may indicate:

• Scheduler failure
• API server problem
• etcd problem
• Cluster-wide capacity shortage
• Broken scheduler configuration

Scheduler vs Autoscaler

The scheduler and autoscaler have different responsibilities.

Flow:

Pod cannot fit
      │
      ▼
Scheduler marks it unschedulable
      │
      ▼
Cluster Autoscaler detects it
      │
      ▼
New node created
      │
      ▼
Scheduler places Pod

Scheduler vs Descheduler

The descheduler does not directly move a Pod. It evicts it, and then the scheduler assigns the replacement.

Interview Answer

The OpenShift scheduler is responsible for assigning unscheduled Pods to suitable nodes. It watches the API server for Pods that do not have a nodeName, places them in a scheduling queue, filters out nodes that cannot satisfy the Pod’s requirements, scores the remaining eligible nodes, and binds the Pod to the best node.

During filtering, it evaluates resource requests, node readiness, taints and tolerations, node selectors, affinity and anti-affinity, topology spread constraints, host ports, and persistent-volume topology. After the scheduler writes the selected node into the Pod specification, the kubelet on that node uses CRI-O to start the containers.

The scheduler does not run containers and normally does not rebalance already running Pods. It is managed by the Kubernetes Scheduler Operator and runs highly available on the control-plane nodes using leader election. For troubleshooting a Pending Pod, I first use oc describe pod and inspect the FailedScheduling event before checking node capacity, labels, taints, affinity rules, storage topology, and finally the scheduler Operator if the problem affects many workloads.

Ingress Operator in OpenShift

The Ingress Operator manages how HTTP and HTTPS traffic enters an OpenShift cluster.

Its primary job is to deploy and maintain the OpenShift router, which receives external requests and forwards them to the correct application Service and Pods.

The simplest traffic flow is:

Client
  │
  ▼
External Load Balancer
  │
  ▼
OpenShift Router
  │
  ▼
Route
  │
  ▼
Service
  │
  ▼
Application Pods

The Ingress Operator runs in:

openshift-ingress-operator

The router Pods normally run in:

openshift-ingress

Ingress Operator vs Router

These are not the same component.

Ingress Operator
       │
       ▼
IngressController CR
       │
       ▼
Router Deployment
       │
       ▼
Router Pods
       │
       ▼
Routes and Services

The Operator manages the routers. The routers process the actual application traffic.

Main Responsibilities

1. Deploying router Pods

The Ingress Operator creates and maintains router workloads.

Check them with:

oc get pods -n openshift-ingress -o wide

Typical router Pods:

router-default-7d8b5c9d4f-abc12
router-default-7d8b5c9d4f-def34

The Operator ensures the desired number of router replicas is running.

Desired replicas: 2
Actual replicas: 1
        │
        ▼
Ingress Operator detects drift
        │
        ▼
Creates replacement router Pod

2. Managing the IngressController resource

The main configuration object is:

oc get ingresscontroller -n openshift-ingress-operator

The default object is usually:

default

Inspect it:

oc get ingresscontroller default \
  -n openshift-ingress-operator \
  -o yaml

A simplified example:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  replicas: 3
  domain: apps.cluster.example.com

The Operator watches this resource and reconciles the router configuration.

3. Managing the wildcard application domain

Applications commonly use names such as:

payments.apps.cluster.example.com
mobile.apps.cluster.example.com
api.apps.cluster.example.com

The wildcard DNS record:

*.apps.cluster.example.com

normally points to the ingress load balancer.

*.apps.cluster.example.com
             │
             ▼
      External Load Balancer
             │
             ▼
         Router Pods

The IngressController defines the domain that its routers serve.

4. Managing Route traffic

An OpenShift Route exposes a Service externally.

Example:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: payments
  namespace: banking
spec:
  host: payments.apps.cluster.example.com
  to:
    kind: Service
    name: payments-service
  port:
    targetPort: https

Traffic flow:

payments.apps.cluster.example.com
              │
              ▼
          Router Pod
              │
              ▼
        Route: payments
              │
              ▼
   Service: payments-service
              │
              ▼
        Ready payment Pods

The router watches Routes and EndpointSlices and updates its proxy configuration dynamically.

TLS Termination

The Ingress Operator manages router configuration for different TLS models.

Edge termination

Client ──HTTPS──> Router ──HTTP──> Application

The router terminates TLS.

Example:

tls:
  termination: edge

Use this when encryption between the router and application is not required or the internal network is trusted.

Re-encrypt termination

Client ──HTTPS──> Router ──HTTPS──> Application

The router terminates the client connection and creates a new TLS connection to the backend.

Example:

tls:
  termination: reencrypt
  destinationCACertificate: |
    ...

This is commonly used for sensitive enterprise applications.

Passthrough termination

Client ──HTTPS──> Router ──HTTPS──> Application

The router does not decrypt the request. TLS terminates inside the application.

Example:

tls:
  termination: passthrough

Because the router cannot inspect HTTP content, passthrough routing relies primarily on TLS SNI.

Default Ingress Certificate

The default router normally presents a wildcard certificate for:

*.apps.cluster.example.com

The Ingress Operator manages the router’s default certificate configuration.

You can configure a custom certificate through a Secret:

oc create secret tls custom-apps-certificate \
  --cert=apps.crt \
  --key=apps.key \
  -n openshift-ingress

Then reference it:

oc patch ingresscontroller default \
  -n openshift-ingress-operator \
  --type=merge \
  -p '{
    "spec": {
      "defaultCertificate": {
        "name": "custom-apps-certificate"
      }
    }
  }'

The Operator rolls out the router configuration with the new certificate.

Ingress Publishing Strategies

The Ingress Operator supports different ways to expose routers depending on the infrastructure.

LoadBalancerService

Common in public clouds:

Cloud Load Balancer
        │
        ▼
Router Service
        │
        ▼
Router Pods

The Operator creates a Service of type LoadBalancer.

HostNetwork

Router Pods bind directly to ports on the node.

Client
  │
  ▼
Node IP:80/443
  │
  ▼
Router Pod using host network

This is often used in bare-metal or controlled on-premises environments.

NodePortService

The router is exposed through NodePorts.

External Load Balancer
        │
        ▼
NodeIP:NodePort
        │
        ▼
Router Service

An external F5, HAProxy, or NetScaler can send traffic to the NodePorts.

Private or internal load balancer

An ingress controller can be exposed only internally.

Example architecture:

Internet users
      │
      ▼
Public LB
      │
      ▼
Public IngressController
Corporate users
      │
      ▼
Internal LB
      │
      ▼
Private IngressController

This is valuable in banking and enterprise environments.

Multiple Ingress Controllers

You are not limited to the default router.

You might create:

default
public
internal
partner
pci

Example separation:

Public applications
       │
       ▼
public.apps.cluster.example.com
       │
       ▼
Public routers
Internal applications
       │
       ▼
internal.apps.cluster.example.com
       │
       ▼
Internal routers

Each ingress controller can have its own:

• Domain
• Certificate
• Node placement
• Replica count
• Publishing strategy
• Route selector
• Namespace selector
• Load balancer