In OpenShift (OCP) and Kubernetes, Node Selectors and Node Affinity are mechanisms used to attract pods to specific nodes based on key-value labels assigned to those nodes.
While Taints are used to repel pods, Node Selectors and Node Affinity actively tell the OpenShift scheduler where your applications should be placed.
1. Node Selectors (Simple & Direct)
A Node Selector is the simplest way to constrain pods to nodes with specific labels. You label a node, and then add a matching nodeSelector key-value pair to your Pod or Deployment specification.
Best For: Simple, binary placement requirements (e.g., “put this pod on SSD storage”).
Limitation: It only supports hard AND logic and exact string matches (key=value). It cannot do “OR” conditions, regex, or soft preferences.
CLI Example – Labeling a node:
Bash
oc label node worker-1 storage=fast-ssd
YAML Example – Assigning a Pod via nodeSelector:
YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: database-app
spec:
template:
spec:
nodeSelector:
storage: fast-ssd # Pod will ONLY land on nodes with this exact label
containers:
- name: postgres
image: quay.io/postgresql/postgresql-13:latest
2. Node Affinity (Advanced & Flexible)
Node Affinity expands on Node Selectors by introducing expressive rules, logical operators (e.g., In, NotIn, Exists, DoesNotExist, Gt, Lt), and soft preferences.
Node Affinity offers two distinct rules:
requiredDuringSchedulingIgnoredDuringExecution (Hard Rule): The scheduler must find a node matching the rule to place the pod. If no matching node exists, the pod remains in a Pending state.
preferredDuringSchedulingIgnoredDuringExecution (Soft Rule): The scheduler tries to find a node matching the criteria. If no matching node is available, it places the pod on an alternative node anyway.
What does “IgnoredDuringExecution” mean?
If a node’s labels change after a pod is already running on it, OpenShift will not evict or move the pod. It only evaluates the rule during the initial scheduling phase.
YAML Example – Advanced Node Affinity Placement:
YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: analytics-processor
spec:
template:
spec:
affinity:
nodeAffinity:
# HARD RULE: Must be in us-east-1a OR us-east-1b
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-1a
- us-east-1b
# SOFT RULE: Prefers high-memory nodes, but falls back if unavailable
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80 # Priority score (1-100)
preference:
matchExpressions:
- key: node-role.kubernetes.io/high-mem
operator: Exists
containers:
- name: spark-worker
image: quay.io/analytics/spark:latest
3. Node Selectors vs. Node Affinity vs. Taints
Feature
Node Selector
Node Affinity
Taints & Tolerations
Direction
Attracts pods to nodes.
Attracts pods to nodes.
Repels pods from nodes.
Complexity
Simple exact match (key=value).
Complex expressions (In, Exists, weights).
Key-Value-Effect match (key=value:Effect).
Flexibility
Hard requirement only.
Supports both Hard and Soft (preferred) rules.
Supports Hard (NoSchedule, NoExecute) and Soft (PreferNoSchedule).
Isolating control plane nodes, reserving GPUs, maintaining nodes.
Pro Tip: For complex enterprise deployments, combine Node Affinity (to group your pods into specific infrastructure zones) with Taints & Tolerations (to prevent unapproved workloads from creeping into those zones).
In OpenShift (OCP) and Kubernetes, a taint is a core scheduling mechanism applied to a node that allows the node to repel pods.
Think of a taint as a “Keep Out” sign placed on a server.By default, the OpenShift scheduler will not place any application pod on a tainted node unless that pod explicitly carries a matching toleration (a “key pass”).
While Node Selectors and Node Affinity are used to attract pods to specific nodes, Taints and Tolerations are used to repel unwanted pods away from nodes.
1. Anatomy of a Taint
A taint consists of three components: a Key, an optional Value, and a Taint Effect:
Taint} = key=value:Effect
The Three Taint Effects:
NoSchedule (Hard Restriction):New pods without a matching toleration will never be scheduled onto this node.However, any pods already running on the node before the taint was applied are left alone.
PreferNoSchedule (Soft Restriction):The scheduler tries to avoid placing pods on this node, but if no other compute resources are available in the cluster, it will place the pod here anyway.
NoExecute (Eviction Restriction):The strongest effect.Any pod currently running on the node that does not tolerate the taint is immediately evicted (killed and rescheduled elsewhere).
2. Common Real-World Use Cases in OpenShift
Dedicated Infrastructure Nodes:Isolating OpenShift Infra components (Ingress Routers, Monitoring, Image Registry) onto dedicated worker nodes so user applications cannot drain their resources.
Specialized Hardware (GPUs / High-RAM):Tainting nodes that have expensive NVIDIA GPUs (gpu=true:NoSchedule) so standard web apps don’t accidentally get scheduled on them, reserving the hardware strictly for AI/ML workloads.
Master Node Isolation:OpenShift automatically places a taint on Control Plane (master) nodes (node-role.kubernetes.io/master=:NoSchedule) so user workloads only run on worker nodes.
Node Maintenance / Failure (Taint-based Eviction):When a node loses network connection or runs out of disk, OpenShift automatically applies temporary system taints like node.kubernetes.io/unreachable:NoExecute or node.kubernetes.io/disk-pressure:NoSchedule.
3. Declarative Setup Example
Step A: Applying a Taint to an OCP Node
Using the CLI, an administrator marks a node designated for GPU processing:
To allow a machine learning pod to run on that node, you add a tolerations block to its PodSpec:
YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: ml-model-trainer
spec:
template:
spec:
containers:
- name: cuda-runner
image: quay.io/ai-lab/cuda-runner:latest
# The Pod MUST have this matching toleration to land on worker-gpu-0
tolerations:
- key: "gpu"
operator: "Equal"
value: "nvidia"
effect: "NoSchedule"
Note: A toleration allows a pod to run on a tainted node, but it does not force it to go there.If you want to force the pod onto that specific GPU node, combine the Toleration with a nodeSelector or NodeAffinity.
OpenShift and AI is one of the fastest-growing areas in enterprise IT. Red Hat has built an entire AI platform around OpenShift, making it possible to run Generative AI, Machine Learning, MLOps, and AI inference on the same Kubernetes platform that hosts enterprise applications.
For an OpenShift Architect interview, you should understand not only AI concepts but also how OpenShift integrates with AI services.
OpenShift AI Ecosystem
Users
│
Web / Mobile Apps
│
OpenShift Routes
│
------------------------
│ │
Business Apps AI Applications
│ │
└──────────────┬───────┘
│
OpenShift Platform
│
------------------------------------------------------
│ │ │ │
RHOAI OpenShift GPU Nodes OpenShift GitOps
Pipelines
│
▼
Kubeflow / Jupyter / Model Serving
│
▼
LLM (Llama, Granite, Mistral, GPT, etc.)
│
▼
Vector Database / AI Search
│
▼
Enterprise Data Sources
What is Red Hat OpenShift AI (RHOAI)?
Red Hat OpenShift AI (formerly Red Hat OpenShift Data Science) is Red Hat’s enterprise AI/ML platform built on OpenShift.
It provides:
Jupyter Notebooks
Model training
Model serving
Pipelines
Experiment tracking
GPU scheduling
AI model deployment
MLOps
LLM serving
Vector database integration
Think of it as:
OpenShift
+
AI Platform
+
MLOps
AI Architecture on OpenShift
Developers
│
Git Repository
│
OpenShift GitOps
│
AI Pipeline (Tekton)
│
------------------------
│ │
Train Model Build Container
│ │
------------------------
│
Model Registry
│
Model Serving
│
REST API
│
Applications
Major AI Components
1. Jupyter Notebooks
Used for:
Data science
Python
TensorFlow
PyTorch
Hugging Face
Experiments
Example:
Data Scientist
↓
Notebook
↓
Python
↓
Train Model
2. Model Serving
Once trained:
Model
↓
Model Server
↓
REST API
↓
Application
Supported technologies include:
KServe
vLLM (commonly used for LLM inference)
NVIDIA Triton Inference Server
Caikit (used in some Red Hat AI scenarios)
3. Pipelines
Uses Kubeflow Pipelines (or Tekton depending on workflow).
Example:
Dataset
↓
Preprocessing
↓
Training
↓
Evaluation
↓
Deploy
↓
Production
Everything becomes repeatable.
GPU Support
OpenShift schedules GPUs like any other resource.
Architecture:
GPU Node
↓
NVIDIA Driver
↓
NVIDIA Device Plugin
↓
Pod Requests GPU
↓
GPU Allocated
Example Pod:
resources:
limits:
nvidia.com/gpu: 1
AI Model Lifecycle
Collect Data
↓
Train Model
↓
Validate
↓
Containerize
↓
Deploy
↓
Monitor
↓
Retrain
This is called MLOps.
OpenShift AI with LLMs
You can deploy models like:
IBM Granite
Llama 3.x
Mistral
DeepSeek (where licensing and hardware permit)
Gemma
Phi
Architecture:
User
↓
Chat Application
↓
OpenShift Route
↓
Model Server
↓
LLM
↓
Response
RAG (Retrieval-Augmented Generation)
This is one of the most common enterprise AI architectures.
User Question
↓
Embedding Model
↓
Vector Database
↓
Relevant Documents
↓
LLM
↓
Answer
Instead of relying only on the model’s knowledge, the LLM searches company documents.
Enterprise Banking AI Example
Customer
↓
Chatbot
↓
OpenShift Route
↓
Authentication
↓
API Gateway
↓
LLM Gateway
↓
RAG
↓
Vector Database
↓
Bank Policies
↓
Azure AI Search /
OpenSearch /
Milvus /
PgVector
↓
Response
This keeps answers grounded in enterprise data rather than only the model’s pretraining.
AI Security
Security is critical.
Authentication
OAuth
OpenID Connect
LDAP
SSO
Authorization
RBAC
Data Scientist
↓
Namespace
↓
Notebook
↓
GPU
Network
NetworkPolicies isolate:
Model servers
Databases
Pipelines
Notebooks
Secrets
Store:
API Keys
OpenAI keys
Hugging Face tokens
Database credentials
using Kubernetes Secrets or an external secret manager such as HashiCorp Vault or cloud-native secret services.
AI Observability
Monitor:
GPU utilization
CPU
Memory
Inference latency
Token generation rate
Request throughput
Error rates
Using:
Prometheus
↓
Grafana
↓
Alertmanager
Logging:
Vector
↓
Loki
or
Splunk
or
Elasticsearch
AI Storage
Training:
S3
Ceph
OpenShift Data Foundation
NFS
Model storage:
Model Registry
↓
Object Storage
AI Networking
Inference traffic:
Client
↓
Route
↓
Model Service
↓
LLM
Training traffic:
Notebook
↓
Object Storage
↓
GPU Worker
AI Scaling
Model serving uses Kubernetes autoscaling.
Traffic
↓
HPA
↓
2 Pods
↓
5 Pods
↓
20 Pods
For LLMs, scaling decisions often also consider GPU availability and model loading time.
GitOps for AI
Everything is stored in Git.
Git
↓
ArgoCD
↓
Notebook
↓
Pipeline
↓
Model
↓
Serving
AI CI/CD
Git Push
↓
Tekton
↓
Train
↓
Test
↓
Build
↓
Deploy
AI Monitoring
Monitor:
GPU
Inference
Latency
Memory
Model Accuracy
Token Usage
Failures
In production, you should also monitor:
Model drift
Data drift
Hallucination rates (where measurable)
Business KPIs
AI Governance
Enterprise AI requires:
Model versioning
Approval workflows
Audit logging
Data lineage
Dataset versioning
Explainability where required
Compliance controls
AI Operators
Common Operators include:
NVIDIA GPU Operator
Red Hat OpenShift AI Operator
OpenShift Pipelines Operator
OpenShift GitOps Operator
Service Mesh Operator (optional)
OpenTelemetry Operator
AI + OpenShift Architecture
Users
│
Web / Mobile
│
OpenShift Route
│
API Gateway
│
Authentication (OAuth)
│
-----------------------
│ │
▼ ▼
Business APIs AI Inference API
│
Model Serving
│
------------------------
│ │
LLM (Granite/Llama) Embedding Model
│ │
└──────────┬───────────┘
▼
Vector Database
│
Enterprise Documents
│
S3 / ODF / Database
OpenShift AI vs Azure OpenAI
OpenShift AI
Azure OpenAI
Run models on your infrastructure
Managed AI service
Full Kubernetes control
Microsoft-managed
GPU management required
No GPU management
Supports multiple open models
Microsoft-hosted models
Air-gapped deployments possible
Cloud service
Better for hybrid/on-prem
Better for Azure-native workloads
Many enterprises use both:
Azure OpenAI for managed GPT models.
OpenShift AI for on-premises inference, data sovereignty, or running open models.
Interview Questions
What is OpenShift AI?
Enterprise AI platform built on OpenShift for model development, training, deployment, and lifecycle management.
Why OpenShift for AI?
Kubernetes-native
GPU orchestration
MLOps
Security
Multi-cloud
GitOps
Scalability
Enterprise support
How do you deploy an LLM?
Deploy GPU Operator.
Configure GPU nodes.
Deploy the model server (e.g., vLLM).
Download or mount the model.
Expose via a Service and Route.
Monitor latency and resource usage.
How would you secure AI?
RBAC
NetworkPolicies
OAuth/OIDC
Secrets management
Image signing
Signed models where supported
Audit logging
TLS everywhere
Enterprise Banking AI Architecture
Customer
│
Mobile App
│
OpenShift Route
│
API Gateway
│
Authentication (OAuth/OIDC)
│
AI Gateway Service
│
------------------------
│ │
▼ ▼
LLM Inference Embedding Model
│ │
└──────────┬───────────┘
▼
Vector Database
│
----------------------------
│ │ │
▼ ▼ ▼
Banking Docs Policies Knowledge Base
Interview Answer (2 Minutes)
“OpenShift provides an enterprise platform for building, deploying, and operating AI applications through Red Hat OpenShift AI. It supports the full MLOps lifecycle, including Jupyter notebooks for development, pipelines for model training, model registries, and scalable model serving. For generative AI, I typically design a Retrieval-Augmented Generation architecture where applications authenticate through OpenShift, call an API or model-serving layer, retrieve relevant enterprise documents from a vector database, and then pass that context to an LLM such as IBM Granite or Llama. OpenShift handles GPU scheduling through the NVIDIA GPU Operator, while GitOps, RBAC, NetworkPolicies, Secrets, and the monitoring stack provide secure and repeatable operations. For regulated industries like banking, I recommend combining RAG, centralized audit logging, strong identity controls, and model governance so that AI responses are based on approved enterprise knowledge and meet compliance requirements.”
The DNS Operator deploys, configures, and continuously manages CoreDNS, which provides DNS resolution and Kubernetes Service discovery for Pods inside an OpenShift cluster.
A simple way to remember it:
The DNS Operator manages the DNS platform; CoreDNS answers the DNS queries.
DNS Operator
│
▼
CoreDNS DaemonSet
│
▼
DNS Service IP
│
▼
Pods resolve Services and external names
The DNS Operator is installed automatically during OpenShift installation. It runs as a Deployment, while CoreDNS runs as a DaemonSet managed by that Operator. (Red Hat Documentation)
DNS Operator vs CoreDNS
These are separate components:
Component
Responsibility
DNS Operator
Manages DNS configuration and CoreDNS lifecycle
CoreDNS
Processes DNS queries
DNS Service
Provides a stable ClusterIP for DNS queries
kubelet
Configures each Pod to use the cluster DNS Service
Node resolver
Maintains node hostname entries where required
DNS Operator
│
▼
Creates and manages
│
├── CoreDNS DaemonSet
├── DNS Service
├── CoreDNS ConfigMap
└── Node resolver DaemonSet
The Operator implements the cluster-scoped DNS API in the operator.openshift.io API group. (Red Hat Documentation)
Where the Components Run
The DNS Operator normally runs in:
openshift-dns-operator
CoreDNS and node-resolver components normally run in:
openshift-dns
Check the Operator:
oc get deployment -n openshift-dns-operator
Check DNS Pods:
oc get pods -n openshift-dns -o wide
Check DaemonSets:
oc get daemonset -n openshift-dns
Typical output includes:
dns-default
node-resolver
Main DNS Architecture
Application Pod
│
│ DNS query
▼
Cluster DNS Service IP
│
▼
CoreDNS Pod
│
├── Internal cluster name?
│ └── Answer from Kubernetes API data
│
└── External name?
└── Forward to upstream DNS resolver
The default internal cluster domain is:
cluster.local
CoreDNS provides resolution for internal names such as:
The exact DNS Service IP depends on the cluster Service network. OpenShift commonly assigns the DNS Service a stable IP from the Service CIDR. (Red Hat Documentation)
You can check it with:
oc get dns.operator/default -o yaml
Look under:
status:
clusterDomain: cluster.local
clusterIP: 172.30.0.10
Why CoreDNS Runs as a DaemonSet
CoreDNS normally runs through the dns-default DaemonSet.
Node 1 Node 2 Node 3
│ │ │
CoreDNS CoreDNS CoreDNS
Benefits:
DNS capacity scales as nodes are added.
DNS is distributed across the cluster.
A single DNS Pod failure does not stop name resolution.
Requests can be handled close to workloads.
DNS remains available if individual nodes fail.
The DNS Operator creates the CoreDNS DaemonSet and exposes it through a Service with a stable IP. (Red Hat Documentation)
Internal Name Resolution
CoreDNS watches the Kubernetes API for Services, EndpointSlices, namespaces, and other relevant resources.
For a Service query:
Pod asks for:
payments-api.banking.svc.cluster.local
│
▼
CoreDNS queries its Kubernetes data
│
▼
Returns the Service ClusterIP
For a headless Service:
spec:
clusterIP: None
CoreDNS can return Pod or endpoint IPs instead of one Service ClusterIP.
This is useful for:
StatefulSets
Databases
Cluster members
Direct endpoint discovery
External DNS Resolution
If a Pod asks for an external name:
api.example.com
CoreDNS normally forwards the query to an upstream resolver.
Application Pod
│
▼
CoreDNS
│
▼
Corporate or cloud DNS
│
▼
External DNS result
Upstream resolvers normally come from node resolver configuration or from explicit DNS Operator settings.
DNS Forwarding
The DNS Operator supports forwarding particular DNS zones to specific resolvers.
Example requirement:
*.corp.bank.local
│
▼
Corporate DNS servers
A simplified configuration is:
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
name: default
spec:
servers:
-name: corporate-dns
zones:
- corp.bank.local
forwardPlugin:
upstreams:
- 10.20.30.10
- 10.20.30.11
Flow:
Query: database.corp.bank.local
│
▼
CoreDNS matches corp.bank.local
│
▼
Forwards to 10.20.30.10 / 10.20.30.11
Queries that do not match a configured zone fall back to the configured upstream resolvers. (Red Hat Documentation)
Edit DNS configuration with:
oc edit dns.operator/default
Do not manually edit the Operator-generated CoreDNS ConfigMap as the primary configuration method.
DNS Caching
CoreDNS caches successful and unsuccessful DNS responses.
This reduces:
Query latency
Load on upstream DNS servers
Repeated external lookups
Network traffic
OpenShift supports configuring positive and negative cache TTLs through the DNS Operator resource. (Red Hat Documentation)
Example:
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
name: default
spec:
cache:
positiveTTL: 1h
negativeTTL: 30s
Be careful with TTL tuning:
Very low TTLs increase DNS load.
Very high TTLs can retain stale results longer.
Negative caching can make a recently created record appear unavailable until the negative TTL expires.
Node Resolver
The DNS Operator also manages a node-resolver DaemonSet.
Check it with:
oc get daemonset node-resolver -n openshift-dns
The node-resolver component helps maintain node-level name resolution information, including managed entries in the node’s /etc/hosts where required.
DNS Operator
│
├── dns-default DaemonSet
│ └── Cluster DNS
│
└── node-resolver DaemonSet
└── Node hostname resolution support
Reconciliation Loop
The DNS Operator continuously compares desired DNS configuration with the actual resources.
DNS resource: default
│
▼
DNS Operator reads desired state
│
▼
Checks DaemonSets, Service and ConfigMap
│
▼
Difference found?
┌──┴──┐
│ │
No Yes
│ │
▼ ▼
Wait Recreate or update resources
│
▼
Validate DNS availability
│
▼
Update Operator status
Examples that trigger reconciliation:
CoreDNS Pod fails.
DNS forwarding configuration changes.
Cache settings change.
OpenShift is upgraded.
A managed ConfigMap changes.
Node placement is modified.
A DaemonSet does not match the desired configuration.
DNS Operator Health
Check the ClusterOperator:
oc get clusteroperator dns
Healthy output:
NAME AVAILABLE PROGRESSING DEGRADED
dns True False False
Interpretation:
Condition
Meaning
Available=True
DNS service is operational
Progressing=True
DNS resources are being changed
Degraded=True
A DNS component or configuration is failing
Detailed information:
oc describe clusteroperator dns
The Operator considers DNS available when the DNS Service has a ClusterIP and at least one CoreDNS Pod is available. (Red Hat Documentation)
DNS can resolve a Service even if its backend Pods are unhealthy.
DNS resolution does not prove the application itself is reachable.
EndpointSlices affect application traffic, not necessarily the existence of the Service DNS record.
Scenario 2: External Names Fail but Internal Names Work
For example:
payments-api.banking.svc.cluster.local → works
example.com → fails
This usually indicates an upstream-forwarding issue.
Check:
Upstream DNS server availability
DNS Operator forwarding configuration
Firewall access to UDP/TCP port 53
Node /etc/resolv.conf
Corporate DNS reachability
Egress restrictions
Forwarding loops
Review CoreDNS logs:
oc logs -n openshift-dns <dns-default-pod> \
-c dns
Scenario 3: One Node Has DNS Problems
If Pods on one node fail DNS while other nodes work:
oc get pods -n openshift-dns -o wide
Check whether the affected node has a healthy CoreDNS Pod.
Then examine:
Node networking
OVN connectivity
DNS DaemonSet Pod
kubelet configuration
Service routing
Firewall rules
MTU problems
Node resource pressure
Test from Pods scheduled on both a healthy and affected node.
Scenario 4: DNS Query Is Slow
Possible causes:
Slow upstream resolver
DNS forwarding loop
Packet loss
Excessive query volume
Too-low cache TTL
CoreDNS CPU throttling
Node network problems
Search-domain expansion caused by ndots
External queries being tried as multiple internal names first
Measure lookup time:
time getent hosts external.example.com
Compare internal and external queries separately.
Scenario 5: CoreDNS Pods Are Pending
Check:
oc describe pod <dns-pod> -n openshift-dns
Possible causes:
Node selector mismatch
Missing toleration
Insufficient CPU or memory
Node taints
Scheduling restrictions
Image pull problem
CoreDNS and node-resolver placement can be controlled using node selectors and tolerations in the DNS Operator configuration. (Red Hat Documentation)
Common DNS Errors
Could not resolve host
Possible causes:
CoreDNS unavailable
Wrong Pod resolver configuration
Upstream DNS failure
NetworkPolicy or firewall blocking DNS
Typographical error
SERVFAIL
Possible causes:
Upstream resolver failure
DNS forwarding loop
Invalid zone configuration
DNSSEC or upstream issue
NXDOMAIN
Means the requested name does not exist according to the resolver.
Check:
Service name
Namespace
DNS zone
Negative cache
External record creation
DNS works but connection fails
DNS only returned an IP address. Check:
Service port
EndpointSlices
Pod readiness
NetworkPolicy
Application process
TLS configuration
DNS Operator vs DNS Operator Configuration
Avoid directly editing:
CoreDNS DaemonSet
dns-default Service
generated dns-default ConfigMap
These are Operator-managed and changes may be reverted.
Configure DNS using:
oc edit dns.operator/default
The Operator then generates the appropriate CoreDNS configuration and performs reconciliation.
Relationship with Other Operators
Cluster Version Operator
│
▼
DNS Operator
│
├── CoreDNS DaemonSet
├── Node Resolver DaemonSet
├── DNS Service
└── DNS ConfigMap
Dependencies include:
Component
Relationship
Network Operator
Provides connectivity to DNS Pods and Service IP
kubelet
Places cluster DNS information into Pod resolver configuration
API Server
Provides Service and Endpoint data
Ingress Operator
Depends on external wildcard DNS for application routes
Monitoring
Collects DNS component metrics and alerts
CVO
Installs and upgrades the DNS Operator
Interview Answer
The OpenShift DNS Operator deploys and manages CoreDNS to provide internal name resolution and Kubernetes Service discovery. The Operator runs as a Deployment in openshift-dns-operator, while it manages the dns-default CoreDNS DaemonSet, the node-resolver DaemonSet, a DNS Service with a stable ClusterIP, and the generated CoreDNS configuration in openshift-dns.
Pods send DNS queries to the DNS Service IP configured in their /etc/resolv.conf. CoreDNS resolves internal names such as service.namespace.svc.cluster.local using Kubernetes API information and forwards external or configured private-zone queries to upstream resolvers. The Operator continuously reconciles these resources and supports configuration through the cluster-scoped dns.operator/default object, including forwarding, caching, and node placement.
For troubleshooting, I start with oc get co dns, inspect the DNS Operator and CoreDNS Pods, check the dns-default Service and ConfigMap, test DNS from a Pod, and determine whether the failure affects internal names, external names, or only Pods on a particular node. I then check upstream resolvers, OVN connectivity, NetworkPolicies, firewall rules and CoreDNS logs.
In OCP, the control plane components are not system services (systemd units). They run as static pods — and kubelet is the only thing that can run static pods.
Without kubelet on masters:
❌ etcd cannot start
❌ kube-apiserver cannot start
❌ kube-controller-manager cannot start
❌ kube-scheduler cannot start
kubelet IS the bootstrap mechanism for the entire control plane.
Static Pods — The Core Concept
kubelet watches one directory constantly:
/etc/kubernetes/manifests/
├── etcd-pod.yaml
├── kube-apiserver-pod.yaml
├── kube-controller-manager-pod.yaml
└── kube-scheduler-pod.yaml
Any .yaml dropped here → kubelet starts it as a pod. Any .yaml removed → kubelet stops and removes it.
No API server needed. kubelet reads these files directly from disk. This is exactly why cluster-restore.sh works by moving manifests in and out of this directory — it’s controlling the control plane via kubelet’s static pod mechanism.
So while kubelet on masters CAN run any pod, the scheduler won’t place regular workloads there unless you explicitly tolerate the taint. The taint does NOT affect static pods — kubelet runs those directly, bypassing the scheduler entirely.
kubelet Startup Sequence on a Master
This is what happens when a master node boots:
1. RHCOS boots
│
▼
2. systemd starts kubelet.service
│
▼
3. kubelet reads /etc/kubernetes/manifests/
│
├── etcd-pod.yaml → starts etcd containers
├── kube-apiserver-pod.yaml → starts API server containers
├── kube-controller-manager-pod.yaml
└── kube-scheduler-pod.yaml
│
▼
4. etcd becomes healthy (has quorum)
│
▼
5. kube-apiserver connects to etcd, starts serving
│
▼
6. kubelet registers THIS master node with the API server
│
▼
7. Cluster operators come up (CVO, etcd-operator, etc.)
Notice: kubelet starts etcd, and etcd enables the API, and the API is what kubelet later registers with. kubelet bootstraps its own control plane and then registers with it.
├── etcd container keeps running (CRI-O manages it independently)
├── But if etcd crashes → kubelet not there to restart it
├── kube-apiserver keeps running (same)
│
└── After 5 min: node shows NotReady (no heartbeat to API)
│
└── etcd operator detects degraded member
└── alerts fire, but cluster may still function
if other 2 masters are healthy
Key insight: CRI-O keeps containers alive after kubelet dies — but kubelet is the only thing that will restart a crashed container. Without kubelet, a crashed etcd stays dead.
The Dependency Chain in One View
RHCOS
└── systemd
└── kubelet.service ← runs on master as a systemd unit
└── reads /etc/kubernetes/manifests/
├── etcd-pod.yaml
│ └── CRI-O runs etcd containers
│ └── etcd cluster (the database)
│
├── kube-apiserver-pod.yaml
│ └── CRI-O runs API server containers
│ └── Kubernetes/OCP API
│
├── kube-controller-manager-pod.yaml
│ └── CRI-O runs controller-manager
│
└── kube-scheduler-pod.yaml
└── CRI-O runs scheduler
The entire OCP control plane is just kubelet reading files from a directory and telling CRI-O what to run. This simplicity is what makes the restore procedure work — you control the control plane by controlling what kubelet sees in /etc/kubernetes/manifests/.
Taints prevent general workloads from being scheduled.
Example:
oc adm taint node gpu-1 dedicated=gpu:NoSchedule
A Pod must have a matching toleration.
tolerations:
-key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
This is used for:
GPU nodes
Infrastructure nodes
Storage nodes
Special security zones
High-performance workloads
Worker Pools
MachineConfigPools group nodes with the same operating-system configuration.
Typical pools:
master
worker
infra
gpu
storage
Check:
oc get mcp
Example:
NAME UPDATED UPDATING DEGRADED
worker True False False
infra True False False
Custom worker pools can receive separate:
Kubelet settings
Kernel arguments
CRI-O settings
Systemd units
OS files
Worker Node Failure
If a worker fails:
Worker node unavailable
│
▼
Node becomes NotReady
│
▼
Controller detects failure
│
▼
Pods are recreated elsewhere
│
▼
Scheduler selects healthy workers
This depends on:
Multiple replicas
Storage accessibility
Pod disruption controls
Node failure detection
Sufficient spare capacity
A single-replica application can experience downtime.
Common Worker Problems
Node NotReady
Check:
oc get nodes
oc describe node <worker>
Then:
oc debug node/<worker>
chroot /host
systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio
Possible causes:
kubelet failure
CRI-O failure
Network failure
Certificate problem
Disk pressure
Memory pressure
Node OS issue
API server connectivity issue
DiskPressure
Check:
df -h
df -i
du -sh /var/lib/containers/*
du -sh /var/log/*
Possible causes:
Large container logs
Image accumulation
Container storage exhaustion
Inode exhaustion
Failed garbage collection
MemoryPressure
Check:
free -h
top
oc adm topnode
Possible causes:
Oversized workloads
Incorrect requests and limits
Memory leak
Too many Pods
Platform agents consuming resources
Pods stuck in ContainerCreating
Investigate:
oc describe pod <pod> -n <namespace>
Possible causes:
CRI-O failure
Image pull problem
CNI/OVN failure
CSI mount failure
SCC or permission issue
Node disk pressure
ImagePullBackOff
Check:
Image name
Registry access
ImagePullSecret
DNS
Proxy configuration
Registry certificate trust
CRI-O logs
journalctl -u crio
crictl images
Useful Worker Troubleshooting Commands
oc get nodes
oc describe node <worker>
oc adm top nodes
oc get pods -A-o wide --field-selector spec.nodeName=<worker>
Node-level access:
oc debug node/<worker>
chroot /host
Inside the host:
systemctl status kubelet
systemctl status crio
journalctl -u kubelet
journalctl -u crio
df -h
df -i
free -h
iostat -x110
sar -u110
pidstat -d110
Runtime inspection:
crictl ps
crictl pods
crictl images
crictl info
Control Plane vs Worker Components
Control plane
Worker node
kube-apiserver
kubelet
etcd
CRI-O
scheduler
OCI runtime
controller manager
OVN node components
Cluster Operators
Machine Config Daemon
Stores desired state
Runs workloads
Makes placement decisions
Executes assigned Pods
Control plane says:
"Run this Pod on worker-2"
Worker-2 says:
"I will pull the image, configure networking, mount storage, and start it"
Interview Answer
An OpenShift worker node is responsible for executing application workloads. It normally runs RHCOS, the kubelet, CRI-O, an OCI runtime such as crun, OVN-Kubernetes networking components, the Machine Config Daemon, and node-level DaemonSets such as Node Exporter, CoreDNS, logging collectors, and CSI plugins.
The kubelet registers the worker with the API server and watches for Pods assigned by the scheduler. It asks CRI-O to create the Pod sandbox and start containers, OVN configures networking, and CSI drivers mount persistent storage. The Machine Config Daemon keeps the RHCOS configuration synchronized with the desired MachineConfigPool state.
For troubleshooting, I start with oc get nodes and oc describe node, inspect node conditions such as Ready, MemoryPressure, and DiskPressure, then use oc debug node and chroot /host to check kubelet, CRI-O, storage, memory, network, and system logs.
The kube-apiserver is the main entry point into the cluster.
All administrative and platform operations go through it.
Examples:
oc get pods
oc apply -f deployment.yaml
oc delete pod mypod
Request flow:
oc client
│
▼
API load balancer
│
▼
kube-apiserver
│
├── Authentication
├── Authorization
├── Admission controls
├── Resource validation
└── etcd read/write
The API server handles:
Kubernetes API requests
Authentication
RBAC authorization
Admission webhooks
Object validation
Communication with etcd
It listens on:
TCP 6443
Check it with:
oc get pods -n openshift-kube-apiserver -o wide
oc get co kube-apiserver
2. etcd
etcd is the distributed key-value database that stores the authoritative cluster state.
It stores:
Deployments
Pods and desired state
Services
Secrets
ConfigMaps
Routes
RBAC
Nodes
Operators
CRDs
MachineConfig objects
API Server
│
▼
etcd
│
▼
Cluster state
A standard OpenShift cluster normally has three etcd members:
master-0
master-1
master-2
Quorum requirement:
3 members → 2 required for quorum
If one member fails, the cluster can usually continue.
If two members fail, etcd loses quorum and control-plane operations stop.
Check etcd:
oc get co etcd
oc get pods -n openshift-etcd -o wide
3. kube-scheduler
The scheduler decides which node should run a new Pod.
It evaluates:
CPU requests
Memory requests
Node selectors
Taints and tolerations
Affinity and anti-affinity
Topology spread constraints
Persistent-volume topology
Host ports
Node readiness
Pending Pod
│
▼
Scheduler filters nodes
│
▼
Scheduler scores eligible nodes
│
▼
Pod assigned to worker-2
│
▼
kubelet starts Pod
The scheduler does not start the container. It only assigns the Pod to a node.
Check it with:
oc get co kube-scheduler
oc get pods -n openshift-kube-scheduler -o wide
4. kube-controller-manager
The kube-controller-manager runs multiple Kubernetes controllers.
Controllers continuously compare:
Desired state
vs
Actual state
and take action to correct differences.
Important controllers include:
Deployment controller
ReplicaSet controller
Node controller
Job controller
Service account controller
EndpointSlice controller
Namespace controller
Persistent-volume controller
Example:
Deployment requests 3 replicas
│
▼
Controller sees only 2 Pods
│
▼
Creates another Pod
Check it with:
oc get co kube-controller-manager
oc get pods -n openshift-kube-controller-manager -o wide
5. OpenShift Controller Manager
OpenShift also includes OpenShift-specific controllers.
These manage platform-specific resources and behavior beyond standard Kubernetes.
Examples include:
OpenShift project behavior
Build-related resources
Image resources
OpenShift authorization functions
Platform-specific reconciliation
It runs separately from the Kubernetes controller manager.
Check:
oc get co openshift-controller-manager
oc get pods -n openshift-controller-manager
6. OpenShift API Server
The OpenShift API Server provides OpenShift-specific APIs that extend Kubernetes.
Examples include APIs related to:
Projects
Routes
Builds
Images
OpenShift-specific authorization
Security extensions
Architecture:
Client
│
▼
Kubernetes API aggregation layer
│
├── Kubernetes APIs
└── OpenShift APIs
Check:
oc get co openshift-apiserver
oc get pods -n openshift-apiserver
7. Cluster Version Operator
The Cluster Version Operator, or CVO, manages the overall OpenShift release version.
It is responsible for:
Installing platform components
Coordinating upgrades
Applying release manifests
Monitoring ClusterOperators
Ensuring components match the desired release
New OpenShift release
│
▼
Cluster Version Operator
│
▼
Platform Operators upgraded
│
▼
Control-plane and worker updates
Check:
oc get clusterversion
oc get co
8. Machine Config Operator
The Machine Config Operator, or MCO, manages the operating-system configuration of RHCOS nodes.
It controls:
RHCOS updates
CRI-O configuration
kubelet configuration
Kernel arguments
Systemd units
CA certificates
Registry configuration
Node files
MachineConfig
│
▼
Machine Config Operator
│
▼
Machine Config Daemon
│
▼
Drain → Apply → Reboot → Ready
Check:
oc get mcp
oc get machineconfig
oc get co machine-config
9. Authentication Operator
The Authentication Operator manages OpenShift OAuth and login services.
It handles:
Identity providers
OAuth server
Login flow
Authentication certificates
Token configuration
Example:
User
│
▼
Corporate identity provider
│
▼
OpenShift OAuth
│
▼
OpenShift access token
Check:
oc get co authentication
oc get oauth cluster -o yaml
10. Ingress Operator
The Ingress Operator manages the OpenShift router.
It controls:
Router Pods
IngressControllers
Wildcard certificates
Router replicas
Publishing strategy
Public and private ingress
External client
│
▼
Load balancer
│
▼
Router Pods
│
▼
Route
│
▼
Service
│
▼
Application Pods
Check:
oc get co ingress
oc get ingresscontroller -n openshift-ingress-operator
oc get pods -n openshift-ingress
11. DNS Operator
The DNS Operator manages CoreDNS for cluster Service discovery.
It allows Pods to resolve names such as:
payments-api.banking.svc.cluster.local
Architecture:
Application Pod
│
▼
DNS Service IP
│
▼
CoreDNS
│
├── Internal Service names
└── External upstream DNS
Check:
oc get co dns
oc get pods -n openshift-dns
oc get dns.operator/default -o yaml
12. Network Operator
The Cluster Network Operator manages the OpenShift network plugin, usually OVN-Kubernetes.
It manages:
Pod networks
Service networks
OVN components
GENEVE tunnels
MTU
Egress features
NetworkPolicies
Node networking components
Pod A
│
▼
OVN virtual network
│
▼
Pod B
Check:
oc get co network
oc get network.operator cluster -o yaml
oc get pods -n openshift-ovn-kubernetes
13. Cloud Controller Manager
On cloud platforms, the cloud controller integrates OpenShift with AWS, Azure, or GCP.
It manages functions such as:
Cloud node information
Load balancers
Routes
Instance metadata
Cloud volumes, depending on the driver architecture
Example:
Service type LoadBalancer
│
▼
Cloud Controller
│
▼
AWS / Azure / GCP load balancer
14. Machine API Operator
The Machine API Operator manages infrastructure machines on supported platforms.
It handles:
Machine objects
MachineSets
MachineHealthChecks
Worker creation
Worker replacement
Autoscaling integration
MachineSet replicas: 5
│
▼
Machine API Operator
│
▼
Create cloud VMs
│
▼
New OpenShift workers join
Check:
oc get machines -A
oc get machinesets -A
oc get machinehealthchecks -A
15. Monitoring Components
The control plane is monitored by the OpenShift monitoring stack.
Main components include:
Prometheus
Alertmanager
kube-state-metrics
Node Exporter
Prometheus Operator
Thanos components
They monitor:
API latency
etcd latency
Scheduler health
Operator status
Node health
Resource utilization
Check:
oc get pods -n openshift-monitoring
Static Pods on Control-Plane Nodes
Several critical control-plane components run as static Pods:
etcd
kube-apiserver
kube-controller-manager
kube-scheduler
A static Pod is managed directly by the kubelet on the node.
Static Pod manifest
│
▼
kubelet reads manifest
│
▼
Control-plane Pod starts
This allows core components to start even when the scheduler is unavailable.
Control Plane Request Flow
When you run:
oc create deployment nginx --image=nginx
the full sequence is:
1. oc sends request to API load balancer
2. Load balancer selects a kube-apiserver
3. API server authenticates the user
4. RBAC authorizes the request
5. Admission controls validate the object
6. Deployment is stored in etcd
7. Controller Manager creates a ReplicaSet
8. ReplicaSet controller creates a Pod
9. Scheduler selects a worker node
10. kubelet asks CRI-O to start the container
11. OVN configures Pod networking
12. Pod becomes Running
High Availability
A production OpenShift control plane normally uses three control-plane nodes.
master-0
master-1
master-2
Availability is maintained through:
Three API server instances
Three etcd members
Scheduler leader election
Controller-manager leader election
API load balancing
Rolling upgrades
Static Pods
Operator reconciliation
Only one scheduler and controller-manager instance is active as leader at a time, while others remain ready to take over.
Control Plane vs Worker Nodes
Control plane
Worker nodes
Runs API servers
Runs application Pods
Runs etcd
Runs kubelet
Runs scheduler
Runs CRI-O
Runs controllers
Runs OVN node components
Manages desired state
Executes workloads
Stores cluster state
Hosts applications
Control Plane
│
▼
Decides what should run
Worker Node
│
▼
Runs the actual workload
Important Commands
oc get nodes
oc get clusteroperators
oc get clusterversion
oc get pods -A
Control-plane Pods:
oc get pods -n openshift-etcd
oc get pods -n openshift-kube-apiserver
oc get pods -n openshift-kube-controller-manager
oc get pods -n openshift-kube-scheduler
Operator health:
oc get co
Healthy status:
AVAILABLE=True
PROGRESSING=False
DEGRADED=False
Troubleshooting Control Plane
Use this sequence:
API unavailable or slow
│
▼
Check API load balancer and DNS
│
▼
Check kube-apiserver
│
▼
Check etcd health and latency
│
▼
Check control-plane nodes
│
▼
Check scheduler and controllers
│
▼
Check ClusterOperators
Useful commands:
oc get co
oc get nodes
oc get--raw='/readyz?verbose'
oc get pods -n openshift-etcd -o wide
oc get pods -n openshift-kube-apiserver -o wide
For node-level investigation:
oc debug node/<control-plane-node>
chroot /host
systemctl status kubelet
journalctl -u kubelet
iostat -x110
Interview Answer
The OpenShift control plane is responsible for managing the cluster’s desired state and making all scheduling, API, and lifecycle decisions. Its key components are the kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and OpenShift-specific API and controller services.
The API server receives all requests, authenticates and authorizes them, and stores the resulting state in etcd. The controller managers continuously reconcile resources, while the scheduler selects suitable worker nodes for new Pods. OpenShift Operators such as the Cluster Version Operator, Machine Config Operator, Ingress Operator, DNS Operator, Network Operator, and Authentication Operator manage the platform components around the core Kubernetes control plane.
In a highly available cluster, these components run across three control-plane nodes. The API servers are load balanced, etcd maintains quorum, and the scheduler and controller managers use leader election. For troubleshooting, I start with oc get co, check API readiness, etcd health, control-plane Pods and nodes, and then review Operator conditions and logs.
In OpenShift, WAL fsync normally refers to how quickly etcd can safely write changes to its Write-Ahead Log on disk.
A simple definition:
Before etcd confirms an important cluster-state change, it writes the change to its WAL and requests that the operating system physically persist it to storage using fsync or fdatasync.
This protects the cluster state if an etcd process, control-plane node, or operating system suddenly fails.
What is a WAL?
WAL means Write-Ahead Log.
etcd records a change in the log before applying it to its main backend database.
API change
│
▼
etcd receives proposal
│
▼
Write proposal to WAL
│
▼
fsync to persistent storage
│
▼
Replicate through Raft
│
▼
Commit transaction
│
▼
API request succeeds
Examples of changes written through etcd include:
Creating a Pod
Updating a Deployment
Creating or changing a Secret
Updating a ConfigMap
Changing node status
Updating EndpointSlices
Operator status updates
Creating or deleting Kubernetes resources
What does fsync do?
When an application writes data, the operating system may initially place it in memory cache.
Application write
│
▼
Operating-system page cache
│
▼
Storage device later
That is fast, but data still in memory can be lost during a sudden power or operating-system failure.
fsync tells the operating system:
Do not acknowledge this operation until the data
has been flushed to persistent storage.
For etcd:
WAL entry
│
▼
fsync
│
▼
Disk confirms persistence
│
▼
etcd continues the commit
Therefore, WAL fsync latency is directly influenced by the storage system.
Why OpenShift depends on it
The Kubernetes API server stores cluster state in etcd.
oc apply
│
▼
kube-apiserver
│
▼
etcd leader
│
▼
WAL fsync and Raft replication
│
▼
Commit
│
▼
API response
If WAL fsync becomes slow, etcd writes become slow. That can make the OpenShift API slow because API write operations depend on etcd committing changes.
Red Hat notes that slow storage or competing disk activity can cause high fsync latency, API slowness, request timeouts, missed heartbeats, and temporary etcd leader loss. (Red Hat Documentation)
Example of a normal write
Suppose you run:
oc scale deployment payments-api --replicas=5
The flow is:
1. oc sends PATCH request
2. API server validates and authorizes it
3. API server sends the change to etcd
4. etcd leader writes the proposal to WAL
5. WAL is synchronized to storage
6. Proposal is replicated to other etcd members
7. A majority acknowledges it
8. The change is committed
9. API server returns success
If step 5 takes 2 ms, the transaction can proceed quickly.
If step 5 takes 200 ms or more, API requests accumulate and controllers begin reconciling more slowly.
WAL vs etcd backend database
etcd uses both a WAL and a backend database.
Incoming change
│
▼
WAL
Durable sequential record
│
▼
Raft commit
│
▼
Backend database
Current key-value state
They have separate performance metrics:
Metric
Meaning
etcd_disk_wal_fsync_duration_seconds
Time required to persist the WAL
etcd_disk_backend_commit_duration_seconds
Time required to commit the backend database transaction
Interpretation:
High WAL fsync latency
→ synchronous WAL storage problem
High backend commit latency
→ backend database storage or database-pressure issue
Both high
→ general disk contention, throttling, or storage degradation
This shows the WAL fsync duration below which approximately 99% of observations occurred during the selected period.
Breaking it down by instance helps identify one slow control-plane node:
master-0 → 0.004 seconds
master-1 → 0.120 seconds
master-2 → 0.005 seconds
Here, master-1 is the likely problem.
Red Hat identifies WAL fsync duration, backend commit duration, and leader changes as important metrics for evaluating etcd storage performance. (Red Hat Documentation)
What is an acceptable value?
The exact alert threshold can vary by OpenShift release, workload, and test method.
Current Red Hat guidance for validating etcd storage includes checking the p99 fsync result produced by its supported fio-based performance test. Some current documentation uses a threshold below 10 ms, while other recent release documentation and test output refer to 20 ms. Use the guidance and alerts supplied for your exact OpenShift version rather than applying one universal value. (Red Hat Documentation)
As a practical operational interpretation:
A few milliseconds
→ healthy low-latency storage
Consistent tens of milliseconds
→ investigate
Large or recurring spikes
→ likely to affect etcd and API performance
Focus on:
p99 behavior
Sustained duration
Differences between members
Correlation with API latency
Leader changes and timeouts
A single isolated spike is less concerning than repeated or sustained high latency.
Causes of high WAL fsync latency
1. Slow storage
Examples:
Mechanical disks
Slow SAN
High-latency network-backed block storage
Poorly configured virtual disks
Underperforming SSDs
2. IOPS or throughput throttling
Cloud disks and virtual machines can have:
Disk IOPS limits
Throughput limits
Burst-credit exhaustion
Instance-wide storage limits
3. Noisy neighbours
The etcd virtual disk might share physical infrastructure with:
Other virtual machines
Databases
Backup jobs
Storage replication
Large image operations
4. Snapshots and backups
Hypervisor snapshots or storage backups can temporarily increase latency.
5. Control-plane processes producing I/O
Examples:
Logging agents
Security scanners
Backup agents
Excessive journal writes
Container image operations
6. Device or filesystem errors
Examples:
Storage path failures
NVMe timeouts
SAN multipath issues
Filesystem problems
Disk nearly full
Red Hat recommends low-latency block storage for etcd and advises against sharing its underlying I/O infrastructure with competing I/O-intensive workloads. (Red Hat Documentation)
Symptoms in OpenShift
High WAL fsync latency can produce:
Slow oc commands
API request timeouts
Slow application deployments
Delayed Operator reconciliation
Pods remaining Pending longer
ClusterOperators becoming degraded
etcd slow fdatasync messages
Missed Raft heartbeats
Increased leader elections
Web console delays
Slow disk
│
▼
Slow WAL fsync
│
▼
Slow etcd commits
│
▼
Slow API server
│
▼
Slow controllers and Operators
│
▼
Cluster-wide control-plane impact
Existing application containers might continue processing traffic, but control-plane changes become slow or fail.
Identify the slow member using Prometheus, then debug its control-plane node:
oc debug node/master-1
Enter the host:
chroot /host
Check where etcd data resides:
findmnt /var/lib/etcd
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
df -h
df -i
Check real-time disk performance:
iostat -x110
Important fields:
Field
Meaning
await
Average I/O latency
w_await
Write latency
aqu-sz
Average disk queue depth
%util
Device busy time
w/s
Writes per second
wkB/s
Write throughput
Look for:
High w_await
Growing aqu-sz
Sustained device pressure
Spikes matching etcd latency
%util alone is not sufficient for modern parallel devices. Latency and queue depth provide better context.
Find competing processes
Use:
pidstat -d110
Possible output:
PID kB_rd/s kB_wr/s COMMAND
2100 0.00 800.00 etcd
7350 0.00 9000.00 backup-agent
This suggests that backup-agent may be competing with etcd.
Historical statistics:
sar -d110
sar -u110
sar -q110
Check kernel storage errors:
journalctl -k--since"1 hour ago" |
grep-Ei'I/O error|timeout|reset|nvme|scsi|xfs'
Check etcd logs
List etcd Pods:
oc get pods -n openshift-etcd -o wide
Review one member:
oc logs -n openshift-etcd \
<etcd-pod> \
-c etcd \
--since=1h
Search for likely symptoms:
oc logs -n openshift-etcd \
<etcd-pod> \
-c etcd \
--since=1h |
grep-Ei \
'slow fdatasync|took too long|timeout|heartbeat|leader|election'
Do not confuse WAL fsync with network replication
A write requires both disk durability and Raft consensus.
etcd leader
│
├── WAL fsync on local disk
├── Replication to peers
└── Majority acknowledgement
Therefore, a slow transaction can come from:
Local disk fsync latency
Peer disk latency
Network round-trip latency
CPU starvation
etcd overload
Always compare disk and network metrics before concluding that storage is the only problem.
Remediation
Immediate
Stop or reschedule competing backup or scanning jobs.
Resolve storage path failures.
Fix disk or VM throttling.
Reduce runaway API writes.
Pause nonessential bulk deployments.
Remove unrelated I/O-intensive activity from control-plane storage.
Permanent
Use dedicated low-latency SSD or NVMe-backed storage.
Use guaranteed IOPS rather than burst-only storage.
Ensure the VM instance supports sufficient total storage throughput.
Isolate control-plane nodes from noisy neighbours.
Avoid putting application workloads on control-plane nodes.
Monitor p99 WAL fsync continuously.
Validate storage with the Red Hat-supported fio procedure.
Do not react by restarting every etcd member. That could destroy quorum.
Also, do not use defragmentation as the first solution: it may reduce internal database fragmentation, but it does not repair slow physical storage.
Interview answer
WAL fsync in OpenShift is the time etcd takes to durably persist a Raft proposal into its Write-Ahead Log. When an API operation changes cluster state, the API server sends it to etcd. The etcd leader records the proposal in the WAL, synchronizes it to persistent storage, replicates it to the other members, and commits it after a majority acknowledges the proposal.
Because fsync waits for storage durability, slow storage directly increases etcd transaction latency and therefore OpenShift API latency. Sustained high fsync latency can cause request timeouts, pending proposals, missed Raft heartbeats, and leader changes. I monitor the p99 value of etcd_disk_wal_fsync_duration_seconds_bucket, compare it across members, and correlate it with backend commit latency, peer network RTT, leader changes, and API latency. At the node level, I use oc debug node, iostat, sar, and pidstat to identify disk latency, queueing, throttling, or competing processes. The permanent solution is isolated, low-latency storage with guaranteed IOPS—not repeatedly restarting etcd or treating defragmentation as a disk-performance fix.
Leader Changes and Pending Proposals in OpenShift etcd
Both metrics describe the stability and performance of the etcd cluster, which stores OpenShift control-plane state.
kube-apiserver
│
▼
etcd leader
│
├── Writes WAL locally
├── Replicates proposal to followers
└── Waits for quorum
A healthy etcd cluster should have:
A stable leader
Very few unexpected leader changes
Pending proposals normally close to zero
Low WAL fsync latency
Low peer-network latency
1. What is an etcd leader?
etcd uses the Raft consensus algorithm. In a standard three-member OpenShift etcd cluster:
master-0: etcd leader
master-1: etcd follower
master-2: etcd follower
The leader handles cluster-state writes.
For example:
oc scale deployment payments --replicas=5
The write process is:
API server sends update
│
▼
etcd leader creates proposal
│
▼
Leader writes proposal to WAL
│
▼
Proposal replicated to followers
│
▼
Majority acknowledges
│
▼
Proposal committed
│
▼
API request succeeds
The leader keeps followers synchronized and commits a write only after quorum acknowledges it. (Red Hat Documentation)
2. What is a leader change?
A leader change occurs when the current leader stops being leader and another etcd member is elected.
Before:
master-0 = leader
master-1 = follower
master-2 = follower
Leader heartbeat lost
│
▼
Election timeout reached
│
▼
New election
After:
master-0 = follower/unavailable
master-1 = leader
master-2 = follower
Leader election is a normal high-availability mechanism. The problem is not an occasional leader change during planned maintenance; the problem is frequent or unexpected elections.
Why does the leader change?
Common causes include:
Slow WAL fsync
The leader cannot persist its Raft log quickly enough.
Slow disk
│
▼
Slow WAL fsync
│
▼
Heartbeat processing delayed
│
▼
Followers assume leader failed
│
▼
New election
High network latency or packet loss
Followers do not receive heartbeats in time.
Leader heartbeat
│
X packet loss or delay
│
Follower election timeout
│
▼
Leader election
CPU or memory starvation
The etcd process cannot schedule enough CPU time to send or process heartbeats.
Control-plane node restart
A reboot or static-pod restart can trigger a legitimate election.
Storage or node failure
Examples include:
Cloud-disk throttling
SAN congestion
Datastore latency
Failed storage path
Hypervisor pause
Node hardware failure
Slow storage and competing disk activity can cause long fsync times, missed heartbeats, request timeouts, and temporary leader loss. (Red Hat Documentation)
Leader-change metric
Use:
etcd_server_leader_changes_seen_total
This is a cumulative counter. To see recent changes:
Red Hat identifies WAL fsync duration, backend commit latency, leader changes, and peer RTT as important etcd performance signals. (Red Hat Documentation)
Troubleshooting procedure
Step 1: Check etcd health
oc get co etcd
oc describe co etcd
oc get pods -n openshift-etcd -o wide
Look for:
Degraded conditions
Pod restarts
One unhealthy member
Revision rollout problems
Step 2: Identify the current leader
oc get pods -n openshift-etcd -o wide
Enter a healthy etcd Pod and run:
etcdctl endpoint status --cluster-w table
The output shows which member is the leader. Use the supported certificate environment and command procedure for your OCP version.
Step 3: Check etcd logs
oc logs -n openshift-etcd \
<etcd-pod> \
-c etcd \
--since=1h |
grep-Ei \
'leader|election|heartbeat|slow fdatasync|timeout|proposal|took too long'
Look for messages indicating:
leader changed
lost leader
elected leader
failed to send heartbeat
slow fdatasync
request timed out
apply request took too long
Step 4: Check disk performance
On the affected control-plane node:
oc debug node/<master-node>
chroot /host
Then run:
iostat -x110
sar -d110
pidstat -d110
df -h
df -i
Check:
await and w_await
aqu-sz
Storage utilization
Competing backup or logging processes
Disk capacity and inodes
Hypervisor or cloud-disk throttling
Step 5: Check network latency
Compare peer round-trip latency between etcd members.
Also investigate:
Packet loss
Firewall state
MTU mismatch
NIC errors
Hypervisor networking
Cross-site or cross-zone latency
Network jitter
Step 6: Check excessive API writers
Use API metrics:
sum by (verb, resource) (
rate(apiserver_request_total[5m])
)
Find high-volume clients:
topk(
20,
sum by (user_agent, verb) (
rate(apiserver_request_total[5m])
)
)
Look for unexpected increases in:
POST
PUT
PATCH
DELETE
Example production incident
02:00 backup starts on control-plane datastore
│
▼
Disk write latency increases
│
▼
WAL fsync rises to 150 ms
│
├── Proposals queue
│ └── pending proposals = 60
│
└── Heartbeats delayed
└── 4 leader elections
│
▼
API becomes slow
Resolution:
Stop or reschedule the competing backup.
Move control-plane storage to isolated low-latency disks.
Verify cloud or datastore IOPS and throughput.
Confirm pending proposals return near zero.
Confirm no new leader changes.
Verify API latency and ClusterOperator health.
What not to do
Avoid:
Restarting all etcd members simultaneously
Restarting all control-plane nodes together
Manually deleting etcd data
Removing members without the supported procedure
Changing election timers as the first fix
Defragmenting repeatedly to solve physical disk latency
OpenShift uses validated etcd timer values for each platform. Changing timers can hide symptoms rather than correct the underlying disk or network problem. (Red Hat Documentation)
Interview answer
An etcd leader change occurs when the current Raft leader becomes unavailable or fails to deliver heartbeats within the election timeout, causing another member to be elected. Occasional leader changes during maintenance can be expected, but frequent changes indicate instability caused by disk fsync latency, network delay, packet loss, CPU starvation, or control-plane node failures. I monitor this using increase(etcd_server_leader_changes_seen_total[15m]).
A pending proposal is an etcd write request that has been accepted but has not yet been committed through Raft consensus. The metric etcd_server_proposals_pending should normally stay close to zero. A sustained increase means etcd cannot persist or replicate writes as fast as they arrive, commonly because of slow WAL storage, high peer latency, resource pressure, or excessive API write activity.
I correlate both metrics with WAL fsync latency, backend commit latency, peer RTT, proposal failures, API request rates, and etcd logs. If leader changes and pending proposals increase together, I treat it as a serious control-plane performance problem and investigate the affected member’s storage, network, CPU, and competing processes.
The etcd Operator manages the lifecycle, configuration, health, certificates, and membership of the etcd cluster that stores OpenShift’s control-plane state.
In a standard highly available OpenShift cluster, etcd runs on the three control-plane nodes:
The etcd Operator continually observes the cluster, compares the current state with the required state, and corrects differences through the Kubernetes and etcd management APIs. (Red Hat Documentation)
Why etcd Is Critical
etcd is the authoritative database for Kubernetes and OpenShift.
It stores objects such as:
Deployments
Pods and their desired state
Services
Routes
Secrets
ConfigMaps
RBAC
Nodes
CRDs and Custom Resources
Operator configuration
MachineConfig objects
Cluster configuration
The runtime contents of containers and application databases are not stored in etcd.
oc apply -f deployment.yaml
│
▼
kube-apiserver
│
▼
etcd
│
▼
Deployment object stored
│
▼
Controllers create Pods
If etcd becomes unavailable, existing containers can often continue running temporarily, but:
New Pods cannot be scheduled.
Configuration changes cannot be saved.
Operators cannot reconcile normally.
oc commands that require the API begin failing.
Cluster recovery and automation stop functioning correctly.
etcd Operator vs etcd
These are different components:
Component
Responsibility
etcd
Stores Kubernetes and OpenShift state
etcd Operator
Deploys, configures, monitors, and maintains etcd
kube-apiserver
Reads and writes objects to etcd
Cluster Version Operator
Installs and upgrades the etcd Operator
Cluster Version Operator
│
▼
etcd Operator
│
▼
etcd members
│
▼
Cluster state database
The Operator itself does not store the cluster state. It manages the etcd processes that do.
Location and Resources
The Operator normally runs in:
openshift-etcd-operator
The etcd static Pods run in:
openshift-etcd
Check them:
oc get pods -n openshift-etcd-operator
oc get pods -n openshift-etcd -o wide
Check the ClusterOperator:
oc get clusteroperator etcd
The cluster-scoped configuration resource is:
oc get etcd cluster -o yaml
The etcd cluster Operator provides the cluster-scoped etcds.operator.openshift.io API and is configured through the etcd/cluster object. (Red Hat Documentation)
Main Responsibilities of the etcd Operator
1. Deploying etcd as static Pods
On each control-plane node, etcd runs as a static Pod.
Static Pod manifest
│
▼
kubelet on master node
│
▼
etcd Pod starts
Typical Pods:
oc get pods -n openshift-etcd -o wide
Example:
etcd-master-0
etcd-master-1
etcd-master-2
Static Pods are managed directly by the kubelet, not by a Deployment.
This is important because core control-plane services must be able to start even when normal Kubernetes scheduling is unavailable.
2. Maintaining etcd membership
A three-member etcd cluster normally has:
Member 1: master-0
Member 2: master-1
Member 3: master-2
The Operator monitors whether the expected members match the available control-plane nodes.
When a control-plane node is properly replaced, the Operator can:
Generate certificates for the new member
Add the replacement member to etcd
Remove stale membership
Reconcile the new topology
Red Hat documents that when a lost control-plane node is replaced, the etcd cluster Operator handles generating new TLS certificates and adding the new node as an etcd member. (Red Hat Documentation)
3. Preserving quorum
etcd uses the Raft consensus algorithm.
For three members:
Members: 3
Required quorum: 2
Maximum simultaneous failures: 1
For five members:
Members: 5
Required quorum: 3
Maximum simultaneous failures: 2
A standard OpenShift control plane normally uses three members.
master-0 master-1 master-2
Healthy Healthy Failed
\ /
Quorum remains
If two of three members are lost:
master-0 master-1 master-2
Healthy Failed Failed
│
▼
No quorum
The Operator cannot simply recreate lost authoritative state when quorum is gone. You must follow the documented disaster-recovery procedure and restore from a valid backup. Red Hat explicitly distinguishes single-member replacement from loss of the majority of control-plane hosts. (Red Hat Documentation)
4. Managing certificates
etcd communication is secured with TLS.
Certificates include:
Peer certificates for member-to-member communication
Server certificates
Client certificates for API server access
Certificate authority bundles
etcd-0 ←── mutual TLS ──→ etcd-1
│ │
└────── mutual TLS ────────→ etcd-2
The Operator manages the certificate resources and rolls out new static-Pod revisions when certificates rotate.
It also ensures the kube-apiserver has the required trust and client credentials to connect to etcd securely.
5. Managing static-Pod revisions
Configuration changes are rolled out using versioned revisions.
Current revision 20
│
Configuration changes
▼
New revision 21 generated
│
▼
Install on control-plane nodes
│
▼
Validate member health
You can inspect revision-related resources:
oc get configmaps -n openshift-etcd
oc get secrets -n openshift-etcd
The Operator ensures the expected configuration, certificates, and manifests are synchronized across the control-plane nodes.
6. Monitoring cluster health
The Operator monitors:
Member availability
Quorum
Endpoint health
Static-Pod revisions
Certificate status
Member synchronization
Leader stability
Backup and defragmentation-related conditions
Storage and API-visible health signals
Check its high-level condition:
oc get co etcd
A healthy status is:
AVAILABLE True
PROGRESSING False
DEGRADED False
Detailed conditions:
oc describe co etcd
7. Supporting member recovery
If one etcd member fails but quorum remains, the recovery process depends on the failure type:
The control-plane machine is stopped.
The node is NotReady.
The etcd Pod is crash-looping.
The underlying machine was permanently lost.
The certificates are invalid.
The Operator can reconcile the member when a temporarily unavailable node returns. For permanent loss, replacement must follow the supported procedure.
Red Hat recommends taking an etcd backup before replacing an unhealthy member. (Red Hat Documentation)
8. Automating defragmentation and maintenance
etcd receives a high number of small updates and deletions. Deleted data can leave unused space inside the backend database.
Objects created and updated
│
▼
Objects deleted
│
▼
Unused internal database pages
│
▼
Fragmentation
The etcd Operator performs supported maintenance activities, including automatic defragmentation behavior in current OpenShift releases.
However:
Defragmentation is not a fix for slow physical storage.
If WAL fsync latency is high because the disk is saturated, the solution is usually faster or isolated storage, not repeated defragmentation.
Reconciliation Loop
The etcd Operator follows the normal Operator control-loop model:
Observe control-plane nodes and etcd state
│
▼
Read desired configuration
│
▼
Compare desired state with actual state
│
┌───────┴────────┐
│ │
Matches Difference
│ │
▼ ▼
No change Reconcile resources
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Update static Rotate certs Fix membership
Pod revision
│
▼
Validate health
│
▼
Update status
Examples that trigger reconciliation include:
A control-plane node is replaced.
A certificate needs rotation.
A new OpenShift release changes etcd.
Static-Pod configuration differs.
An expected member is missing.
A member returns after temporary failure.
How an API Request Uses etcd
For example:
oc apply -f app.yaml
The flow is:
oc client
│
▼
API load balancer
│
▼
kube-apiserver
│
├── Authentication
├── Authorization
├── Admission
└── Validation
│
▼
etcd
│
▼
Persist Kubernetes object
The etcd Operator is not in the request data path. It ensures the etcd cluster receiving the request remains healthy and correctly configured.
Correlate node storage metrics with etcd WAL fsync and backend commit latency.
Backups
The Operator manages etcd operation, but the administrator must maintain a tested backup strategy.
A control-plane backup contains:
etcd snapshot
+
static Kubernetes resources
Run the documented backup script from a healthy control-plane node and copy the resulting files to secure off-cluster storage.
Backups should be:
Automated
Encrypted
Stored off-cluster
Access controlled
Tested regularly
Matched to documented recovery procedures
An etcd snapshot does not replace application database or persistent-volume backups.
What Not to Do
Avoid:
Deleting /var/lib/etcd
Manually editing static-Pod manifests
Manually removing etcd members without the supported procedure
Restarting all control-plane nodes together
Restarting all etcd members simultaneously
Copying a data directory between members
Restoring a snapshot into a live healthy cluster
Treating defragmentation as the first fix for disk contention
Editing Operator-managed resources directly
Unsafe etcd changes can cause permanent loss of cluster state.
Interview Answer
The etcd Operator manages the OpenShift control-plane etcd cluster. etcd itself stores the authoritative Kubernetes state, while the Operator deploys and maintains the etcd static Pods, certificates, configuration revisions, cluster membership and health. It continuously compares the desired state with the actual state and reconciles differences.
In a standard highly available cluster, etcd runs as three members on the control-plane nodes and requires two members for quorum. If one member fails, the cluster can continue operating while the member is recovered or replaced. If the majority is lost, the Operator cannot recreate the missing state, and the cluster must be restored using the documented disaster-recovery process and a valid etcd snapshot.
For troubleshooting, I begin with oc get co etcd, inspect the Operator conditions and logs, check the etcd static Pods and endpoint health, verify control-plane nodes, and examine WAL fsync, backend commit, peer latency, leader changes and pending proposals. I also verify disk latency and capacity because etcd depends on fast, low-latency storage.
