Role of the Scheduler in OpenShift

The scheduler decides which node should run a newly created Pod.

It does not start containers itself. Its job is to select the best eligible node and assign the Pod to it.

User creates Pod
      │
      ▼
API Server stores Pod
      │
      ▼
Pod has no nodeName
      │
      ▼
Scheduler evaluates nodes
      │
      ▼
Scheduler selects one node
      │
      ▼
kubelet starts the Pod

Where the Scheduler Runs

In a standard OpenShift cluster, the Kubernetes scheduler runs on the control-plane nodes as a static Pod.

Check it with:

oc get pods -n openshift-kube-scheduler -o wide

The scheduler is managed by the Kubernetes Scheduler Operator.

Check its status:

oc get co kube-scheduler

Healthy status:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

Scheduler vs Scheduler Operator

These are different components.

Scheduler Operator
        │
        ▼
Manages kube-scheduler
        │
        ▼
kube-scheduler assigns Pods

Scheduling Flow

Suppose a Pod is created:

apiVersion: v1
kind: Pod
metadata:
  name: payments-api
spec:
  containers:
    - name: app
      image: registry.example.com/payments-api:1.0

Initially, the Pod has no node assignment:

spec:
  nodeName: ""

The scheduler watches the API server for these unscheduled Pods.

Step 1: Watch for Pending Pods

The scheduler detects:

Pod: payments-api
Status: Pending
Node: none

It adds the Pod to its scheduling queue.

Step 2: Filter Nodes

The scheduler eliminates nodes that cannot run the Pod.

This is called the filtering phase.

Example:

Available nodes:
worker-1
worker-2
worker-3
worker-4

The scheduler checks:

• CPU availability
• Memory availability
• Node readiness
• Taints and tolerations
• Node selectors
• Node affinity
• Pod affinity and anti-affinity
• Persistent volume topology
• Host ports
• Pod count limits
• Resource constraints

After filtering:

worker-1 → eligible
worker-2 → insufficient memory
worker-3 → taint not tolerated
worker-4 → nodeSelector mismatch

Only worker-1 remains.

Step 3: Score Nodes

If several nodes are eligible, the scheduler scores them.

Example:

worker-1 → score 85
worker-2 → score 72
worker-3 → score 91

The scheduler selects:

worker-3

Scoring can consider:

• Resource balance
• Node affinity preferences
• Pod spreading
• Image locality
• Existing workload distribution
• Topology preferences

Step 4: Bind the Pod

The scheduler updates the Pod through the API server:

spec:
  nodeName: worker-3

This is called binding.

Step 5: kubelet Starts the Pod

The kubelet on worker-3 sees the assignment.

kubelet
   │
   ▼
CRI-O
   │
   ▼
Pull image
   │
   ▼
Configure networking
   │
   ▼
Mount volumes
   │
   ▼
Start container

The scheduler is no longer involved after the node assignment unless the Pod is recreated.

Important Point: Scheduler Does Not Move Running Pods

The scheduler normally handles only Pods that do not yet have a node assignment.

It does not automatically move a running Pod from one node to another just because another node becomes less busy.

Running Pod on worker-1
          │
          ▼
Scheduler does not rebalance it automatically

To redistribute workloads, you may use:

• Descheduler
• Node drain
• Pod eviction
• Deployment rollout
• Cluster autoscaling
• Manual rescheduling

Resource Requests

The scheduler uses resource requests, not actual current consumption, when deciding placement.

Example:

resources:
  requests:
    cpu: "2"
    memory: 4Gi
  limits:
    cpu: "4"
    memory: 8Gi

The scheduler reserves:

2 CPU
4 GiB memory

It does not schedule based on the Pod’s current real-time usage.

This is why incorrect requests can cause poor scheduling.

Requests too high

Pod remains Pending
Insufficient CPU or memory

Requests too low

Node becomes overloaded
Pods compete for resources

Node Selectors

A node selector forces a Pod onto nodes with matching labels.

Example:

spec:
  nodeSelector:
    workload-type: payments

Only nodes labeled:

oc label node worker-3 workload-type=payments

are eligible.

Node Affinity

Node affinity provides more expressive placement rules.

Example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - zone-a
                - zone-b

This means the Pod must run in zone-a or zone-b.

Pod Affinity

Pod affinity places Pods near other Pods.

Example use case:

Application Pod
      near
Cache Pod

This can reduce latency but may reduce fault isolation.

Pod Anti-Affinity

Pod anti-affinity spreads Pods apart.

Example:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: payments-api
        topologyKey: kubernetes.io/hostname

This prevents two payments-api replicas from running on the same node.

worker-1 → payments-api-1
worker-2 → payments-api-2
worker-3 → payments-api-3

This improves high availability.

Topology Spread Constraints

Topology spread constraints distribute Pods across zones or nodes.

Example:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payments-api

This helps distribute replicas evenly across availability zones.

Zone A → 2 Pods
Zone B → 2 Pods
Zone C → 2 Pods

Taints and Tolerations

A taint prevents Pods from being scheduled unless they tolerate it.

Example taint:

oc adm taint nodes worker-3 dedicated=payments:NoSchedule

Only Pods with this toleration can run there:

tolerations:
  - key: dedicated
    operator: Equal
    value: payments
    effect: NoSchedule

Typical OpenShift uses include:

• Control-plane protection
• Infrastructure nodes
• GPU nodes
• Storage nodes
• Dedicated application nodes

Control-Plane Nodes

Control-plane nodes normally have taints that prevent ordinary application workloads.

node-role.kubernetes.io/master:NoSchedule

or:

node-role.kubernetes.io/control-plane:NoSchedule

Platform Pods have the required tolerations, but regular application Pods do not.

Storage-Aware Scheduling

For Pods using persistent storage, the scheduler must consider volume topology.

Example:

PVC is available only in zone-a
         │
         ▼
Pod must be scheduled to node in zone-a

The scheduler evaluates:

• StorageClass
• PersistentVolume
• Volume node affinity
• Availability zone
• CSI driver constraints

A Pod may remain Pending if no eligible node can access the volume.

Scheduler Profiles in OpenShift

OpenShift supports scheduler profiles that influence how workloads are placed.

Common profiles include:

• LowNodeUtilization
• HighNodeUtilization
• NoScoring

A simplified interpretation:

Check scheduler configuration:

oc get scheduler cluster -o yaml

Example:

apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  name: cluster
spec:
  profile: LowNodeUtilization

Use supported configuration through the cluster Scheduler resource rather than editing static Pod manifests.

High Availability

The scheduler runs on control-plane nodes, but only one instance is active as leader at a time.

scheduler-master-0 → leader
scheduler-master-1 → standby
scheduler-master-2 → standby

If the leader fails:

Leader unavailable
       │
       ▼
Leader election
       │
       ▼
Another scheduler becomes active

Existing Pods continue running during a scheduler outage, but new Pods cannot be assigned until scheduling resumes.

What Happens if the Scheduler Is Down?

Existing workloads generally continue running.

However:

• New Pods stay Pending.
• Failed Pods cannot be placed on another node.
• Deployments cannot scale successfully.
• New Jobs remain unscheduled.
• Node drain replacements may remain Pending.
• Cluster upgrades can be affected.

Example:

Deployment replicas desired: 5
Running: 3
Pending: 2

Pending Pod Troubleshooting

Start with:

oc get pod <pod-name> -n <namespace>

Then:

oc describe pod <pod-name> -n <namespace>

Look at Events.

Common messages:

0/10 nodes are available:
3 insufficient memory
2 node(s) had untolerated taint
4 node(s) didn't match node selector
1 node(s) had volume node affinity conflict

This message is often the fastest way to identify the scheduling problem.

Common Scheduling Failures

Insufficient CPU

0/5 nodes are available:
5 Insufficient cpu

Check:

oc adm top nodes
oc describe node <node>

Remember that scheduling uses requested CPU, not actual CPU usage.

Insufficient memory

0/5 nodes are available:
5 Insufficient memory

Check allocated requests:

oc describe node <node>

Look under:

Allocated resources

Untolerated taint

node(s) had untolerated taint

Check:

oc describe node <node> | grep -i taint

Then verify the Pod tolerations.

Node selector mismatch

node(s) didn't match Pod's node affinity/selector

Check:

oc get nodes --show-labels

Compare with:

oc get pod <pod> -o yaml

Pod anti-affinity conflict

The placement rules may be too strict for the available number of nodes.

Example:

3 replicas
Only 2 eligible nodes
Required anti-affinity

The third Pod remains Pending.

Volume node affinity conflict

The Pod and volume are tied to different zones.

Check:

oc describe pod <pod>
oc get pv <pv-name> -o yaml
oc get pvc -n <namespace>

Too many Pods on a node

The node may have reached its Pod capacity.

Check:

oc describe node <node>

Look for:

pods: 250

and the number currently allocated.

Scheduler Operator Troubleshooting

Check the ClusterOperator:

oc get co kube-scheduler

Describe it:

oc describe co kube-scheduler

Check scheduler Pods:

oc get pods -n openshift-kube-scheduler -o wide

Check logs:

oc logs -n openshift-kube-scheduler \
  <scheduler-pod> \
  -c kube-scheduler

Check the Operator:

oc get pods -n openshift-kube-scheduler-operator

oc logs -n openshift-kube-scheduler-operator \
  deployment/openshift-kube-scheduler-operator

Check API readiness:

oc get --raw='/readyz?verbose'

The scheduler depends on a healthy API server and etcd.

Scheduling Troubleshooting Flow

Pod Pending
    │
    ▼
oc describe pod
    │
    ▼
Read FailedScheduling event
    │
    ├── Insufficient resources
    ├── Taints/tolerations
    ├── Node selectors
    ├── Affinity rules
    ├── Storage topology
    └── Pod capacity
           │
           ▼
Check scheduler Operator only if many unrelated Pods are affected

A single Pending Pod usually indicates a workload placement issue.

Many unrelated Pending Pods across the cluster may indicate:

• Scheduler failure
• API server problem
• etcd problem
• Cluster-wide capacity shortage
• Broken scheduler configuration

Scheduler vs Autoscaler

The scheduler and autoscaler have different responsibilities.

Flow:

Pod cannot fit
      │
      ▼
Scheduler marks it unschedulable
      │
      ▼
Cluster Autoscaler detects it
      │
      ▼
New node created
      │
      ▼
Scheduler places Pod

Scheduler vs Descheduler

The descheduler does not directly move a Pod. It evicts it, and then the scheduler assigns the replacement.

Interview Answer

The OpenShift scheduler is responsible for assigning unscheduled Pods to suitable nodes. It watches the API server for Pods that do not have a nodeName, places them in a scheduling queue, filters out nodes that cannot satisfy the Pod’s requirements, scores the remaining eligible nodes, and binds the Pod to the best node.

During filtering, it evaluates resource requests, node readiness, taints and tolerations, node selectors, affinity and anti-affinity, topology spread constraints, host ports, and persistent-volume topology. After the scheduler writes the selected node into the Pod specification, the kubelet on that node uses CRI-O to start the containers.

The scheduler does not run containers and normally does not rebalance already running Pods. It is managed by the Kubernetes Scheduler Operator and runs highly available on the control-plane nodes using leader election. For troubleshooting a Pending Pod, I first use oc describe pod and inspect the FailedScheduling event before checking node capacity, labels, taints, affinity rules, storage topology, and finally the scheduler Operator if the problem affects many workloads.

Ingress Operator in OpenShift

The Ingress Operator manages how HTTP and HTTPS traffic enters an OpenShift cluster.

Its primary job is to deploy and maintain the OpenShift router, which receives external requests and forwards them to the correct application Service and Pods.

The simplest traffic flow is:

Client
  │
  ▼
External Load Balancer
  │
  ▼
OpenShift Router
  │
  ▼
Route
  │
  ▼
Service
  │
  ▼
Application Pods

The Ingress Operator runs in:

openshift-ingress-operator

The router Pods normally run in:

openshift-ingress

Ingress Operator vs Router

These are not the same component.

Ingress Operator
       │
       ▼
IngressController CR
       │
       ▼
Router Deployment
       │
       ▼
Router Pods
       │
       ▼
Routes and Services

The Operator manages the routers. The routers process the actual application traffic.

Main Responsibilities

1. Deploying router Pods

The Ingress Operator creates and maintains router workloads.

Check them with:

oc get pods -n openshift-ingress -o wide

Typical router Pods:

router-default-7d8b5c9d4f-abc12
router-default-7d8b5c9d4f-def34

The Operator ensures the desired number of router replicas is running.

Desired replicas: 2
Actual replicas: 1
        │
        ▼
Ingress Operator detects drift
        │
        ▼
Creates replacement router Pod

2. Managing the IngressController resource

The main configuration object is:

oc get ingresscontroller -n openshift-ingress-operator

The default object is usually:

default

Inspect it:

oc get ingresscontroller default \
  -n openshift-ingress-operator \
  -o yaml

A simplified example:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  replicas: 3
  domain: apps.cluster.example.com

The Operator watches this resource and reconciles the router configuration.

3. Managing the wildcard application domain

Applications commonly use names such as:

payments.apps.cluster.example.com
mobile.apps.cluster.example.com
api.apps.cluster.example.com

The wildcard DNS record:

*.apps.cluster.example.com

normally points to the ingress load balancer.

*.apps.cluster.example.com
             │
             ▼
      External Load Balancer
             │
             ▼
         Router Pods

The IngressController defines the domain that its routers serve.

4. Managing Route traffic

An OpenShift Route exposes a Service externally.

Example:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: payments
  namespace: banking
spec:
  host: payments.apps.cluster.example.com
  to:
    kind: Service
    name: payments-service
  port:
    targetPort: https

Traffic flow:

payments.apps.cluster.example.com
              │
              ▼
          Router Pod
              │
              ▼
        Route: payments
              │
              ▼
   Service: payments-service
              │
              ▼
        Ready payment Pods

The router watches Routes and EndpointSlices and updates its proxy configuration dynamically.

TLS Termination

The Ingress Operator manages router configuration for different TLS models.

Edge termination

Client ──HTTPS──> Router ──HTTP──> Application

The router terminates TLS.

Example:

tls:
  termination: edge

Use this when encryption between the router and application is not required or the internal network is trusted.

Re-encrypt termination

Client ──HTTPS──> Router ──HTTPS──> Application

The router terminates the client connection and creates a new TLS connection to the backend.

Example:

tls:
  termination: reencrypt
  destinationCACertificate: |
    ...

This is commonly used for sensitive enterprise applications.

Passthrough termination

Client ──HTTPS──> Router ──HTTPS──> Application

The router does not decrypt the request. TLS terminates inside the application.

Example:

tls:
  termination: passthrough

Because the router cannot inspect HTTP content, passthrough routing relies primarily on TLS SNI.

Default Ingress Certificate

The default router normally presents a wildcard certificate for:

*.apps.cluster.example.com

The Ingress Operator manages the router’s default certificate configuration.

You can configure a custom certificate through a Secret:

oc create secret tls custom-apps-certificate \
  --cert=apps.crt \
  --key=apps.key \
  -n openshift-ingress

Then reference it:

oc patch ingresscontroller default \
  -n openshift-ingress-operator \
  --type=merge \
  -p '{
    "spec": {
      "defaultCertificate": {
        "name": "custom-apps-certificate"
      }
    }
  }'

The Operator rolls out the router configuration with the new certificate.

Ingress Publishing Strategies

The Ingress Operator supports different ways to expose routers depending on the infrastructure.

LoadBalancerService

Common in public clouds:

Cloud Load Balancer
        │
        ▼
Router Service
        │
        ▼
Router Pods

The Operator creates a Service of type LoadBalancer.

HostNetwork

Router Pods bind directly to ports on the node.

Client
  │
  ▼
Node IP:80/443
  │
  ▼
Router Pod using host network

This is often used in bare-metal or controlled on-premises environments.

NodePortService

The router is exposed through NodePorts.

External Load Balancer
        │
        ▼
NodeIP:NodePort
        │
        ▼
Router Service

An external F5, HAProxy, or NetScaler can send traffic to the NodePorts.

Private or internal load balancer

An ingress controller can be exposed only internally.

Example architecture:

Internet users
      │
      ▼
Public LB
      │
      ▼
Public IngressController
Corporate users
      │
      ▼
Internal LB
      │
      ▼
Private IngressController

This is valuable in banking and enterprise environments.

Multiple Ingress Controllers

You are not limited to the default router.

You might create:

default
public
internal
partner
pci

Example separation:

Public applications
       │
       ▼
public.apps.cluster.example.com
       │
       ▼
Public routers
Internal applications
       │
       ▼
internal.apps.cluster.example.com
       │
       ▼
Internal routers

Each ingress controller can have its own:

• Domain
• Certificate
• Node placement
• Replica count
• Publishing strategy
• Route selector
• Namespace selector
• Load balancer

Route Selection

A custom ingress controller can serve only selected Routes.

Example using a route selector:

spec:
  routeSelector:
    matchLabels:
      ingress: internal

Then label a Route:

oc label route payments ingress=internal -n banking

Only the matching ingress controller serves that Route.

Namespace Selection

An ingress controller can also serve only selected namespaces.

Example:

spec:
  namespaceSelector:
    matchLabels:
      exposure: internal

Label the namespace:

oc label namespace banking exposure=internal

This provides stronger organizational separation.

Router Placement on Infrastructure Nodes

In production, router Pods are often placed on dedicated infrastructure nodes.

Application workers
  └── Business workloads
Infrastructure workers
  ├── Router Pods
  ├── Registry
  ├── Monitoring
  └── Logging

Example node placement:

spec:
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/infra: ""
    tolerations:
      - key: node-role.kubernetes.io/infra
        operator: Exists
        effect: NoSchedule

Benefits include:

• Isolating ingress traffic from application workloads
• Predictable capacity
• Easier network segmentation
• Reduced noisy-neighbour risk
• Better operational control

High Availability

A production ingress controller should have multiple router replicas.

External Load Balancer
        │
  ┌─────┼─────┐
  ▼     ▼     ▼
Router1 Router2 Router3

If one router fails:

Router1 readiness fails
        │
        ▼
Endpoint removed
        │
        ▼
Traffic continues through Router2 and Router3

For high availability:

• Use at least two replicas.
• Spread routers across nodes and zones.
• Use anti-affinity or topology controls.
• Ensure the external load balancer removes unhealthy backends.
• Maintain sufficient capacity during upgrades.

Router Load-Balancing Algorithms

The router can distribute traffic to application Pods using algorithms such as:

• roundrobin
• leastconn
• source

Example annotation:

metadata:
  annotations:
    haproxy.router.openshift.io/balance: leastconn

Round robin

Requests rotate across backend Pods.

Least connections

New requests go to the backend with fewer active connections.

Source

Uses the client source to provide a form of persistence.

Applications should ideally be stateless rather than relying heavily on sticky sessions.

Request Processing

A request follows this path:

1. DNS resolves application hostname
2. Client connects to external load balancer
3. Load balancer selects a router endpoint
4. Router evaluates hostname and Route
5. Router performs TLS processing
6. Router selects the target Service
7. Router selects a ready endpoint
8. Request reaches the application Pod

The router generally sends traffic only to endpoints considered ready.

Ingress Operator Reconciliation

The Ingress Operator continuously compares desired and actual state.

IngressController spec
          │
          ▼
Operator observes router state
          │
          ▼
Compare desired and actual state
          │
    ┌─────┴─────┐
    │           │
 Matches      Drift
    │           │
    ▼           ▼
 No action    Create/update resources
                    │
                    ▼
             Check router health
                    │
                    ▼
              Update status

Examples of events that trigger reconciliation:

• Replica count changes
• Certificate changes
• Domain changes
• Node-placement changes
• Router Pod failure
• OpenShift upgrade
• Publishing-strategy changes

Relationship with Other Components

Cluster Version Operator
        │
        ▼
Ingress Operator
        │
        ├── IngressController CR
        ├── Router Deployment
        ├── Router Service
        ├── Certificates
        └── Status

It also depends on:

Useful Commands

Check ClusterOperator status

oc get co ingress

Healthy state:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

More detail:

oc describe co ingress

Check Ingress Operator Pods

oc get pods -n openshift-ingress-operator

View logs:

oc logs -n openshift-ingress-operator \
  deployment/ingress-operator

Check IngressControllers

oc get ingresscontroller \
  -n openshift-ingress-operator

Describe the default controller:

oc describe ingresscontroller default \
  -n openshift-ingress-operator

Check router Pods

oc get pods -n openshift-ingress -o wide

Check router logs:

oc logs -n openshift-ingress <router-pod>

Check router Service

oc get svc -n openshift-ingress

Check Routes

oc get routes -A

Describe one Route:

oc describe route payments -n banking

Check Service endpoints

oc get svc payments-service -n banking
oc get endpointslices -n banking

Troubleshooting a Degraded Ingress Operator

Use this sequence:

ClusterOperator
      ↓
IngressController
      ↓
Operator logs
      ↓
Router Deployment and Pods
      ↓
Service and external load balancer
      ↓
DNS
      ↓
Route
      ↓
Application Service and endpoints

Step 1: Check Operator condition

oc get co ingress
oc describe co ingress

Look for messages about:

• Router deployment unavailable
• Load balancer provisioning failure
• Certificate problem
• DNS failure
• Insufficient replicas
• Node-placement failure

Step 2: Check IngressController status

oc get ingresscontroller default \
  -n openshift-ingress-operator \
  -o yaml

Review conditions such as:

• Available
• Progressing
• Degraded
• LoadBalancerManaged
• Admitted

Step 3: Check router Pods

oc get pods -n openshift-ingress -o wide

For a failing Pod:

oc describe pod <router-pod> -n openshift-ingress
oc logs <router-pod> -n openshift-ingress

Possible causes:

• Image pull failure
• Node selector mismatch
• Missing toleration
• Port conflict
• Certificate failure
• Insufficient CPU or memory
• Readiness probe failure

Step 4: Check DNS

dig payments.apps.cluster.example.com
dig '*.apps.cluster.example.com'

The application hostname should resolve to the correct ingress VIP or load balancer.

Step 5: Check external access

curl -vk https://payments.apps.cluster.example.com

This helps identify:

• DNS errors
• TCP failures
• TLS failures
• Router responses
• HTTP status codes

Step 6: Check the Route

oc get route payments -n banking
oc describe route payments -n banking

Verify:

• Correct hostname
• Correct Service
• Correct target port
• Correct TLS termination
• Route is admitted by the expected router

Step 7: Check Service and endpoints

oc get svc payments-service -n banking
oc get endpointslices -n banking
oc get pods -n banking

A Route with no ready backend endpoints commonly returns:

503 Service Unavailable

Common Errors

503 Service Unavailable

Usually means the router cannot reach a healthy backend.

Check:

• Pod readiness
• Service selector
• EndpointSlices
• Target port
• Application listening port
• NetworkPolicy

Application route does not resolve

Likely causes:

• Missing wildcard DNS
• Wrong domain
• Wrong load-balancer address
• Public/private DNS mismatch

TLS certificate error

Check:

• Certificate validity
• Certificate chain
• Hostname/SAN
• Secret name
• Route-specific certificate
• IngressController default certificate

Router Pods Pending

Check:

• Node selector
• Taints and tolerations
• Resource pressure
• Pod anti-affinity
• SCC or permissions
• Host port availability

Banking Architecture Example

Internet
   │
   ▼
DDoS protection
   │
   ▼
WAF
   │
   ▼
Public Load Balancer
   │
   ▼
Public IngressController
   │
   ▼
Internet-facing applications
Corporate network
   │
   ▼
Internal Load Balancer
   │
   ▼
Private IngressController
   │
   ▼
Internal banking applications

Recommended controls:

• Separate public and internal ingress controllers
• Dedicated infrastructure nodes
• Re-encrypt or passthrough TLS
• WAF in front of public ingress
• Multiple replicas across failure domains
• Central router access logging
• NetworkPolicies behind the router
• Separate certificates and DNS zones
• Route and namespace selectors
• No direct public access to internal routers

Important Interview Distinction

The Ingress Operator does not directly forward application requests. It manages the IngressController and router resources. The router Pods perform the actual Layer 7 traffic routing.

Ingress Operator
      │
      ▼
Manages Router
      │
      ▼
Router processes traffic

Interview Answer

The OpenShift Ingress Operator manages the lifecycle and configuration of the cluster’s ingress controllers. It creates and maintains the router Deployments, Services, certificates, replica count, publishing strategy, and node placement. The default ingress controller serves the wildcard application domain, normally *.apps.<cluster-domain>, and the router Pods receive HTTP or HTTPS traffic from an external load balancer.

When a request arrives, the router matches the hostname to an OpenShift Route, performs edge, re-encrypt, or passthrough TLS handling, and forwards the request to the target Service and ready Pod endpoints. The Operator continuously reconciles the desired IngressController configuration with the running router resources and replaces failed Pods or rolls out configuration changes. For production environments, I normally use multiple router replicas on dedicated infrastructure nodes and separate public and internal ingress controllers. When troubleshooting, I check oc get co ingress, the IngressController conditions, Operator logs, router Pods, external load balancer, wildcard DNS, Route configuration, Service, EndpointSlices, and application readiness.

API Server Operator in OpenShift

The Kubernetes API Server Operator manages the lifecycle and configuration of the Kubernetes API server in OpenShift.

Its main responsibility is to make sure the kube-apiserver is:

• Installed
• Correctly configured
• Highly available
• Using valid certificates
• Running the version required by OpenShift
• Automatically recovered if a component fails
• Safely rolled out during upgrades

The Operator runs in:

openshift-kube-apiserver-operator

It manages API server instances in:

openshift-kube-apiserver

The Operator is installed and updated through the Cluster Version Operator. Its cluster-scoped configuration resource is KubeAPIServer, named cluster. (Red Hat Documentation)

Where It Fits

Users, kubelets, Operators and controllers
                    │
                    ▼
        API load balancer :6443
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
   master-0      master-1      master-2
 kube-apiserver kube-apiserver kube-apiserver
       ▲            ▲            ▲
       └────────────┼────────────┘
                    │
       Kubernetes API Server Operator
                    │
                    ▼
       Configuration, rollout and health
                    │
                    ▼
                   etcd

The API server is the main entry point into the cluster. Commands such as:

oc get pods
oc create deployment
oc apply -f app.yaml

all pass through the Kubernetes API server.

Kubernetes API Server vs API Server Operator

These are different components.

The flow is:

API Server Operator
        │
        ▼
Manages kube-apiserver
        │
        ▼
kube-apiserver handles API requests

Kubernetes API Server vs OpenShift API Server

OpenShift has two related API server layers.

Kubernetes API server

Handles standard Kubernetes resources:

• Pods
• Deployments
• Services
• Secrets
• ConfigMaps
• Nodes
• RBAC
• StatefulSets

OpenShift API server

Handles OpenShift-specific APIs, such as certain:

• Projects
• Routes-related platform integrations
• Security and authorization extensions
• OpenShift-specific resources

Client request
      │
      ▼
Kubernetes API aggregation layer
      │
      ├── Kubernetes APIs
      │
      └── OpenShift-specific APIs

The Kubernetes API Server Operator manages kube-apiserver, while the OpenShift API Server Operator installs and maintains openshift-apiserver. (Red Hat Documentation)

Main Responsibilities

1. Deploying API server static pods

On each control-plane node, the API server runs as a static pod.

/etc/kubernetes/manifests/
        │
        ▼
kubelet detects manifest
        │
        ▼
kube-apiserver pod starts

Typical API server pods can be viewed with:

oc get pods -n openshift-kube-apiserver -o wide

Example:

kube-apiserver-master-0
kube-apiserver-master-1
kube-apiserver-master-2

Static pods are controlled by the kubelet directly rather than by a Deployment.

2. Managing revisions

The Operator creates versioned API server revisions.

Revision 12
     │
     ▼
Configuration changed
     │
     ▼
Revision 13 created
     │
     ▼
Control-plane nodes updated gradually

You can see revision resources and related configuration in the API server namespace:

oc get configmaps -n openshift-kube-apiserver
oc get secrets -n openshift-kube-apiserver

Revision-based management allows the Operator to roll out a consistent configuration and diagnose which revision each node is running.

3. Rolling updates

The Operator avoids replacing every API server simultaneously.

master-0
Update → Ready
          │
          ▼
master-1
Update → Ready
          │
          ▼
master-2
Update → Ready

This preserves API availability as long as:

• The load balancer has healthy backends.
• Enough control-plane nodes remain available.
• etcd retains quorum.
• The new revision becomes healthy.

During an OpenShift upgrade, the CVO delivers the new release state and the API Server Operator reconciles the API server toward that version.

4. Certificate management

The API server requires several certificates for:

• External API access
• Internal API access
• Communication with etcd
• Authentication of clients
• Communication with aggregated APIs
• Service-network endpoints

The Operator helps manage and rotate API server certificates.

Certificate approaches rotation point
               │
               ▼
Operator creates updated certificate resources
               │
               ▼
New static-pod revision
               │
               ▼
API servers roll out incrementally

This reduces the chance of an API outage caused by expired certificates.

5. API server configuration

The cluster-scoped resource is:

oc get kubeapiserver cluster -o yaml

The corresponding API is:

operator.openshift.io/v1
kind: KubeAPIServer
metadata:
  name: cluster

The resource provides configuration for the Operator that manages kube-apiserver. (Red Hat Documentation)

A simplified example:

apiVersion: operator.openshift.io/v1
kind: KubeAPIServer
metadata:
  name: cluster
spec:
  audit:
    profile: Default

Do not add unsupported fields or manually edit generated static-pod manifests.

Use the supported cluster API:

oc edit kubeapiserver cluster

Red Hat identifies oc edit kubeapiserver as the configuration interface for the Kubernetes API Server Operator. (Red Hat Documentation)

6. Audit policy configuration

The Operator applies the configured API audit profile.

Audit records may include:

• User identity
• Service account identity
• API resource
• Operation or verb
• Source IP
• Request result
• Timestamp

Example operations:

create
update
patch
delete
get
list

Audit configuration should be changed through the KubeAPIServer resource, not by directly modifying the API server static-pod command arguments.

7. etcd connectivity

The API server reads and writes cluster state in etcd.

oc request
    │
    ▼
kube-apiserver
    │
    ▼
Authentication and authorization
    │
    ▼
Admission controls
    │
    ▼
etcd read/write
    │
    ▼
Response

The Operator manages the API server configuration required to connect securely to etcd, but the etcd cluster itself is maintained by the etcd Operator.

A slow etcd backend directly affects API performance:

Slow etcd disk
      │
      ▼
Slow etcd transaction
      │
      ▼
Slow API response
      │
      ▼
Slow oc commands and controller reconciliation

8. Health monitoring

The Operator monitors whether the API server is:

• Available
• Progressing
• Degraded
• Running the expected revision
• Responding to health checks

Check the ClusterOperator:

oc get clusteroperator kube-apiserver

Healthy status:

AVAILABLE     True
PROGRESSING   False
DEGRADED      False

Detailed information:

oc describe clusteroperator kube-apiserver

Reconciliation Process

The Operator continuously performs this loop:

Observe desired configuration
             │
             ▼
Inspect current API server revision
             │
             ▼
Compare desired and actual state
             │
       ┌─────┴─────┐
       │           │
   Matches      Difference
       │           │
       ▼           ▼
 No action    Create new revision
                   │
                   ▼
            Roll out static pods
                   │
                   ▼
              Check readiness
                   │
                   ▼
              Update status

Example: the audit profile changes.

Administrator updates KubeAPIServer CR
                    │
                    ▼
Operator notices configuration change
                    │
                    ▼
New revision generated
                    │
                    ▼
API server nodes updated incrementally
                    │
                    ▼
Operator reports Available

Request Processing Through the API Server

A request normally passes through several stages:

oc apply
   │
   ▼
Load balancer
   │
   ▼
kube-apiserver
   │
   ├── TLS validation
   ├── Authentication
   ├── Authorization
   ├── Admission control
   ├── Resource validation
   └── etcd persistence
          │
          ▼
       Response

For example:

oc create deployment nginx --image=nginx

The API server:

1. Authenticates the user.
2. Checks RBAC authorization.
3. Runs admission controls.
4. Validates the Deployment.
5. Stores it in etcd.
6. Returns the result.
7. Controllers later create the ReplicaSet and Pods.

The Operator maintains the API server that performs these steps; it does not process user API requests itself.

Useful Commands

Check ClusterOperator status

oc get co kube-apiserver
oc describe co kube-apiserver

Check Operator pods

oc get pods -n openshift-kube-apiserver-operator

Check Operator logs

oc logs -n openshift-kube-apiserver-operator \
  deployment/kube-apiserver-operator

Check API server pods

oc get pods -n openshift-kube-apiserver -o wide

Check a specific API server pod

oc describe pod -n openshift-kube-apiserver \
  kube-apiserver-master-0

Check API readiness

oc get --raw='/readyz?verbose'

Check API liveness

oc get --raw='/livez?verbose'

Check configuration

oc get kubeapiserver cluster -o yaml

Check recent events

oc get events -n openshift-kube-apiserver \
  --sort-by='.lastTimestamp'

Troubleshooting a Degraded API Server Operator

Use this sequence:

ClusterOperator
      ↓
Operator conditions
      ↓
Operator pod and logs
      ↓
API server static pods
      ↓
Node health
      ↓
etcd health
      ↓
Certificates and load balancer

Step 1: Check status

oc get co kube-apiserver
oc describe co kube-apiserver

Look at the condition messages under:

• Available
• Progressing
• Degraded

The condition message usually points to the affected revision or node.

Step 2: Check the Operator

oc get pods -n openshift-kube-apiserver-operator

oc logs -n openshift-kube-apiserver-operator \
  deployment/kube-apiserver-operator \
  --since=1h

Look for:

• Revision installation failure
• Certificate errors
• Missing ConfigMaps or Secrets
• Static-pod rollout timeout
• Node installer failure

Step 3: Check API server pods

oc get pods -n openshift-kube-apiserver -o wide

Check containers in a failing static pod:

oc describe pod -n openshift-kube-apiserver \
  <kube-apiserver-pod>

oc logs -n openshift-kube-apiserver \
  <kube-apiserver-pod> \
  -c kube-apiserver \
  --since=1h

Step 4: Check etcd

oc get co etcd
oc get pods -n openshift-etcd -o wide

API server symptoms can be caused by:

• etcd disk latency
• Lost etcd quorum
• Slow network between control-plane nodes
• etcd certificate errors
• etcd database pressure

Step 5: Check the affected node

oc get nodes
oc describe node <master-node>

For host-level investigation:

oc debug node/<master-node>
chroot /host
systemctl status kubelet
journalctl -u kubelet --since "1 hour ago"

Step 6: Check the load balancer

Test the cluster endpoint:

curl -k https://api.<cluster-domain>:6443/readyz

Test each control-plane backend separately:

curl -k https://<master-0>:6443/readyz
curl -k https://<master-1>:6443/readyz
curl -k https://<master-2>:6443/readyz

Possible problems:

• An unhealthy master remains in the pool.
• Health check is incorrect.
• Port 6443 is blocked.
• TLS inspection interferes with API traffic.
• DNS resolves to the wrong VIP.

What Not to Do

Avoid:

Editing static-pod manifests manually
Deleting API server certificates
Restarting all control-plane nodes together
Deleting revision resources without Red Hat guidance
Editing Operator-managed Deployments directly
Changing etcd data manually

Operator-managed resources will often be restored, and unsafe changes could make the API unavailable.

Always use supported configuration resources and preserve etcd quorum.

Relationship with Other Operators

Cluster Version Operator
          │
          ▼
Kubernetes API Server Operator
          │
          ├── works with etcd Operator
          ├── relies on Machine Config Operator
          ├── exposes health to Monitoring
          └── provides APIs used by all other Operators

Interview Answer

The Kubernetes API Server Operator manages and updates the kube-apiserver instances running as static pods on the OpenShift control-plane nodes. It is installed through the Cluster Version Operator and continuously reconciles the API server’s desired configuration with its actual state. It manages versioned static-pod revisions, certificates, audit configuration, etcd connectivity and rolling updates across the control-plane nodes.

The API Server Operator itself does not process application requests; the kube-apiserver does that. The Operator makes sure the API servers remain correctly configured and highly available. During a configuration change or upgrade, it creates a new revision and rolls it out incrementally, checking readiness before completing the transition. For troubleshooting, I start with oc get co kube-apiserver, inspect its conditions, check the Operator logs and API server static pods, then validate etcd, control-plane node health, certificates and the external API load balancer.

This is one of the most common OpenShift Architect interview questions.

The interviewer wants to know whether you understand how OpenShift manages itself. The key concept is that almost every major platform component is managed by an Operator.

What is an Operator?

An Operator is a Kubernetes controller that extends Kubernetes using Custom Resource Definitions (CRDs) to automate the lifecycle of applications and platform components.

An Operator continuously performs a reconciliation loop:

Desired State (CR)
        ↓
Operator watches
        ↓
Actual State
        ↓
Difference?
        ↓
Take Action
        ↓
Cluster matches Desired State

Unlike a traditional administrator, an Operator works continuously to maintain the desired state.

OpenShift Operator Architecture

                    Cluster Version Operator
                              │
       ┌──────────────────────┼──────────────────────┐
       │                      │                      │
       ▼                      ▼                      ▼
API Server Operator      Machine Config       Ingress Operator
                              Operator
       │                      │                      │
       ▼                      ▼                      ▼
 Authentication          etcd Operator        DNS Operator

Each Operator manages one specific component.

Major Operators

1. Cluster Version Operator (CVO)

Most important Operator

Responsible for:

• Cluster upgrades
• Platform version management
• Installing platform Operators
• Coordinating upgrades
• Ensuring all Operators reach the desired version

Think of it as the conductor of the orchestra.

New OpenShift Release
↓
Cluster Version Operator
↓
Updates Operators
↓
Cluster Updated

Useful commands:

oc get clusterversion
oc adm upgrade
oc get co

2. Machine Config Operator (MCO)

Responsible for:

• RHCOS updates
• Kernel arguments
• CRI-O configuration
• kubelet configuration
• SSH keys
• OS files
• Systemd services

Architecture:

MachineConfig
↓
Machine Config Controller
↓
Machine Config Daemon
↓
Node Updated

During upgrades:

Worker
↓
Drain
↓
OS Update
↓
Reboot
↓
Ready

Commands:

oc get mcp
oc describe mcp worker
oc get machineconfig

3. kube-apiserver Operator

Manages:

• API Server Pods
• API certificates
• API configuration
• Audit configuration
• Encryption
• Scaling
• Rolling updates

If an API Server crashes:

API Server Down
↓
Operator Detects
↓
New Pod
↓
API Restored

Commands:

oc get pods -n openshift-kube-apiserver

4. etcd Operator

Manages:

• etcd cluster
• Membership
• Certificates
• Health
• Scaling
• Recovery

Monitors:

Leader
Followers
Quorum
Certificates
Storage

If etcd fails:

Member Failed
↓
Operator Detects
↓
Recovery

Commands:

oc get pods -n openshift-etcd
oc get co etcd

5. Ingress Operator

Responsible for:

• Router Pods
• Wildcard certificates
• Route publishing
• IngressControllers
• Load balancing

Architecture:

Route
↓
Ingress Operator
↓
Router Pods
↓
Applications

Commands:

oc get ingresscontroller -n openshift-ingress-operator
oc get pods -n openshift-ingress

6. Authentication Operator

Manages:

• OAuth
• Identity Providers
• Login page
• OAuth certificates
• Token configuration

Example:

LDAP
↓
Authentication Operator
↓
OAuth
↓
OpenShift Login

Commands:

oc get oauth cluster -o yaml

7. DNS Operator

Responsible for:

• CoreDNS
• Cluster DNS
• Service discovery
• DNS forwarding
• Stub domains

Application
↓
DNS
↓
CoreDNS
↓
Service IP

Commands:

oc get dns.operator/default

8. Network Operator

Configures:

• OVN-Kubernetes
• Cluster Network
• Service Network
• Pod CIDRs
• Egress IP
• MTU

Architecture:

Network CR
↓
Network Operator
↓
OVN
↓
Pod Networking

Commands:

oc get network.operator cluster

9. Image Registry Operator

Responsible for:

• Internal Registry
• Storage backend
• Registry certificates
• Scaling

Flow:

Image Push
↓
Registry Operator
↓
Storage

Commands:

oc get configs.imageregistry.operator.openshift.io cluster

10. Monitoring Operator

Deploys:

• Prometheus
• Alertmanager
• Node Exporter
• kube-state-metrics
• Grafana (optional/external)

Architecture:

Monitoring Operator
↓
Prometheus
↓
Alertmanager
↓
Metrics

Commands:

oc get pods -n openshift-monitoring

11. Console Operator

Manages:

• Web Console
• Console plugins
• Branding
• Console route

Console Operator
↓
Console Pods
↓
https://console-openshift-console.apps.cluster

12. Machine API Operator

Cloud platforms only.

Responsible for:

• Creating VMs
• Deleting VMs
• Auto Scaling
• MachineSets
• MachineHealthChecks

Flow:

MachineSet
↓
Machine API
↓
AWS EC2
Azure VM
GCP VM

Commands:

oc get machines -A
oc get machinesets -A

13. Cluster Storage Operator

Depends on platform.

Can manage:

• CSI Drivers
• Storage Classes
• Persistent Volumes

Example:

PVC
↓
CSI Driver
↓
Cloud Disk
↓
Pod

14. Insights Operator

Collects:

• Cluster health
• Recommendations
• Telemetry
• Upgrade risks

Cluster
↓
Insights
↓
Red Hat Portal

15. Operator Lifecycle Manager (OLM)

Installs Operators.

Responsible for:

• OperatorHub
• CSVs
• Subscriptions
• InstallPlans

Architecture:

OperatorHub
↓
Subscription
↓
CSV
↓
Operator Installed

Commands:

oc get csv -A
oc get subscriptions -A

Relationship Between Operators

                     Cluster Version Operator

          ┌──────────────┼──────────────┐

          ▼              ▼              ▼

     API Server       Machine        Ingress

       Operator       Config         Operator

                        │

                        ▼

                 Machine Config

                     Daemon

                        │

                        ▼

                     RHCOS

Cluster Operators Status

The most common interview command:

oc get co

Example:

NAME              AVAILABLE   PROGRESSING   DEGRADED
authentication    True        False         False
console           True        False         False
dns               True        False         False
etcd              True        False         False
ingress           True        False         False
monitoring        True        False         False

Healthy cluster:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

How Operators Work Together During an Upgrade

Suppose OpenShift upgrades from 4.18 → 4.19.

CVO
↓
API Server Operator
↓
etcd Operator
↓
MCO
↓
Ingress Operator
↓
Monitoring
↓
Authentication
↓
Console
↓
Completed

Each Operator reports status back to the CVO.

The CVO waits until each Operator is healthy before moving to the next stage.

Troubleshooting Operators

Check Operators

oc get co

Describe an Operator

oc describe co ingress

View Events

oc get events -A

Check Pods

Example:

oc get pods -n openshift-etcd
oc get pods -n openshift-kube-apiserver
oc get pods -n openshift-machine-config-operator

View Logs

oc logs <pod>

Real Interview Scenario

Question:

“The Ingress Operator is degraded. How would you troubleshoot?”

Answer:

1. Check Operator status:

oc get co ingress

2. Describe it:

oc describe co ingress

3. Check IngressController:

oc get ingresscontroller -n openshift-ingress-operator

4. Check router Pods:

oc get pods -n openshift-ingress

5. Verify:

• Router readiness
• Certificates
• Routes
• Load balancer
• DNS
• Node placement

Operator Responsibilities Summary

Interview Answer (2 Minutes)

**OpenShift is built around the Operator pattern, where each major platform component is managed by its own Operator. The Cluster Version Operator orchestrates upgrades and ensures every platform Operator reaches the desired version. The Machine Config Operator manages RHCOS configuration, operating-system updates, CRI-O settings, and node reboots. The kube-apiserver Operator manages the API servers, while the etcd Operator manages the distributed key-value store, including certificates, membership, and health. The Ingress Operator manages router Pods and Routes, the Authentication Operator manages OAuth and identity providers, the Network Operator configures OVN-Kubernetes, the DNS Operator manages CoreDNS, and the Monitoring Operator deploys Prometheus, Alertmanager, and node-exporter. During normal operation, each Operator continuously reconciles the desired state with the actual state, automatically correcting drift. When troubleshooting, I start with oc get co; a healthy cluster shows Available=True, Progressing=False, and Degraded=False for its ClusterOperators.

What is a DaemonSet in OpenShift?

A DaemonSet is a Kubernetes workload that ensures exactly one copy of a Pod runs on every selected node in the cluster.

Unlike a Deployment, where you specify the number of replicas, a DaemonSet automatically creates and removes Pods as nodes are added or removed.

Think of a DaemonSet as “one Pod per node.”

Why DaemonSets are Important in OpenShift

Many OpenShift platform services must run on every node because they collect metrics, logs, or provide node-level networking.

For example:

                OpenShift Cluster

      ┌──────────────┬──────────────┬──────────────┐
      │              │              │
      ▼              ▼              ▼
   Worker-1       Worker-2       Worker-3
      │              │              │
      ▼              ▼              ▼
 Node Exporter   Node Exporter   Node Exporter

Each node has its own Node Exporter Pod.

If a new node joins:

Worker-4
↓
DaemonSet automatically creates
↓
Node Exporter Pod

No manual action is required.

DaemonSet vs Deployment

Example:

Deployment

Cluster
10 Nodes
↓
Deployment replicas = 3
↓
Only 3 Pods

DaemonSet

Cluster
10 Nodes
↓
DaemonSet
↓
10 Pods
(1 on each node)

How DaemonSet Works

New Node Added
        │
        ▼
DaemonSet Controller detects node
        │
        ▼
Creates Pod
        │
        ▼
Pod scheduled on new node

If a node is removed:

Node Deleted
↓
DaemonSet Pod Deleted

Real OpenShift DaemonSets

Run:

oc get daemonset -A

Typical output:

NAMESPACE                     NAME
openshift-monitoring          node-exporter
openshift-logging             collector
openshift-ovn-kubernetes      ovnkube-node
openshift-multus              multus
openshift-machine-config      machine-config-daemon

These are all critical platform components.

1. Node Exporter

Every Node
↓
Node Exporter
↓
CPU
Memory
Disk
Filesystem
Network

Used by Prometheus for infrastructure monitoring.

2. Machine Config Daemon

MachineConfig
↓
Machine Config Operator
↓
Machine Config Daemon
↓
Every Node

The Machine Config Daemon applies operating system changes on each RHCOS node.

3. OVN-Kubernetes

Every node participates in cluster networking.

Node
↓
OVN Pod
↓
Programs Open vSwitch
↓
Pod Networking

Without it, Pods cannot communicate.

4. Vector Collector

OpenShift Logging deploys Vector as a DaemonSet.

Every Node
↓
Vector
↓
Collect container logs
↓
Loki
Splunk
Elasticsearch

Each node collects its own logs locally and forwards them.

DaemonSet Architecture

                  DaemonSet Controller
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
        ▼                 ▼                 ▼
     Worker-1         Worker-2         Worker-3
        │                 │                 │
        ▼                 ▼                 ▼
    One Pod          One Pod          One Pod

Scheduling

The DaemonSet controller watches nodes.

Whenever a new node appears:

Node Ready
↓
Matches selector
↓
Create Pod
↓
Schedule Pod

Node Selectors

DaemonSets can target only specific nodes.

Example:

spec:
  template:
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: ""

Only worker nodes receive the Pod.

Taints and Tolerations

Control-plane nodes are often tainted.

Example:

master-0
↓
NoSchedule

To run there, the DaemonSet must tolerate the taint.

Example:

tolerations:
- operator: Exists

Without the toleration:

Master Node
×
No Pod

Example DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
      - name: node-exporter
        image: prom/node-exporter

You don’t specify replicas.

The controller calculates the number automatically.

DaemonSet Lifecycle

Cluster
3 Nodes
↓
DaemonSet
↓
3 Pods

Add two nodes:

Cluster
5 Nodes
↓
DaemonSet
↓
5 Pods

Remove one node:

Cluster
4 Nodes
↓
DaemonSet
↓
4 Pods

Common Use Cases

Monitoring

Node Exporter
↓
Metrics

Logging

Vector
↓
Collect logs

Networking

OVN
↓
Configure networking

Storage

CSI Node Plugin

Every Node
↓
Attach Volumes

Security

Examples:

• Falco
• Security agents
• Endpoint protection
• File integrity monitoring

DaemonSet vs StatefulSet

DaemonSet vs Deployment

Useful Commands

List DaemonSets:

oc get daemonset -A

Describe:

oc describe daemonset node-exporter -n openshift-monitoring

Check Pods:

oc get pods -o wide -n openshift-monitoring

Watch rollout:

oc rollout status daemonset/node-exporter -n openshift-monitoring

Edit:

oc edit daemonset node-exporter -n openshift-monitoring

Note: Do not edit OpenShift-managed DaemonSets (such as node-exporter, machine-config-daemon, or ovnkube-node) directly. They are managed by Operators, and your changes will typically be overwritten. Use the supported Operator configuration instead.

Troubleshooting DaemonSets

Missing Pod on a Node

Check:

oc get daemonset -A

Then:

oc describe daemonset <name>

Verify:

• Node Selector
• Taints
• Tolerations
• Resource availability
• Events

Pod Pending

Check:

oc describe pod <pod>

Common causes:

• No CPU
• No memory
• Taint not tolerated
• Missing image
• Failed image pull

New Node Has No Pod

Verify:

oc get nodes

Then:

oc get daemonset

Look for:

Desired Number Scheduled
Current Number Scheduled

If they differ:

Desired = 10
Current = 9

Investigate why one node isn’t running the DaemonSet Pod.

Real Production Example

A new worker node joins the cluster.

Within seconds:

New Worker
↓
OVN Pod
↓
Machine Config Daemon
↓
Node Exporter
↓
Vector
↓
CSI Driver

The node becomes fully integrated into the cluster without manual intervention.

Interview Answer (2 Minutes)

A DaemonSet is a Kubernetes workload that ensures one Pod runs on every selected node in the cluster. Unlike a Deployment, where you specify a replica count, a DaemonSet automatically creates a Pod on each matching node and removes it when the node leaves the cluster. In OpenShift, DaemonSets are used for node-level services such as the Machine Config Daemon, OVN-Kubernetes, Node Exporter, Vector log collectors, and CSI storage plugins because these components must run on every node. As the cluster scales, the DaemonSet scales automatically with it. When troubleshooting, I check the DaemonSet status, node selectors, taints and tolerations, events, and whether the number of scheduled Pods matches the number of eligible nodes. One important operational point is that OpenShift-managed DaemonSets are controlled by Operators, so they should be configured through the Operator rather than edited directly.

You need a central log collection and storage solution, but it does not necessarily have to be ELK or Splunk.

OpenShift nodes retain local container logs only temporarily. For centralized searching, retention, alerting, auditing and compliance, install the Red Hat OpenShift Logging Operator and send logs to one or more supported backends.

For a new OpenShift deployment, the main choices are:

1. Vector + LokiStack
2. Vector + Splunk
3. Vector + external Elasticsearch/OpenSearch
4. Vector + multiple destinations

OpenShift Logging collects three primary categories:

• Application logs — logs from application containers
• Infrastructure logs — OpenShift, Kubernetes, CRI-O and node-service logs
• Audit logs — Kubernetes API, OpenShift API, Linux audit and OVN audit logs

The collector runs as a DaemonSet on cluster nodes and forwards selected logs using ClusterLogForwarder. Red Hat’s current logging architecture uses Vector as the collector; Fluentd is legacy/deprecated. (Red Hat Documentation)

Recommended Native OpenShift Solution

For most new OpenShift environments, use:

OpenShift nodes
      │
      ▼
Vector collector
      │
      ▼
LokiStack
      │
      ▼
OpenShift web console

Components

Red Hat OpenShift Logging Operator
        │
        ├── Vector collectors
        │       └── One collector pod per node
        │
        ├── ClusterLogForwarder
        │       └── Routing and filtering rules
        │
        └── LokiStack
                └── Central log storage

Red Hat recommends LokiStack as the supported on-cluster log store for newer deployments. The old OpenShift-managed Elasticsearch and Kibana logging stack is no longer the strategic choice; Kibana is not supported for OpenShift logging beginning with OCP 4.16, and the OpenShift Elasticsearch Operator is no longer supported as the default logging storage solution. (Red Hat Documentation)

When LokiStack is sufficient

Choose LokiStack when:

• You need operational troubleshooting.
• You want logs visible in the OpenShift console.
• You want a Red Hat-supported OpenShift-native solution.
• Your developers primarily search logs by namespace, pod, container and labels.
• You do not need a full enterprise SIEM.
• You want lower infrastructure complexity than Elasticsearch.

A typical implementation:

Application logs ──────┐
Infrastructure logs ───┼──> Vector ──> LokiStack
Audit logs ────────────┘

For sensitive environments, I would normally separate audit-log retention from short-term operational logging.

Splunk Solution

Use Splunk when the organization already uses Splunk as its centralized logging or SIEM platform.

OpenShift
    │
    ▼
Vector
    │ TLS
    ▼
Splunk HEC
    │
    ▼
Splunk indexers
    │
    ▼
Search / alerts / SIEM

OpenShift Logging supports forwarding directly to Splunk HTTP Event Collector, using Vector and a ClusterLogForwarder resource. (Red Hat Documentation)

Splunk is preferable when you need

• Enterprise SIEM integration
• Security correlation across OpenShift, firewalls, IAM and endpoints
• Long audit retention
• SOC dashboards
• Threat detection
• Compliance reporting
• Integration with Splunk Enterprise Security
• Existing Splunk operational teams and licenses

Advantages

• Mature enterprise searching and alerting
• Strong security analytics
• Central correlation across many technologies
• Long-term indexed retention
• Mature RBAC and compliance functions

Disadvantages

• Licensing can be expensive, especially when based on ingested volume.
• Kubernetes logs can generate very high daily volumes.
• Poor filtering can send unnecessary debug logs into costly indexes.
• Splunk becomes an external dependency for operational troubleshooting.

ELK or External Elasticsearch

You can forward OpenShift logs to an externally managed Elasticsearch deployment:

OpenShift nodes
      │
      ▼
Vector
      │
      ▼
External Elasticsearch
      │
      ▼
Kibana

Supported OpenShift logging versions have provided Elasticsearch output from ClusterLogForwarder; however, this should be an external Elasticsearch service that your organization operates, rather than relying on the old OpenShift Elasticsearch Operator for new log-storage deployments. (Red Hat Documentation)

ELK is appropriate when

• Your organization already operates Elasticsearch.
• Application teams require detailed full-text searching.
• You need Kibana dashboards.
• You want more control over indices and data models.
• You can operate Elasticsearch clusters reliably.
• Splunk licensing is not justified.

Additional components you must manage

• Elasticsearch sizing
• Data nodes and master nodes
• Storage capacity
• Shard counts
• Index lifecycle management
• Kibana
• TLS and authentication
• Backup and restore
• Version upgrades
• Cluster health
• Index mappings
• Disk watermarks

ELK can cost less in licensing than Splunk, but it carries more operational responsibility.

Loki vs ELK vs Splunk

Recommended Enterprise Architecture

For a bank or regulated organization, I would use a dual-destination design:

                     OpenShift nodes
                           │
                           ▼
                        Vector
                    ┌──────┴───────┐
                    │              │
                    ▼              ▼
                LokiStack       Splunk HEC
                    │              │
             Operational logs   Audit/security logs
                    │              │
              7–30 days         1–7 years

Example routing:

This gives the platform team fast OpenShift-native log access while the security team receives relevant logs in the enterprise SIEM.

OpenShift supports pipelines that select application, infrastructure and audit inputs and route them to chosen external outputs. (Red Hat Documentation)

Example Splunk Forwarding Design

The exact API fields can vary by OpenShift Logging Operator release, so validate them against your installed Operator version.

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: splunk-forwarder
  namespace: openshift-logging
spec:
  serviceAccount:
    name: log-collector
  outputs:
    - name: splunk
      type: splunk
      splunk:
        url: https://splunk-hec.example.com:8088
        authentication:
          token:
            key: hecToken
            secretName: splunk-hec-secret
  pipelines:
    - name: audit-to-splunk
      inputRefs:
        - audit
      outputRefs:
        - splunk
    - name: infrastructure-to-splunk
      inputRefs:
        - infrastructure
      outputRefs:
        - splunk

Store the HEC token in a Secret, use TLS certificate validation and do not place the token directly in the YAML.

Sizing Considerations

Before selecting the product, estimate:

Daily volume =
nodes
× containers per node
× average log rate
× 86,400 seconds

For example:

50 nodes
× 40 containers
× 1 KB/second
≈ 173 GB per day before filtering

Actual volume varies significantly, but this demonstrates why filtering is critical.

Consider:

• Number of nodes
• Number of pods
• Average events per second
• Average event size
• Retention period
• Replication factor
• Compression
• Audit volume
• Debug logging
• Search concurrency
• Availability requirements

Do not send every log to long-term expensive storage by default.

Practical Recommendation

Small or medium OpenShift platform

Vector + LokiStack

Use a 7–30-day retention period and forward only audit/security logs elsewhere when required.

Existing enterprise ELK platform

Vector + external Elasticsearch

Do not deploy the legacy OpenShift Elasticsearch Operator for a new solution.

Existing Splunk/SOC environment

Vector + LokiStack + Splunk

Use LokiStack for platform operations and Splunk for audit, security and compliance.

Regulated production environment

My preferred design is:

Application logs      → LokiStack
Infrastructure logs   → LokiStack + Splunk
Audit logs            → Splunk
Critical app logs     → LokiStack + Splunk

This balances operational usability, compliance and ingestion cost.

Interview Answer

OpenShift does not require Splunk or ELK specifically, but production clusters normally require centralized log storage because node-local container logs are temporary. I would install the Red Hat OpenShift Logging Operator and use Vector collectors deployed as a DaemonSet. For a new native OpenShift logging implementation, I would use LokiStack rather than the legacy Elasticsearch and Kibana stack.

In an enterprise already using Splunk, I would use ClusterLogForwarder to send audit and security-relevant infrastructure logs to Splunk HEC, while keeping application and operational logs in LokiStack. This avoids sending all high-volume container logs to Splunk and reduces licensing costs. External Elasticsearch remains an option when the organization already operates ELK, but its capacity, lifecycle, security and upgrades must be managed separately.

It is generally not recommended to install the standalone Prometheus Node Exporter on OpenShift (RHCOS) nodes.

OpenShift already includes node-level monitoring as part of the platform, and adding another Node Exporter can create duplicate metrics, unnecessary resource usage, and management complexity.

What OpenShift Already Provides

OpenShift installs a complete monitoring stack:

                   OpenShift Monitoring

           Prometheus (Platform)
                    │
    ┌───────────────┼─────────────────┐
    │               │                 │
    ▼               ▼                 ▼
 kube-state    kubelet/cAdvisor   node-exporter
 metrics          metrics          (DaemonSet)

The platform monitoring stack includes:

• Prometheus
• Alertmanager
• Grafana (developer preview or external)
• Prometheus Operator
• kube-state-metrics
• kubelet metrics
• Node Exporter (managed by OpenShift)
• Telemetry components

So, Node Exporter is already deployed as a DaemonSet in the openshift-monitoring namespace.

You can verify it:

oc get daemonset -n openshift-monitoring

Example:

NAME
node-exporter

Or:

oc get pods -n openshift-monitoring | grep node-exporter

What Metrics Does It Collect?

The built-in Node Exporter collects host metrics such as:

• CPU utilization
• Memory usage
• Disk I/O
• Filesystem usage
• Network traffic
• Load average
• Context switches
• Processes
• Kernel statistics
• Filesystem inodes
• Disk latency
• NUMA information

Examples:

node_cpu_seconds_total

node_memory_MemAvailable_bytes

node_filesystem_avail_bytes

node_disk_io_time_seconds_total

node_network_receive_bytes_total

Why You Shouldn’t Install Another Node Exporter

If you deploy another Node Exporter yourself:

OpenShift
Node Exporter (built-in)
+
Custom Node Exporter

Problems include:

• Duplicate metrics (node_cpu_seconds_total, etc.)
• Metric name collisions
• Higher Prometheus cardinality
• Extra CPU and memory usage
• Additional ports to manage (typically 9100)
• Unsupported configuration drift
• More maintenance during upgrades

When Would You Install Your Own?

There are valid exceptions.

1. External Prometheus

Suppose you have:

Corporate Monitoring
Prometheus
↓
OpenShift Cluster

If the external Prometheus cannot scrape the OpenShift-managed Node Exporter due to network or security constraints, you might deploy a separate exporter specifically for that monitoring system.

2. Air-Gapped Monitoring

Some organizations maintain a completely separate monitoring platform that does not rely on OpenShift’s built-in monitoring.

3. Non-OpenShift Servers

For example:

Linux VM
Windows Server
Oracle DB Server
Load Balancer
Storage Appliance

Those systems can run standalone Node Exporter (or equivalent exporters) because they are not OpenShift nodes.

How OpenShift Collects Node Metrics

RHCOS Node
│
├── kubelet
├── CRI-O
├── node-exporter
└── cAdvisor
↓
Prometheus
↓
Alertmanager
↓
Grafana

Node Exporter complements kubelet and cAdvisor:

Can You Customize Node Exporter?

Not by editing the DaemonSet directly.

In OpenShift, monitoring components are managed by the Cluster Monitoring Operator (CMO). Direct modifications are overwritten.

Supported customization is done through the cluster monitoring configuration, for example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config

Enterprise Best Practice

For production OpenShift clusters:

Use OpenShift Monitoring
↓
Use built-in Node Exporter
↓
Do NOT deploy another Node Exporter

For the rest of your infrastructure:

Linux Servers
↓
Node Exporter
↓
Prometheus

A common enterprise architecture looks like this:

                    Enterprise Monitoring

                   Grafana
                      │
               Thanos / Prometheus
                      │
      ┌───────────────┴────────────────┐
      │                                │
      ▼                                ▼
OpenShift Monitoring           Linux VMs
(Node Exporter built-in)       (Node Exporter installed)

Interview Answer

I would not install a standalone Node Exporter on OpenShift worker or control-plane nodes because OpenShift already deploys and manages Node Exporter as part of the Cluster Monitoring Operator. The built-in exporter collects host metrics such as CPU, memory, filesystem, disk I/O, and network statistics, which are scraped by the platform Prometheus. Installing a second Node Exporter would create duplicate metrics, increase cardinality, and complicate supportability. If I need to monitor external Linux servers, I install Node Exporter there. For OpenShift itself, I rely on the Red Hat-managed monitoring stack and customize it only through supported configuration mechanisms.

This is the approach recommended for production OpenShift environments and is what most enterprise customers follow.

If you’re interviewing for an OpenShift Architect/SRE role, knowing the oc commands for OVN-Kubernetes is extremely valuable. Below are the commands I would expect a senior OpenShift engineer to know.

1. Verify the Network Operator

oc get clusteroperator network

Healthy output:

NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED
network   4.18.10   True        False         False

Detailed status:

oc describe clusteroperator network

2. Verify Network Operator Pods

oc get pods -n openshift-network-operator

Example:

network-operator-xxxxx Running

3. Check OVN Pods

oc get pods -n openshift-ovn-kubernetes

Typical output:

ovnkube-master
ovnkube-node
ovnkube-control-plane
ovnkube-db

4. Show OVN Pods on Each Node

oc get pods -n openshift-ovn-kubernetes -o wide

This confirms every worker has an ovnkube-node pod.

5. Check OVN DaemonSet

oc get daemonset -n openshift-ovn-kubernetes

Example:

ovnkube-node

6. Check OVN Deployment

oc get deployment -n openshift-ovn-kubernetes

7. View OVN Logs

Node agent:

oc logs -n openshift-ovn-kubernetes <ovnkube-node-pod>

Master:

oc logs -n openshift-ovn-kubernetes <ovnkube-master-pod>

Previous container logs:

oc logs --previous

8. Describe an OVN Pod

oc describe pod <ovnkube-node-pod> \
-n openshift-ovn-kubernetes

Useful for:

• Restarts
• Readiness
• Events
• Image versions

9. Check Node Network Status

oc get node

Detailed:

oc describe node worker-1

Look for:

NetworkUnavailable=False
Ready=True

10. Verify Pod IP Addresses

oc get pods -A -o wide

Example:

NAMESPACE
POD
IP
NODE

Confirms OVN allocated IPs correctly.

11. Check Cluster Network

oc get network.config cluster -o yaml

Example:

clusterNetwork:
- cidr: 10.128.0.0/14

12. View Network Operator Configuration

oc get networks.operator.openshift.io cluster -o yaml

Shows:

• MTU
• Geneve
• Service CIDR
• Cluster CIDR

13. Check Node MTU

Debug into a node:

oc debug node/<node-name>

Then:

chroot /host

Check:

ip link

or

ip addr

14. Check Geneve Interface

ip link | grep genev

Usually:

genev_sys_6081

15. Check Routing Table

ip route

16. Check OVS Bridges

ovs-vsctl show

Shows:

br-int
br-ex

17. Show OVS Interfaces

ovs-vsctl list interface

18. Show Open vSwitch Ports

ovs-vsctl show

or

ovs-ofctl show br-int

19. Verify Geneve Tunnel

ovs-vsctl show

Look for:

type=geneve

20. Verify Encapsulation

ovn-sbctl list encap

Should display:

geneve

21. Check OVN Northbound DB

ovn-nbctl show

Displays:

• Logical switches
• Routers
• ACLs

22. Check Southbound Database

ovn-sbctl show

Displays:

• Chassis
• Encapsulation
• Tunnel information

23. Verify Chassis Registration

ovn-sbctl list chassis

Every worker node should appear.

24. Check Pod Connectivity

oc exec -it <pod> -- ping <other-pod-ip>

25. DNS Test

oc exec -it <pod> -- nslookup kubernetes.default

26. Test Service

oc exec <pod> -- curl http://service-name

27. Check Network Policies

oc get networkpolicy -A

Describe one:

oc describe networkpolicy <policy>

28. Check Egress IP

oc get egressip -A

29. Check Egress Firewall

oc get egressfirewall -A

30. Observe Events

oc get events -A --sort-by=.metadata.creationTimestamp

31. Debug a Node

oc debug node/<node>

Then:

chroot /host

Useful commands:

journalctl -u ovnkube-node
journalctl -u ovs-vswitchd
journalctl -u ovsdb-server
ip route
ip addr
ovs-vsctl show

32. Collect Network Must-Gather

oc adm must-gather

Network-focused:

oc adm must-gather \
--image=registry.redhat.io/openshift4/network-tools-rhel8

Common Interview Scenario

Question: Pods on different worker nodes cannot communicate. How do you troubleshoot?

A structured approach is:

1. Verify node health:oc get nodes
2. Check OVN components:oc get pods -n openshift-ovn-kubernetes
3. Review logs:oc logs -n openshift-ovn-kubernetes <ovnkube-node-pod>
4. Test pod-to-pod connectivity:oc exec <pod> -- ping <remote-pod-ip>
5. Inspect Geneve tunnels:ovs-vsctl show
6. Confirm all chassis are registered:ovn-sbctl list chassis
7. Verify there are no blocking NetworkPolicy, EgressIP, or EgressFirewall resources.
8. If the issue persists, gather diagnostics with:oc adm must-gather

These commands cover the majority of day-to-day OVN-Kubernetes troubleshooting tasks in OpenShift and are commonly discussed in senior platform engineering and architect interviews.

Below is a practical OpenShift oc command checklist for etcd troubleshooting. Most commands require cluster-admin.

1. Check overall cluster health

oc get clusterversion
oc get clusteroperators

Focus on these operators:

oc get co etcd kube-apiserver authentication

Healthy state:

AVAILABLE   PROGRESSING   DEGRADED
True        False         False

Get detailed etcd Operator conditions:

oc describe co etcd

Or extract only the conditions:

oc get co etcd \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"  "}{.message}{"\n"}{end}'

2. Check control-plane nodes

oc get nodes -l node-role.kubernetes.io/master -o wide

On newer clusters, also try:

oc get nodes -l node-role.kubernetes.io/control-plane -o wide

Check node conditions:

oc describe node <control-plane-node>

Compact view:

oc get nodes \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,NETWORK:.status.conditions[?(@.type=="NetworkUnavailable")].status'

A three-member etcd cluster needs a majority—normally at least two healthy members—to make progress.

3. Check etcd pods

oc get pods -n openshift-etcd -o wide

Useful label-based view:

oc get pods -n openshift-etcd -l app=etcd -o wide

Watch for:

oc get pods -n openshift-etcd -w

Check restarts and container readiness:

oc get pods -n openshift-etcd \
  -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,READY:.status.containerStatuses[*].ready'

Describe a failing pod:

oc describe pod <etcd-pod> -n openshift-etcd

4. Check etcd endpoint health

Select one healthy etcd pod:

ETCD_POD=$(oc get pods -n openshift-etcd \
  -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}')
echo "$ETCD_POD"

Run endpoint health:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint health --cluster -w table

Expected result: every endpoint reports true or “is healthy.”

Red Hat documents etcdctl endpoint health and endpoint status as primary checks for etcd health and consensus latency. (Red Hat Documentation)

5. Check endpoint status, leader and database size

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster -w table

This shows:

• member ID
• etcd version
• database size
• leader status
• Raft term and index
• applied index
• errors

JSON output:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster -w json

Check only the database size:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster \
  -w fields

Compare the RAFT INDEX and RAFT APPLIED INDEX values across members. A member significantly behind the others may have disk, network, or synchronization problems.

6. Check etcd membership

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl member list -w table

Confirm that:

• All expected control-plane members exist.
• Each member is started.
• Peer and client URLs match the correct nodes.
• No stale or duplicate member remains.

Do not run etcdctl member remove during normal diagnosis. Member removal is a recovery operation and should follow the documented unhealthy-member replacement procedure. (Red Hat Documentation)

7. Check etcd alarms

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl alarm list

A common critical alarm is:

NOSPACE

When etcd exhausts its quota, writes can fail. Red Hat recommends treating low-space alerts urgently rather than merely clearing the alarm. (Red Hat Documentation)

Do not blindly run:

etcdctl alarm disarm

First correct the underlying database-size, quota, compaction, or storage problem.

8. Review etcd logs

Current logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --tail=200

Follow logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd -f

Previous crashed container:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --previous

Logs from all etcd pods:

for pod in $(oc get pods -n openshift-etcd \
  -l app=etcd -o name); do
    echo "===== $pod ====="
    oc logs -n openshift-etcd "$pod" -c etcd \
      --since=30m 2>&1 |
      grep -Ei 'error|warn|timeout|leader|slow|unhealthy|corrupt|space|fsync'
done

Important messages include:

leader changed
leader failed
request timed out
context deadline exceeded
apply request took too long
failed to send out heartbeat
database space exceeded
wal
corrupt

9. Check static-pod revisions

etcd runs as static pods on control-plane nodes.

oc get pods -n openshift-etcd \
  -l app=etcd \
  -L revision

Check the revision-pruner and installer pods:

oc get pods -n openshift-etcd | grep -E 'installer|revision-pruner'

Check etcd Operator configuration:

oc get kubeapiserver cluster -o yaml
oc get etcd cluster -o yaml

Check etcd resource conditions:

oc describe etcd cluster

Compact conditions:

oc get etcd cluster \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"  "}{.reason}{"  "}{.message}{"\n"}{end}'

10. Check recent events

oc get events -n openshift-etcd \
  --sort-by=.metadata.creationTimestamp

Cluster-wide warning events:

oc get events -A \
  --field-selector type=Warning \
  --sort-by=.metadata.creationTimestamp

Recent events only:

oc get events -n openshift-etcd \
  --sort-by=.lastTimestamp | tail -30

11. Check etcd-related alerts

oc get prometheusrules -A | grep -i etcd

Query active alerts through the monitoring API:

oc -n openshift-monitoring exec \
  prometheus-k8s-0 -c prometheus -- \
  curl -s 'http://localhost:9090/api/v1/alerts'

Filter with jq:

oc -n openshift-monitoring exec \
  prometheus-k8s-0 -c prometheus -- \
  curl -s 'http://localhost:9090/api/v1/alerts' |
jq -r '
  .data.alerts[]
  | select(.labels.alertname | test("etcd|Etcd"; "i"))
  | [.labels.alertname, .state, .annotations.message]
  | @tsv

Typical alerts include:

etcdMembersDown
etcdInsufficientMembers
etcdHighNumberOfLeaderChanges
etcdNoLeader
etcdHighFsyncDurations
etcdHighCommitDurations
etcdDatabaseQuotaLowSpace

12. Query important etcd metrics

Create access to Prometheus:

oc -n openshift-monitoring port-forward \
  svc/prometheus-k8s 9090:9090

Then query locally using an authenticated method appropriate for your environment, or use the OpenShift web console metrics page.

Useful PromQL queries:

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Members without a leader

etcd_server_has_leader

Expected value:

1

WAL fsync latency p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

Backend commit latency p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Database size

etcd_mvcc_db_total_size_in_bytes

Quota usage ratio

etcd_mvcc_db_total_size_in_bytes
/
etcd_server_quota_backend_bytes

etcd is highly sensitive to storage latency. Slow fsync operations can cause missed heartbeats, proposal delays, API timeouts, and temporary leader loss. (Red Hat Documentation)

13. Inspect the control-plane host

Start a debug shell:

oc debug node/<control-plane-node>

Enter the host filesystem:

chroot /host

Check disk utilization:

df -h
df -i

Check the etcd data directory:

du -sh /var/lib/etcd
du -sh /var/lib/etcd/member/*

Check block devices:

lsblk
findmnt /var/lib/etcd

Check I/O pressure:

iostat -xz 1 10

Check CPU, memory and load:

uptime
free -h
vmstat 1 10

Check kernel storage errors:

journalctl -k --since "1 hour ago" |
grep -Ei 'error|timeout|reset|nvme|scsi|blk|I/O'

Check kubelet and CRI-O:

journalctl -u kubelet --since "1 hour ago"
journalctl -u crio --since "1 hour ago"

Exit:

exit
exit

14. Check network connectivity between etcd members

etcd normally uses:

• TCP 2379: client traffic
• TCP 2380: peer communication

From a control-plane debug session:

nc -vz <other-control-plane-ip> 2379
nc -vz <other-control-plane-ip> 2380

Check listening sockets:

ss -lntp | grep -E ':2379|:2380'

Look for packet loss or latency between control-plane nodes:

ping -c 10 <other-control-plane-ip>

In production, also verify firewalls, security groups, load balancers, MTU, and the underlying network path.

15. Check API latency symptoms

Because the Kubernetes API stores its persistent state in etcd, etcd latency often appears first as API slowness.

time oc get nodes
time oc get pods -A >/dev/null
time oc get co

Check API server pods:

oc get pods -n openshift-kube-apiserver -o wide

Check API server logs for etcd timeouts:

oc logs -n openshift-kube-apiserver \
  <kube-apiserver-pod> -c kube-apiserver \
  --since=30m |
grep -Ei 'etcd|timeout|deadline|slow'

16. Verify backups

Locate a control-plane node:

oc get nodes -l node-role.kubernetes.io/master

Open a debug shell:

oc debug node/<control-plane-node>
chroot /host

Create a supported backup:

/usr/local/bin/cluster-backup.sh /home/core/assets/backup

Verify files:

ls -lh /home/core/assets/backup

Typical output includes:

snapshot_*.db
static_kuberesources_*.tar.gz

Take one cluster backup from one healthy control-plane node—not a separate backup from every member. Backup and restore procedures must follow the documentation for the exact OpenShift release. (Red Hat Documentation)

Fast troubleshooting workflow

API slow or unavailable
        |
        v
oc get co etcd
        |
        v
Check control-plane nodes and etcd pods
        |
        v
etcdctl endpoint health --cluster
        |
        v
endpoint status + member list + alarm list
        |
        v
Check leader changes and fsync latency
        |
        v
Inspect disk, I/O and network on each control-plane node
        |
        v
Review etcd and kube-apiserver logs
        |
        v
Take backup and must-gather before invasive recovery

Compact command bundle

oc get co etcd
oc get etcd cluster
oc get nodes -l node-role.kubernetes.io/master -o wide
oc get pods -n openshift-etcd -l app=etcd -o wide
ETCD_POD=$(oc get pods -n openshift-etcd \
  -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint health --cluster -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl member list -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl alarm list
oc logs -n openshift-etcd "$ETCD_POD" \
  -c etcd --since=30m
oc get events -n openshift-etcd \
  --sort-by=.metadata.creationTimestamp

Before member removal, manual defragmentation, quota changes, or snapshot restoration, collect diagnostics:

oc adm must-gather

Those operations can affect quorum or temporarily block a member and should not be treated as routine diagnostic commands.

Below is a practical OpenShift oc command reference for OADP/Velero backup and restore troubleshooting.

OADP is the Red Hat-supported Operator that deploys and manages Velero components for backing up application Kubernetes objects, persistent volumes, internal images, and supported OpenShift Virtualization workloads. (Red Hat Customer Portal)

1. Check the OADP Operator installation

Find the Operator:

oc get csv -A | grep -i oadp

Check subscriptions:

oc get subscription -A | grep -i oadp

Typical OADP namespace:

openshift-adp

Check Operator objects:

oc get csv,subscription,installplan -n openshift-adp

Detailed ClusterServiceVersion status:

oc describe csv -n openshift-adp \
  $(oc get csv -n openshift-adp -o name | grep oadp | head -1)

Check the Operator deployment:

oc get deployment -n openshift-adp

Check Operator logs:

oc logs -n openshift-adp \
  deployment/openshift-adp-controller-manager \
  -c manager --tail=200

Follow logs:

oc logs -n openshift-adp \
  deployment/openshift-adp-controller-manager \
  -c manager -f

2. Check the DataProtectionApplication

The DataProtectionApplication, commonly abbreviated as DPA, is the main OADP configuration resource.

oc get dataprotectionapplication -n openshift-adp

Short form:

oc get dpa -n openshift-adp

View configuration:

oc get dpa -n openshift-adp -o yaml

Describe the DPA:

oc describe dpa <dpa-name> -n openshift-adp

Extract its conditions:

oc get dpa <dpa-name> -n openshift-adp \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"  "}{.reason}{"  "}{.message}{"\n"}{end}'

Look for:

Reconciled=True

Also inspect:

• Backup storage configuration
• Cloud provider plugin
• Credential Secret
• CSI configuration
• Data mover configuration
• Node agent settings

3. Check Velero and node-agent pods

oc get pods -n openshift-adp -o wide

Common components include:

velero
node-agent
openshift-adp-controller-manager

Depending on the OADP version and configuration, the filesystem backup component can appear as node-agent; older environments might refer to Restic.

Check Velero deployment:

oc get deployment velero -n openshift-adp

Check node-agent DaemonSet:

oc get daemonset -n openshift-adp

Verify that a node-agent pod is running on each applicable node:

oc get pods -n openshift-adp \
  -l name=node-agent -o wide

Inspect pod restarts:

oc get pods -n openshift-adp \
  -o custom-columns='POD:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[*].restartCount'

Describe a failing pod:

oc describe pod <pod-name> -n openshift-adp

4. Check Velero logs

Current Velero logs:

oc logs -n openshift-adp deployment/velero \
  --tail=300

Follow logs:

oc logs -n openshift-adp deployment/velero -f

Previous crashed container: