Choosing the Right OpenShift Logging Solution

You need a central log collection and storage solution, but it does not necessarily have to be ELK or Splunk.

OpenShift nodes retain local container logs only temporarily. For centralized searching, retention, alerting, auditing and compliance, install the Red Hat OpenShift Logging Operator and send logs to one or more supported backends.

For a new OpenShift deployment, the main choices are:

1. Vector + LokiStack
2. Vector + Splunk
3. Vector + external Elasticsearch/OpenSearch
4. Vector + multiple destinations

OpenShift Logging collects three primary categories:

  • Application logs — logs from application containers
  • Infrastructure logs — OpenShift, Kubernetes, CRI-O and node-service logs
  • Audit logs — Kubernetes API, OpenShift API, Linux audit and OVN audit logs

The collector runs as a DaemonSet on cluster nodes and forwards selected logs using ClusterLogForwarder. Red Hat’s current logging architecture uses Vector as the collector; Fluentd is legacy/deprecated. (Red Hat Documentation)

Recommended Native OpenShift Solution

For most new OpenShift environments, use:

OpenShift nodes
Vector collector
LokiStack
OpenShift web console

Components

Red Hat OpenShift Logging Operator
├── Vector collectors
│ └── One collector pod per node
├── ClusterLogForwarder
│ └── Routing and filtering rules
└── LokiStack
└── Central log storage

Red Hat recommends LokiStack as the supported on-cluster log store for newer deployments. The old OpenShift-managed Elasticsearch and Kibana logging stack is no longer the strategic choice; Kibana is not supported for OpenShift logging beginning with OCP 4.16, and the OpenShift Elasticsearch Operator is no longer supported as the default logging storage solution. (Red Hat Documentation)

When LokiStack is sufficient

Choose LokiStack when:

  • You need operational troubleshooting.
  • You want logs visible in the OpenShift console.
  • You want a Red Hat-supported OpenShift-native solution.
  • Your developers primarily search logs by namespace, pod, container and labels.
  • You do not need a full enterprise SIEM.
  • You want lower infrastructure complexity than Elasticsearch.

A typical implementation:

Application logs ──────┐
Infrastructure logs ───┼──> Vector ──> LokiStack
Audit logs ────────────┘

For sensitive environments, I would normally separate audit-log retention from short-term operational logging.


Splunk Solution

Use Splunk when the organization already uses Splunk as its centralized logging or SIEM platform.

OpenShift
Vector
│ TLS
Splunk HEC
Splunk indexers
Search / alerts / SIEM

OpenShift Logging supports forwarding directly to Splunk HTTP Event Collector, using Vector and a ClusterLogForwarder resource. (Red Hat Documentation)

Splunk is preferable when you need
  • Enterprise SIEM integration
  • Security correlation across OpenShift, firewalls, IAM and endpoints
  • Long audit retention
  • SOC dashboards
  • Threat detection
  • Compliance reporting
  • Integration with Splunk Enterprise Security
  • Existing Splunk operational teams and licenses
Advantages
  • Mature enterprise searching and alerting
  • Strong security analytics
  • Central correlation across many technologies
  • Long-term indexed retention
  • Mature RBAC and compliance functions
Disadvantages
  • Licensing can be expensive, especially when based on ingested volume.
  • Kubernetes logs can generate very high daily volumes.
  • Poor filtering can send unnecessary debug logs into costly indexes.
  • Splunk becomes an external dependency for operational troubleshooting.

ELK or External Elasticsearch

You can forward OpenShift logs to an externally managed Elasticsearch deployment:

OpenShift nodes
Vector
External Elasticsearch
Kibana

Supported OpenShift logging versions have provided Elasticsearch output from ClusterLogForwarder; however, this should be an external Elasticsearch service that your organization operates, rather than relying on the old OpenShift Elasticsearch Operator for new log-storage deployments. (Red Hat Documentation)

ELK is appropriate when
  • Your organization already operates Elasticsearch.
  • Application teams require detailed full-text searching.
  • You need Kibana dashboards.
  • You want more control over indices and data models.
  • You can operate Elasticsearch clusters reliably.
  • Splunk licensing is not justified.
Additional components you must manage
  • Elasticsearch sizing
  • Data nodes and master nodes
  • Storage capacity
  • Shard counts
  • Index lifecycle management
  • Kibana
  • TLS and authentication
  • Backup and restore
  • Version upgrades
  • Cluster health
  • Index mappings
  • Disk watermarks

ELK can cost less in licensing than Splunk, but it carries more operational responsibility.


Loki vs ELK vs Splunk

AreaLokiStackExternal ELKSplunk
OpenShift-nativeExcellentExternal integrationExternal integration
Red Hat-supported on-cluster storeYesNo for new managed ES deploymentsForwarding supported
Operational troubleshootingExcellentExcellentExcellent
Full-text analyticsMore limitedStrongStrong
SIEM capabilityLimitedRequires additional security toolsExcellent
Infrastructure complexityLow to mediumHighMedium externally
Licensing costGenerally lowerProduct-dependentUsually highest
Kubernetes label searchesExcellentGoodGood
Long-term audit retentionPossibleGoodExcellent
SOC integrationLimitedModerateExcellent
Best usePlatform operationsCustom analyticsEnterprise security

Recommended Enterprise Architecture

For a bank or regulated organization, I would use a dual-destination design:

                     OpenShift nodes
                           │
                           ▼
                        Vector
                    ┌──────┴───────┐
                    │              │
                    ▼              ▼
                LokiStack       Splunk HEC
                    │              │
             Operational logs   Audit/security logs
                    │              │
              7–30 days         1–7 years

Example routing:

Log typeDestinationPurpose
ApplicationLokiStackDeveloper and operational troubleshooting
InfrastructureLokiStack and SplunkOperations and security correlation
AuditSplunkCompliance and security monitoring
Selected critical applicationsLokiStack and SplunkOperational and business-security analysis
Debug logsLokiStack onlyAvoid expensive Splunk ingestion

This gives the platform team fast OpenShift-native log access while the security team receives relevant logs in the enterprise SIEM.

OpenShift supports pipelines that select application, infrastructure and audit inputs and route them to chosen external outputs. (Red Hat Documentation)


Example Splunk Forwarding Design

The exact API fields can vary by OpenShift Logging Operator release, so validate them against your installed Operator version.

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: splunk-forwarder
namespace: openshift-logging
spec:
serviceAccount:
name: log-collector
outputs:
- name: splunk
type: splunk
splunk:
url: https://splunk-hec.example.com:8088
authentication:
token:
key: hecToken
secretName: splunk-hec-secret
pipelines:
- name: audit-to-splunk
inputRefs:
- audit
outputRefs:
- splunk
- name: infrastructure-to-splunk
inputRefs:
- infrastructure
outputRefs:
- splunk

Store the HEC token in a Secret, use TLS certificate validation and do not place the token directly in the YAML.


Sizing Considerations

Before selecting the product, estimate:

Daily volume =
nodes
× containers per node
× average log rate
× 86,400 seconds

For example:

50 nodes
× 40 containers
× 1 KB/second
≈ 173 GB per day before filtering

Actual volume varies significantly, but this demonstrates why filtering is critical.

Consider:

  • Number of nodes
  • Number of pods
  • Average events per second
  • Average event size
  • Retention period
  • Replication factor
  • Compression
  • Audit volume
  • Debug logging
  • Search concurrency
  • Availability requirements

Do not send every log to long-term expensive storage by default.


Practical Recommendation

Small or medium OpenShift platform
Vector + LokiStack

Use a 7–30-day retention period and forward only audit/security logs elsewhere when required.

Existing enterprise ELK platform
Vector + external Elasticsearch

Do not deploy the legacy OpenShift Elasticsearch Operator for a new solution.

Existing Splunk/SOC environment
Vector + LokiStack + Splunk

Use LokiStack for platform operations and Splunk for audit, security and compliance.

Regulated production environment

My preferred design is:

Application logs → LokiStack
Infrastructure logs → LokiStack + Splunk
Audit logs → Splunk
Critical app logs → LokiStack + Splunk

This balances operational usability, compliance and ingestion cost.

Interview Answer

OpenShift does not require Splunk or ELK specifically, but production clusters normally require centralized log storage because node-local container logs are temporary. I would install the Red Hat OpenShift Logging Operator and use Vector collectors deployed as a DaemonSet. For a new native OpenShift logging implementation, I would use LokiStack rather than the legacy Elasticsearch and Kibana stack.

In an enterprise already using Splunk, I would use ClusterLogForwarder to send audit and security-relevant infrastructure logs to Splunk HEC, while keeping application and operational logs in LokiStack. This avoids sending all high-volume container logs to Splunk and reduces licensing costs. External Elasticsearch remains an option when the organization already operates ELK, but its capacity, lifecycle, security and upgrades must be managed separately.

Understanding OpenShift’s Built-in Monitoring

It is generally not recommended to install the standalone Prometheus Node Exporter on OpenShift (RHCOS) nodes.

OpenShift already includes node-level monitoring as part of the platform, and adding another Node Exporter can create duplicate metrics, unnecessary resource usage, and management complexity.


What OpenShift Already Provides

OpenShift installs a complete monitoring stack:

                   OpenShift Monitoring

           Prometheus (Platform)
                    │
    ┌───────────────┼─────────────────┐
    │               │                 │
    ▼               ▼                 ▼
 kube-state    kubelet/cAdvisor   node-exporter
 metrics          metrics          (DaemonSet)

The platform monitoring stack includes:

  • Prometheus
  • Alertmanager
  • Grafana (developer preview or external)
  • Prometheus Operator
  • kube-state-metrics
  • kubelet metrics
  • Node Exporter (managed by OpenShift)
  • Telemetry components

So, Node Exporter is already deployed as a DaemonSet in the openshift-monitoring namespace.

You can verify it:

oc get daemonset -n openshift-monitoring

Example:

NAME
node-exporter

Or:

oc get pods -n openshift-monitoring | grep node-exporter

What Metrics Does It Collect?

The built-in Node Exporter collects host metrics such as:

  • CPU utilization
  • Memory usage
  • Disk I/O
  • Filesystem usage
  • Network traffic
  • Load average
  • Context switches
  • Processes
  • Kernel statistics
  • Filesystem inodes
  • Disk latency
  • NUMA information

Examples:

node_cpu_seconds_total
node_memory_MemAvailable_bytes
node_filesystem_avail_bytes
node_disk_io_time_seconds_total
node_network_receive_bytes_total

Why You Shouldn’t Install Another Node Exporter

If you deploy another Node Exporter yourself:

OpenShift
Node Exporter (built-in)
+
Custom Node Exporter

Problems include:

  • Duplicate metrics (node_cpu_seconds_total, etc.)
  • Metric name collisions
  • Higher Prometheus cardinality
  • Extra CPU and memory usage
  • Additional ports to manage (typically 9100)
  • Unsupported configuration drift
  • More maintenance during upgrades

When Would You Install Your Own?

There are valid exceptions.

1. External Prometheus

Suppose you have:

Corporate Monitoring
Prometheus
OpenShift Cluster

If the external Prometheus cannot scrape the OpenShift-managed Node Exporter due to network or security constraints, you might deploy a separate exporter specifically for that monitoring system.


2. Air-Gapped Monitoring

Some organizations maintain a completely separate monitoring platform that does not rely on OpenShift’s built-in monitoring.


3. Non-OpenShift Servers

For example:

Linux VM
Windows Server
Oracle DB Server
Load Balancer
Storage Appliance

Those systems can run standalone Node Exporter (or equivalent exporters) because they are not OpenShift nodes.


How OpenShift Collects Node Metrics

RHCOS Node
├── kubelet
├── CRI-O
├── node-exporter
└── cAdvisor
Prometheus
Alertmanager
Grafana

Node Exporter complements kubelet and cAdvisor:

ComponentMetrics
Node ExporterHost OS (CPU, memory, disks, network)
kubeletPod lifecycle and node health
cAdvisorContainer CPU, memory, filesystem and network
kube-state-metricsKubernetes object state

Can You Customize Node Exporter?

Not by editing the DaemonSet directly.

In OpenShift, monitoring components are managed by the Cluster Monitoring Operator (CMO). Direct modifications are overwritten.

Supported customization is done through the cluster monitoring configuration, for example:

apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config

Enterprise Best Practice

For production OpenShift clusters:

Use OpenShift Monitoring
Use built-in Node Exporter
Do NOT deploy another Node Exporter

For the rest of your infrastructure:

Linux Servers
Node Exporter
Prometheus

A common enterprise architecture looks like this:

                    Enterprise Monitoring

                   Grafana
                      │
               Thanos / Prometheus
                      │
      ┌───────────────┴────────────────┐
      │                                │
      ▼                                ▼
OpenShift Monitoring           Linux VMs
(Node Exporter built-in)       (Node Exporter installed)

Interview Answer

I would not install a standalone Node Exporter on OpenShift worker or control-plane nodes because OpenShift already deploys and manages Node Exporter as part of the Cluster Monitoring Operator. The built-in exporter collects host metrics such as CPU, memory, filesystem, disk I/O, and network statistics, which are scraped by the platform Prometheus. Installing a second Node Exporter would create duplicate metrics, increase cardinality, and complicate supportability. If I need to monitor external Linux servers, I install Node Exporter there. For OpenShift itself, I rely on the Red Hat-managed monitoring stack and customize it only through supported configuration mechanisms.

This is the approach recommended for production OpenShift environments and is what most enterprise customers follow.

Essential OC Commands for OpenShift OVN-Kubernetes Troubleshooting

If you’re interviewing for an OpenShift Architect/SRE role, knowing the oc commands for OVN-Kubernetes is extremely valuable. Below are the commands I would expect a senior OpenShift engineer to know.


1. Verify the Network Operator

oc get clusteroperator network

Healthy output:

NAME VERSION AVAILABLE PROGRESSING DEGRADED
network 4.18.10 True False False

Detailed status:

oc describe clusteroperator network

2. Verify Network Operator Pods

oc get pods -n openshift-network-operator

Example:

network-operator-xxxxx Running

3. Check OVN Pods

oc get pods -n openshift-ovn-kubernetes

Typical output:

ovnkube-master
ovnkube-node
ovnkube-control-plane
ovnkube-db

4. Show OVN Pods on Each Node

oc get pods -n openshift-ovn-kubernetes -o wide

This confirms every worker has an ovnkube-node pod.


5. Check OVN DaemonSet

oc get daemonset -n openshift-ovn-kubernetes

Example:

ovnkube-node

6. Check OVN Deployment

oc get deployment -n openshift-ovn-kubernetes

7. View OVN Logs

Node agent:

oc logs -n openshift-ovn-kubernetes <ovnkube-node-pod>

Master:

oc logs -n openshift-ovn-kubernetes <ovnkube-master-pod>

Previous container logs:

oc logs --previous

8. Describe an OVN Pod

oc describe pod <ovnkube-node-pod> \
-n openshift-ovn-kubernetes

Useful for:

  • Restarts
  • Readiness
  • Events
  • Image versions

9. Check Node Network Status

oc get node

Detailed:

oc describe node worker-1

Look for:

NetworkUnavailable=False
Ready=True

10. Verify Pod IP Addresses

oc get pods -A -o wide

Example:

NAMESPACE
POD
IP
NODE

Confirms OVN allocated IPs correctly.


11. Check Cluster Network

oc get network.config cluster -o yaml

Example:

clusterNetwork:
- cidr: 10.128.0.0/14

12. View Network Operator Configuration

oc get networks.operator.openshift.io cluster -o yaml

Shows:

  • MTU
  • Geneve
  • Service CIDR
  • Cluster CIDR

13. Check Node MTU

Debug into a node:

oc debug node/<node-name>

Then:

chroot /host

Check:

ip link

or

ip addr

14. Check Geneve Interface

ip link | grep genev

Usually:

genev_sys_6081

15. Check Routing Table

ip route

16. Check OVS Bridges

ovs-vsctl show

Shows:

br-int
br-ex

17. Show OVS Interfaces

ovs-vsctl list interface

18. Show Open vSwitch Ports

ovs-vsctl show

or

ovs-ofctl show br-int

19. Verify Geneve Tunnel

ovs-vsctl show

Look for:

type=geneve

20. Verify Encapsulation

ovn-sbctl list encap

Should display:

geneve

21. Check OVN Northbound DB

ovn-nbctl show

Displays:

  • Logical switches
  • Routers
  • ACLs

22. Check Southbound Database

ovn-sbctl show

Displays:

  • Chassis
  • Encapsulation
  • Tunnel information

23. Verify Chassis Registration

ovn-sbctl list chassis

Every worker node should appear.


24. Check Pod Connectivity

oc exec -it <pod> -- ping <other-pod-ip>

25. DNS Test

oc exec -it <pod> -- nslookup kubernetes.default

26. Test Service

oc exec <pod> -- curl http://service-name

27. Check Network Policies

oc get networkpolicy -A

Describe one:

oc describe networkpolicy <policy>

28. Check Egress IP

oc get egressip -A

29. Check Egress Firewall

oc get egressfirewall -A

30. Observe Events

oc get events -A --sort-by=.metadata.creationTimestamp

31. Debug a Node

oc debug node/<node>

Then:

chroot /host

Useful commands:

journalctl -u ovnkube-node
journalctl -u ovs-vswitchd
journalctl -u ovsdb-server
ip route
ip addr
ovs-vsctl show

32. Collect Network Must-Gather

oc adm must-gather

Network-focused:

oc adm must-gather \
--image=registry.redhat.io/openshift4/network-tools-rhel8

Common Interview Scenario

Question: Pods on different worker nodes cannot communicate. How do you troubleshoot?

A structured approach is:

  1. Verify node health:oc get nodes
  2. Check OVN components:oc get pods -n openshift-ovn-kubernetes
  3. Review logs:oc logs -n openshift-ovn-kubernetes <ovnkube-node-pod>
  4. Test pod-to-pod connectivity:oc exec <pod> -- ping <remote-pod-ip>
  5. Inspect Geneve tunnels:ovs-vsctl show
  6. Confirm all chassis are registered:ovn-sbctl list chassis
  7. Verify there are no blocking NetworkPolicy, EgressIP, or EgressFirewall resources.
  8. If the issue persists, gather diagnostics with:oc adm must-gather

These commands cover the majority of day-to-day OVN-Kubernetes troubleshooting tasks in OpenShift and are commonly discussed in senior platform engineering and architect interviews.

Essential OpenShift Commands for etcd Troubleshooting

Below is a practical OpenShift oc command checklist for etcd troubleshooting. Most commands require cluster-admin.

1. Check overall cluster health

oc get clusterversion
oc get clusteroperators

Focus on these operators:

oc get co etcd kube-apiserver authentication

Healthy state:

AVAILABLE PROGRESSING DEGRADED
True False False

Get detailed etcd Operator conditions:

oc describe co etcd

Or extract only the conditions:

oc get co etcd \
-o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.message}{"\n"}{end}'

2. Check control-plane nodes

oc get nodes -l node-role.kubernetes.io/master -o wide

On newer clusters, also try:

oc get nodes -l node-role.kubernetes.io/control-plane -o wide

Check node conditions:

oc describe node <control-plane-node>

Compact view:

oc get nodes \
-o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,NETWORK:.status.conditions[?(@.type=="NetworkUnavailable")].status'

A three-member etcd cluster needs a majority—normally at least two healthy members—to make progress.

3. Check etcd pods

oc get pods -n openshift-etcd -o wide

Useful label-based view:

oc get pods -n openshift-etcd -l app=etcd -o wide

Watch for:

oc get pods -n openshift-etcd -w

Check restarts and container readiness:

oc get pods -n openshift-etcd \
-o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,READY:.status.containerStatuses[*].ready'

Describe a failing pod:

oc describe pod <etcd-pod> -n openshift-etcd

4. Check etcd endpoint health

Select one healthy etcd pod:

ETCD_POD=$(oc get pods -n openshift-etcd \
-l app=etcd \
-o jsonpath='{.items[0].metadata.name}')
echo "$ETCD_POD"

Run endpoint health:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl endpoint health --cluster -w table

Expected result: every endpoint reports true or “is healthy.”

Red Hat documents etcdctl endpoint health and endpoint status as primary checks for etcd health and consensus latency. (Red Hat Documentation)

5. Check endpoint status, leader and database size

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl endpoint status --cluster -w table

This shows:

  • member ID
  • etcd version
  • database size
  • leader status
  • Raft term and index
  • applied index
  • errors

JSON output:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl endpoint status --cluster -w json

Check only the database size:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl endpoint status --cluster \
-w fields

Compare the RAFT INDEX and RAFT APPLIED INDEX values across members. A member significantly behind the others may have disk, network, or synchronization problems.

6. Check etcd membership

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl member list -w table

Confirm that:

  • All expected control-plane members exist.
  • Each member is started.
  • Peer and client URLs match the correct nodes.
  • No stale or duplicate member remains.

Do not run etcdctl member remove during normal diagnosis. Member removal is a recovery operation and should follow the documented unhealthy-member replacement procedure. (Red Hat Documentation)

7. Check etcd alarms

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl alarm list

A common critical alarm is:

NOSPACE

When etcd exhausts its quota, writes can fail. Red Hat recommends treating low-space alerts urgently rather than merely clearing the alarm. (Red Hat Documentation)

Do not blindly run:

etcdctl alarm disarm

First correct the underlying database-size, quota, compaction, or storage problem.

8. Review etcd logs

Current logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --tail=200

Follow logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd -f

Previous crashed container:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --previous

Logs from all etcd pods:

for pod in $(oc get pods -n openshift-etcd \
-l app=etcd -o name); do
echo "===== $pod ====="
oc logs -n openshift-etcd "$pod" -c etcd \
--since=30m 2>&1 |
grep -Ei 'error|warn|timeout|leader|slow|unhealthy|corrupt|space|fsync'
done

Important messages include:

leader changed
leader failed
request timed out
context deadline exceeded
apply request took too long
failed to send out heartbeat
database space exceeded
wal
corrupt

9. Check static-pod revisions

etcd runs as static pods on control-plane nodes.

oc get pods -n openshift-etcd \
-l app=etcd \
-L revision

Check the revision-pruner and installer pods:

oc get pods -n openshift-etcd | grep -E 'installer|revision-pruner'

Check etcd Operator configuration:

oc get kubeapiserver cluster -o yaml
oc get etcd cluster -o yaml

Check etcd resource conditions:

oc describe etcd cluster

Compact conditions:

oc get etcd cluster \
-o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}'

10. Check recent events

oc get events -n openshift-etcd \
--sort-by=.metadata.creationTimestamp

Cluster-wide warning events:

oc get events -A \
--field-selector type=Warning \
--sort-by=.metadata.creationTimestamp

Recent events only:

oc get events -n openshift-etcd \
--sort-by=.lastTimestamp | tail -30

11. Check etcd-related alerts

oc get prometheusrules -A | grep -i etcd

Query active alerts through the monitoring API:

oc -n openshift-monitoring exec \
prometheus-k8s-0 -c prometheus -- \
curl -s 'http://localhost:9090/api/v1/alerts'

Filter with jq:

oc -n openshift-monitoring exec \
prometheus-k8s-0 -c prometheus -- \
curl -s 'http://localhost:9090/api/v1/alerts' |
jq -r '
.data.alerts[]
| select(.labels.alertname | test("etcd|Etcd"; "i"))
| [.labels.alertname, .state, .annotations.message]
| @tsv

Typical alerts include:

etcdMembersDown
etcdInsufficientMembers
etcdHighNumberOfLeaderChanges
etcdNoLeader
etcdHighFsyncDurations
etcdHighCommitDurations
etcdDatabaseQuotaLowSpace

12. Query important etcd metrics

Create access to Prometheus:

oc -n openshift-monitoring port-forward \
svc/prometheus-k8s 9090:9090

Then query locally using an authenticated method appropriate for your environment, or use the OpenShift web console metrics page.

Useful PromQL queries:

Leader changes
increase(etcd_server_leader_changes_seen_total[15m])
Members without a leader
etcd_server_has_leader

Expected value:

1
WAL fsync latency p99
histogram_quantile(
0.99,
sum by (instance, le) (
rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
)
)
Backend commit latency p99
histogram_quantile(
0.99,
sum by (instance, le) (
rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
)
)
Database size
etcd_mvcc_db_total_size_in_bytes
Quota usage ratio
etcd_mvcc_db_total_size_in_bytes
/
etcd_server_quota_backend_bytes

etcd is highly sensitive to storage latency. Slow fsync operations can cause missed heartbeats, proposal delays, API timeouts, and temporary leader loss. (Red Hat Documentation)

13. Inspect the control-plane host

Start a debug shell:

oc debug node/<control-plane-node>

Enter the host filesystem:

chroot /host

Check disk utilization:

df -h
df -i

Check the etcd data directory:

du -sh /var/lib/etcd
du -sh /var/lib/etcd/member/*

Check block devices:

lsblk
findmnt /var/lib/etcd

Check I/O pressure:

iostat -xz 1 10

Check CPU, memory and load:

uptime
free -h
vmstat 1 10

Check kernel storage errors:

journalctl -k --since "1 hour ago" |
grep -Ei 'error|timeout|reset|nvme|scsi|blk|I/O'

Check kubelet and CRI-O:

journalctl -u kubelet --since "1 hour ago"
journalctl -u crio --since "1 hour ago"

Exit:

exit
exit

14. Check network connectivity between etcd members

etcd normally uses:

  • TCP 2379: client traffic
  • TCP 2380: peer communication

From a control-plane debug session:

nc -vz <other-control-plane-ip> 2379
nc -vz <other-control-plane-ip> 2380

Check listening sockets:

ss -lntp | grep -E ':2379|:2380'

Look for packet loss or latency between control-plane nodes:

ping -c 10 <other-control-plane-ip>

In production, also verify firewalls, security groups, load balancers, MTU, and the underlying network path.

15. Check API latency symptoms

Because the Kubernetes API stores its persistent state in etcd, etcd latency often appears first as API slowness.

time oc get nodes
time oc get pods -A >/dev/null
time oc get co

Check API server pods:

oc get pods -n openshift-kube-apiserver -o wide

Check API server logs for etcd timeouts:

oc logs -n openshift-kube-apiserver \
<kube-apiserver-pod> -c kube-apiserver \
--since=30m |
grep -Ei 'etcd|timeout|deadline|slow'

16. Verify backups

Locate a control-plane node:

oc get nodes -l node-role.kubernetes.io/master

Open a debug shell:

oc debug node/<control-plane-node>
chroot /host

Create a supported backup:

/usr/local/bin/cluster-backup.sh /home/core/assets/backup

Verify files:

ls -lh /home/core/assets/backup

Typical output includes:

snapshot_*.db
static_kuberesources_*.tar.gz

Take one cluster backup from one healthy control-plane node—not a separate backup from every member. Backup and restore procedures must follow the documentation for the exact OpenShift release. (Red Hat Documentation)

Fast troubleshooting workflow

API slow or unavailable
|
v
oc get co etcd
|
v
Check control-plane nodes and etcd pods
|
v
etcdctl endpoint health --cluster
|
v
endpoint status + member list + alarm list
|
v
Check leader changes and fsync latency
|
v
Inspect disk, I/O and network on each control-plane node
|
v
Review etcd and kube-apiserver logs
|
v
Take backup and must-gather before invasive recovery

Compact command bundle

oc get co etcd
oc get etcd cluster
oc get nodes -l node-role.kubernetes.io/master -o wide
oc get pods -n openshift-etcd -l app=etcd -o wide
ETCD_POD=$(oc get pods -n openshift-etcd \
-l app=etcd \
-o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl endpoint health --cluster -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl endpoint status --cluster -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl member list -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
etcdctl alarm list
oc logs -n openshift-etcd "$ETCD_POD" \
-c etcd --since=30m
oc get events -n openshift-etcd \
--sort-by=.metadata.creationTimestamp

Before member removal, manual defragmentation, quota changes, or snapshot restoration, collect diagnostics:

oc adm must-gather

Those operations can affect quorum or temporarily block a member and should not be treated as routine diagnostic commands.

OpenShift OADP Command Guide for Backup and Restore

Below is a practical OpenShift oc command reference for OADP/Velero backup and restore troubleshooting.

OADP is the Red Hat-supported Operator that deploys and manages Velero components for backing up application Kubernetes objects, persistent volumes, internal images, and supported OpenShift Virtualization workloads. (Red Hat Customer Portal)

1. Check the OADP Operator installation

Find the Operator:

oc get csv -A | grep -i oadp

Check subscriptions:

oc get subscription -A | grep -i oadp

Typical OADP namespace:

openshift-adp

Check Operator objects:

oc get csv,subscription,installplan -n openshift-adp

Detailed ClusterServiceVersion status:

oc describe csv -n openshift-adp \
$(oc get csv -n openshift-adp -o name | grep oadp | head -1)

Check the Operator deployment:

oc get deployment -n openshift-adp

Check Operator logs:

oc logs -n openshift-adp \
deployment/openshift-adp-controller-manager \
-c manager --tail=200

Follow logs:

oc logs -n openshift-adp \
deployment/openshift-adp-controller-manager \
-c manager -f

2. Check the DataProtectionApplication

The DataProtectionApplication, commonly abbreviated as DPA, is the main OADP configuration resource.

oc get dataprotectionapplication -n openshift-adp

Short form:

oc get dpa -n openshift-adp

View configuration:

oc get dpa -n openshift-adp -o yaml

Describe the DPA:

oc describe dpa <dpa-name> -n openshift-adp

Extract its conditions:

oc get dpa <dpa-name> -n openshift-adp \
-o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}'

Look for:

Reconciled=True

Also inspect:

  • Backup storage configuration
  • Cloud provider plugin
  • Credential Secret
  • CSI configuration
  • Data mover configuration
  • Node agent settings

3. Check Velero and node-agent pods

oc get pods -n openshift-adp -o wide

Common components include:

velero
node-agent
openshift-adp-controller-manager

Depending on the OADP version and configuration, the filesystem backup component can appear as node-agent; older environments might refer to Restic.

Check Velero deployment:

oc get deployment velero -n openshift-adp

Check node-agent DaemonSet:

oc get daemonset -n openshift-adp

Verify that a node-agent pod is running on each applicable node:

oc get pods -n openshift-adp \
-l name=node-agent -o wide

Inspect pod restarts:

oc get pods -n openshift-adp \
-o custom-columns='POD:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[*].restartCount'

Describe a failing pod:

oc describe pod <pod-name> -n openshift-adp

4. Check Velero logs

Current Velero logs:

oc logs -n openshift-adp deployment/velero \
--tail=300

Follow logs:

oc logs -n openshift-adp deployment/velero -f

Previous crashed container:

oc logs -n openshift-adp deployment/velero \
--previous

Search for common failures:

oc logs -n openshift-adp deployment/velero \
--since=1h |
grep -Ei 'error|failed|warning|timeout|credential|access denied|snapshot|repository'

Check node-agent logs:

oc logs -n openshift-adp <node-agent-pod> \
--tail=300

Logs from every node-agent pod:

for pod in $(oc get pods -n openshift-adp \
-l name=node-agent -o name); do
echo "===== $pod ====="
oc logs -n openshift-adp "$pod" \
--since=1h 2>&1 |
grep -Ei 'error|failed|timeout|repository|volume|snapshot'
done

5. Check BackupStorageLocation

A BackupStorageLocation, or BSL, defines the object-storage destination such as AWS S3, Azure Blob Storage, Google Cloud Storage, or an S3-compatible endpoint.

oc get backupstoragelocation -n openshift-adp

Short form:

oc get bsl -n openshift-adp

Detailed view:

oc describe bsl <bsl-name> -n openshift-adp

YAML:

oc get bsl <bsl-name> -n openshift-adp -o yaml

Check availability:

oc get bsl -n openshift-adp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,LAST-VALIDATED:.status.lastValidationTime,MESSAGE:.status.message'

Expected phase:

Available

Common failure states include:

Unavailable
Unknown

Typical causes:

  • Invalid cloud credentials
  • Incorrect bucket name
  • Incorrect region
  • Invalid S3 endpoint
  • Missing object-storage permissions
  • Certificate trust problems
  • Network or proxy failures

6. Check VolumeSnapshotLocation

oc get volumesnapshotlocation -n openshift-adp

Short form:

oc get vsl -n openshift-adp

Describe it:

oc describe vsl <vsl-name> -n openshift-adp

View YAML:

oc get vsl <vsl-name> -n openshift-adp -o yaml

Check whether the configured provider and region match the persistent volumes being protected.

7. Check backup credentials

List Secrets:

oc get secrets -n openshift-adp

Check which Secret is referenced by the DPA:

oc get dpa <dpa-name> -n openshift-adp -o yaml |
grep -A5 credential

Inspect Secret metadata:

oc describe secret <credentials-secret> -n openshift-adp

Do not print credential values into shared terminals, tickets, or chat logs.

Check whether the key exists without displaying its contents:

oc get secret <credentials-secret> -n openshift-adp \
-o jsonpath='{.data}' |
jq 'keys'

A common expected key is:

cloud

Verify that the Velero service account can read the Secret:

oc auth can-i get secret/<credentials-secret> \
--as system:serviceaccount:openshift-adp:velero \
-n openshift-adp

8. List backups

oc get backups.velero.io -n openshift-adp

Short form:

oc get backup -n openshift-adp

Detailed table:

oc get backup -n openshift-adp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,COMPLETED:.status.completionTimestamp,ERRORS:.status.errors,WARNINGS:.status.warnings,EXPIRES:.status.expiration'

Watch backup progress:

oc get backup -n openshift-adp -w

Describe a backup:

oc describe backup <backup-name> -n openshift-adp

View the complete Backup CR:

oc get backup <backup-name> \
-n openshift-adp -o yaml

Important phases:

New
InProgress
Completed
PartiallyFailed
Failed
Deleting

9. Create a simple namespace backup

Create a backup manifest:

cat <<'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Backup
metadata:
name: myapp-backup
namespace: openshift-adp
spec:
includedNamespaces:
- myapp
storageLocation: default
ttl: 720h0m0s
EOF

Monitor it:

oc get backup myapp-backup \
-n openshift-adp -w

Inspect the result:

oc describe backup myapp-backup \
-n openshift-adp

10. Back up selected resource types

cat <<'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Backup
metadata:
name: myapp-resources
namespace: openshift-adp
spec:
includedNamespaces:
- myapp
includedResources:
- deployments
- services
- configmaps
- secrets
- persistentvolumeclaims
storageLocation: default
ttl: 720h0m0s
EOF

Be careful when excluding cluster-scoped resources, because applications might depend on:

  • CustomResourceDefinitions
  • ClusterRoles
  • ClusterRoleBindings
  • StorageClasses
  • SecurityContextConstraints
  • Operators

11. Back up resources by label

cat <<'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Backup
metadata:
name: frontend-backup
namespace: openshift-adp
spec:
includedNamespaces:
- myapp
labelSelector:
matchLabels:
app: frontend
storageLocation: default
EOF

Verify labels before starting:

oc get all,pvc,configmap,secret \
-n myapp -l app=frontend

12. Check backup details using the Velero CLI

When the Velero CLI is installed:

velero backup get

Detailed backup information:

velero backup describe <backup-name> --details

Download backup logs:

velero backup logs <backup-name>

Save them:

velero backup logs <backup-name> \
> <backup-name>.log

Velero supports both CLI commands and Kubernetes custom resources; in an OADP-managed environment, oc get backup and oc describe backup remain useful even when the Velero CLI is unavailable. (Velero)

13. Check pod volume backups

For filesystem-based persistent-volume backups:

oc get podvolumebackups -n openshift-adp

Short form, where supported:

oc get pvb -n openshift-adp

Detailed table:

oc get podvolumebackups -n openshift-adp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,POD:.spec.pod.name,VOLUME:.spec.volume,START:.status.startTimestamp,COMPLETED:.status.completionTimestamp,MESSAGE:.status.message'

Describe a failed object:

oc describe podvolumebackup <name> \
-n openshift-adp

List failed or partially failed items:

oc get podvolumebackups -n openshift-adp \
--field-selector status.phase=Failed

14. Check CSI snapshots

List CSI snapshot classes:

oc get volumesnapshotclass

List snapshots across all namespaces:

oc get volumesnapshot -A

Check snapshot contents:

oc get volumesnapshotcontent

Describe the snapshot:

oc describe volumesnapshot <snapshot-name> \
-n <application-namespace>

Check readiness:

oc get volumesnapshot -A \
-o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.readyToUse,SOURCE-PVC:.spec.source.persistentVolumeClaimName,ERROR:.status.error.message'

Verify the CSI driver:

oc get csidriver
oc get csinode

Check StorageClass and PVC:

oc get pvc -n <namespace>
oc describe pvc <pvc-name> -n <namespace>
oc get storageclass

15. Check DataUpload and DataDownload objects

For OADP data mover workflows:

oc get datauploads -n openshift-adp
oc get datadownloads -n openshift-adp

Detailed status:

oc get datauploads -n openshift-adp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,COMPLETED:.status.completionTimestamp,MESSAGE:.status.message'
oc get datadownloads -n openshift-adp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,COMPLETED:.status.completionTimestamp,MESSAGE:.status.message'

Describe failures:

oc describe dataupload <name> -n openshift-adp
oc describe datadownload <name> -n openshift-adp

The exact data-movement resources available depend on the OADP release and DPA configuration.

16. List restores

oc get restores.velero.io -n openshift-adp

Short form:

oc get restore -n openshift-adp

Detailed table:

oc get restore -n openshift-adp \
-o custom-columns='NAME:.metadata.name,BACKUP:.spec.backupName,PHASE:.status.phase,START:.status.startTimestamp,COMPLETED:.status.completionTimestamp,ERRORS:.status.errors,WARNINGS:.status.warnings'

Watch restore progress:

oc get restore -n openshift-adp -w

Describe a restore:

oc describe restore <restore-name> \
-n openshift-adp

17. Restore an entire backup

cat <<'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Restore
metadata:
name: myapp-restore
namespace: openshift-adp
spec:
backupName: myapp-backup
EOF

Monitor:

oc get restore myapp-restore \
-n openshift-adp -w

Inspect:

oc describe restore myapp-restore \
-n openshift-adp

Velero restore behavior is controlled by the Restore custom resource, including namespace mappings, resource filters, label selectors and existing-resource policies. (Velero)

18. Restore into a different namespace

cat <<'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Restore
metadata:
name: myapp-restore-test
namespace: openshift-adp
spec:
backupName: myapp-backup
namespaceMapping:
myapp: myapp-restore-test
EOF

Verify:

oc get all,pvc,configmap,secret \
-n myapp-restore-test

Namespace mapping is useful for disaster-recovery testing, but hardcoded references to the original namespace may require application-specific changes.

19. Restore selected resources

cat <<'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Restore
metadata:
name: myapp-config-restore
namespace: openshift-adp
spec:
backupName: myapp-backup
includedNamespaces:
- myapp
includedResources:
- configmaps
- secrets
EOF

20. Check pod volume restores

oc get podvolumerestores -n openshift-adp

Detailed status:

oc get podvolumerestores -n openshift-adp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,POD:.spec.pod.name,VOLUME:.spec.volume,START:.status.startTimestamp,COMPLETED:.status.completionTimestamp,MESSAGE:.status.message'

Describe a failed restore:

oc describe podvolumerestore <name> \
-n openshift-adp

21. Check restore logs with Velero CLI

velero restore get

Detailed status:

velero restore describe <restore-name> --details

Logs:

velero restore logs <restore-name>

Save the logs:

velero restore logs <restore-name> \
> <restore-name>.log

22. Check backup schedules

oc get schedules.velero.io -n openshift-adp

Short form:

oc get schedule -n openshift-adp

Detailed view:

oc get schedule -n openshift-adp \
-o custom-columns='NAME:.metadata.name,SCHEDULE:.spec.schedule,PAUSED:.spec.paused,LAST-BACKUP:.status.lastBackup,PHASE:.status.phase'

Describe:

oc describe schedule <schedule-name> \
-n openshift-adp

A Velero Schedule is a repeating backup request based on cron notation. (Velero)

23. Create a daily schedule

Example: run every day at 02:00:

cat <<'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: myapp-daily
namespace: openshift-adp
spec:
schedule: "0 2 * * *"
template:
includedNamespaces:
- myapp
storageLocation: default
ttl: 720h0m0s
EOF

Check generated backups:

oc get backup -n openshift-adp \
-l velero.io/schedule-name=myapp-daily

Trigger an immediate backup based on a schedule using the Velero CLI:

velero backup create \
--from-schedule myapp-daily

Creating a manual backup from a schedule does not alter the recurring schedule. (Velero)

24. Pause and resume a schedule

Pause:

oc patch schedule myapp-daily \
-n openshift-adp \
--type merge \
-p '{"spec":{"paused":true}}'

Resume:

oc patch schedule myapp-daily \
-n openshift-adp \
--type merge \
-p '{"spec":{"paused":false}}'

Confirm:

oc get schedule myapp-daily \
-n openshift-adp \
-o jsonpath='{.spec.paused}{"\n"}'

25. Delete backups safely

Delete through the Velero request mechanism:

velero backup delete <backup-name> --confirm

Or create a deletion request:

cat <<EOF | oc apply -f -
apiVersion: velero.io/v1
kind: DeleteBackupRequest
metadata:
generateName: <backup-name>-
namespace: openshift-adp
spec:
backupName: <backup-name>
EOF

Check deletion requests:

oc get deletebackuprequest -n openshift-adp

Avoid relying only on:

oc delete backup <backup-name> -n openshift-adp

Deleting only the Kubernetes Backup CR might not perform the intended cleanup of associated backup data in object storage.

26. Check backup repository health

oc get backuprepositories -n openshift-adp

Detailed status:

oc get backuprepositories -n openshift-adp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,LAST-MAINTENANCE:.status.lastMaintenanceTime,MESSAGE:.status.message'

Describe:

oc describe backuprepository <name> \
-n openshift-adp

Common repository issues include:

  • Repository not ready
  • Incorrect encryption password
  • Object-storage access failure
  • Stale repository lock
  • Node-agent connectivity problem
  • Repository maintenance failure

27. Check OADP-related events

Namespace events:

oc get events -n openshift-adp \
--sort-by=.metadata.creationTimestamp

Warning events:

oc get events -n openshift-adp \
--field-selector type=Warning \
--sort-by=.metadata.creationTimestamp

Application namespace events:

oc get events -n <application-namespace> \
--sort-by=.metadata.creationTimestamp

CSI snapshot events:

oc get events -A \
--field-selector type=Warning |
grep -Ei 'snapshot|volume|velero|backup|restore'

28. Check RBAC and SCC

Check Velero service accounts:

oc get serviceaccount -n openshift-adp

Check cluster roles and bindings:

oc get clusterrole,clusterrolebinding |
grep -Ei 'velero|oadp'

Check whether Velero can read application resources:

oc auth can-i get pods \
--as system:serviceaccount:openshift-adp:velero \
-n myapp

Check PVC access:

oc auth can-i get persistentvolumeclaims \
--as system:serviceaccount:openshift-adp:velero \
-n myapp

Check SCC authorization for the node-agent:

oc auth can-i use scc/privileged \
--as system:serviceaccount:openshift-adp:velero \
-n openshift-adp

The exact service account used by node-agent should be confirmed from the pod:

oc get pod <node-agent-pod> -n openshift-adp \
-o jsonpath='{.spec.serviceAccountName}{"\n"}'

29. Check whether application PVCs were included

List application PVCs:

oc get pvc -n myapp

Inspect the backup resource list:

velero backup describe myapp-backup --details

Check related volume backup objects:

oc get podvolumebackups -n openshift-adp
oc get volumesnapshot -A
oc get datauploads -n openshift-adp

A backup can report Completed while application consistency is still not guaranteed. Database applications may require backup hooks, quiescing, native database backups, or operator-specific procedures.

30. Check backup and restore hooks

Inspect hooks in a Backup:

oc get backup <backup-name> \
-n openshift-adp \
-o jsonpath='{.spec.hooks}' |
jq

Check pod annotations:

oc get pods -n <namespace> -o yaml |
grep -i -A5 -B5 backup.velero.io

Common annotations include volume backup selection and pre/post backup behavior, depending on the configured backup method.

31. Inventory all OADP resources

oc api-resources |
grep -Ei 'velero|oadp'

List the common resources:

for resource in \
dataprotectionapplications \
backupstoragelocations \
volumesnapshotlocations \
backups \
restores \
schedules \
podvolumebackups \
podvolumerestores \
backuprepositories \
datauploads \
datadownloads; do
echo
echo "===== $resource ====="
oc get "$resource" -n openshift-adp 2>/dev/null ||
echo "Resource unavailable or none found"
done

32. Quick health-check script

#!/usr/bin/env bash
set -u
NS="${1:-openshift-adp}"
echo "===== OADP CSV ====="
oc get csv -n "$NS" 2>/dev/null | grep -i oadp || true
echo
echo "===== DPA ====="
oc get dpa -n "$NS" -o wide 2>/dev/null || true
echo
echo "===== OADP PODS ====="
oc get pods -n "$NS" -o wide
echo
echo "===== BACKUP STORAGE LOCATIONS ====="
oc get bsl -n "$NS" \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message' \
2>/dev/null || true
echo
echo "===== VOLUME SNAPSHOT LOCATIONS ====="
oc get vsl -n "$NS" 2>/dev/null || true
echo
echo "===== RECENT BACKUPS ====="
oc get backup -n "$NS" \
--sort-by=.metadata.creationTimestamp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARNINGS:.status.warnings,START:.status.startTimestamp' \
2>/dev/null | tail -20
echo
echo "===== RECENT RESTORES ====="
oc get restore -n "$NS" \
--sort-by=.metadata.creationTimestamp \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARNINGS:.status.warnings,START:.status.startTimestamp' \
2>/dev/null | tail -20
echo
echo "===== SCHEDULES ====="
oc get schedule -n "$NS" 2>/dev/null || true
echo
echo "===== FAILED POD VOLUME BACKUPS ====="
oc get podvolumebackups -n "$NS" \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message' \
2>/dev/null | grep -E 'NAME|Failed|PartiallyFailed' || true
echo
echo "===== WARNING EVENTS ====="
oc get events -n "$NS" \
--field-selector type=Warning \
--sort-by=.metadata.creationTimestamp \
2>/dev/null | tail -30
echo
echo "===== RECENT VELERO ERRORS ====="
oc logs -n "$NS" deployment/velero \
--since=30m 2>/dev/null |
grep -Ei 'error|failed|timeout|denied|unavailable' |
tail -50 || true

Run it:

chmod +x oadp-health.sh
./oadp-health.sh

Fast troubleshooting workflow

Backup or restore fails
|
v
Check OADP Operator and DPA
|
v
Check Velero and node-agent pods
|
v
Validate BSL and VSL
|
v
Describe Backup or Restore CR
|
v
Read Velero and node-agent logs
|
v
Inspect PVB/PVR, snapshots or DataUpload/DataDownload
|
v
Check credentials, permissions and network access
|
v
Validate restored application and data

The most useful first commands are:

oc get dpa -n openshift-adp
oc get pods -n openshift-adp -o wide
oc get bsl,vsl -n openshift-adp
oc get backup,restore,schedule -n openshift-adp
oc describe backup <backup-name> -n openshift-adp
oc logs deployment/velero -n openshift-adp --since=1h
oc get events -n openshift-adp --sort-by=.metadata.creationTimestamp

Understanding OpenShift Load Balancers

OpenShift Load Balancers Explained

In OpenShift, “load balancer” can refer to several different layers:

External client traffic
External Load Balancer
OpenShift Ingress Router
Service
Application Pods

There are also dedicated load balancers for the OpenShift API and machine configuration endpoints.


Main Load Balancers in OpenShift

A standard OpenShift cluster normally needs these external endpoints:

EndpointPortPurpose
api.<cluster>.<domain>6443Kubernetes and OpenShift API
api-int.<cluster>.<domain>6443Internal API communication
*.apps.<cluster>.<domain>80/443Application routes
Machine Config Server22623Node ignition and machine configuration

The most important distinction is:

API load balancer
Application ingress load balancer

They serve different traffic and normally use different backend pools.


High-Level Architecture

                          Administrators
                               │
                               ▼
                    api.cluster.example.com
                               │
                     API Load Balancer
                               │
                 ┌─────────────┼─────────────┐
                 ▼             ▼             ▼
             master-0       master-1       master-2
               :6443          :6443          :6443


                         Application Users
                               │
                               ▼
                 app1.apps.cluster.example.com
                               │
                  Ingress Load Balancer
                               │
                 ┌─────────────┼─────────────┐
                 ▼             ▼             ▼
            router pod     router pod     router pod
                 │             │             │
                 └─────────────┼─────────────┘
                               ▼
                         OpenShift Service
                               │
                         Application Pods

1. API Load Balancer

The API load balancer provides highly available access to the OpenShift control plane.

Clients include:

  • oc
  • OpenShift web console
  • kubelets
  • Operators
  • Controllers
  • Automation pipelines
  • Monitoring systems

Traffic path:

oc command
api.cluster.example.com:6443
External load balancer
├── master-0:6443
├── master-1:6443
└── master-2:6443

The load balancer distributes requests across all healthy API servers.

Recommended behavior

Use:

  • Layer 4 TCP load balancing
  • TCP health checks or HTTPS health checks
  • No application-level rewriting
  • Source connection stability where required
  • All control-plane nodes as backends

Example HAProxy configuration:

frontend api-server
bind *:6443
mode tcp
default_backend api-server-backend
backend api-server-backend
mode tcp
balance roundrobin
option tcp-check
server master0 10.10.10.10:6443 check
server master1 10.10.10.11:6443 check
server master2 10.10.10.12:6443 check

2. Internal API Load Balancer

The internal API name is commonly:

api-int.<cluster>.<domain>

It is used by nodes and internal cluster components.

Worker kubelet
api-int.cluster.example.com:6443
Internal load balancer
Control-plane nodes

In some designs, the same load balancer serves both public and internal API names. In more restricted environments, separate internal and external virtual IPs are used.


3. Machine Config Server Load Balancer

The Machine Config Server listens on port 22623.

It provides Ignition and machine configuration during installation and node provisioning.

New OpenShift node
api-int.cluster.example.com:22623
Load balancer
Machine Config Server

Typical backend targets:

master-0:22623
master-1:22623
master-2:22623

Example HAProxy configuration:

frontend machine-config-server
bind *:22623
mode tcp
default_backend machine-config-backend
backend machine-config-backend
mode tcp
balance roundrobin
server master0 10.10.10.10:22623 check
server master1 10.10.10.11:22623 check
server master2 10.10.10.12:22623 check

4. Application Ingress Load Balancer

The application load balancer handles traffic for OpenShift Routes.

Example DNS:

payments.apps.cluster.example.com
mobile.apps.cluster.example.com
banking.apps.cluster.example.com

These normally resolve to the ingress load balancer.

Client
payments.apps.cluster.example.com
External load balancer
OpenShift router pods
Service
Application pods

The router pods are usually HAProxy-based and managed by the Ingress Operator.


OpenShift Router

The OpenShift router is not the same as the external load balancer.

External Load Balancer
Ingress Router Pods
OpenShift Services
Application Pods

The external load balancer only forwards traffic to the router nodes or router pods.

The router then:

  • Matches the hostname
  • Matches the route
  • Handles TLS
  • Selects the backend service
  • Load-balances traffic to pods

Route Traffic Flow

Suppose the application route is:

https://payments.apps.cluster.example.com

The request path is:

Browser
DNS
Ingress load balancer
Router pod
Route object
Service
Pod endpoint

OpenShift Route Example

apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: payments
namespace: banking
spec:
host: payments.apps.cluster.example.com
to:
kind: Service
name: payments-service
port:
targetPort: https
tls:
termination: edge

The router watches Route objects and dynamically updates its routing configuration.


Load Balancing Inside the Cluster

OpenShift Services provide internal load balancing.

Example:

apiVersion: v1
kind: Service
metadata:
name: payments-service
spec:
selector:
app: payments
ports:
- port: 443
targetPort: 8443

Traffic flow:

payments-service
├── payments-pod-1
├── payments-pod-2
└── payments-pod-3

The service exposes a stable virtual IP and distributes traffic to healthy pod endpoints.


Service Types

ClusterIP

Default service type.

Available only inside the cluster
spec:
type: ClusterIP

Use it for internal service-to-service communication.


NodePort

Exposes a port on every node.

Client
NodeIP:NodePort
Service
Pods
spec:
type: NodePort

NodePort is usually not preferred as the primary application exposure method in enterprise OpenShift. Routes are more common.


LoadBalancer

Requests an external load balancer from the cloud provider or supported infrastructure integration.

spec:
type: LoadBalancer

Example flow in AWS:

Service type LoadBalancer
Cloud Controller Manager
AWS NLB or ELB created
OpenShift nodes or pods

This is frequently used for non-HTTP protocols or applications that need their own external load balancer.


ExternalName

Maps a service to an external DNS name.

spec:
type: ExternalName
externalName: database.example.com

It does not create a real load balancer.


Ingress Controller Placement

By default, ingress router pods are scheduled according to the Ingress Controller configuration.

For enterprise environments, dedicated infrastructure nodes are recommended:

Worker nodes
├── Application workloads
└── Batch workloads
Infrastructure nodes
├── Ingress routers
├── Registry
├── Monitoring
└── Logging

Example node labels:

oc label node infra-0 node-role.kubernetes.io/infra=""
oc label node infra-1 node-role.kubernetes.io/infra=""
oc label node infra-2 node-role.kubernetes.io/infra=""

The Ingress Controller can then use a node placement policy.


Typical Bare-Metal Architecture

                         Corporate Network
                               │
                     F5 / HAProxy / NetScaler
                     ┌─────────┴──────────┐
                     │                    │
                  API VIP             Apps VIP
                     │                    │
          ┌──────────┼──────────┐    ┌────┼────┐
          ▼          ▼          ▼    ▼    ▼    ▼
       master-0   master-1   master-2  infra nodes
       :6443      :6443      :6443     :80/:443

Example VIPs:

API VIP: 10.10.20.10
Apps VIP: 10.10.20.20

DNS:

api.ocp.example.com → 10.10.20.10
api-int.ocp.example.com → 10.10.20.10
*.apps.ocp.example.com → 10.10.20.20

Typical Cloud Architecture

In AWS, Azure, or GCP, OpenShift can create and manage cloud load balancers.

Internet
Cloud Load Balancer
Ingress Router Service
Router Pods
Application Services

Possible cloud load balancers include:

  • AWS Network Load Balancer
  • Azure Load Balancer
  • Google Cloud Load Balancer

The exact implementation depends on:

  • Installation method
  • Platform integration
  • Ingress Controller configuration
  • Service annotations
  • Internal or external exposure

Internal vs External Ingress

A cluster can have multiple Ingress Controllers.

Example:

Public Ingress Controller
└── *.apps.ocp.example.com
Internal Ingress Controller
└── *.internal.apps.ocp.example.com

This is useful for banking environments:

Internet-facing applications
WAF → Public LB → Public Router
Internal banking applications
Internal LB → Private Router

You can separate them by:

  • DNS domain
  • Node placement
  • Route labels
  • Namespace selectors
  • Network zones
  • Load balancer scope

TLS Termination

OpenShift Routes support several TLS models.

Edge termination
Client ──HTTPS──> Router ──HTTP──> Pod

The router terminates TLS.


Re-encrypt termination
Client ──HTTPS──> Router ──HTTPS──> Pod

The router decrypts and creates a new TLS connection to the backend.

This is commonly preferred for sensitive applications.


Passthrough termination
Client ──HTTPS──> Router ──HTTPS──> Pod

The router does not decrypt the traffic. TLS terminates in the application pod.


Load Balancer Health Checks

API health check

A common check is:

TCP 6443

or an HTTPS readiness endpoint:

https://<master>:6443/readyz
Ingress health check

Common checks include:

TCP 80
TCP 443

or the router health endpoint, depending on platform configuration.

The load balancer should remove unhealthy endpoints automatically.


Session Persistence

Most OpenShift applications should be stateless.

However, if an application requires sticky sessions, Routes support session affinity through cookies.

Example annotation:

metadata:
annotations:
haproxy.router.openshift.io/balance: source

Other algorithms include:

roundrobin
leastconn
source

Avoid depending heavily on session persistence when applications can instead store session state externally.


Load Balancing Algorithms

External load balancer

Typical algorithms:

  • Round robin
  • Least connections
  • Source IP
  • Weighted round robin
OpenShift router

Common options:

  • Round robin
  • Least connections
  • Source-based persistence
Service load balancing

Kubernetes and OVN-Kubernetes distribute service traffic across available endpoints.


Failure Scenarios

One API server fails
master-1 fails
LB health check fails
master-1 removed from pool
Traffic continues to master-0 and master-2

One router pod fails
router pod fails
Load balancer or Service removes endpoint
Traffic continues through remaining router pods

One application pod fails
Application pod fails
Readiness probe fails
Endpoint removed from Service
Traffic goes to healthy pods

This shows the three load-balancing layers:

External LB health
Router readiness
Application pod readiness

Troubleshooting API Load Balancer

Test DNS:

dig api.cluster.example.com

Test the API:

curl -k https://api.cluster.example.com:6443/readyz

Test individual control-plane nodes:

curl -k https://master-0.example.com:6443/readyz
curl -k https://master-1.example.com:6443/readyz
curl -k https://master-2.example.com:6443/readyz

Check API pods:

oc get pods -n openshift-kube-apiserver -o wide

Look for:

  • Incorrect backend ports
  • Failed health checks
  • Missing master node
  • TLS inspection
  • Idle timeout too low
  • Load balancer SNAT exhaustion
  • DNS pointing to the wrong VIP
  • Firewall blocking port 6443

Troubleshooting Application Load Balancer

Check DNS:

dig payments.apps.cluster.example.com

Test the route:

curl -vk https://payments.apps.cluster.example.com

Check route:

oc get route -n banking
oc describe route payments -n banking

Check router pods:

oc get pods -n openshift-ingress -o wide

Check Ingress Controller:

oc get ingresscontroller -n openshift-ingress-operator
oc describe ingresscontroller default \
-n openshift-ingress-operator

Check the application service and endpoints:

oc get svc -n banking
oc get endpoints -n banking
oc get endpointslices -n banking

Check pod readiness:

oc get pods -n banking
oc describe pod <pod-name> -n banking

The full troubleshooting path is:

DNS
External load balancer
Router
Route
Service
EndpointSlice
Pod readiness
Application

Common Problems

Route returns 503

Usually means the router cannot find a healthy backend.

Check:

oc get endpointslices -n <namespace>
oc get pods -n <namespace>
oc describe route <route> -n <namespace>

Common causes:

  • No ready pods
  • Wrong Service selector
  • Wrong target port
  • Failed readiness probe
  • Application not listening

API intermittently unavailable

Possible causes:

  • Load balancer sending traffic to unhealthy master
  • Incorrect health check
  • Too-short timeout
  • Control-plane API latency
  • Network packet loss
  • etcd latency
  • TLS inspection device interference

Route works internally but not externally

Check:

  • Wildcard DNS
  • Firewall
  • External VIP
  • Load balancer pool
  • Router node placement
  • Port 80/443
  • Public versus private ingress configuration

Banking Best Practices

For a regulated banking environment, I would use:

API traffic:
Admin network → Private API LB → Control-plane nodes
Public traffic:
Internet → DDoS protection → WAF → Public LB → Public routers
Internal traffic:
Corporate network → Internal LB → Internal routers

Additional controls:

  • Separate API and application VIPs
  • Private API endpoint
  • Dedicated infra nodes for routers
  • Multiple router replicas across failure domains
  • Re-encrypt or passthrough TLS for sensitive applications
  • WAF in front of public ingress
  • Mutual TLS for partner applications
  • Centralized load balancer and router access logs
  • NetworkPolicies behind the router
  • Health checks based on readiness
  • Avoid TLS interception on OpenShift API traffic
  • Monitor connection count, latency and backend health

Important Interview Distinction

An OpenShift architect should distinguish these three layers:

LayerComponentPurpose
ExternalF5, HAProxy, cloud LBSends traffic into the cluster
IngressOpenShift routerMatches Routes and sends traffic to Services
InternalKubernetes ServiceDistributes traffic to application pods
External Load Balancer
OpenShift Router
Kubernetes Service
Pods

Interview Answer

OpenShift uses load balancing at multiple layers. The API load balancer exposes port 6443 and distributes administrative and internal Kubernetes API traffic across the control-plane nodes. Port 22623 is used for the Machine Config Server during node provisioning. Application traffic is normally sent through a separate ingress load balancer on ports 80 and 443 to OpenShift router pods. The routers evaluate Route objects, terminate or pass through TLS, and forward requests to Kubernetes Services, which then distribute traffic to healthy pod endpoints.

On bare metal, the external load balancer might be F5, HAProxy or NetScaler. In cloud environments, OpenShift integrates with the cloud provider’s load-balancing services. For production, I would use separate API and application VIPs, redundant router replicas across failure domains, proper health checks, dedicated infrastructure nodes and separate public and internal Ingress Controllers. When troubleshooting, I follow the traffic path from DNS to the external load balancer, router, Route, Service, EndpointSlice, pod readiness and finally the application.

Understanding CRI-O: The Default OpenShift Container Runtime

CRI-O is the default container runtime in OpenShift Container Platform (OCP). It is a lightweight, Kubernetes-native runtime designed specifically to run containers according to the Open Container Initiative (OCI) standards. Unlike Docker, CRI-O is not a full container platform—it only provides the functionality Kubernetes needs to start, stop, and manage containers.

For OpenShift interviews, CRI-O is one of the most important platform components to understand.


Where CRI-O Fits in OpenShift

                    User
                      │
                  oc CLI / API
                      │
               kube-apiserver
                      │
                 Scheduler
                      │
                 kubelet
                      │
                CRI (gRPC API)
                      │
                  CRI-O
          ┌───────────┴────────────┐
          │                        │
     runc / crun             Image Management
          │                        │
          └───────────┬────────────┘
                      │
               Linux Kernel
          (Namespaces + cgroups + SELinux)

Key point: Kubernetes never talks directly to containers—it communicates with CRI-O through the Container Runtime Interface (CRI).


Why Red Hat Uses CRI-O Instead of Docker

Originally Kubernetes supported Docker through Dockershim.

Kubernetes
Dockershim
Docker

Problems:

  • Docker included many features Kubernetes didn’t need.
  • Additional translation layer (Dockershim).
  • Higher resource consumption.
  • Larger attack surface.
  • More maintenance.

When Dockershim was removed from Kubernetes, OpenShift adopted CRI-O because it is:

  • Kubernetes-native
  • OCI-compliant
  • Lightweight
  • Easier to secure
  • Easier to maintain

OpenShift Node Components

Every worker node typically runs:

Worker Node
RHCOS
├── kubelet
├── CRI-O
├── Machine Config Daemon
├── Network (OVN)
├── Node Exporter
└── Monitoring Agents

CRI-O is the service responsible for running all containers on the node.


Container Startup Flow

Suppose you create a Deployment:

oc create deployment nginx --image=nginx

The sequence is:

Deployment
ReplicaSet
Pod
Scheduler
Worker Node
kubelet
CRI-O
runc/crun
Linux Kernel
Container Running

What Happens Internally

Imagine the pod is scheduled to Worker-1.

Step 1

Scheduler assigns:

Pod
Worker-1

Step 2

kubelet notices:

Desired Pod
Need container

Step 3

kubelet calls CRI-O:

CreateContainer()
StartContainer()

through the CRI gRPC API.


Step 4

CRI-O:

  • pulls the image
  • creates filesystem
  • prepares networking
  • mounts volumes
  • creates namespaces
  • configures cgroups
  • applies SELinux labels

Step 5

CRI-O launches:

runc
or
crun

Step 6

The OCI runtime asks the Linux kernel to create:

  • PID namespace
  • Network namespace
  • Mount namespace
  • User namespace (if configured)
  • IPC namespace
  • cgroups

The application process then starts.


Components of CRI-O

CRI-O
├── CRI Server
├── Image Manager
├── Runtime Manager
├── Storage Manager
├── Networking
├── Logging
└── OCI Runtime

1. CRI Server

Receives requests from kubelet.

Examples:

RunPodSandbox()
CreateContainer()
StartContainer()
StopContainer()
RemoveContainer()

2. Image Manager

Responsible for:

Pull Image
Verify Image
Store Image
Reuse Cached Image

Uses:

  • Quay
  • Internal registry
  • Docker Hub
  • Other OCI registries

3. Storage

Uses the Linux OverlayFS storage driver.

Typical storage location:

/var/lib/containers/storage

Example:

Image Layers
OverlayFS
Writable Layer
Container

4. OCI Runtime

CRI-O does not execute containers directly.

It launches:

runc
or
crun

These create the container using Linux kernel primitives.


Why OpenShift Prefers crun

Recent OpenShift versions prefer crun because it offers:

  • Faster startup
  • Lower memory usage
  • Better cgroup v2 support
  • Better performance at scale

Networking

CRI-O does not configure networking itself.

It requests networking from Kubernetes.

CRI-O
CNI Plugin
OVN-Kubernetes
Pod IP
Network Ready

Storage

When a pod uses a PVC:

PVC
CSI Driver
StorageClass
Volume
CRI-O Mount
Container

CRI-O mounts the volume before starting the container.


Security

One reason Red Hat chose CRI-O is security.

Every container starts with:

  • SELinux labels
  • cgroups
  • namespaces
  • seccomp profiles
  • SCC restrictions
  • capabilities dropped
  • read-only root filesystem (when configured)

Example:

Container
SELinux
cgroups
Namespaces
Kernel

Logging

CRI-O captures stdout/stderr from containers.

Logs are typically stored under:

/var/log/containers/

These are then collected by logging agents such as Vector or Fluentd.

Flow:

Application
stdout
CRI-O
Node Log
Logging Stack

Image Pull

When an image is not cached:

Pod
CRI-O
Registry
Download
Verify
Store
Start Container

If cached:

Pod
CRI-O
Cached Image
Container Starts

CRI-O vs Docker

FeatureDockerCRI-O
Kubernetes optimizedNoYes
OCI compliantYesYes
Requires DockershimYes (historically)No
LightweightNoYes
Container runtime onlyNoYes
Image build capabilityYesNo
Native Kubernetes runtimeNoYes
Default in OpenShiftNoYes

Useful Commands

Check the CRI-O service:

systemctl status crio

View logs:

journalctl -u crio

Check running containers:

crictl ps

List images:

crictl images

Inspect pods:

crictl pods

Inspect a container:

crictl inspect <container-id>

Runtime information:

crictl info

Troubleshooting CRI-O

Pod stuck in ContainerCreating

Check:

oc describe pod <pod>

Then inspect:

  • Image pull errors
  • Volume mount failures
  • Network setup failures
  • CRI-O logs
journalctl -u crio

ImagePullBackOff

Verify:

crictl images

Check:

  • Registry availability
  • ImagePullSecrets
  • DNS resolution
  • Authentication

CrashLoopBackOff

Check:

oc logs <pod>
oc describe pod

Then verify:

  • Application startup
  • Memory limits
  • Exit codes
  • Liveness/readiness probes

CRI-O service down
systemctl status crio

If stopped:

systemctl restart crio

Then verify:

systemctl is-active crio

CRI-O and OpenShift Operators

The Machine Config Operator (MCO) manages CRI-O configuration across the cluster.

Example workflow:

MachineConfig
Machine Config Operator
Machine Config Daemon
Update /etc/crio/
Restart CRI-O (if required)
Node Ready

This ensures every worker node has a consistent runtime configuration.


CRI-O Integration with Other OpenShift Components

               OpenShift Cluster

                    API Server
                        │
                    Scheduler
                        │
                     kubelet
                        │
                     CRI-O
        ┌───────────────┼────────────────┐
        │               │                │
     Image Pull     Storage Mount     Networking
        │               │                │
     Registry         CSI Driver      OVN-Kubernetes
        │               │                │
        └───────────────┼────────────────┘
                        │
                    OCI Runtime
                  (crun / runc)
                        │
                  Linux Kernel

Interview Answer (2-Minute Version)

CRI-O is OpenShift’s default container runtime and implements the Kubernetes Container Runtime Interface (CRI). Instead of Kubernetes talking directly to Docker, the kubelet communicates with CRI-O over gRPC. CRI-O is responsible for pulling OCI-compliant images, preparing the container filesystem, mounting storage, configuring networking through the CNI plugins, applying security settings such as SELinux labels, cgroups, seccomp profiles, and Security Context Constraints, and finally launching the container using an OCI runtime such as crun or runc. Unlike Docker, CRI-O is purpose-built for Kubernetes, making it smaller, faster, and more secure. In OpenShift, its configuration is managed centrally by the Machine Config Operator, ensuring consistent runtime settings across all worker nodes. For troubleshooting, I typically start with oc describe pod, then check journalctl -u crio, use crictl ps, crictl images, and crictl inspect, and verify image pulls, storage mounts, networking, and runtime health.

Ignition: The First-Boot Provisioning Tool for OpenShift Nodes

What is Ignition in OpenShift?

Ignition is the first-boot provisioning engine used by Red Hat Enterprise Linux CoreOS (RHCOS) to configure OpenShift nodes.

Think of Ignition as the tool that builds the operating system configuration before Kubernetes starts.

A simple way to remember it is:

Ignition provisions a new node only once, during its first boot. After that, the Machine Config Operator (MCO) manages ongoing configuration changes.


Where Ignition Fits in OpenShift

                   New Server / VM
                          │
                          ▼
                 Boot RHCOS ISO/PXE
                          │
                          ▼
                 Download Ignition File
                          │
                          ▼
                    Ignition Runs
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
 Configure disks      Configure users   Configure files
 Configure network    Install SSH keys  Configure systemd
                          │
                          ▼
                  Reboot (if needed)
                          │
                          ▼
                kubelet starts
                          │
                          ▼
              Node joins OpenShift
                          │
                          ▼
         Machine Config Operator takes over

Why Ignition Exists

Before Kubernetes can run, the operating system must be prepared.

A new node initially knows nothing about:

  • The cluster
  • SSH keys
  • Storage layout
  • Network configuration
  • Certificates
  • kubelet
  • CRI-O configuration

Ignition performs all of this automatically during the first boot.

Without Ignition, every node would require manual configuration before joining the cluster.


What Ignition Configures

During first boot, Ignition can configure:

1. Disk Partitioning

Example:

Disk
├── EFI
├── Boot
├── Root
└── Data

It can:

  • Partition disks
  • Create filesystems
  • Format volumes
  • Mount storage

2. Users

Creates users such as:

core

Configures:

  • SSH authorized keys
  • User groups
  • Password hash (if specified)

Example:

passwd:
users:
- name: core
sshAuthorizedKeys:
- ssh-rsa AAAA...

3. Files

Ignition can create files like:

/etc/motd
/etc/sysctl.conf
/etc/containers/registries.conf

Example:

storage:
files:
- path: /etc/example.conf
contents:
source: data:text/plain;base64,...

4. Directories

Example:

/opt/company/
/etc/custom/

5. systemd Services

Ignition can enable:

systemd

Example:

Enable service
Start at boot

6. Certificates

Can install:

  • CA certificates
  • Internal PKI
  • Registry certificates

7. kubelet Bootstrap

Creates the initial kubelet configuration so that it can join the cluster.


Ignition During Installation

A typical OpenShift installation looks like this:

openshift-install
Generate Ignition Files
┌──────┼──────────┐
│ │ │
▼ ▼ ▼
bootstrap.ign
master.ign
worker.ign

Each node type receives its own Ignition configuration.


Bootstrap Node

The bootstrap node receives:

bootstrap.ign

Its job is to:

  • Start temporary control plane
  • Bootstrap etcd
  • Start Kubernetes
  • Create permanent control-plane nodes

After installation:

Bootstrap node
Removed

Master Nodes

Master nodes receive:

master.ign

This configures:

  • kubelet
  • CRI-O
  • certificates
  • etcd membership
  • control plane services

Worker Nodes

Workers receive:

worker.ign

This prepares:

  • kubelet
  • CRI-O
  • networking
  • certificates

Workers then register with the cluster.


Example Boot Process

Imagine a brand-new worker VM.

Step 1

Power on

Boot RHCOS


Step 2

Downloads:

worker.ign

Step 3

Ignition starts


Step 4

Creates:

Users
Files
Directories
Storage
SSH Keys
systemd

Step 5

Starts kubelet


Step 6

kubelet connects:

api-int.cluster.example.com

Step 7

Node joins cluster


Ignition Runs Only Once

One of the most important interview points.

First Boot
Ignition
Never runs again

After first boot:

Machine Config Operator
Machine Config Daemon
Node Updates

Ignition vs Machine Config Operator

Many interviewers ask this.

IgnitionMachine Config Operator
Runs onceRuns throughout cluster life
First bootOngoing configuration
Before KubernetesAfter Kubernetes
Initial provisioningConfiguration management
No cluster requiredCluster already running
Creates OSMaintains OS

Think of it like:

Ignition
Build the house
Machine Config Operator
Maintain the house

Ignition File Format

Ignition files are JSON.

Example:

{
"ignition": {
"version": "3.4.0"
}
}

Generated automatically by:

openshift-install create ignition-configs

Normally administrators do not edit these JSON files manually.


Where Ignition Comes From

During installation:

Install Config
openshift-install
Ignition Generator
bootstrap.ign
master.ign
worker.ign

Relationship with RHCOS

New RHCOS Node
Ignition
Configure OS
Start kubelet
Join OpenShift
Machine Config Operator

Relationship with Machine Config

Later in cluster life:

Administrator
MachineConfig
Machine Config Operator
Machine Config Daemon
Node Updated

Ignition is not involved anymore.


Security

Ignition can securely provision:

  • SSH keys
  • Certificates
  • Registry trust
  • Users
  • Files
  • Kernel arguments

Since it runs before Kubernetes starts, it establishes the initial trusted configuration.


Typical Files Created

Examples include:

/etc/hostname
/etc/containers/
systemd units
authorized_keys
CA certificates
kubelet configuration

Common Troubleshooting

Node never joins cluster

Check:

journalctl -b

Look for:

Ignition failed
network unreachable
cannot download ignition
certificate errors

Verify Ignition

During installation:

openshift-install wait-for bootstrap-complete

If bootstrap never completes:

Often:

  • bootstrap.ign incorrect
  • network issue
  • DNS
  • load balancer
  • certificates

Machine Config changes not applied

Remember:

Ignition
DOES NOT RUN AGAIN

The issue is almost certainly with:

  • Machine Config Operator
  • Machine Config Daemon
  • MachineConfigPool

Ignition vs Cloud-init

A common interview comparison.

IgnitionCloud-init
RHCOS/OpenShiftUbuntu, RHEL, cloud VMs
Runs onceRuns during instance initialization
DeclarativeMostly scripts and configuration
Creates OS stateGeneral VM initialization
JSONYAML
Purpose-built for CoreOSGeneral-purpose provisioning

Best Practices

  • Never manually edit generated .ign files unless you fully understand the implications.
  • Use openshift-install to generate them.
  • Use Ignition only for initial provisioning.
  • Use MachineConfig for day-2 operating-system changes.
  • Avoid manual changes to RHCOS that bypass the Machine Config Operator.

Interview Answer (2 Minutes)

Ignition is the first-boot provisioning engine used by RHCOS in OpenShift. During installation, the openshift-install utility generates three Ignition files—bootstrap.ign, master.ign, and worker.ign—which are consumed when each node boots for the first time. Ignition configures the operating system by creating users, installing SSH keys and certificates, partitioning disks, creating files and directories, configuring systemd services, and preparing the kubelet so the node can join the cluster.

A key point is that Ignition runs only once, before Kubernetes starts. After the node joins the cluster, ongoing operating-system configuration is managed by the Machine Config Operator and the Machine Config Daemon through MachineConfig resources. In interviews, I emphasize that Ignition is for day-0 provisioning, while the Machine Config Operator handles day-2 lifecycle management, updates, and configuration changes across the cluster.

Understanding RHCOS Boot Process for OpenShift

Booting new VMs with Red Hat Enterprise Linux CoreOS (RHCOS) is a critical part of the OpenShift installation process. During installation, the installer provisions the VMs and boots them from an immutable RHCOS image. These nodes then join the cluster and are managed automatically.


OpenShift Boot Process with RHCOS

                User runs openshift-install
                           │
                           ▼
               Ignition configuration generated
                           │
                           ▼
         Infrastructure creates new RHCOS VM
                           │
                           ▼
              VM boots immutable RHCOS image
                           │
                           ▼
             Ignition downloads machine config
                           │
                           ▼
          Configure hostname, networking, SSH
                           │
                           ▼
               kubelet service starts
                           │
                           ▼
        kubelet connects to Kubernetes API
                           │
                           ▼
      CSR generated and approved automatically
                           │
                           ▼
          Node joins the OpenShift cluster
                           │
                           ▼
Machine Config Operator manages the node

Step 1 – Installer Creates Ignition Files

The installer generates three Ignition configurations:

bootstrap.ign
master.ign
worker.ign

Example:

openshift-install create ignition-configs

Each Ignition file tells the node:

  • hostname
  • SSH keys
  • certificates
  • kubelet configuration
  • pull secret
  • networking
  • MachineConfig information

Step 2 – VM Boots RHCOS

The VM boots using the Red Hat CoreOS image.

Examples:

  • VMware
  • KVM
  • Hyper-V
  • AWS EC2
  • Azure VM
  • GCP Compute Engine

Unlike traditional Linux:

No kickstart
No cloud-init
No Ansible required

Instead it boots directly into RHCOS.


Step 3 – Ignition Runs (First Boot Only)

During the first boot:

systemd
Ignition

Ignition configures:

Filesystem

/var
/etc

Users

core

SSH Keys

Certificates

Networking

Kubelet configuration

CRI-O configuration

Machine Config

This happens only once.


Step 4 – Immutable Operating System

RHCOS is immutable.

/

is read-only.

Applications never modify the OS.

Instead they use

/var

Only Machine Config Operator changes the OS.


Step 5 – kubelet Starts

Systemd starts:

crio

then

kubelet

Example:

systemctl status kubelet

Output:

Active: active (running)

Step 6 – kubelet Contacts API Server

kubelet connects to

https://api.cluster.example.com:6443

It authenticates using bootstrap credentials.


Step 7 – CSR Created

Each node creates a Certificate Signing Request.

View pending requests:

oc get csr

Example:

csr-abc123 Pending
csr-def456 Pending

Normally OpenShift approves them automatically.

Approve manually:

oc adm certificate approve csr-abc123

Step 8 – Node Joins Cluster

Check:

oc get nodes

Example:

master-0 Ready
master-1 Ready
master-2 Ready
worker-0 Ready
worker-1 Ready

Step 9 – Machine Config Operator Takes Over

After joining:

Machine Config Operator

manages:

Kernel arguments

OS updates

Certificates

Kubelet config

CRI-O config

Network files

SSH keys

You never manually patch RHCOS.


What Actually Boots?

RHCOS includes:

Linux Kernel
systemd
CRI-O
kubelet
podman
rpm-ostree
Ignition
NetworkManager
SELinux
OpenShift components

Boot Sequence Inside RHCOS

BIOS / UEFI
GRUB
Linux Kernel
initramfs
Ignition
systemd
NetworkManager
CRI-O
kubelet
API Server
Node Ready

Troubleshooting RHCOS Boot

View Ignition logs
journalctl -b -u ignition

Check kubelet
journalctl -u kubelet

Check CRI-O
journalctl -u crio

Verify Machine Config
oc get mcp

Healthy output:

NAME UPDATED
master True
worker True

Check Machine Config Daemon
oc get pods -n openshift-machine-config-operator

Check Ignition file retrieval

On the node:

journalctl -b | grep ignition

Check node status
oc describe node worker-0

Interview Answer (2-minute version)

In OpenShift, new VMs boot from an immutable Red Hat CoreOS image rather than a traditional Linux installation. During installation, the openshift-install utility generates Ignition configuration files for bootstrap, control plane, and worker nodes. On first boot, the Ignition service configures networking, SSH keys, certificates, kubelet, and other system settings. After that, systemd starts CRI-O and the kubelet, which contacts the Kubernetes API server, generates a certificate signing request (CSR), and joins the cluster once approved. From that point onward, the Machine Config Operator (MCO) manages all operating system configuration and updates using rpm-ostree, ensuring every RHCOS node remains consistent, immutable, and centrally managed. This design improves security, simplifies lifecycle management, and reduces configuration drift across the cluster.

Troubleshooting OpenShift Node Issues with oc debug

For a Senior OpenShift Platform Engineer/Architect interview, using oc debug node is the standard and supported method to troubleshoot node-level issues because SSH access to RHCOS nodes is often restricted.

Scenario

Problem:
Users report:

  • oc get pods is taking 10-20 seconds.
  • Pods remain in ContainerCreating.
  • API server latency is increasing.
  • etcd reports slow fsync operations.

You suspect disk contention on one of the control-plane nodes.


Step 1 – Identify the affected node

First determine which master has the issue.

oc get nodes

Example:

NAME STATUS
master-0 Ready
master-1 Ready
master-2 Ready

From Grafana you notice:

master-1
etcd_disk_wal_fsync_duration_seconds
P99 = 180 ms

That immediately suggests storage latency on master-1.


Step 2 – Debug the node

Instead of SSH:

oc debug node/master-1

Output:

Creating debug namespace...
Starting pod...
To use host binaries, run:
chroot /host

Enter the host OS:

chroot /host

Now you’re inside the RHCOS operating system.


Step 3 – Check disk utilization (iostat)

First verify the disks.

lsblk

Example:

NAME SIZE
sda 300G
├─sda1
├─sda2
└─sda3

Run:

iostat -x 1 10

Example output:

Device r/s w/s rkB/s wkB/s await svctm %util aqu-sz
sda 12 520 300 9500 65.2 0.9 100.0 8.5

How to interpret
await
65 ms

Average I/O latency.

Healthy SSD:

<5 ms

Good:

<10 ms

Bad:

30-100 ms

Very bad:

>100 ms

%util
100%

Means the device is busy almost continuously.

High utilization together with high latency indicates storage saturation.


aqu-sz
8.5

Average queue depth.

Healthy:

0-1

Large values mean requests are waiting.


w/s

520 writes/sec

High write rates combined with elevated latency can affect etcd because it performs synchronous writes to its WAL.


Step 4 – Historical performance (sar)

Current activity is only part of the picture.

Run:

sar -d 1 10

Example:

DEV tps rkB/s wkB/s await
sda 480 50 9200 72

This confirms sustained storage latency.

Check CPU wait:

sar -u 1 10

Example:

CPU %user %system %iowait
all 15 12 28

%iowait

28%

Healthy:

<5%

High values indicate CPUs are idle waiting for storage.


Check system load:

sar -q 1 10

Example:

runq-sz
3
plist-sz
2100
ldavg-1
18

A high load average together with elevated I/O wait often points to storage rather than CPU as the bottleneck.


Step 5 – Find which process is causing I/O (pidstat)

Run:

pidstat -d 1 10

Example:

UID PID COMMAND kB_rd/s kB_wr/s
0 2145 etcd 0 2200
0 8120 backupd 0 6500
0 4211 rsync 0 4500
0 5001 fluentd 0 1100

Immediately you can see:

backupd
6500 KB/sec

That is a strong indication the backup process is competing with etcd for disk I/O.


Another example:

PID COMMAND
crio

Large writes from CRI-O could indicate:

  • image extraction
  • image garbage collection
  • many pod creations
  • heavy logging

Step 6 – Check etcd

Verify etcd health:

oc get pods -n openshift-etcd

Then:

oc rsh -n openshift-etcd <etcd-pod>
etcdctl endpoint health --cluster

Review logs:

oc logs -n openshift-etcd <pod> -c etcd

Typical messages:

slow fdatasync
leader changed
request timed out
apply request took too long

These correlate well with disk contention.


Step 7 – Look for filesystem errors

Run:

dmesg | grep -i error

or

journalctl -k

Look for:

I/O errors
NVMe timeout
SCSI timeout
filesystem errors

Step 8 – Check disk space

df -h
Filesystem
/
95%

Nearly full filesystems can increase write latency, especially if log or container storage is consuming most of the space.

Also check inodes:

df -i

Step 9 – Check mounted filesystems

findmnt

Confirm where /var/lib/etcd resides and whether it shares storage with other high-I/O workloads.


Step 10 – Correlate with Prometheus

Compare host observations with metrics such as:

etcd_disk_wal_fsync_duration_seconds
etcd_disk_backend_commit_duration_seconds
node_disk_io_time_seconds_total
node_disk_io_time_weighted_seconds_total

If iostat shows high latency and the etcd WAL fsync metric spikes at the same time, the storage subsystem is the likely bottleneck.


Real Production Example

Imagine every night at 02:00:

API latency increases
etcd WAL latency rises
Users cannot deploy applications
Pods remain Pending

Using pidstat:

backup-agent
12 MB/sec writes

Using iostat:

await = 140 ms
util = 100%

Using Grafana:

etcd WAL fsync
150 ms

The root cause is a scheduled backup saturating the datastore used by the control-plane nodes.

Resolution:

  • Move backups to another storage system or reschedule them.
  • Isolate etcd on dedicated low-latency SSD/NVMe storage.
  • Ensure sufficient IOPS and throughput.
  • Validate recovery by confirming WAL fsync latency, API response times, and ClusterOperator health return to normal.

Interview Answer (2 Minutes)

“When troubleshooting suspected storage-related issues in OpenShift, I start with oc debug node because direct SSH access to RHCOS nodes is often restricted. After entering the host with chroot /host, I use iostat -x to examine disk latency, queue depth, and device utilization. I’m particularly interested in await, %util, and aqu-sz, as high values indicate storage contention. Next, I use sar -d and sar -u to determine whether the problem is sustained over time and whether CPU is spending excessive time in I/O wait. Then I run pidstat -d to identify which processes are generating heavy disk activity, such as backup software, rsync, CRI-O image extraction, or logging agents. I correlate these findings with Prometheus metrics like etcd_disk_wal_fsync_duration_seconds and API latency. If they align, I’ve confirmed a storage bottleneck affecting etcd. The remediation is to eliminate competing disk workloads, provide dedicated low-latency storage for etcd, and then verify recovery by checking API latency, etcd health, and cluster operator status.”