Essential OpenShift Commands for etcd Troubleshooting

techhadoop OCP

Below is a practical OpenShift oc command checklist for etcd troubleshooting. Most commands require cluster-admin.

1. Check overall cluster health

oc get clusterversion
oc get clusteroperators

Focus on these operators:

oc get co etcd kube-apiserver authentication

Healthy state:

AVAILABLE   PROGRESSING   DEGRADED
True        False         False

Get detailed etcd Operator conditions:

oc describe co etcd

Or extract only the conditions:

oc get co etcd \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"  "}{.message}{"\n"}{end}'

2. Check control-plane nodes

oc get nodes -l node-role.kubernetes.io/master -o wide

On newer clusters, also try:

oc get nodes -l node-role.kubernetes.io/control-plane -o wide

Check node conditions:

oc describe node <control-plane-node>

Compact view:

oc get nodes \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,NETWORK:.status.conditions[?(@.type=="NetworkUnavailable")].status'

A three-member etcd cluster needs a majority—normally at least two healthy members—to make progress.

3. Check etcd pods

oc get pods -n openshift-etcd -o wide

Useful label-based view:

oc get pods -n openshift-etcd -l app=etcd -o wide

Watch for:

oc get pods -n openshift-etcd -w

Check restarts and container readiness:

oc get pods -n openshift-etcd \
  -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,READY:.status.containerStatuses[*].ready'

Describe a failing pod:

oc describe pod <etcd-pod> -n openshift-etcd

4. Check etcd endpoint health

Select one healthy etcd pod:

ETCD_POD=$(oc get pods -n openshift-etcd \
  -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}')
echo "$ETCD_POD"

Run endpoint health:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint health --cluster -w table

Expected result: every endpoint reports true or “is healthy.”

Red Hat documents etcdctl endpoint health and endpoint status as primary checks for etcd health and consensus latency. (Red Hat Documentation)

5. Check endpoint status, leader and database size

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster -w table

This shows:

  • member ID
  • etcd version
  • database size
  • leader status
  • Raft term and index
  • applied index
  • errors

JSON output:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster -w json

Check only the database size:

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster \
  -w fields

Compare the RAFT INDEX and RAFT APPLIED INDEX values across members. A member significantly behind the others may have disk, network, or synchronization problems.

6. Check etcd membership

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl member list -w table

Confirm that:

  • All expected control-plane members exist.
  • Each member is started.
  • Peer and client URLs match the correct nodes.
  • No stale or duplicate member remains.

Do not run etcdctl member remove during normal diagnosis. Member removal is a recovery operation and should follow the documented unhealthy-member replacement procedure. (Red Hat Documentation)

7. Check etcd alarms

oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl alarm list

A common critical alarm is:

NOSPACE

When etcd exhausts its quota, writes can fail. Red Hat recommends treating low-space alerts urgently rather than merely clearing the alarm. (Red Hat Documentation)

Do not blindly run:

etcdctl alarm disarm

First correct the underlying database-size, quota, compaction, or storage problem.

8. Review etcd logs

Current logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --tail=200

Follow logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd -f

Previous crashed container:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --previous

Logs from all etcd pods:

for pod in $(oc get pods -n openshift-etcd \
  -l app=etcd -o name); do
    echo "===== $pod ====="
    oc logs -n openshift-etcd "$pod" -c etcd \
      --since=30m 2>&1 |
      grep -Ei 'error|warn|timeout|leader|slow|unhealthy|corrupt|space|fsync'
done

Important messages include:

leader changed
leader failed
request timed out
context deadline exceeded
apply request took too long
failed to send out heartbeat
database space exceeded
wal
corrupt

9. Check static-pod revisions

etcd runs as static pods on control-plane nodes.

oc get pods -n openshift-etcd \
  -l app=etcd \
  -L revision

Check the revision-pruner and installer pods:

oc get pods -n openshift-etcd | grep -E 'installer|revision-pruner'

Check etcd Operator configuration:

oc get kubeapiserver cluster -o yaml
oc get etcd cluster -o yaml

Check etcd resource conditions:

oc describe etcd cluster

Compact conditions:

oc get etcd cluster \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"  "}{.reason}{"  "}{.message}{"\n"}{end}'

10. Check recent events

oc get events -n openshift-etcd \
  --sort-by=.metadata.creationTimestamp

Cluster-wide warning events:

oc get events -A \
  --field-selector type=Warning \
  --sort-by=.metadata.creationTimestamp

Recent events only:

oc get events -n openshift-etcd \
  --sort-by=.lastTimestamp | tail -30

11. Check etcd-related alerts

oc get prometheusrules -A | grep -i etcd

Query active alerts through the monitoring API:

oc -n openshift-monitoring exec \
  prometheus-k8s-0 -c prometheus -- \
  curl -s 'http://localhost:9090/api/v1/alerts'

Filter with jq:

oc -n openshift-monitoring exec \
  prometheus-k8s-0 -c prometheus -- \
  curl -s 'http://localhost:9090/api/v1/alerts' |
jq -r '
  .data.alerts[]
  | select(.labels.alertname | test("etcd|Etcd"; "i"))
  | [.labels.alertname, .state, .annotations.message]
  | @tsv

Typical alerts include:

etcdMembersDown
etcdInsufficientMembers
etcdHighNumberOfLeaderChanges
etcdNoLeader
etcdHighFsyncDurations
etcdHighCommitDurations
etcdDatabaseQuotaLowSpace

12. Query important etcd metrics

Create access to Prometheus:

oc -n openshift-monitoring port-forward \
  svc/prometheus-k8s 9090:9090

Then query locally using an authenticated method appropriate for your environment, or use the OpenShift web console metrics page.

Useful PromQL queries:

Leader changes
increase(etcd_server_leader_changes_seen_total[15m])
Members without a leader
etcd_server_has_leader

Expected value:

1
WAL fsync latency p99
histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)
Backend commit latency p99
histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)
Database size
etcd_mvcc_db_total_size_in_bytes
Quota usage ratio
etcd_mvcc_db_total_size_in_bytes
/
etcd_server_quota_backend_bytes

etcd is highly sensitive to storage latency. Slow fsync operations can cause missed heartbeats, proposal delays, API timeouts, and temporary leader loss. (Red Hat Documentation)

13. Inspect the control-plane host

Start a debug shell:

oc debug node/<control-plane-node>

Enter the host filesystem:

chroot /host

Check disk utilization:

df -h
df -i

Check the etcd data directory:

du -sh /var/lib/etcd
du -sh /var/lib/etcd/member/*

Check block devices:

lsblk
findmnt /var/lib/etcd

Check I/O pressure:

iostat -xz 1 10

Check CPU, memory and load:

uptime
free -h
vmstat 1 10

Check kernel storage errors:

journalctl -k --since "1 hour ago" |
grep -Ei 'error|timeout|reset|nvme|scsi|blk|I/O'

Check kubelet and CRI-O:

journalctl -u kubelet --since "1 hour ago"
journalctl -u crio --since "1 hour ago"

Exit:

exit
exit

14. Check network connectivity between etcd members

etcd normally uses:

  • TCP 2379: client traffic
  • TCP 2380: peer communication

From a control-plane debug session:

nc -vz <other-control-plane-ip> 2379
nc -vz <other-control-plane-ip> 2380

Check listening sockets:

ss -lntp | grep -E ':2379|:2380'

Look for packet loss or latency between control-plane nodes:

ping -c 10 <other-control-plane-ip>

In production, also verify firewalls, security groups, load balancers, MTU, and the underlying network path.

15. Check API latency symptoms

Because the Kubernetes API stores its persistent state in etcd, etcd latency often appears first as API slowness.

time oc get nodes
time oc get pods -A >/dev/null
time oc get co

Check API server pods:

oc get pods -n openshift-kube-apiserver -o wide

Check API server logs for etcd timeouts:

oc logs -n openshift-kube-apiserver \
  <kube-apiserver-pod> -c kube-apiserver \
  --since=30m |
grep -Ei 'etcd|timeout|deadline|slow'

16. Verify backups

Locate a control-plane node:

oc get nodes -l node-role.kubernetes.io/master

Open a debug shell:

oc debug node/<control-plane-node>
chroot /host

Create a supported backup:

/usr/local/bin/cluster-backup.sh /home/core/assets/backup

Verify files:

ls -lh /home/core/assets/backup

Typical output includes:

snapshot_*.db
static_kuberesources_*.tar.gz

Take one cluster backup from one healthy control-plane node—not a separate backup from every member. Backup and restore procedures must follow the documentation for the exact OpenShift release. (Red Hat Documentation)

Fast troubleshooting workflow

API slow or unavailable
        |
        v
oc get co etcd
        |
        v
Check control-plane nodes and etcd pods
        |
        v
etcdctl endpoint health --cluster
        |
        v
endpoint status + member list + alarm list
        |
        v
Check leader changes and fsync latency
        |
        v
Inspect disk, I/O and network on each control-plane node
        |
        v
Review etcd and kube-apiserver logs
        |
        v
Take backup and must-gather before invasive recovery

Compact command bundle

oc get co etcd
oc get etcd cluster
oc get nodes -l node-role.kubernetes.io/master -o wide
oc get pods -n openshift-etcd -l app=etcd -o wide
ETCD_POD=$(oc get pods -n openshift-etcd \
  -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint health --cluster -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl endpoint status --cluster -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl member list -w table
oc exec -n openshift-etcd -c etcd "$ETCD_POD" -- \
  etcdctl alarm list
oc logs -n openshift-etcd "$ETCD_POD" \
  -c etcd --since=30m
oc get events -n openshift-etcd \
  --sort-by=.metadata.creationTimestamp

Before member removal, manual defragmentation, quota changes, or snapshot restoration, collect diagnostics:

oc adm must-gather

Those operations can affect quorum or temporarily block a member and should not be treated as routine diagnostic commands.

Leave a Reply