Troubleshooting Intermittent API Latency Caused by etcd Disk Contention

The troubleshooting path is:

Slow oc/API requests
        ↓
Confirm API latency
        ↓
Correlate with etcd request latency
        ↓
Check WAL fsync and backend commit latency
        ↓
Identify the affected control-plane node
        ↓
Find competing disk activity or storage throttling
        ↓
Remove contention or move etcd to faster storage
        ↓
Validate etcd and API recovery

etcd must durably write consensus proposals to its write-ahead log before acknowledging them. Slow storage or competing processes can increase fsync duration, causing request timeouts, missed heartbeats and potentially temporary leader loss. During leader elections, API operations that change cluster state can stall. (Red Hat Documentation)

1. Confirm the API latency

First determine whether the issue affects:

• All API operations
• Only write operations
• One API server
• One control-plane node
• A particular resource type
• A particular time window

Test API readiness:

oc get --raw='/readyz?verbose'

Test simple reads:

time oc get nodes
time oc get namespaces
time oc get pods -A --request-timeout=10s

Test the API directly:

curl -k -w '\nDNS: %{time_namelookup}
Connect: %{time_connect}
TLS: %{time_appconnect}
TTFB: %{time_starttransfer}
Total: %{time_total}\n' \
  https://api.cluster.example.com:6443/readyz

This helps distinguish:

DNS or load-balancer latency
TLS connection latency
API processing latency
etcd-backed request latency

Check cluster health at the same time:

oc get clusteroperators
oc get nodes
oc get pods -n openshift-etcd -o wide
oc get pods -n openshift-kube-apiserver -o wide

Look for:

etcd Degraded=True
kube-apiserver Degraded=True
control-plane nodes NotReady
frequent etcd pod restarts
API readiness failures

2. Correlate API latency with etcd latency

Open the OpenShift console and inspect:

Observe
  → Dashboards
  → etcd

Also inspect:

Observe
  → Dashboards
  → Kubernetes / API server

The most important correlation is:

API request latency rises
            +
etcd fsync latency rises
            +
node disk latency rises

If all three rise at the same time, disk contention is a strong hypothesis.

3. Check the key etcd disk metrics

WAL fsync latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

This measures how long etcd takes to persist write-ahead-log records.

A sustained increase indicates:

• Slow underlying disks
• Storage throttling
• Storage queue saturation
• Competing writes
• Hypervisor storage contention
• Cloud disk credit exhaustion

The WAL fsync metric is one of the primary etcd metrics affected by storage I/O performance. (Red Hat Documentation)

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

This measures latency committing the etcd backend database.

Interpretation:

High WAL fsync only
→ WAL device or synchronous-write problem
High backend commit only
→ Backend database write or fragmentation pressure
Both high
→ General disk contention or storage degradation

etcd request latency

histogram_quantile(
  0.99,
  sum by (operation, type, le) (
    rate(etcd_request_duration_seconds_bucket[5m])
  )
)

Look for slow:

• PUT
• POST
• Transactions
• Range requests
• Lease operations

Write operations are usually affected most strongly by slow WAL storage.

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Frequent leader changes during disk-latency spikes suggest etcd members are missing heartbeats or taking too long to process consensus traffic.

Proposal failures

rate(etcd_server_proposals_failed_total[5m])

Also inspect pending proposals:

etcd_server_proposals_pending

A rising pending-proposal count means etcd cannot commit work as quickly as it receives it.

Database size and quota

etcd_mvcc_db_total_size_in_bytes

etcd_mvcc_db_total_size_in_use_in_bytes

etcd_server_quota_backend_bytes

Calculate reclaimable fragmented space:

(
  etcd_mvcc_db_total_size_in_bytes
-
  etcd_mvcc_db_total_size_in_use_in_bytes
) / 1024 / 1024

A large difference indicates internal fragmentation, but fragmentation and disk contention are different problems. Defragmentation may reduce database size; it does not fix fundamentally slow or saturated storage. Red Hat notes that defragmentation blocks the member while it runs, so it must be handled carefully and members must be allowed to recover between operations. (Red Hat Documentation)

4. Identify the affected etcd member

Break metrics down by instance.

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

Example:

master-0: normal
master-1: fsync spikes to 150 ms
master-2: normal

This strongly suggests that the storage attached to master-1 is the problem.

Map etcd pods to nodes:

oc get pods -n openshift-etcd \
  -l app=etcd \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase

Check which member is leader:

oc -n openshift-etcd rsh \
  $(oc get pod -n openshift-etcd \
    -l app=etcd \
    -o name | head -1) \
  etcdctl endpoint status --cluster -w table

In some OpenShift releases, the label can be k8s-app=etcd; verify with:

oc get pods -n openshift-etcd --show-labels

5. Inspect etcd health and logs

Check etcd endpoint health:

ETCD_POD=$(oc get pods -n openshift-etcd \
  -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}')
oc rsh -n openshift-etcd "$ETCD_POD" \
  etcdctl endpoint health --cluster

Check endpoint status:

oc rsh -n openshift-etcd "$ETCD_POD" \
  etcdctl endpoint status --cluster -w table

Look for:

• Slow endpoint response
• Unexpected database-size differences
• Raft index divergence
• Missing members
• A single slow member
• Frequent leadership changes

Review logs:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --since=2h

Search for:

oc logs -n openshift-etcd "$ETCD_POD" -c etcd --since=2h |
  grep -Ei \
  'slow fdatasync|slow request|took too long|leader|election|heartbeat|timeout|apply request'

Typical symptoms include messages resembling:

slow fdatasync
request timed out
leader changed
apply request took too long
failed to send out heartbeat

Also check the Operator:

oc logs -n openshift-etcd-operator \
  deployment/etcd-operator \
  --since=2h

6. Inspect host disk performance

Debug the affected control-plane node:

oc debug node/master-1

Enter the host filesystem:

chroot /host

Check block devices and mounts:

lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
findmnt /var/lib/etcd
df -h
df -i

Check disk utilization:

iostat -x 1 10

Important fields:

Warning signs include:

High await or w_await
Persistently high %util
Growing aqu-sz
Latency spikes matching API incidents

Depending on the device and kernel, %util=100 does not always prove saturation, especially for modern parallel storage. Latency and queue depth are more important.

Check historical activity:

sar -d 1 10
sar -q 1 10
sar -u 1 10

Check processes generating I/O:

pidstat -d 1 10

If installed and approved:

iotop -oPa

Do not install arbitrary tools or modify control-plane hosts during an incident unless this follows the bank’s change process and Red Hat support guidance.

7. Find the source of contention

Common causes include:

Shared-disk contention

etcd storage might share an underlying datastore with:

• VM snapshots
• Backup jobs
• Antivirus or filesystem scanning
• Monitoring agents
• Log collectors
• Container image activity
• Hypervisor migration
• Storage replication
• Other high-I/O virtual machines

A periodic latency spike at the same time each day often points to scheduled backup, snapshot, replication or scanning activity.

Cloud disk throttling

Check the cloud provider for:

• IOPS limits
• Throughput limits
• Queue depth
• Burst-credit exhaustion
• Disk latency
• Instance-level I/O limits

Increasing only the disk IOPS may not help when the VM instance itself has a lower aggregate storage throughput limit.

Virtualized storage contention

For virtualized control-plane nodes, investigate:

OpenShift node
    ↓
Virtual disk
    ↓
Datastore
    ↓
Storage controller
    ↓
Physical disks

The guest might show high latency even though its own I/O rate is low because another VM is saturating the datastore.

Correlate OpenShift data with:

• VMware datastore latency
• vSAN congestion
• SAN controller latency
• HBA queue depth
• Storage path failovers
• Multipath errors
• Hypervisor snapshots

Local filesystem pressure

Check:

journalctl -k --since "2 hours ago" |
  grep -Ei 'I/O error|timeout|reset|nvme|scsi|blk|xfs'

Check for filesystem or device errors:

dmesg -T |
  grep -Ei 'I/O error|timeout|reset|nvme|scsi|xfs'

Check system journals consuming excessive I/O:

journalctl --disk-usage

Do not manually remove /var/lib/etcd, WAL files or etcd database files.

8. Rule out network latency

Disk and network issues can produce similar etcd symptoms.

Check peer round-trip latency:

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Interpretation:

High fsync + normal peer RTT
→ Disk or storage issue
Normal fsync + high peer RTT
→ Network issue
Both high
→ Broader node, hypervisor or infrastructure contention

etcd replication performance depends on network latency, and high peer latency can trigger disruptive leader elections. (Red Hat Documentation)

9. Check whether API workload is contributing

Disk contention may be worsened by excessive API writes.

Identify expensive or high-volume clients through API metrics and audit logs.

Useful metrics include:

sum by (verb, resource) (
  rate(apiserver_request_total[5m])
)

topk(
  20,
  sum by (user_agent, verb) (
    rate(apiserver_request_total[5m])
  )
)

Look for:

• Operators stuck in fast reconciliation loops
• Controllers continuously updating status
• Excessive Events
• CI/CD repeatedly creating and deleting objects
• Monitoring systems making expensive list requests
• Large numbers of Secrets or ConfigMaps
• Broken automation generating thousands of API changes

Compare read and write pressure:

sum by (verb) (
  rate(apiserver_request_total[5m])
)

A sharp increase in POST, PUT, PATCH or DELETE operations can increase etcd write pressure.

The permanent solution may require both:

Faster isolated storage
          +
Reduction of unnecessary API writes

10. Remediation

Immediate containment

Depending on the confirmed cause:

• Stop or reschedule a competing backup or scan.
• Remove unrelated high-I/O workloads from the datastore.
• Resolve a failed storage path.
• Restore cloud disk IOPS or throughput capacity.
• Fix a runaway controller or automation loop.
• Pause nonessential bulk deployments.
• Escalate storage degradation to the infrastructure team.

Do not restart all control-plane nodes or etcd members together.

Permanent storage remediation

Use:

• Dedicated low-latency SSD or NVMe storage
• Guaranteed rather than burst-only IOPS
• Adequate throughput
• Dedicated datastore or storage policy
• No noisy-neighbour workloads
• Sufficient host-level I/O capacity
• Redundant, healthy storage paths

Red Hat recommends low-latency block storage for etcd because slow disks and other disk activity can directly increase WAL fsync latency. (Red Hat Documentation)

Database fragmentation

First determine whether automatic defragmentation is operating successfully:

oc logs -n openshift-etcd-operator \
  deployment/etcd-operator |
  grep -i defrag

Modern OpenShift releases automatically perform etcd defragmentation based on fragmentation thresholds. Manual defragmentation should not be the first response to disk contention and should follow the procedure for the exact OpenShift release. (Red Hat Documentation)

If Red Hat support or the documented procedure requires manual defragmentation:

• Take a valid etcd backup first.
• Verify quorum and endpoint health.
• Process one member at a time.
• Defragment the leader last.
• Wait for the member and cluster to recover between operations.

Defragmentation is blocking for the member on which it runs. (Red Hat Documentation)

11. Validate recovery

After remediation, compare the same incident metrics:

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

increase(etcd_server_leader_changes_seen_total[15m])

Then verify:

oc get --raw='/readyz?verbose'
oc get clusteroperators
oc get nodes
oc get pods -n openshift-etcd
oc get pods -n openshift-kube-apiserver

Success criteria:

WAL fsync latency returns to baseline
Backend commit latency returns to baseline
No new leader elections
No pending proposals
All etcd endpoints healthy
API p99 latency returns to SLO
ClusterOperators are healthy

Production Runbook Summary

Interview Answer

“I would first confirm that the latency is inside the API processing path rather than DNS, TLS or the load balancer. Then I would correlate API server latency with etcd’s WAL fsync and backend commit metrics, broken down by member. If one member shows elevated etcd_disk_wal_fsync_duration_seconds while peer network latency remains normal, that points to disk contention on that control-plane node.

I would inspect etcd endpoint health, leadership changes, pending proposals and logs for slow fdatasync, request timeouts or heartbeat failures. On the affected node, I would use oc debug node, iostat, sar and pidstat to examine disk latency, queue depth and competing processes. I would also check the cloud disk, VMware datastore or SAN layer for IOPS throttling, burst-credit exhaustion, snapshots, backup jobs or noisy neighbours.

Immediate remediation would be to stop the competing I/O or reduce excessive API writes while preserving etcd quorum. The permanent correction would be dedicated low-latency SSD or NVMe storage with guaranteed IOPS and sufficient host-level throughput. I would not restart all etcd members or manually defragment them as an initial reaction. After remediation, I would verify WAL fsync latency, backend commit latency, leader stability, endpoint health, API p99 latency and ClusterOperator status.”