Leader Changes and Pending Proposals in OpenShift etcd

Both metrics describe the stability and performance of the etcd cluster, which stores OpenShift control-plane state.

kube-apiserver
      │
      ▼
etcd leader
      │
      ├── Writes WAL locally
      ├── Replicates proposal to followers
      └── Waits for quorum

A healthy etcd cluster should have:

• A stable leader
• Very few unexpected leader changes
• Pending proposals normally close to zero
• Low WAL fsync latency
• Low peer-network latency

1. What is an etcd leader?

etcd uses the Raft consensus algorithm. In a standard three-member OpenShift etcd cluster:

master-0: etcd leader
master-1: etcd follower
master-2: etcd follower

The leader handles cluster-state writes.

For example:

oc scale deployment payments --replicas=5

The write process is:

API server sends update
        │
        ▼
etcd leader creates proposal
        │
        ▼
Leader writes proposal to WAL
        │
        ▼
Proposal replicated to followers
        │
        ▼
Majority acknowledges
        │
        ▼
Proposal committed
        │
        ▼
API request succeeds

The leader keeps followers synchronized and commits a write only after quorum acknowledges it. (Red Hat Documentation)

2. What is a leader change?

A leader change occurs when the current leader stops being leader and another etcd member is elected.

Before:
master-0 = leader
master-1 = follower
master-2 = follower
Leader heartbeat lost
        │
        ▼
Election timeout reached
        │
        ▼
New election
After:
master-0 = follower/unavailable
master-1 = leader
master-2 = follower

Leader election is a normal high-availability mechanism. The problem is not an occasional leader change during planned maintenance; the problem is frequent or unexpected elections.

Why does the leader change?

Common causes include:

Slow WAL fsync

The leader cannot persist its Raft log quickly enough.

Slow disk
   │
   ▼
Slow WAL fsync
   │
   ▼
Heartbeat processing delayed
   │
   ▼
Followers assume leader failed
   │
   ▼
New election

High network latency or packet loss

Followers do not receive heartbeats in time.

Leader heartbeat
       │
       X packet loss or delay
       │
Follower election timeout
       │
       ▼
Leader election

CPU or memory starvation

The etcd process cannot schedule enough CPU time to send or process heartbeats.

Control-plane node restart

A reboot or static-pod restart can trigger a legitimate election.

Storage or node failure

Examples include:

• Cloud-disk throttling
• SAN congestion
• Datastore latency
• Failed storage path
• Hypervisor pause
• Node hardware failure

Slow storage and competing disk activity can cause long fsync times, missed heartbeats, request timeouts, and temporary leader loss. (Red Hat Documentation)

Leader-change metric

Use:

etcd_server_leader_changes_seen_total

This is a cumulative counter. To see recent changes:

increase(etcd_server_leader_changes_seen_total[15m])

Interpretation:

0 during normal operation
→ Stable leader
1 during a planned master reboot
→ Usually expected
Repeated changes in a short period
→ Investigate immediately

Because each member may expose the counter, inspect it by instance:

sum by (instance) (
  increase(etcd_server_leader_changes_seen_total[15m])
)

Impact of frequent leader changes

During an election, etcd briefly has no active leader.

Leader lost
    │
    ▼
Election in progress
    │
    ▼
Writes temporarily pause
    │
    ▼
API write requests wait or time out

Symptoms can include:

• Slow oc apply, oc create, or oc delete
• API request timeouts
• Operators reconciling slowly
• Delayed node status updates
• Pods taking longer to create
• ClusterOperators becoming degraded
• request timed out messages
• Temporary control-plane instability

During leader loss and reelection, Kubernetes API requests that cause state changes can be interrupted or delayed. (Red Hat Documentation)

3. What is an etcd proposal?

A proposal is a requested change to etcd state that must pass through Raft consensus.

Examples include:

Create Deployment
Update Secret
Delete Pod
Modify ConfigMap
Update Node status
Change Route
Update Operator status

Simplified flow:

API write
   │
   ▼
Proposal created
   │
   ▼
WAL persisted
   │
   ▼
Replicated to followers
   │
   ▼
Quorum reached
   │
   ▼
Proposal committed

4. What are pending proposals?

A pending proposal is a proposal that etcd has received but has not yet committed.

The metric is:

etcd_server_proposals_pending

This is a gauge showing the current number of outstanding proposals.

Normally:

Pending proposals ≈ 0

Brief small increases during bursts can be normal.

A sustained or rising value indicates that etcd is receiving changes faster than it can persist, replicate, and commit them.

Incoming proposals
        │
        ▼
etcd processing capacity insufficient
        │
        ▼
Queue grows
        │
        ▼
Pending proposals increase

Why do proposals remain pending?

Slow disk writes

Proposal
   │
   ▼
Waiting for WAL fsync
   │
   ▼
Proposal remains pending

Slow peer replication

Leader sends proposal
       │
       ▼
Follower response delayed
       │
       ▼
Waiting for quorum

Leader election

Proposals can pause while a new leader is elected.

Excessive Kubernetes API writes

A runaway Operator or automation process may generate more writes than etcd can process.

Examples:

• Controller updating status continuously
• CI/CD loop repeatedly creating resources
• Excessive Kubernetes Events
• Large bulk deployments
• Frequent ConfigMap or Secret updates
• Broken automation repeatedly patching objects

CPU pressure

The etcd member cannot process requests promptly.

Oversized API objects

Large objects require more disk, network, and serialization work.

Pending-proposal interpretation

etcd_server_proposals_pending

Example:

0–a few, briefly
→ Usually normal during a write burst
Continuously above zero
→ etcd is falling behind
Steadily increasing
→ Severe processing, disk, or network bottleneck

Also check failed proposals:

rate(etcd_server_proposals_failed_total[5m])

Interpret together:

Pending rising, failures zero
→ Requests are delayed but may eventually commit
Pending rising, failures rising
→ etcd cannot successfully process part of the workload

Relationship between the two metrics

Leader changes and pending proposals often appear together.

Slow disk or network
        │
        ├── WAL commits slow
        │       └── Pending proposals rise
        │
        └── Heartbeats delayed
                └── Leader changes rise

Example:

WAL fsync p99:         180 ms
Pending proposals:     75
Leader changes:        6 in 15 minutes
API requests:          timing out

This strongly suggests etcd instability, often caused by disk or network latency.

Important correlation matrix

Metrics to examine together

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Failed proposals

rate(etcd_server_proposals_failed_total[5m])

WAL fsync p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

Backend commit p99

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Peer RTT p99

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Red Hat identifies WAL fsync duration, backend commit latency, leader changes, and peer RTT as important etcd performance signals. (Red Hat Documentation)

Troubleshooting procedure

Step 1: Check etcd health

oc get co etcd
oc describe co etcd
oc get pods -n openshift-etcd -o wide

Look for:

• Degraded conditions
• Pod restarts
• One unhealthy member
• Revision rollout problems

Step 2: Identify the current leader

oc get pods -n openshift-etcd -o wide

Enter a healthy etcd Pod and run:

etcdctl endpoint status --cluster -w table

The output shows which member is the leader. Use the supported certificate environment and command procedure for your OCP version.

Step 3: Check etcd logs

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h |
grep -Ei \
  'leader|election|heartbeat|slow fdatasync|timeout|proposal|took too long'

Look for messages indicating:

leader changed
lost leader
elected leader
failed to send heartbeat
slow fdatasync
request timed out
apply request took too long

Step 4: Check disk performance

On the affected control-plane node:

oc debug node/<master-node>
chroot /host

Then run:

iostat -x 1 10
sar -d 1 10
pidstat -d 1 10
df -h
df -i

Check:

• await and w_await
• aqu-sz
• Storage utilization
• Competing backup or logging processes
• Disk capacity and inodes
• Hypervisor or cloud-disk throttling

Step 5: Check network latency

Compare peer round-trip latency between etcd members.

Also investigate:

• Packet loss
• Firewall state
• MTU mismatch
• NIC errors
• Hypervisor networking
• Cross-site or cross-zone latency
• Network jitter

Step 6: Check excessive API writers

Use API metrics:

sum by (verb, resource) (
  rate(apiserver_request_total[5m])
)

Find high-volume clients:

topk(
  20,
  sum by (user_agent, verb) (
    rate(apiserver_request_total[5m])
  )
)

Look for unexpected increases in:

POST
PUT
PATCH
DELETE

Example production incident

02:00 backup starts on control-plane datastore
             │
             ▼
Disk write latency increases
             │
             ▼
WAL fsync rises to 150 ms
             │
             ├── Proposals queue
             │       └── pending proposals = 60
             │
             └── Heartbeats delayed
                     └── 4 leader elections
             │
             ▼
API becomes slow

Resolution:

• Stop or reschedule the competing backup.
• Move control-plane storage to isolated low-latency disks.
• Verify cloud or datastore IOPS and throughput.
• Confirm pending proposals return near zero.
• Confirm no new leader changes.
• Verify API latency and ClusterOperator health.

What not to do

Avoid:

• Restarting all etcd members simultaneously
• Restarting all control-plane nodes together
• Manually deleting etcd data
• Removing members without the supported procedure
• Changing election timers as the first fix
• Defragmenting repeatedly to solve physical disk latency

OpenShift uses validated etcd timer values for each platform. Changing timers can hide symptoms rather than correct the underlying disk or network problem. (Red Hat Documentation)

Interview answer

An etcd leader change occurs when the current Raft leader becomes unavailable or fails to deliver heartbeats within the election timeout, causing another member to be elected. Occasional leader changes during maintenance can be expected, but frequent changes indicate instability caused by disk fsync latency, network delay, packet loss, CPU starvation, or control-plane node failures. I monitor this using increase(etcd_server_leader_changes_seen_total[15m]).

A pending proposal is an etcd write request that has been accepted but has not yet been committed through Raft consensus. The metric etcd_server_proposals_pending should normally stay close to zero. A sustained increase means etcd cannot persist or replicate writes as fast as they arrive, commonly because of slow WAL storage, high peer latency, resource pressure, or excessive API write activity.

I correlate both metrics with WAL fsync latency, backend commit latency, peer RTT, proposal failures, API request rates, and etcd logs. If leader changes and pending proposals increase together, I treat it as a serious control-plane performance problem and investigate the affected member’s storage, network, CPU, and competing processes.