WAL fsync in OpenShift

In OpenShift, WAL fsync normally refers to how quickly etcd can safely write changes to its Write-Ahead Log on disk.

A simple definition:

Before etcd confirms an important cluster-state change, it writes the change to its WAL and requests that the operating system physically persist it to storage using fsync or fdatasync.

This protects the cluster state if an etcd process, control-plane node, or operating system suddenly fails.

What is a WAL?

WAL means Write-Ahead Log.

etcd records a change in the log before applying it to its main backend database.

API change
   │
   ▼
etcd receives proposal
   │
   ▼
Write proposal to WAL
   │
   ▼
fsync to persistent storage
   │
   ▼
Replicate through Raft
   │
   ▼
Commit transaction
   │
   ▼
API request succeeds

Examples of changes written through etcd include:

• Creating a Pod
• Updating a Deployment
• Creating or changing a Secret
• Updating a ConfigMap
• Changing node status
• Updating EndpointSlices
• Operator status updates
• Creating or deleting Kubernetes resources

What does fsync do?

When an application writes data, the operating system may initially place it in memory cache.

Application write
       │
       ▼
Operating-system page cache
       │
       ▼
Storage device later

That is fast, but data still in memory can be lost during a sudden power or operating-system failure.

fsync tells the operating system:

Do not acknowledge this operation until the data
has been flushed to persistent storage.

For etcd:

WAL entry
   │
   ▼
fsync
   │
   ▼
Disk confirms persistence
   │
   ▼
etcd continues the commit

Therefore, WAL fsync latency is directly influenced by the storage system.

Why OpenShift depends on it

The Kubernetes API server stores cluster state in etcd.

oc apply
   │
   ▼
kube-apiserver
   │
   ▼
etcd leader
   │
   ▼
WAL fsync and Raft replication
   │
   ▼
Commit
   │
   ▼
API response

If WAL fsync becomes slow, etcd writes become slow. That can make the OpenShift API slow because API write operations depend on etcd committing changes.

Red Hat notes that slow storage or competing disk activity can cause high fsync latency, API slowness, request timeouts, missed heartbeats, and temporary etcd leader loss. (Red Hat Documentation)

Example of a normal write

Suppose you run:

oc scale deployment payments-api --replicas=5

The flow is:

1. oc sends PATCH request
2. API server validates and authorizes it
3. API server sends the change to etcd
4. etcd leader writes the proposal to WAL
5. WAL is synchronized to storage
6. Proposal is replicated to other etcd members
7. A majority acknowledges it
8. The change is committed
9. API server returns success

If step 5 takes 2 ms, the transaction can proceed quickly.

If step 5 takes 200 ms or more, API requests accumulate and controllers begin reconciling more slowly.

WAL vs etcd backend database

etcd uses both a WAL and a backend database.

Incoming change
      │
      ▼
WAL
Durable sequential record
      │
      ▼
Raft commit
      │
      ▼
Backend database
Current key-value state

They have separate performance metrics:

Interpretation:

High WAL fsync latency
→ synchronous WAL storage problem
High backend commit latency
→ backend database storage or database-pressure issue
Both high
→ general disk contention, throttling, or storage degradation

Main Prometheus metric

The key histogram metric is:

etcd_disk_wal_fsync_duration_seconds_bucket

To calculate the p99 latency:

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

This shows the WAL fsync duration below which approximately 99% of observations occurred during the selected period.

Breaking it down by instance helps identify one slow control-plane node:

master-0 → 0.004 seconds
master-1 → 0.120 seconds
master-2 → 0.005 seconds

Here, master-1 is the likely problem.

Red Hat identifies WAL fsync duration, backend commit duration, and leader changes as important metrics for evaluating etcd storage performance. (Red Hat Documentation)

What is an acceptable value?

The exact alert threshold can vary by OpenShift release, workload, and test method.

Current Red Hat guidance for validating etcd storage includes checking the p99 fsync result produced by its supported fio-based performance test. Some current documentation uses a threshold below 10 ms, while other recent release documentation and test output refer to 20 ms. Use the guidance and alerts supplied for your exact OpenShift version rather than applying one universal value. (Red Hat Documentation)

As a practical operational interpretation:

A few milliseconds
→ healthy low-latency storage
Consistent tens of milliseconds
→ investigate
Large or recurring spikes
→ likely to affect etcd and API performance

Focus on:

• p99 behavior
• Sustained duration
• Differences between members
• Correlation with API latency
• Leader changes and timeouts

A single isolated spike is less concerning than repeated or sustained high latency.

Causes of high WAL fsync latency

1. Slow storage

Examples:

• Mechanical disks
• Slow SAN
• High-latency network-backed block storage
• Poorly configured virtual disks
• Underperforming SSDs

2. IOPS or throughput throttling

Cloud disks and virtual machines can have:

• Disk IOPS limits
• Throughput limits
• Burst-credit exhaustion
• Instance-wide storage limits

3. Noisy neighbours

The etcd virtual disk might share physical infrastructure with:

• Other virtual machines
• Databases
• Backup jobs
• Storage replication
• Large image operations

4. Snapshots and backups

Hypervisor snapshots or storage backups can temporarily increase latency.

5. Control-plane processes producing I/O

Examples:

• Logging agents
• Security scanners
• Backup agents
• Excessive journal writes
• Container image operations

6. Device or filesystem errors

Examples:

• Storage path failures
• NVMe timeouts
• SAN multipath issues
• Filesystem problems
• Disk nearly full

Red Hat recommends low-latency block storage for etcd and advises against sharing its underlying I/O infrastructure with competing I/O-intensive workloads. (Red Hat Documentation)

Symptoms in OpenShift

High WAL fsync latency can produce:

• Slow oc commands
• API request timeouts
• Slow application deployments
• Delayed Operator reconciliation
• Pods remaining Pending longer
• ClusterOperators becoming degraded
• etcd slow fdatasync messages
• Missed Raft heartbeats
• Increased leader elections
• Web console delays

Slow disk
   │
   ▼
Slow WAL fsync
   │
   ▼
Slow etcd commits
   │
   ▼
Slow API server
   │
   ▼
Slow controllers and Operators
   │
   ▼
Cluster-wide control-plane impact

Existing application containers might continue processing traffic, but control-plane changes become slow or fail.

Related metrics

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Failed proposals

rate(etcd_server_proposals_failed_total[5m])

Peer network latency

histogram_quantile(
  0.99,
  sum by (instance, To, le) (
    rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])
  )
)

Interpret the metrics together:

High fsync + normal peer RTT
→ storage issue
Normal fsync + high peer RTT
→ network issue
High fsync + leader changes
→ storage may be destabilizing Raft
High fsync + pending proposals
→ etcd cannot commit writes fast enough

Node-level troubleshooting

Identify the slow member using Prometheus, then debug its control-plane node:

oc debug node/master-1

Enter the host:

chroot /host

Check where etcd data resides:

findmnt /var/lib/etcd
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
df -h
df -i

Check real-time disk performance:

iostat -x 1 10

Important fields:

Look for:

High w_await
Growing aqu-sz
Sustained device pressure
Spikes matching etcd latency

%util alone is not sufficient for modern parallel devices. Latency and queue depth provide better context.

Find competing processes

Use:

pidstat -d 1 10

Possible output:

PID    kB_rd/s    kB_wr/s    COMMAND
2100      0.00      800.00    etcd
7350      0.00     9000.00    backup-agent

This suggests that backup-agent may be competing with etcd.

Historical statistics:

sar -d 1 10
sar -u 1 10
sar -q 1 10

Check kernel storage errors:

journalctl -k --since "1 hour ago" |
  grep -Ei 'I/O error|timeout|reset|nvme|scsi|xfs'

Check etcd logs

List etcd Pods:

oc get pods -n openshift-etcd -o wide

Review one member:

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h

Search for likely symptoms:

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h |
grep -Ei \
  'slow fdatasync|took too long|timeout|heartbeat|leader|election'

Do not confuse WAL fsync with network replication

A write requires both disk durability and Raft consensus.

etcd leader
   │
   ├── WAL fsync on local disk
   ├── Replication to peers
   └── Majority acknowledgement

Therefore, a slow transaction can come from:

• Local disk fsync latency
• Peer disk latency
• Network round-trip latency
• CPU starvation
• etcd overload

Always compare disk and network metrics before concluding that storage is the only problem.

Remediation

Immediate

• Stop or reschedule competing backup or scanning jobs.
• Resolve storage path failures.
• Fix disk or VM throttling.
• Reduce runaway API writes.
• Pause nonessential bulk deployments.
• Remove unrelated I/O-intensive activity from control-plane storage.

Permanent

• Use dedicated low-latency SSD or NVMe-backed storage.
• Use guaranteed IOPS rather than burst-only storage.
• Ensure the VM instance supports sufficient total storage throughput.
• Isolate control-plane nodes from noisy neighbours.
• Avoid putting application workloads on control-plane nodes.
• Monitor p99 WAL fsync continuously.
• Validate storage with the Red Hat-supported fio procedure.

Do not react by restarting every etcd member. That could destroy quorum.

Also, do not use defragmentation as the first solution: it may reduce internal database fragmentation, but it does not repair slow physical storage.

Interview answer

WAL fsync in OpenShift is the time etcd takes to durably persist a Raft proposal into its Write-Ahead Log. When an API operation changes cluster state, the API server sends it to etcd. The etcd leader records the proposal in the WAL, synchronizes it to persistent storage, replicates it to the other members, and commits it after a majority acknowledges the proposal.

Because fsync waits for storage durability, slow storage directly increases etcd transaction latency and therefore OpenShift API latency. Sustained high fsync latency can cause request timeouts, pending proposals, missed Raft heartbeats, and leader changes. I monitor the p99 value of etcd_disk_wal_fsync_duration_seconds_bucket, compare it across members, and correlate it with backend commit latency, peer network RTT, leader changes, and API latency. At the node level, I use oc debug node, iostat, sar, and pidstat to identify disk latency, queueing, throttling, or competing processes. The permanent solution is isolated, low-latency storage with guaranteed IOPS—not repeatedly restarting etcd or treating defragmentation as a disk-performance fix.