etcd Operator in OpenShift

The etcd Operator manages the lifecycle, configuration, health, certificates, and membership of the etcd cluster that stores OpenShift’s control-plane state.

In a standard highly available OpenShift cluster, etcd runs on the three control-plane nodes:

                  OpenShift API
                       │
                       ▼
                kube-apiserver
                       │
                       ▼
                etcd cluster
          ┌────────────┼────────────┐
          ▼            ▼            ▼
      master-0      master-1      master-2
       etcd-0        etcd-1        etcd-2
          ▲            ▲            ▲
          └────────────┼────────────┘
                       │
                  etcd Operator

The etcd Operator continually observes the cluster, compares the current state with the required state, and corrects differences through the Kubernetes and etcd management APIs. (Red Hat Documentation)

Why etcd Is Critical

etcd is the authoritative database for Kubernetes and OpenShift.

It stores objects such as:

• Deployments
• Pods and their desired state
• Services
• Routes
• Secrets
• ConfigMaps
• RBAC
• Nodes
• CRDs and Custom Resources
• Operator configuration
• MachineConfig objects
• Cluster configuration

The runtime contents of containers and application databases are not stored in etcd.

oc apply -f deployment.yaml
             │
             ▼
       kube-apiserver
             │
             ▼
            etcd
             │
             ▼
 Deployment object stored
             │
             ▼
 Controllers create Pods

If etcd becomes unavailable, existing containers can often continue running temporarily, but:

• New Pods cannot be scheduled.
• Configuration changes cannot be saved.
• Operators cannot reconcile normally.
• oc commands that require the API begin failing.
• Cluster recovery and automation stop functioning correctly.

etcd Operator vs etcd

These are different components:

Cluster Version Operator
          │
          ▼
      etcd Operator
          │
          ▼
      etcd members
          │
          ▼
   Cluster state database

The Operator itself does not store the cluster state. It manages the etcd processes that do.

Location and Resources

The Operator normally runs in:

openshift-etcd-operator

The etcd static Pods run in:

openshift-etcd

Check them:

oc get pods -n openshift-etcd-operator
oc get pods -n openshift-etcd -o wide

Check the ClusterOperator:

oc get clusteroperator etcd

The cluster-scoped configuration resource is:

oc get etcd cluster -o yaml

The etcd cluster Operator provides the cluster-scoped etcds.operator.openshift.io API and is configured through the etcd/cluster object. (Red Hat Documentation)

Main Responsibilities of the etcd Operator

1. Deploying etcd as static Pods

On each control-plane node, etcd runs as a static Pod.

Static Pod manifest
       │
       ▼
kubelet on master node
       │
       ▼
etcd Pod starts

Typical Pods:

oc get pods -n openshift-etcd -o wide

Example:

etcd-master-0
etcd-master-1
etcd-master-2

Static Pods are managed directly by the kubelet, not by a Deployment.

This is important because core control-plane services must be able to start even when normal Kubernetes scheduling is unavailable.

2. Maintaining etcd membership

A three-member etcd cluster normally has:

Member 1: master-0
Member 2: master-1
Member 3: master-2

The Operator monitors whether the expected members match the available control-plane nodes.

When a control-plane node is properly replaced, the Operator can:

• Generate certificates for the new member
• Add the replacement member to etcd
• Remove stale membership
• Reconcile the new topology

Red Hat documents that when a lost control-plane node is replaced, the etcd cluster Operator handles generating new TLS certificates and adding the new node as an etcd member. (Red Hat Documentation)

3. Preserving quorum

etcd uses the Raft consensus algorithm.

For three members:

Members: 3
Required quorum: 2
Maximum simultaneous failures: 1

For five members:

Members: 5
Required quorum: 3
Maximum simultaneous failures: 2

A standard OpenShift control plane normally uses three members.

master-0    master-1    master-2
 Healthy     Healthy     Failed
      \         /
       Quorum remains

If two of three members are lost:

master-0    master-1    master-2
 Healthy     Failed      Failed
      │
      ▼
  No quorum

The Operator cannot simply recreate lost authoritative state when quorum is gone. You must follow the documented disaster-recovery procedure and restore from a valid backup. Red Hat explicitly distinguishes single-member replacement from loss of the majority of control-plane hosts. (Red Hat Documentation)

4. Managing certificates

etcd communication is secured with TLS.

Certificates include:

• Peer certificates for member-to-member communication
• Server certificates
• Client certificates for API server access
• Certificate authority bundles

etcd-0  ←── mutual TLS ──→  etcd-1
   │                           │
   └────── mutual TLS ────────→ etcd-2

The Operator manages the certificate resources and rolls out new static-Pod revisions when certificates rotate.

It also ensures the kube-apiserver has the required trust and client credentials to connect to etcd securely.

5. Managing static-Pod revisions

Configuration changes are rolled out using versioned revisions.

Current revision 20
        │
 Configuration changes
        ▼
New revision 21 generated
        │
        ▼
Install on control-plane nodes
        │
        ▼
Validate member health

You can inspect revision-related resources:

oc get configmaps -n openshift-etcd
oc get secrets -n openshift-etcd

The Operator ensures the expected configuration, certificates, and manifests are synchronized across the control-plane nodes.

6. Monitoring cluster health

The Operator monitors:

• Member availability
• Quorum
• Endpoint health
• Static-Pod revisions
• Certificate status
• Member synchronization
• Leader stability
• Backup and defragmentation-related conditions
• Storage and API-visible health signals

Check its high-level condition:

oc get co etcd

A healthy status is:

AVAILABLE     True
PROGRESSING   False
DEGRADED      False

Detailed conditions:

oc describe co etcd

7. Supporting member recovery

If one etcd member fails but quorum remains, the recovery process depends on the failure type:

• The control-plane machine is stopped.
• The node is NotReady.
• The etcd Pod is crash-looping.
• The underlying machine was permanently lost.
• The certificates are invalid.

The Operator can reconcile the member when a temporarily unavailable node returns. For permanent loss, replacement must follow the supported procedure.

Red Hat recommends taking an etcd backup before replacing an unhealthy member. (Red Hat Documentation)

8. Automating defragmentation and maintenance

etcd receives a high number of small updates and deletions. Deleted data can leave unused space inside the backend database.

Objects created and updated
          │
          ▼
Objects deleted
          │
          ▼
Unused internal database pages
          │
          ▼
Fragmentation

The etcd Operator performs supported maintenance activities, including automatic defragmentation behavior in current OpenShift releases.

However:

Defragmentation is not a fix for slow physical storage.

If WAL fsync latency is high because the disk is saturated, the solution is usually faster or isolated storage, not repeated defragmentation.

Reconciliation Loop

The etcd Operator follows the normal Operator control-loop model:

Observe control-plane nodes and etcd state
                     │
                     ▼
       Read desired configuration
                     │
                     ▼
 Compare desired state with actual state
                     │
             ┌───────┴────────┐
             │                │
          Matches          Difference
             │                │
             ▼                ▼
         No change      Reconcile resources
                              │
               ┌──────────────┼──────────────┐
               ▼              ▼              ▼
          Update static   Rotate certs   Fix membership
          Pod revision
                              │
                              ▼
                       Validate health
                              │
                              ▼
                       Update status

Examples that trigger reconciliation include:

• A control-plane node is replaced.
• A certificate needs rotation.
• A new OpenShift release changes etcd.
• Static-Pod configuration differs.
• An expected member is missing.
• A member returns after temporary failure.

How an API Request Uses etcd

For example:

oc apply -f app.yaml

The flow is:

oc client
    │
    ▼
API load balancer
    │
    ▼
kube-apiserver
    │
    ├── Authentication
    ├── Authorization
    ├── Admission
    └── Validation
           │
           ▼
          etcd
           │
           ▼
  Persist Kubernetes object

The etcd Operator is not in the request data path. It ensures the etcd cluster receiving the request remains healthy and correctly configured.

etcd Leader and Followers

One etcd member acts as the Raft leader.

                etcd leader
                    │
             Replicates writes
          ┌─────────┴─────────┐
          ▼                   ▼
      follower             follower

A write is committed after a majority acknowledges it:

API write
   │
   ▼
Leader writes WAL
   │
   ├── replicate to follower 1
   └── replicate to follower 2
             │
             ▼
      Majority acknowledges
             │
             ▼
         Commit write

This is why etcd requires:

• Low-latency storage
• Reliable networking
• Low latency between control-plane nodes
• Accurate time synchronization
• Stable control-plane resources

etcd continuously persists many small changes, making fast, low-latency I/O especially important. (Red Hat Documentation)

Useful Troubleshooting Commands

Check ClusterOperator status

oc get co etcd
oc describe co etcd

Check etcd Operator

oc get pods -n openshift-etcd-operator

oc logs -n openshift-etcd-operator \
  deployment/etcd-operator \
  --since=1h

Check etcd Pods

oc get pods -n openshift-etcd -o wide

Inspect Pod containers

oc describe pod -n openshift-etcd <etcd-pod>

oc logs -n openshift-etcd \
  <etcd-pod> \
  -c etcd \
  --since=1h

Check configuration

oc get etcd cluster -o yaml

Check nodes

oc get nodes
oc describe node <control-plane-node>

Checking Endpoint Health

First identify the etcd Pods:

oc get pods -n openshift-etcd --show-labels

Then enter a healthy etcd Pod:

oc rsh -n openshift-etcd <etcd-pod>

Depending on the OpenShift version and container environment, run the provided etcdctl command with the appropriate certificates:

etcdctl endpoint health --cluster
etcdctl endpoint status --cluster -w table

The status output helps identify:

• Member ID
• Endpoint
• etcd version
• Database size
• Leader
• Raft term and index
• Errors

Use the commands and certificate paths documented for the exact OpenShift version rather than inventing or replacing TLS parameters manually.

Important etcd Metrics

WAL fsync latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
  )
)

High values indicate slow synchronous writes.

Backend commit latency

histogram_quantile(
  0.99,
  sum by (instance, le) (
    rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])
  )
)

Leader changes

increase(etcd_server_leader_changes_seen_total[15m])

Pending proposals

etcd_server_proposals_pending

Database size

etcd_mvcc_db_total_size_in_bytes

These help distinguish:

High WAL latency
      → Storage issue
High peer RTT
      → Network issue
Frequent leader changes
      → Storage, network, or resource instability
Increasing pending proposals
      → etcd cannot process writes quickly enough

Common Failure Scenarios

Scenario 1: One member is unavailable

Three members
   │
   ├── Two healthy
   └── One failed

Result:

• Quorum remains.
• API usually continues functioning.
• The Operator reports degradation.
• Investigate and replace or recover the unhealthy member using the supported procedure.

Do not immediately delete etcd data or membership manually.

Scenario 2: Two members are unavailable

Three members
   │
   ├── One healthy
   └── Two failed

Result:

• Quorum is lost.
• Writes stop.
• API availability is severely affected.
• Normal Operator reconciliation cannot restore the authoritative state.
• Perform control-plane disaster recovery from a valid etcd snapshot.

Scenario 3: etcd Pod is CrashLoopBackOff

Check:

oc describe pod -n openshift-etcd <pod>
oc logs -n openshift-etcd <pod> -c etcd --previous

Possible causes:

• Corrupt or unavailable storage
• Certificate failure
• Invalid member state
• Static-Pod revision problem
• Disk full
• File permissions
• Network or peer connectivity
• Node-level failure

Scenario 4: Slow API caused by etcd

Symptoms:

• Slow oc commands
• API timeouts
• Operators become degraded
• Leader changes
• Slow fdatasync messages

Investigate:

oc debug node/<master-node>
chroot /host
iostat -x 1 10
sar -d 1 10
pidstat -d 1 10
df -h
df -i

Correlate node storage metrics with etcd WAL fsync and backend commit latency.

Backups

The Operator manages etcd operation, but the administrator must maintain a tested backup strategy.

A control-plane backup contains:

etcd snapshot
       +
static Kubernetes resources

Run the documented backup script from a healthy control-plane node and copy the resulting files to secure off-cluster storage.

Backups should be:

• Automated
• Encrypted
• Stored off-cluster
• Access controlled
• Tested regularly
• Matched to documented recovery procedures

An etcd snapshot does not replace application database or persistent-volume backups.

What Not to Do

Avoid:

• Deleting /var/lib/etcd
• Manually editing static-Pod manifests
• Manually removing etcd members without the supported procedure
• Restarting all control-plane nodes together
• Restarting all etcd members simultaneously
• Copying a data directory between members
• Restoring a snapshot into a live healthy cluster
• Treating defragmentation as the first fix for disk contention
• Editing Operator-managed resources directly

Unsafe etcd changes can cause permanent loss of cluster state.

Interview Answer

The etcd Operator manages the OpenShift control-plane etcd cluster. etcd itself stores the authoritative Kubernetes state, while the Operator deploys and maintains the etcd static Pods, certificates, configuration revisions, cluster membership and health. It continuously compares the desired state with the actual state and reconciles differences.

In a standard highly available cluster, etcd runs as three members on the control-plane nodes and requires two members for quorum. If one member fails, the cluster can continue operating while the member is recovered or replaced. If the majority is lost, the Operator cannot recreate the missing state, and the cluster must be restored using the documented disaster-recovery process and a valid etcd snapshot.

For troubleshooting, I begin with oc get co etcd, inspect the Operator conditions and logs, check the etcd static Pods and endpoint health, verify control-plane nodes, and examine WAL fsync, backend commit, peer latency, leader changes and pending proposals. I also verify disk latency and capacity because etcd depends on fast, low-latency storage.