OpenShift Disaster Recovery: Restore Control Plane Accurately

techhadoop Uncategorized

This is one of the hardest OpenShift interview questions. It tests whether you understand etcd, static pods, Machine Config Operator, Operators, backups, and disaster recovery.

Scenario: All three control plane (master) nodes are lost due to a datacenter failure, storage corruption, or accidental deletion. Worker nodes still exist, but the Kubernetes API is unavailable.

A strong answer should emphasize that the critical asset is the etcd backup. Without a valid etcd backup, you cannot restore the cluster’s Kubernetes state (objects such as Deployments, Secrets, ConfigMaps, CRDs, Routes, etc.).

Recovery Strategy Overview

                Disaster
                    │
                    ▼
        All Control Plane Lost
                    │
                    ▼
     Provision New Control Plane VMs
                    │
                    ▼
      Install Matching RHCOS Version
                    │
                    ▼
 Restore etcd from Snapshot + Static Pod Resources
                    │
                    ▼
 API Server Available Again
                    │
                    ▼
 Control Plane Operators Recover
                    │
                    ▼
 Workers Reconnect
                    │
                    ▼
 Applications Recover

Step 1 – Assess the Failure

First determine:

  • Is etcd data still available?
  • Were all control plane VMs lost?
  • Are worker nodes still running?
  • Is the load balancer intact?
  • Is the infrastructure (DNS, storage, networking) still available?

Remember:

Workers do not contain the cluster state.

The authoritative cluster state is stored in etcd.

Step 2 – Verify Backups

OpenShift supports backing up:

Platform

  • etcd snapshot
  • static pod resources (required alongside the snapshot)

Applications

  • OADP/Velero
  • CSI snapshots
  • Database-native backups

A typical backup includes:

etcd snapshot
+
static_kuberesources.tar.gz

The static Kubernetes resources archive contains the manifests and certificates needed by the control plane.

Step 3 – Rebuild the Infrastructure

Provision replacement control plane nodes with:

  • Same OpenShift version
  • Same RHCOS version
  • Similar CPU/RAM sizing
  • Correct networking
  • Same DNS names (or update infrastructure accordingly)

Example:

master-1
master-2
master-3

Step 4 – Restore the First Control Plane Node

You restore the cluster from one control plane node first.

Typical high-level process:

  • Boot RHCOS.
  • Place the etcd snapshot and static resources on the node.
  • Use the documented OpenShift restore procedure (cluster-restore.sh in supported versions) to restore etcd and recreate the static control plane components.
  • Start the control plane services.

This restores:

  • Kubernetes objects
  • Secrets
  • ConfigMaps
  • CRDs
  • RBAC
  • Routes
  • Operator state

Step 5 – Recover the API Server

Once etcd is restored:

etcd
↓
API Server
↓
Controller Manager
↓
Scheduler

The API should become available again.

Verify:

oc get nodes

Initially, only the restored control plane may appear Ready.

Step 6 – Rejoin Remaining Control Plane Nodes

Provision the remaining control plane nodes so they join the restored cluster.

OpenShift rebuilds:

  • kube-apiserver
  • controller-manager
  • scheduler
  • etcd members

One node at a time until the control plane regains quorum and high availability.

Step 7 – Cluster Operators Reconcile

After the API is available, Operators begin reconciling automatically.

Examples:

  • Authentication Operator
  • DNS Operator
  • Monitoring Operator
  • Network Operator
  • Ingress Operator
  • Image Registry Operator
  • Machine Config Operator

Check:

oc get co

The goal is:

Available=True
Progressing=False
Degraded=False

Step 8 – Worker Nodes Reconnect

Worker kubelets continuously attempt to reconnect.

Worker
↓
API Server Restored
↓
TLS Authentication
↓
Node Ready

Verify:

oc get nodes

Step 9 – Restore Applications (If Needed)

If persistent storage or application data was also lost:

Restore using:

  • OADP / Velero
  • CSI snapshots
  • Database backups

Typical order:

  1. Storage
  2. Databases
  3. Stateful applications
  4. Stateless applications

Step 10 – Validate the Cluster

Check:

oc get clusterversion
oc get co
oc get mcp
oc get nodes
oc get pods -A

Confirm:

  • All Operators healthy
  • No degraded MachineConfigPools
  • Nodes Ready
  • Applications running
  • Routes responding

What Happens Internally?

When etcd is restored:

etcd Snapshot
↓
API Objects
↓
Deployments
↓
ReplicaSets
↓
Pods
↓
Services
↓
Routes

Kubernetes controllers and Operators recreate the runtime state from the restored desired state.

If Workers Were Running During the Outage

During API downtime:

  • Existing containers generally continue running.
  • kubelets continue managing local pods.
  • No new scheduling occurs.
  • No configuration changes can be applied.
  • Controllers cannot reconcile state.

After the API returns:

API Available
↓
Workers Reconnect
↓
Status Updated
↓
Normal Scheduling Resumes

What If etcd Is Lost and No Backup Exists?

This is effectively a cluster rebuild.

You can recreate:

  • Control plane
  • Worker nodes
  • Operators

However, you cannot recover Kubernetes objects such as:

  • Deployments
  • Secrets
  • ConfigMaps
  • Routes
  • CRDs
  • RBAC
  • Persistent resource definitions

Applications must be redeployed from GitOps, manifests, Helm charts, or other deployment artifacts, and application data must come from separate backups.

Best Practices

  • Back up etcd regularly and verify the backups.
  • Store etcd snapshots off-cluster and off-site.
  • Back up the required static pod resources together with the snapshot.
  • Use OADP/Velero for application-level backup and recovery.
  • Test disaster recovery procedures periodically in a non-production environment.
  • Maintain infrastructure as code (Terraform/Ansible) to rebuild the underlying infrastructure consistently.
  • Use GitOps (for example, Argo CD) so application manifests can be redeployed quickly after platform recovery.

Interview Answer (2-Minute Version)

“If all control plane nodes are lost, my first priority is to recover the Kubernetes control plane from a valid etcd snapshot and the associated static pod resources. I would provision replacement control plane nodes running the same OpenShift and RHCOS versions, restore the etcd snapshot on the first control plane node using Red Hat’s documented restore procedure, and bring the API server back online. After the API is available, I would add the remaining control plane nodes back into the cluster to restore high availability. The Cluster Operators then reconcile the platform automatically, and worker nodes reconnect using their existing kubelet certificates. If application storage was also lost, I would restore it separately using OADP/Velero, CSI snapshots, or database-native backups. Finally, I’d validate ClusterOperators, MachineConfigPools, node health, and application functionality. Without a valid etcd backup, the platform must be rebuilt and Kubernetes objects cannot be recovered, so regular tested backups are essential.”

Leave a Reply