Understanding cluster-restore.sh: A Full Overview

Let me explain every section in detail.

`cluster-restore.sh` — Complete Internals Explained

Invocation

			
# Run on the recovery master node only
sudo /usr/local/bin/cluster-restore.sh /home/core/backup

The backup directory must contain both files from cluster-backup.sh:

			
/home/core/backup/
  ├── snapshot_2025-03-01_143022.db
  └── static_kuberesources_2025-03-01_143022.tar.gz

Section 1 — Shell Safety + Root Check

			
set -o errexit    # any command failure = immediate exit
set -o pipefail   # catch failures inside pipes
set -o errtrace   # ERR traps inherited by functions
if [[ $EUID -ne 0 ]]; then
  echo "This script must be run as root"
  exit 1
fi

		

Same pattern as cluster-backup.sh. The restore needs root for:

Writing to /etc/kubernetes/ (manifests, certs)
Writing to /var/lib/etcd/ (etcd data dir)
Moving static pod manifests in/out of /etc/kubernetes/manifests/
Killing and restarting CRI-O containers

Section 2 — Load Environment

			
source_required_dependency \
  /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/etcd.env
source_required_dependency \
  /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/etcd-common-tools

Critical note: these are sourced from the currently on-disk files — the ones that already exist on the recovery node — NOT from the backup yet. They provide key variables:

			
# From etcd.env:
ETCD_DATA_DIR=/var/lib/etcd
ETCD_DATA_DIR_BACKUP=/var/lib/etcd-backup
CONFIG_FILE_DIR=/etc/kubernetes
MANIFEST_DIR=/etc/kubernetes/manifests
MANIFEST_STOPPED_DIR=/etc/kubernetes/manifests-stopped
RESTORE_ETCD_POD_YAML=/etc/kubernetes/static-pod-resources/etcd-certs/.../restore-etcd-pod.yaml
ETCD_REV_JSON=/var/lib/etcd/revision.json
# From etcd-common-tools (functions loaded):
dl_etcdctl()
check_snapshot_status()
mv_static_pods()
wait_for_containers_to_stop()
backup_remaining_etcd_data_dir_contents()
print_restore_completion_message()

		

Section 3 — Locate Backup Files

			
BACKUP_FILE=$(ls -vd "${BACKUP_DIR}"/static_kuberesources*.tar.gz | tail -1)
SNAPSHOT_FILE=$(ls -vd "${BACKUP_DIR}"/snapshot*.db | tail -1)

ls -v sorts version-numerically (handles timestamps correctly), then tail -1 picks the most recent file if multiple backups exist in the same directory. This means:

			
snapshot_2025-03-01_120000.db   ← older, ignored
snapshot_2025-03-01_143022.db   ← newest, SELECTED

⚠️ This is why you should only put one backup set per directory, or the restore might use the wrong snapshot.

Section 4 — Snapshot Validation

			
if [ ! -f "${SNAPSHOT_FILE}" ]; then
  echo "etcd snapshot ${SNAPSHOT_FILE} does not exist"
  exit 1
fi
dl_etcdctl
check_snapshot_status "${SNAPSHOT_FILE}"

		

Before touching anything on the system, the script:

① Downloads matching etcdctl — same version as the cluster’s etcd, from etcd-common-tools.

② Validates the snapshot hash — check_snapshot_status runs:

etcdctl snapshot status "${SNAPSHOT_FILE}" --write-out=table

Output:

			
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 1bf371f1 |   294858 |       5891 |     120 MB |
+----------+----------+------------+------------+

		

If the .db file is corrupt or truncated, the script aborts here — before any destructive action. This is the safety valve.

Section 5 — Choose Restore Mode

			
ETCD_CLIENT="${ETCD_ETCDCTL_BIN+etcdctl}"
if [ -n "${ETCD_ETCDUTL_BIN}" ]; then
  ETCD_CLIENT="${ETCD_ETCDUTL_BIN}"
fi

Two restore tools exist:

etcdctl snapshot restore — older, still works
etcdutl snapshot restore — newer standalone tool (no server connection needed)

The script prefers etcdutl if available. This affects the ETCD_ETCDCTL_RESTORE branch later.

There are also two restore paths controlled by ETCD_ETCDCTL_RESTORE:

			
ETCD_ETCDCTL_RESTORE not set (default) → Full restore via restore-etcd pod
                                          (multi-member clusters, standard DR)
ETCD_ETCDCTL_RESTORE=true              → Direct etcdctl restore
                                          (Single Node OpenShift / SNO,
                                           or re-joining a live cluster)

		

Section 6 — Stop etcd (THE POINT OF NO RETURN)

			
ETCD_STATIC_POD_LIST=("etcd-pod.yaml")
ETCD_STATIC_POD_CONTAINERS=("etcd" "etcdctl" "etcd-metrics" "etcd-readyz" "etcd-rev")
mv_static_pods "${ETCD_STATIC_POD_LIST[@]}"
wait_for_containers_to_stop "${ETCD_STATIC_POD_CONTAINERS[@]}"

This is the point of no return. Two operations happen:

① mv_static_pods — moves the etcd manifest out of the kubelet watch directory:

			
# What mv_static_pods does internally:
mv /etc/kubernetes/manifests/etcd-pod.yaml \
   /etc/kubernetes/manifests-stopped/etcd-pod.yaml

kubelet watches /etc/kubernetes/manifests/ at all times. The moment etcd-pod.yaml disappears from that directory, kubelet signals CRI-O to stop all etcd containers. The move is atomic — no partial state.

② wait_for_containers_to_stop — polls until all 5 etcd containers are truly gone:

			
# Internally polls:
crictl ps | grep -E "etcd|etcdctl|etcd-metrics|etcd-readyz|etcd-rev"
# Waits until output is empty

Why wait? The restore must not write a new data dir while etcd still has a file lock on the old one. Writing to a locked BoltDB = corruption.

			
Timeline:
  mv manifest → kubelet notices → CRI-O sends SIGTERM to containers
             → containers drain → CRI-O removes containers
             → wait_for_containers_to_stop returns
             → SAFE TO PROCEED

		

Section 7 — Back Up the Old Data Directory

			
if [ ! -d "${ETCD_DATA_DIR_BACKUP}" ]; then
  mkdir -p "${ETCD_DATA_DIR_BACKUP}"
fi
if [ -d "${ETCD_DATA_DIR}/member" ]; then
  if [ -d "${ETCD_DATA_DIR_BACKUP}/member" ]; then
    rm -rf "${ETCD_DATA_DIR_BACKUP}"/member
  fi
  mv "${ETCD_DATA_DIR}"/member "${ETCD_DATA_DIR_BACKUP}"/
fi

		

Before overwriting anything, the existing etcd data is moved — not deleted:

/var/lib/etcd/member/        → moved to → /var/lib/etcd-backup/member/

This is a safety net. If the restore fails or produces an inconsistent cluster, you can manually move it back. The old data is never irrecoverably destroyed by this script.

Section 8a — Standard Restore Path (default, multi-master)

			
if [ -z "${ETCD_ETCDCTL_RESTORE}" ]; then
  # 1. Extract static pod resources from backup
  tar -C "${CONFIG_FILE_DIR}" -xzf "${BACKUP_FILE}" static-pod-resources
  # 2. Stage the snapshot where the restore pod can find it
  cp -p "${SNAPSHOT_FILE}" "${ETCD_DATA_DIR_BACKUP}"/snapshot.db
  # 3. Move revision tracking file if it exists
  [ ! -f "${ETCD_REV_JSON}" ] || mv -f "${ETCD_REV_JSON}" "${ETCD_DATA_DIR_BACKUP}"/revision.json
  # 4. Clear anything else from /var/lib/etcd
  backup_remaining_etcd_data_dir_contents
  # 5. Drop in the restore-etcd pod manifest
  echo "starting restore-etcd static pod"
  cp -p "${RESTORE_ETCD_POD_YAML}" "${MANIFEST_DIR}/etcd-pod.yaml"
fi

		

This is the path used in real disaster recovery on a standard 3-master cluster. Here’s what each step does:

① Extract static_kuberesources.tar.gz

			
tar -C /etc/kubernetes -xzf static_kuberesources_<ts>.tar.gz static-pod-resources

This overwrites /etc/kubernetes/static-pod-resources/ with the backed-up revision:

			
/etc/kubernetes/static-pod-resources/
  ├── etcd-pod-3/          ← restored from backup
  ├── kube-apiserver-pod-7/
  ├── kube-controller-manager-pod-8/
  └── kube-scheduler-pod-6/

		

All certificates and manifests are now at the exact state from backup time. This is critical because the etcd snapshot data was encoded/signed with these exact certs.

② Stage the snapshot

cp -p "${SNAPSHOT_FILE}" /var/lib/etcd-backup/snapshot.db

The restore-etcd pod (launched in step ⑤) will read the snapshot from this fixed location. Using cp -p preserves timestamps.

③ Move revision.json

etcd revision tracking is stored in /var/lib/etcd/revision.json. The restore pod uses this to handle revision bump logic — ensuring the restored cluster starts at a revision higher than any peer, preventing Raft confusion.

④ Clear /var/lib/etcd

backup_remaining_etcd_data_dir_contents

The restore pod requires /var/lib/etcd to be completely empty before it runs — it will refuse to restore into a non-empty directory. This function moves any leftover files (fio perf artifacts, stray old snapshots) out of the way.

⑤ Start the restore-etcd static pod

cp -p "${RESTORE_ETCD_POD_YAML}" /etc/kubernetes/manifests/etcd-pod.yaml

This drops a special restore manifest into the kubelet watch directory. kubelet sees it immediately and starts the restore pod. This pod is different from the normal etcd pod — it:

			
1. Runs: etcdutl snapshot restore /var/lib/etcd-backup/snapshot.db \
           --data-dir=/var/lib/etcd \
           --name=master-0 \
           --initial-cluster=master-0=https://192.168.1.10:2380 \
           --initial-cluster-token=<unique-token> \
           --initial-advertise-peer-urls=https://192.168.1.10:2380
           
2. Performs a revision bump (writes a higher revision number)
   so this member is authoritative when other masters re-join
3. Marks etcd as compacted to clean up old revision history
4. Replaces itself with the normal etcd-pod.yaml to start real etcd

		

The revision bump is the key difference vs the ETCD_ETCDCTL_RESTORE path. On a single recovering master, you WANT to bump the revision to be ahead of any lingering peer state.

Section 8b — Direct Restore Path (`ETCD_ETCDCTL_RESTORE=true`)

			
else
  echo "removing etcd data dir..."
  rm -rf "${ETCD_DATA_DIR}"
  mkdir -p "${ETCD_DATA_DIR}"
  echo "starting snapshot restore through etcdctl..."
  if ! ${ETCD_CLIENT} snapshot restore "${SNAPSHOT_FILE}" \
       --data-dir="${ETCD_DATA_DIR}"; then
    echo "Snapshot restore failed. Aborting!"
    exit 1
  fi
  # Restore original etcd pod manifest
  mv "${MANIFEST_STOPPED_DIR}/etcd-pod.yaml" "${MANIFEST_DIR}/etcd-pod.yaml"
fi

		

Used for Single Node OpenShift (SNO) or re-joining a live cluster quorum. Key differences:

No revision bump. The script comment explains exactly why:

We are never going to rev-bump here to ensure we don’t cause a revision split between the remainder of the running cluster and this restore member. Imagine your non-restore quorum members run at rev 100, we would attempt to rev bump this with snapshot at rev 120, now this member is 20 revisions ahead and RAFT is confused.

So on SNO or re-join scenarios: restore the data, don’t bump the revision, let Raft sync naturally.

Direct etcdutl snapshot restore writes the BoltDB data directory directly — no intermediate restore pod needed.

Moves the original etcd-pod.yaml back from manifests-stopped/ to manifests/, restarting the normal etcd static pod against the freshly restored data.

Section 9 — Completion

print_restore_completion_message

Prints instructions for what to do next:

			
Restore completed successfully.
Next steps:
1. Copy this backup to all other control plane nodes
2. On each other control plane node, stop etcd:
   sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp/
3. Remove old etcd data on each other node:
   sudo rm -rf /var/lib/etcd
4. Restart kubelet on all control plane nodes:
   sudo systemctl restart kubelet
5. Wait for etcd operator to re-add members
6. Approve any pending CSRs for worker nodes

		

Complete Flow Diagram

			
cluster-restore.sh /home/core/backup
         │
         ├─ 1. Root check + shell safety
         ├─ 2. source etcd.env + etcd-common-tools
         ├─ 3. Locate newest .db and .tar.gz in backup dir
         │
         ├─ 4. VALIDATE SNAPSHOT (abort if corrupt)
         │       └── dl_etcdctl → check_snapshot_status
         │
         ├─ 5. Detect restore mode (restore-pod vs direct etcdctl)
         │
         ├─ 6. STOP ETCD ← POINT OF NO RETURN
         │       ├── mv etcd-pod.yaml → manifests-stopped/
         │       └── wait for all 5 etcd containers to exit
         │
         ├─ 7. PRESERVE OLD DATA
         │       └── mv /var/lib/etcd/member → /var/lib/etcd-backup/member
         │
         ├─ 8a. STANDARD RESTORE (default)
         │       ├── tar -xzf static_kuberesources.tar.gz → /etc/kubernetes/
         │       ├── cp snapshot.db → /var/lib/etcd-backup/snapshot.db
         │       ├── mv revision.json → /var/lib/etcd-backup/
         │       ├── clear /var/lib/etcd (must be empty)
         │       └── cp restore-etcd-pod.yaml → /etc/kubernetes/manifests/etcd-pod.yaml
         │               │
         │               └── kubelet sees new manifest
         │                       └── restore-etcd pod runs:
         │                             ├── etcdutl snapshot restore
         │                             ├── revision bump
         │                             ├── compaction
         │                             └── swap in real etcd-pod.yaml
         │
         └─ 8b. DIRECT RESTORE (SNO / re-join)
                 ├── rm -rf /var/lib/etcd && mkdir /var/lib/etcd
                 ├── etcdctl snapshot restore → /var/lib/etcd (no rev bump)
                 └── mv etcd-pod.yaml back → /etc/kubernetes/manifests/
                         └── kubelet starts normal etcd on restored data

		

Key Design Decisions — Why It Works This Way

Decision	Reason
Validate snapshot before stopping etcd	Never destroy running state for a corrupt backup
Move old data dir, don’t delete	Safety net if restore fails — rollback possible
Use a restore pod instead of running etcdctl directly	The restore pod runs inside the etcd container image, guaranteeing exact binary version match and correct env
Revision bump on multi-master	Makes the recovering master authoritative; other masters re-join and sync from it via Raft
No revision bump on SNO/re-join	Prevents Raft confusion when live peers already have higher revisions
kubelet manifest swap to trigger start	Leverages the existing static pod mechanism — no new infrastructure needed

Infra Cloud Solutions

Understanding cluster-restore.sh: A Full Overview

`cluster-restore.sh` — Complete Internals Explained

Invocation

Section 1 — Shell Safety + Root Check

Section 2 — Load Environment

Section 3 — Locate Backup Files

Section 4 — Snapshot Validation

Section 5 — Choose Restore Mode

Section 6 — Stop etcd (THE POINT OF NO RETURN)

Section 7 — Back Up the Old Data Directory

Section 8a — Standard Restore Path (default, multi-master)

Section 8b — Direct Restore Path (`ETCD_ETCDCTL_RESTORE=true`)

Section 9 — Completion

Complete Flow Diagram

Key Design Decisions — Why It Works This Way

Like this:

Related

Leave a ReplyCancel reply

cluster-restore.sh — Complete Internals Explained

Invocation

Section 1 — Shell Safety + Root Check

Section 2 — Load Environment

Section 3 — Locate Backup Files

Section 4 — Snapshot Validation

Section 5 — Choose Restore Mode

Section 6 — Stop etcd (THE POINT OF NO RETURN)

Section 7 — Back Up the Old Data Directory

Section 8a — Standard Restore Path (default, multi-master)

Section 8b — Direct Restore Path (ETCD_ETCDCTL_RESTORE=true)

Section 9 — Completion

Complete Flow Diagram

Key Design Decisions — Why It Works This Way

Share this:

Like this:

Related

Leave a ReplyCancel reply

Discover more from Infra Cloud Solutions

`cluster-restore.sh` — Complete Internals Explained

Section 8b — Direct Restore Path (`ETCD_ETCDCTL_RESTORE=true`)