OpenShift: In-Depth Guide to cluster-backup.sh Script

techhadoop OCP

Here’s a deep-dive explanation of every part of the script.

cluster-backup.sh — Complete Internals Explained

The script lives at /usr/local/bin/cluster-backup.sh on every control plane node. It is maintained as a component of the etcd Cluster Operator and is a wrapper around the etcdctl snapshot save command. Here’s every line of it dissected.

Invocation
/usr/local/bin/cluster-backup.sh [--force] <path-to-backup-dir>
# Example
/usr/local/bin/cluster-backup.sh /home/core/backup

It produces two output files:

/home/core/backup/snapshot_2025-03-01_143022.db
/home/core/backup/static_kuberesources_2025-03-01_143022.tar.gz
Section 1 — Shell Safety Flags
set -o errexit   # exit immediately if any command fails
set -o pipefail  # catch failures inside pipes (cmd1 | cmd2)
set -o errtrace  # ERR traps are inherited by functions/subshells

These three together mean: any error anywhere = immediate exit. No silent failures.

Section 2 — Root Check
if [[ $EUID -ne 0 ]]; then
  echo "This script must be run as root"
  exit 1
fi

The script needs access to:

  • /etc/kubernetes/ — TLS certificates and manifests (root-only)
  • /var/lib/etcd/ — etcd data directory (root-only)
  • etcd’s mTLS-protected API endpoint

So it must run as root. On RHCOS you use sudo or oc debug node/....

Section 3 — --force Flag
IS_DIRTY=""
if [ "$1" == "--force" ]; then
  IS_DIRTY="__POSSIBLY_DIRTY__"
  shift
fi

--force skips the operator-health checks (explained below). It marks the output files with __POSSIBLY_DIRTY__ in their filename as a warning:

snapshot__POSSIBLY_DIRTY__.db
static_kuberesources__POSSIBLY_DIRTY__.tar.gz

This is a deliberate signal: “this backup was taken while the cluster was in an uncertain state.” Never use --force backups for production restores unless it’s your only option.

Section 4 — Backup Directory Setup
BACKUP_DIR="$1"
DATESTRING=$(date "+%F_%H%M%S")
BACKUP_TAR_FILE=${BACKUP_DIR}/static_kuberesources_${DATESTRING}${IS_DIRTY}.tar.gz
SNAPSHOT_FILE="${BACKUP_DIR}/snapshot_${DATESTRING}${IS_DIRTY}.db"
trap 'rm -f ${BACKUP_TAR_FILE} ${SNAPSHOT_FILE}' ERR

The trap is critical: if anything fails mid-run, the partial/corrupt output files are automatically deleted. You’ll never be left with a half-written snapshot that looks valid but isn’t.

Section 5 — Load Environment & Certificates
source_required_dependency \
  /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/etcd.env
source_required_dependency \
  /etc/kubernetes/static-pod-resources/etcd-certs/configmaps/etcd-scripts/etcd-common-tools

source_required_dependency is a small guard function that verifies the file exists before sourcing it:

function source_required_dependency {
  local src_path="$1"
  if [ ! -f "${src_path}" ]; then
    echo "required dependencies not found, please ensure this script is run
          on a node with a functional etcd static pod"
    exit 1
  fi
  source "${src_path}"
}

What does etcd.env contain?

It exports the environment variables etcdctl needs to connect to the local etcd member via mTLS:

# Inside etcd.env (approximate content):
export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt
export ETCDCTL_CERT=/etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-master-0.crt
export ETCDCTL_KEY=/etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-master-0.key
export NODE_NODE_ENVVAR_NAME_IP=192.168.1.10   # this node's IP
export CONFIG_FILE_DIR=/etc/kubernetes

What does etcd-common-tools contain?

Shared shell functions used by both backup and restore scripts, including:

  • dl_etcdctl — downloads the correct etcdctl binary
  • check_snapshot_status — validates the snapshot after saving
Section 6 — Certificate Path Fallback
if [ ! -f "${ETCDCTL_CACERT}" ]; then
  echo "Certificate ${ETCDCTL_CACERT} is missing. Checking in different directory"
  export ETCDCTL_CACERT=$(echo ${ETCDCTL_CACERT} | sed -e "s|static-pod-certs|static-pod-resources/etcd-certs|")
  export ETCDCTL_CERT=$(echo ${ETCDCTL_CERT}   | sed -e "s|static-pod-certs|static-pod-resources/etcd-certs|")
  export ETCDCTL_KEY=$(echo ${ETCDCTL_KEY}     | sed -e "s|static-pod-certs|static-pod-resources/etcd-certs|")
  ...
fi

Why is this needed?

The default cert paths in etcd.env point to static-pod-certs/ — a symlink that exists inside the running etcd container. When running the script directly on the host (outside the container), those symlinks don’t resolve. This block remaps the paths to the actual host-side directory static-pod-resources/etcd-certs/, where the same certs are stored.

Inside container:  /etc/kubernetes/static-pod-certs/...
On host directly:  /etc/kubernetes/static-pod-resources/etcd-certs/...
                   (same certs, different path)
Section 7 — Back Up Static Pod Resources
backup_latest_kube_static_resources "${BACKUP_TAR_FILE}"

This is the tar.gz part of the backup. The function:

function backup_latest_kube_static_resources {
  local backup_tar_file="$1"
  local backup_resource_list=("kube-apiserver" "kube-controller-manager" "kube-scheduler" "etcd")
  local latest_resource_dirs=()
  for resource in "${backup_resource_list[@]}"; do
    # 1. Verify the static pod manifest exists
    if [ ! -f "/etc/kubernetes/manifests/${resource}-pod.yaml" ]; then
      echo "error finding manifests for the ${resource} pod."
      exit 1
    fi
    # 2. Find which revision is currently active
    local latest_resource
    latest_resource=$(grep -o -m 1 \
      "/etc/kubernetes/static-pod-resources/${resource}-pod-[0-9]*" \
      "/etc/kubernetes/manifests/${resource}-pod.yaml")
    # 3. Check the operator isn't mid-rollout (unless --force)
    if [ "${IS_DIRTY}" == "" ]; then
      check_if_operator_is_progressing "${resource}"
    fi
    latest_resource_dirs+=("${latest_resource#${CONFIG_FILE_DIR}/}")
  done
  # 4. tar all four resource dirs together
  tar -cpzf "$backup_tar_file" -C "${CONFIG_FILE_DIR}" "${latest_resource_dirs[@]}"
  chmod 600 "$backup_tar_file"
}

Step by step:

① Verify manifests exist — if any of the 4 control plane pods don’t have a manifest file at /etc/kubernetes/manifests/, the node’s kubelet isn’t managing them and the script aborts.

② Find the latest revision — OCP uses a revision-based rollout system. Each time a static pod is updated, a new numbered directory is created:

/etc/kubernetes/static-pod-resources/
  etcd-pod-1/          ← old revision
  etcd-pod-2/          ← old revision
  etcd-pod-3/          ← CURRENT (referenced by manifest)
  kube-apiserver-pod-7/
  kube-controller-manager-pod-8/
  kube-scheduler-pod-6/

The script greps the active manifest to find which numbered revision is in use. This ensures you back up the revision that is actually running, not a stale one.

③ Check operator not progressing — calls:

function check_if_operator_is_progressing {
  local operator="$1"
  if [ ! -f "${KUBECONFIG}" ]; then
    echo "Valid kubeconfig is not found. Exiting!"
    exit 1
  fi
  progressing=$(oc get co "${operator}" \
    -o jsonpath='{.status.conditions[?(@.type=="Progressing")].status}')
  if [ "$progressing" != "False" ]; then
    echo "Currently the $operator operator is progressing.
          A reliable backup requires that a rollout is not in progress. Aborting!"
    exit 1
  fi
}

If any of the 4 operators is mid-rollout (Progressing=True), a backup taken now would capture a half-applied state — inconsistent between etcd data and on-disk manifests. The script refuses unless --force is passed.

④ tar with permissions — packages all 4 resource directories together. chmod 600 restricts the tar to root-only, since it may contain TLS private keys and encryption keys (if etcd encryption at rest is enabled).

What exactly is inside the tar?

static-pod-resources/etcd-pod-3/
  ├── etcd-pod.yaml                  ← pod manifest
  ├── secrets/
  │   ├── etcd-all-certs/            ← peer, server, client certs + keys
  │   └── etcd-metric-client/
  └── configmaps/
      ├── etcd-ca/                   ← CA cert bundle
      └── etcd-scripts/             ← etcd.env, common-tools
static-pod-resources/kube-apiserver-pod-7/
  ├── kube-apiserver-pod.yaml
  ├── secrets/                       ← API server TLS certs + service account keys
  └── configmaps/                    ← kubeconfigs, audit policy, etc.
static-pod-resources/kube-controller-manager-pod-8/  ...
static-pod-resources/kube-scheduler-pod-6/  ...
Section 8 — Download etcdctl
dl_etcdctl

This function (from etcd-common-tools) downloads the etcdctl binary that matches the exact etcd version running in the cluster. It doesn’t use a system-installed etcdctl to avoid version mismatch.

The binary is extracted from the running etcd container image itself via crictl or directly from the image layers, ensuring perfect version alignment.

Section 9 — Take the etcd Snapshot
ETCDCTL_ENDPOINTS="https://${NODE_NODE_ENVVAR_NAME_IP}:2379" \
  etcdctl snapshot save "${SNAPSHOT_FILE}"

This is the core operation. It:

  1. Connects to the local etcd member only (not the cluster endpoint) — important because it avoids redirects and ensures you’re snapshotting from a member that is up
  2. Uses the mTLS certs from etcd.env to authenticate
  3. Calls etcd’s built-in snapshot API — which does a consistent read of the entire BoltDB key-value store
  4. Writes the raw BoltDB file to snapshot_<timestamp>.db

Why local endpoint (NODE_IP:2379) and not a load-balanced endpoint?

  • The snapshot is a point-in-time consistent read — it needs to come from one member
  • etcd’s Raft protocol ensures the member has all committed entries before snapshotting
  • Using a VIP/LB might land on a different member per retry, causing inconsistency

What’s inside the .db file?

It’s a raw BoltDB database file containing every key-value pair in etcd:

/registry/pods/default/my-app-xyz
/registry/deployments/production/frontend
/registry/secrets/kube-system/bootstrap-token-xyz
/registry/clusterroles/cluster-admin
... (every Kubernetes object ever created)

If etcd encryption is enabled, the snapshot also contains the encryption keys for the etcd snapshot data.

Section 10 — Validate the Snapshot
check_snapshot_status "${SNAPSHOT_FILE}"
snapshot_failed=$?
if [[ $snapshot_failed -eq 1 ]]; then
  echo "snapshot failed with exit code ${snapshot_failed}"
  exit 1
fi

check_snapshot_status (from etcd-common-tools) runs:

etcdctl snapshot status "${SNAPSHOT_FILE}" --write-out=table

Which verifies the snapshot’s internal hash and outputs something like:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 1bf371f1 |   294858 |       5891 |     120 MB |
+----------+----------+------------+------------+

The cluster-backup.sh script confirms the snapshot’s validity. If the hash doesn’t match the data, the file is corrupt and gets deleted by the trap.

Complete Flow Summary
cluster-backup.sh /home/core/backup
        │
        ├─ 1. Root check
        ├─ 2. Parse --force flag
        ├─ 3. Create backup dir
        ├─ 4. source etcd.env  → loads ETCDCTL_CACERT/CERT/KEY + NODE_IP
        ├─ 5. source etcd-common-tools  → loads dl_etcdctl, check_snapshot_status
        ├─ 6. Fix cert paths if running outside container
        │
        ├─ 7. backup_latest_kube_static_resources()
        │       ├─ For each of 4 static pods:
        │       │     ├─ Verify manifest exists
        │       │     ├─ Find current revision number
        │       │     └─ Check operator not Progressing
        │       └─ tar -czf static_kuberesources_<ts>.tar.gz (chmod 600)
        │
        ├─ 8. dl_etcdctl  → downloads matching etcdctl binary
        ├─ 9. etcdctl snapshot save  → writes snapshot_<ts>.db
        └─ 10. check_snapshot_status  → validates hash
                    ├─ OK  → "snapshot db and kube resources are successfully saved"
                    └─ FAIL → trap deletes both files, exits 1
Important Operational Rules
RuleWhy
Run on one master onlyetcd snapshot is cluster-wide; duplicates waste space
Wait 24h after install before first backupThe first certificate rotation happens 24 hours after installation; taking a backup before it will contain expired certificates
Run during off-peak hoursAn etcd snapshot has a high I/O cost
Never restore if API is still upIf you can retrieve data using the Kubernetes API server, etcd is available and you should not restore using an etcd backup — it takes a cluster back in time and all clients will experience a conflicting, parallel history
Backup before upgradesYou must use an etcd backup taken from the same z-stream release — for example, a 4.17.5 cluster must use a backup taken from 4.17.5
Store .tar.gz separately if encryption is onIf etcd encryption is enabled, it is recommended to store the static resources file separately from the etcd snapshot for security reasons — however, this file is required to restore from the etcd snapshot

Leave a Reply