Purpose
This script is a pre-flight gate before triggering an OCP upgrade. It runs 7 sequential checks and blocks the upgrade if any critical condition is detected. Think of it as a checklist that a senior SRE would run manually — automated.
1. Verify Active CLI Session & Cluster Admin Access
2. Check ClusterVersion Operator (CVO) Status
What is the CVO?
The Cluster Version Operator is the top-level operator that manages the OCP version and coordinates all upgrades. If it’s unhealthy or busy, no upgrade should start.
What the jsonpath queries extract:
The clusterversion object has a status.conditions array:
yaml
status: conditions: - type: Available status: "True" ← extracted by first query - type: Progressing status: "False" ← extracted by second query - type: Degraded status: "False"
The condition check:
Available | Progressing | Meaning | Action |
|---|---|---|---|
True | False | CVO healthy and idle | ✅ OK to upgrade |
True | True | Upgrade already running | ❌ Block |
False | False | CVO degraded/broken | ❌ Block |
False | True | Upgrade running AND broken | ❌ Block |
Equivalent manual check:
oc get clusterversion# NAME VERSION AVAILABLE PROGRESSING SINCE STATUS# version 4.14.12 True False 5d Cluster version is 4.14.12
3. Audit Core Cluster Operators
4. Check MachineConfigPools (MCP)
What are MCPs?
MachineConfigPools define groups of nodes and the configuration applied to them. During an upgrade, the MCP controller drains and reboots each node to apply the new RHCOS and MachineConfig.
master MCP → controls all 3 master nodesworker MCP → controls all worker nodesinfra MCP → controls infra nodes (if defined)
Degraded MCP check:
A degraded MCP means at least one node in the pool failed to apply its MachineConfig — it didn’t reboot correctly, got stuck, or had a rendering error. Upgrading on top of a degraded MCP compounds the problem.
5. Check Node Readiness
6. Audit PodDisruptionBudgets (PDB) for Potential Deadlocks
What is a PDB?
A PodDisruptionBudget is a policy that limits how many pods of an application can be voluntarily disrupted (evicted) at once:
7. Check Active Critical Alerts
What oc get alerts does:
This uses the OpenShift alerts API resource — a custom OCP resource that surfaces Prometheus AlertManager alerts via the Kubernetes API:
oc get alerts -A# NAMESPACE NAME STATE SEVERITY AGE# openshift-* etcdHighNumberOfFailed firing critical 5m ← caught# openshift-* NodeNotReady pending warning
The 2>/dev/null || echo "":oc get alerts is not available on all OCP versions or configurations. The 2>/dev/null suppresses errors and || echo "" ensures FIRING_CRITICALS is empty (not unset) if the command fails — avoiding the -u unset variable trap.
Why critical alerts block an upgrade:
A firing critical alert means something is actively broken in the cluster. Upgrading on top of an existing critical condition risks:
- Making a broken component worse during its own operator-driven upgrade
- Masking the original problem behind upgrade noise
- A critical alert like
etcdMemberDownmeans your etcd quorum is at risk — the worst time to upgrade
Complete flow:
START
│
├─ [1/7] oc whoami → HARD EXIT if not logged in
├─ [2/7] CVO Available+Idle → ERRORS++ if degraded or progressing
├─ [3/7] Cluster Operators → ERRORS++ if any Degraded or Unavailable
├─ [4/7] MachineConfigPools → ERRORS++ if Degraded / WARN if Paused
├─ [5/7] Node Readiness → ERRORS++ if any NotReady
├─ [6/7] PDB Deadlocks → WARN only (no ERRORS++)
└─ [7/7] Critical Alerts → ERRORS++ if any firing
│
├─ ERRORS == 0 → exit 0 (safe to upgrade)
└─ ERRORS > 0 → exit 1 (do not upgrade)
The script :
#!/usr/bin/env bash## OCP Pre-Upgrade Health Check Automation Script# Validates cluster readiness before initiating an OpenShift platform update.#set -euo pipefail# ANSI Color Codes for Scannable OutputRED='\030[0;31m'GREEN='\033[0;32m'YELLOW='\033[1;33m'NC='\033[0m' # No ColorERRORS=0echo -e "${YELLOW}====================================================${NC}"echo -e "${YELLOW} OpenShift Pre-Upgrade Automated Health Check ${NC}"echo -e "${YELLOW}====================================================${NC}\n"# 1. Verify Active CLI Session & Cluster Admin Accessecho -n "[1/7] Checking OpenShift CLI authentication... "if ! oc whoami &>/dev/null; then echo -e "${RED}[FAILED]${NC} Not logged into an OpenShift cluster. Run 'oc login' first." exit 1fiecho -e "${GREEN}[OK]${NC} Authenticated as $(oc whoami)"# 2. Check ClusterVersion Operator (CVO) Statusecho -n "[2/7] Checking ClusterVersion status... "CVO_STATUS=$(oc get clusterversion -o jsonpath='{.items[0].status.conditions[?(@.type=="Available")].status}')CVO_PROGRESSING=$(oc get clusterversion -o jsonpath='{.items[0].status.conditions[?(@.type=="Progressing")].status}')if [[ "$CVO_STATUS" == "True" && "$CVO_PROGRESSING" == "False" ]]; then echo -e "${GREEN}[OK]${NC} CVO is Available and idle."else echo -e "${RED}[FAILED]${NC} CVO is degraded or an update is already in progress." ((ERRORS++))fi# 3. Audit Core Cluster Operatorsecho "[3/7] Auditing Cluster Operators state..."DEGRADED_OPS=$(oc get clusteroperator -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Degraded" and .status=="True")) | .metadata.name')UNAVAILABLE_OPS=$(oc get clusteroperator -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Available" and .status=="False")) | .metadata.name')if [[ -n "$DEGRADED_OPS" ]]; then echo -e " ${RED}✗ Degraded Operators found:${NC}\n$DEGRADED_OPS" ((ERRORS++))else echo -e " ${GREEN}✓ Zero Degraded operators.${NC}"fiif [[ -n "$UNAVAILABLE_OPS" ]]; then echo -e " ${RED}✗ Unavailable Operators found:${NC}\n$UNAVAILABLE_OPS" ((ERRORS++))else echo -e " ${GREEN}✓ All operators are Available.${NC}"fi# 4. Check MachineConfigPools (MCP)echo "[4/7] Validating MachineConfigPools (MCP)..."DEGRADED_MCP=$(oc get mcp -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Degraded")].status=="True")].metadata.name}')PAUSED_MCP=$(oc get mcp -o jsonpath='{.items[?(@.spec.paused==true)].metadata.name}')if [[ -n "$DEGRADED_MCP" ]]; then echo -e " ${RED}✗ Degraded MachineConfigPools:${NC} $DEGRADED_MCP" ((ERRORS++))else echo -e " ${GREEN}✓ All MachineConfigPools healthy.${NC}"fiif [[ -n "$PAUSED_MCP" ]]; then echo -e " ${YELLOW}! Warning: Paused MachineConfigPools detected:${NC} $PAUSED_MCP"fi# 5. Check Node Readinessecho "[5/7] Checking Node status across cluster..."NOT_READY_NODES=$(oc get nodes --no-headers | awk '$2 != "Ready" {print $1}')if [[ -n "$NOT_READY_NODES" ]]; then echo -e " ${RED}✗ Nodes in NotReady state:${NC}\n$NOT_READY_NODES" ((ERRORS++))else echo -e " ${GREEN}✓ All nodes report Ready status.${NC}"fi# 6. Audit PodDisruptionBudgets (PDB) for Potential Deadlocksecho "[6/7] Checking PodDisruptionBudgets (PDBs) for drain lock risks..."DEADLOCKED_PDBS=$(oc get pdb -A -o json | jq -r '.items[] | select(.status.disruptionsAllowed == 0) | "\(.metadata.namespace)/\(.metadata.name)"')if [[ -n "$DEADLOCKED_PDBS" ]]; then echo -e " ${YELLOW}! PDBs currently allowing 0 disruptions (May block node drains):${NC}" echo "$DEADLOCKED_PDBS" | sed 's/^/ /'else echo -e " ${GREEN}✓ No blocking PDBs detected.${NC}"fi# 7. Check Active Critical Alertsecho "[7/7] Checking Prometheus Alerts..."FIRING_CRITICALS=$(oc get alerts -A -o json 2>/dev/null | jq -r '.items[] | select(.status.state=="firing" and .labels.severity=="critical") | .labels.alertname' || echo "")if [[ -n "$FIRING_CRITICALS" ]]; then echo -e " ${RED}✗ Firing Critical Alerts:${NC}\n$FIRING_CRITICALS" ((ERRORS++))else echo -e " ${GREEN}✓ Zero firing critical alerts.${NC}"fi# Final Summaryecho -e "\n${YELLOW}====================================================${NC}"if [[ $ERRORS -eq 0 ]]; then echo -e "${GREEN} SUCCESS: Cluster passed all health checks. Safe to proceed with upgrade.${NC}" echo -e "${YELLOW}====================================================${NC}" exit 0else echo -e "${RED} FAILURE: Found $ERRORS issue(s). Resolve all errors before upgrading.${NC}" echo -e "${YELLOW}====================================================${NC}" exit 1fi
