OpenShift(OCP) Pre-Upgrade Health Check Script

Purpose

This script is a pre-flight gate before triggering an OCP upgrade. It runs 7 sequential checks and blocks the upgrade if any critical condition is detected. Think of it as a checklist that a senior SRE would run manually — automated.

1. Verify Active CLI Session & Cluster Admin Access

2. Check ClusterVersion Operator (CVO) Status

What is the CVO?
The Cluster Version Operator is the top-level operator that manages the OCP version and coordinates all upgrades. If it’s unhealthy or busy, no upgrade should start.

What the jsonpath queries extract:

The clusterversion object has a status.conditions array:

yaml

			
status:
  conditions:
    - type: Available
      status: "True"       ← extracted by first query
    - type: Progressing
      status: "False"      ← extracted by second query
    - type: Degraded
      status: "False"

		

The condition check:

`Available`	`Progressing`	Meaning	Action
`True`	`False`	CVO healthy and idle	✅ OK to upgrade
`True`	`True`	Upgrade already running	❌ Block
`False`	`False`	CVO degraded/broken	❌ Block
`False`	`True`	Upgrade running AND broken	❌ Block

Equivalent manual check:

			
oc get clusterversion
# NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
# version   4.14.12   True        False         5d      Cluster version is 4.14.12

3. Audit Core Cluster Operators

4. Check MachineConfigPools (MCP)

What are MCPs?

MachineConfigPools define groups of nodes and the configuration applied to them. During an upgrade, the MCP controller drains and reboots each node to apply the new RHCOS and MachineConfig.

			
master MCP → controls all 3 master nodes
worker MCP → controls all worker nodes
infra MCP  → controls infra nodes (if defined)

Degraded MCP check:
A degraded MCP means at least one node in the pool failed to apply its MachineConfig — it didn’t reboot correctly, got stuck, or had a rendering error. Upgrading on top of a degraded MCP compounds the problem.

5. Check Node Readiness

6. Audit PodDisruptionBudgets (PDB) for Potential Deadlocks

What is a PDB?

A PodDisruptionBudget is a policy that limits how many pods of an application can be voluntarily disrupted (evicted) at once:

7. Check Active Critical Alerts

What oc get alerts does:

This uses the OpenShift alerts API resource — a custom OCP resource that surfaces Prometheus AlertManager alerts via the Kubernetes API:

			
oc get alerts -A
# NAMESPACE    NAME                    STATE    SEVERITY   AGE
# openshift-*  etcdHighNumberOfFailed  firing   critical   5m  ← caught
# openshift-*  NodeNotReady            pending  warning

The 2>/dev/null || echo "":
oc get alerts is not available on all OCP versions or configurations. The 2>/dev/null suppresses errors and || echo "" ensures FIRING_CRITICALS is empty (not unset) if the command fails — avoiding the -u unset variable trap.

Why critical alerts block an upgrade:

A firing critical alert means something is actively broken in the cluster. Upgrading on top of an existing critical condition risks:

Making a broken component worse during its own operator-driven upgrade
Masking the original problem behind upgrade noise
A critical alert like etcdMemberDown means your etcd quorum is at risk — the worst time to upgrade

Complete flow:

START
│
├─ [1/7] oc whoami → HARD EXIT if not logged in
├─ [2/7] CVO Available+Idle → ERRORS++ if degraded or progressing
├─ [3/7] Cluster Operators → ERRORS++ if any Degraded or Unavailable
├─ [4/7] MachineConfigPools → ERRORS++ if Degraded / WARN if Paused
├─ [5/7] Node Readiness → ERRORS++ if any NotReady
├─ [6/7] PDB Deadlocks → WARN only (no ERRORS++)
└─ [7/7] Critical Alerts → ERRORS++ if any firing
│
├─ ERRORS == 0 → exit 0 (safe to upgrade)
└─ ERRORS > 0 → exit 1 (do not upgrade)

The script :

			
#!/usr/bin/env bash
#
# OCP Pre-Upgrade Health Check Automation Script
# Validates cluster readiness before initiating an OpenShift platform update.
#
set -euo pipefail
# ANSI Color Codes for Scannable Output
RED='\030[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
ERRORS=0
echo -e "${YELLOW}====================================================${NC}"
echo -e "${YELLOW}   OpenShift Pre-Upgrade Automated Health Check     ${NC}"
echo -e "${YELLOW}====================================================${NC}\n"
# 1. Verify Active CLI Session & Cluster Admin Access
echo -n "[1/7] Checking OpenShift CLI authentication... "
if ! oc whoami &>/dev/null; then
    echo -e "${RED}[FAILED]${NC} Not logged into an OpenShift cluster. Run 'oc login' first."
    exit 1
fi
echo -e "${GREEN}[OK]${NC} Authenticated as $(oc whoami)"
# 2. Check ClusterVersion Operator (CVO) Status
echo -n "[2/7] Checking ClusterVersion status... "
CVO_STATUS=$(oc get clusterversion -o jsonpath='{.items[0].status.conditions[?(@.type=="Available")].status}')
CVO_PROGRESSING=$(oc get clusterversion -o jsonpath='{.items[0].status.conditions[?(@.type=="Progressing")].status}')
if [[ "$CVO_STATUS" == "True" && "$CVO_PROGRESSING" == "False" ]]; then
    echo -e "${GREEN}[OK]${NC} CVO is Available and idle."
else
    echo -e "${RED}[FAILED]${NC} CVO is degraded or an update is already in progress."
    ((ERRORS++))
fi
# 3. Audit Core Cluster Operators
echo "[3/7] Auditing Cluster Operators state..."
DEGRADED_OPS=$(oc get clusteroperator -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Degraded" and .status=="True")) | .metadata.name')
UNAVAILABLE_OPS=$(oc get clusteroperator -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Available" and .status=="False")) | .metadata.name')
if [[ -n "$DEGRADED_OPS" ]]; then
    echo -e "  ${RED}✗ Degraded Operators found:${NC}\n$DEGRADED_OPS"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ Zero Degraded operators.${NC}"
fi
if [[ -n "$UNAVAILABLE_OPS" ]]; then
    echo -e "  ${RED}✗ Unavailable Operators found:${NC}\n$UNAVAILABLE_OPS"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ All operators are Available.${NC}"
fi
# 4. Check MachineConfigPools (MCP)
echo "[4/7] Validating MachineConfigPools (MCP)..."
DEGRADED_MCP=$(oc get mcp -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Degraded")].status=="True")].metadata.name}')
PAUSED_MCP=$(oc get mcp -o jsonpath='{.items[?(@.spec.paused==true)].metadata.name}')
if [[ -n "$DEGRADED_MCP" ]]; then
    echo -e "  ${RED}✗ Degraded MachineConfigPools:${NC} $DEGRADED_MCP"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ All MachineConfigPools healthy.${NC}"
fi
if [[ -n "$PAUSED_MCP" ]]; then
    echo -e "  ${YELLOW}! Warning: Paused MachineConfigPools detected:${NC} $PAUSED_MCP"
fi
# 5. Check Node Readiness
echo "[5/7] Checking Node status across cluster..."
NOT_READY_NODES=$(oc get nodes --no-headers | awk '$2 != "Ready" {print $1}')
if [[ -n "$NOT_READY_NODES" ]]; then
    echo -e "  ${RED}✗ Nodes in NotReady state:${NC}\n$NOT_READY_NODES"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ All nodes report Ready status.${NC}"
fi
# 6. Audit PodDisruptionBudgets (PDB) for Potential Deadlocks
echo "[6/7] Checking PodDisruptionBudgets (PDBs) for drain lock risks..."
DEADLOCKED_PDBS=$(oc get pdb -A -o json | jq -r '.items[] | select(.status.disruptionsAllowed == 0) | "\(.metadata.namespace)/\(.metadata.name)"')
if [[ -n "$DEADLOCKED_PDBS" ]]; then
    echo -e "  ${YELLOW}! PDBs currently allowing 0 disruptions (May block node drains):${NC}"
    echo "$DEADLOCKED_PDBS" | sed 's/^/    /'
else
    echo -e "  ${GREEN}✓ No blocking PDBs detected.${NC}"
fi
# 7. Check Active Critical Alerts
echo "[7/7] Checking Prometheus Alerts..."
FIRING_CRITICALS=$(oc get alerts -A -o json 2>/dev/null | jq -r '.items[] | select(.status.state=="firing" and .labels.severity=="critical") | .labels.alertname' || echo "")
if [[ -n "$FIRING_CRITICALS" ]]; then
    echo -e "  ${RED}✗ Firing Critical Alerts:${NC}\n$FIRING_CRITICALS"
    ((ERRORS++))
else
    echo -e "  ${GREEN}✓ Zero firing critical alerts.${NC}"
fi
# Final Summary
echo -e "\n${YELLOW}====================================================${NC}"
if [[ $ERRORS -eq 0 ]]; then
    echo -e "${GREEN} SUCCESS: Cluster passed all health checks. Safe to proceed with upgrade.${NC}"
    echo -e "${YELLOW}====================================================${NC}"
    exit 0
else
    echo -e "${RED} FAILURE: Found $ERRORS issue(s). Resolve all errors before upgrading.${NC}"
    echo -e "${YELLOW}====================================================${NC}"
    exit 1
fi

		

Infra Cloud Solutions

OpenShift(OCP) Pre-Upgrade Health Check Script

Purpose

Like this:

Related

Leave a ReplyCancel reply

Purpose

Share this:

Like this:

Related

Leave a ReplyCancel reply

Discover more from Infra Cloud Solutions