When an OpenShift ClusterOperator becomes degraded, it typically means one of its managed custom resources, static pods, or underlying dependencies failed its health checks.

Here is the systematic workflow to inspect, isolate, and troubleshoot degraded operators using the oc CLI.

Step 1: Identify Which Operators are Degraded

Run oc get clusteroperator (or co) to get an overview of all system operators:

oc get clusteroperator | grep -E "NAME|True" | grep -v "AVAILABLE"

Or query JSON directly for only degraded items:

oc get clusteroperator -o json | jq -r '.items[] | select(.status.conditions[] | select(.type=="Degraded" and .status=="True")) | .metadata.name'

Step 2: Inspect Operator Conditions & Reason Messages

The ClusterOperator status block contains detailed condition reasons explaining why it is degraded.

# Replace <operator-name> (e.g., ingress, authentication, machine-config, storage)
oc describe clusteroperator <operator-name>

What to look for in the output:

• Check the Status.Conditions section at the bottom.
• Look for Type: Degraded, Status: True, and read the Reason and Message fields. They often pinpoint exact failing resources (e.g., “Deployment degraded: 1/2 replicas available” or “DaemonSet ingress-canary is degraded”).

Step 3: Find the Operator’s Namespace & Pods

Cluster operators usually run inside a dedicated namespace (typically named openshift-<operator-name>).

1. Find the operator’s targeted namespace:
   oc get clusteroperator <operator-name> -o jsonpath='{.spec.relatedObjects[?(@.kind=="Namespace")].name}'
2. Check the status of all pods in that namespace:
   oc get pods -n openshift-<operator-name> -o wide Look for pods in CrashLoopBackOff, ImagePullBackOff, Error, or Pending states.

Step 4: Inspect Failing Pod Logs and Events

Once you identify the problematic pod or deployment inside the operator’s namespace:

1. Check recent namespace events for scheduling or volume mount errors:
   oc get events -n openshift-<operator-name> --sort-by='.metadata.creationTimestamp' | tail -n 20
2. Tail logs of the operator deployment pod:
   oc logs deployment/<operator-deployment-name> -n openshift-<operator-name> --tail=100 If the container restarted, inspect the previous crashed instance:
   oc logs <pod-name> -n openshift-<operator-name> -c <container-name> --previous

Step 5: Common Operator Triage Scenarios

Depending on which operator is degraded, check these specific underlying components:

Scenario A: machine-config Operator Degraded

Usually caused by a node failing to apply a MachineConfig or failing an rpm-ostree update.

# Check MachineConfigPool health
oc get mcp
# Inspect which node is degraded/updating
oc get nodes
# Check MCO daemon logs on the failing node
oc logs -n openshift-machine-config-operator machine-config-daemon-<hash> -c machine-config-daemon

Scenario B: ingress or authentication Operator Degraded

Often linked to router pod failures, missing certificates, or network plugin issues.

# Check IngressController Custom Resources
oc get ingresscontroller -n openshift-ingress-operator
# Inspect router pods
oc get pods -n openshift-ingress

Scenario C: Storage / CSI Operator Degraded

Usually caused by authentication issues with the underlying cloud provider or missing storage class permissions.

# Check storage classes and CSI driver status
oc get storageclass
oc get csidriver

Step 6: Force Reconciliation (If Stuck)

If you resolve the underlying issue (e.g., fixed a network issue or storage permission), but the operator does not automatically clear its degraded state, you can force the operator pod to restart and re-evaluate the cluster:

# Rollout restart the operator deployment
oc rollout restart deployment/<operator-name> -n openshift-<operator-name>