Deploying the LokiStack Operator in OpenShift requires establishing an S3-compatible Object Storage bucket (such as AWS S3, MinIO, or OpenShift Data Foundation NooBaa), storing the bucket credentials in a Kubernetes Secret, and defining a LokiStack Custom Resource.

Step 1: Install the Loki Operator via OLM

Before creating the LokiStack instance, install the Loki Operator from OperatorHub into the openshift-operators-redhat namespace.

YAML

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: loki-operator
  namespace: openshift-operators-redhat
spec:
  channel: stable-5.x # Or the matching channel for your OCP release
  name: loki-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Step 2: Create the Object Storage Credentials Secret

Loki requires access to an S3-compatible object storage bucket to store chunked log data and index files.

1. Create a Secret in the openshift-logging namespace containing your S3 bucket access keys and endpoint configuration:

YAML

apiVersion: v1
kind: Secret
metadata:
  name: logging-loki-s3
  namespace: openshift-logging
stringData:
  access_key_id: "YOUR_S3_ACCESS_KEY"
  access_key_secret: "YOUR_S3_SECRET_KEY"
  bucketnames: "ocp-loki-logs-bucket"
  endpoint: "https://s3.us-east-1.amazonaws.com" # Or internal MinIO/NooBaa endpoint URL
  region: "us-east-1"
type: Opaque

Note for AWS IAM Roles for Service Accounts (IRSA) / STS: If using AWS IRSA or ROSA with STS, you can omit access_key_id and access_key_secret in favor of an AWS IAM role annotation on the service account.

Step 3: Deploy the LokiStack Custom Resource

Define the size and replication footprint of your Loki cluster using the LokiStack CRD. Red Hat provides pre-tuned size profiles (1x.extra-small, 1x.small, 1x.medium, etc.) to automatically calculate resource limits and replica counts.

YAML

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small # Pre-tuned production profile (extra-small, small, medium)
  storage:
    schemas:
      - version: v13
        effectiveDate: "2024-01-01"
    secret:
      name: logging-loki-s3
      type: s3
  storageClassName: fast-ssd # Fast block storage PVCs for Loki write-ahead logs (WAL)
  managementState: Managed
  replication:
    factor: 2 # Number of chunk replicas across Loki ingesters

Apply the manifest:

oc apply -f lokistack.yaml

Step 4: Verify Deployment Health

Check that the Loki components (Ingester, Querier, Query-Frontend, Distributor, Compactor, Gateway) are fully provisioned and healthy:

1. Verify Pod Readiness:
   Bashoc get pods -n openshift-logging -l app.kubernetes.io/name=loki
2. Verify LokiStack CR Status:
   Bashoc get lokistack logging-loki -n openshift-logging -o jsonpath='{.status.conditions}' | jq Ensure all conditions (such as Pending, Ready) transition to Status: "True".

Step 5: Connect Vector to Loki (ClusterLogForwarder)

Now that LokiStack is running, configure the ClusterLogForwarder to collect and stream logs into your new Loki instance using the internal Gateway route.

YAML

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: default-loki-stack
      type: lokiStack
      lokiStack:
        target:
          name: logging-loki
          namespace: openshift-logging
        authentication:
          token:
            from: serviceAccount
      url: 'https://logging-loki-gateway.openshift-logging.svc:8080'
  pipelines:
    - name: all-logs-to-loki
      inputRefs:
        - application
        - infrastructure
        - audit
      outputRefs:
        - default-loki-stack

The Cluster Monitoring Operator (CMO) is a core, default-installed OpenShift operator responsible for deploying, managing, and maintaining the platform’s entire observability stack.

It implements a fully managed Prometheus-based monitoring solution designed to monitor core OpenShift platform components, node infrastructure, and (optionally) user-defined workload applications.

1. Architecture and Key Components

The CMO acts as a high-level orchestrator. It manages custom resources (CRs), deployments, daemonsets, and configurations across the cluster (primarily within the openshift-monitoring and openshift-user-workload-monitoring namespaces).

Plaintext

                               ┌────────────────────────────────┐
                               │  Cluster Monitoring Operator   │
                               │             (CMO)              │
                               └───────────────┬────────────────┘
                                               │ Manages
             ┌─────────────────────────────────┼─────────────────────────────────┐
             ▼                                 ▼                                 ▼
┌─────────────────────────┐       ┌─────────────────────────┐       ┌─────────────────────────┐
│     Prometheus Core     │       │      Alertmanager       │       │      Metrics Exporters  │
│  (Platform Metrics DB)  │       │    (Routing & Alerts)   │       │ (kube-state-metrics,    │
└────────────┬────────────┘       └─────────────────────────┘       │ node-exporter, cAdvisor)│
             │ Scrapes                                              └─────────────────────────┘
             ▼
┌─────────────────────────┐
│ User Workload Monitoring│
│  (Thanos / Prometheus)  │
└─────────────────────────┘

The stack deployed by CMO includes:

• Prometheus: High-performance time-series database configured to scrape platform metrics (control plane services, etcd, OVN-Kubernetes, routers).
• Thanos Querier: Provides a unified query interface (via PromQL) across both platform metrics and user workload metrics.
• Alertmanager: Handles alerting logic, deduplication, grouping, and notification routing (via PagerDuty, Slack, Webhooks, Email).
• Metrics Exporters: Includes node-exporter (host system metrics), kube-state-metrics (Kubernetes object state), and OpenShift component exporters.
• Telemeter Client: Sends a curated, anonymized subset of platform health metrics back to Red Hat Insights for proactive support.

2. Core Responsibilities

1. Automated Configuration & Reconciliation: You do not manage raw Prometheus YAML configs directly. Instead, you modify a central ConfigMap (cluster-monitoring-config), and the CMO reconciles and applies those changes across all underlying Prometheus and Alertmanager instances.
2. Platform Health Enforcement: CMO ensures critical alert rules (e.g., etcd high latency, node disk pressure, API server errors) are always active and cannot be accidentally deleted by cluster users.
3. User Workload Monitoring (UWM): By default, CMO only monitors OpenShift platform components. However, administrators can enable UWM in the CMO configuration to allow developers to deploy ServiceMonitor and PodMonitor CRDs for custom application metrics in their own namespaces.

3. Key Configuration Example

To configure CMO settings (such as setting retention periods, persistent volume claims, or enabling User Workload Monitoring), administrators edit the cluster-monitoring-config ConfigMap in the openshift-monitoring namespace.

YAML

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    # Enable User Workload Monitoring for application metrics
    enableUserWorkload: true
    
    # Configure Platform Prometheus Retention and Storage
    prometheusK8s:
      retention: 15d
      volumeClaimTemplate:
        metadata:
          name: prometheus-pvc
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 100Gi
    
    # Configure Alertmanager Storage
    alertmanagerMain:
      volumeClaimTemplate:
        metadata:
          name: alertmanager-pvc
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 10Gi

4. Useful oc Commands for Troubleshooting CMO

• Check CMO Operator Status:
  oc get clusteroperator monitoring
• View Monitoring Pods:
  oc get pods -n openshift-monitoring
• View User Workload Monitoring Pods (if enabled):
  oc get pods -n openshift-user-workload-monitoring
• Inspect CMO Logs: oc logs -n openshift-monitoring deployment/cluster-monitoring-operator -c cluster-monitoring-operator

While OpenShift Container Platform (OCP) runs dozens of background operators, a select group of essential core and day-2 operators are critical for architecture, security, and cluster operations.

1. The Core Infrastructure Operators (The Engine)

These operators are installed automatically by default and manage the fundamental lifecycle of the cluster itself.

• Cluster Version Operator (CVO): The “master controller” of OpenShift. It checks the update payload image, manages the lifecycle of all other cluster operators, and ensures the cluster stays in its declared version state.
• Machine Config Operator (MCO): Manages the host operating system (Red Hat Enterprise Linux CoreOS / RHCOS). Any change to kernel settings, SSH keys, network configurations, or storage mounts on host nodes is applied declaratively via the MCO.
• Ingress Operator: Manages the HAProxy-based routers that handle incoming HTTP/HTTPS traffic from outside the cluster into internal application services.
• Storage Operator / CSI Operators: Manages persistent storage drivers (AWS EBS, vSphere CSI, ODF) to allow dynamic provisioning of PersistentVolumeClaims (PVCs).

2. Platform Expansion & Management Operators

These operators are typically installed on Day 2 from OperatorHub to manage fleet operations, workloads, and security.

3. Observability & Operations

Operators that provide operational visibility and cluster-wide telemetry.

• Cluster Monitoring Operator (CMO): Deploy and manage the built-in Prometheus, Grafana, and Alertmanager stack to collect cluster and node-level metrics.
• Red Hat OpenShift Logging (Vector / OpenSearch / Loki): Aggregates, parses, and stores logs from cluster components, host nodes, and application containers.
• Node Feature Discovery (NFD) & GPU Operators: Automatically detects specialized hardware (like NVIDIA GPUs or SR-IOV network interfaces) on host nodes and applies corresponding labels for workload scheduling.

Performing an etcd backup is a mandatory step before any major OpenShift Container Platform (OCP) lifecycle operation (such as cluster updates or major infrastructure changes).

OpenShift provides built-in scripts to automate the backup process and restore state directly onto control plane (master) nodes.

Part 1: How to Back Up etcd

The backup script creates a snapshot of the etcd database along with static pod resources, certificates, and API keys.

1.Establish a Debug Session on a Control Plane Node:Prerequisite: Cluster-admin rights required.

Select an active master node and start a debug pod on it:

oc debug node/master-0

2.Set Root Directory Context:

Inside the debug shell, switch your filesystem context to access the host system drives:

chroot /host

3.Execute the Built-in Cluster Backup Script:

Run the automated backup script provided in the RHCOS image, specifying a target output directory:

/usr/local/bin/cluster-backup.sh /var/usr/temp/etcd-backup

Expected Output:

Plaintext

found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-12
etcdctl version: 3.5.x
snapshot saved at /var/usr/temp/etcd-backup/snapshot_2026-08-29_110156.db
...
etcd backup complete!

4.Verify and Save Backup Artifacts:

Check that both the snapshot DB file and static pod resource archives were generated:

ls -lh /var/usr/temp/etcd-backup

Important: Copy these files off the host node to a secure external storage location (S3 bucket, NFS share, or local admin machine) via oc cp or scp.

Part 2: How to Restore etcd (Disaster Recovery)

Use this process if the cluster’s state database suffers unrecoverable corruption or consensus loss across the control plane.

Warning: A restore is a destructive, high-impact action. It resets the cluster state back to the exact point in time when the backup snapshot was taken.

1.Identify the Recovery Master Node:

Select one surviving master node (master-0) to act as the primary recovery host. Transfer your backup files to this node if they aren’t already present on /var/usr/temp/etcd-backup.

2.Stop the Control Plane Static Pods on ALL Master Nodes:

Open SSH or debug sessions to every master node in the cluster and move the static pod manifests out of /etc/kubernetes/manifests/ to stop kube-apiserver, etcd, and kube-controller-manager:

# Run this on master-0, master-1, and master-2
chroot /host
mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp/
mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp/

3.Run the Restore Script on the Recovery Node:

On master-0 (the recovery node only), run the restore script using the path to your backup snapshot directory:

chroot /host
/usr/local/bin/cluster-restore.sh /var/usr/temp/etcd-backup

What this script does:

• Clears corrupted local etcd member data on master-0.
• Restores the snapshot DB.
• Generates a single-member etcd cluster configuration.

4.Restart Kubelet on the Recovery Node:

Restart kubelet on master-0 to re-initialize static pods:

systemctl restart kubelet

5.Force etcd Operator Redeployment:

Exit the host debug shell back to your workstation CLI and patch the Cluster Version / etcd Operator to force quorum reconciliation across the remaining control plane nodes:

# Force the etcd operator to re-evaluate cluster membership
oc patch etcd cluster --type=merge -p '{"spec": {"forceRedeploymentReason": "recovery-'$(date +%s)'"}}'

The etcd-operator will automatically redeploy and scale etcd instances across master-1 and master-2, restoring multi-node HA consensus.

A canary update strategy allows you to test an OpenShift node operating system and config update on a small subset of worker nodes (e.g., a non-critical worker pool) before committing the change across your entire production cluster.

By default, OpenShift updates all nodes in a MachineConfigPool (MCP) automatically once the control plane is updated. Pausing an MCP freezes the Machine Config Operator (MCO) from updating that specific group of nodes.

Step 1: Create a Dedicated Canary MachineConfigPool

Instead of applying updates directly to the default worker pool, isolate 1 or 2 nodes into a custom canary pool.

1. Label a target worker node for Canary testing:

oc label node worker-canary-0.example.com node-role.kubernetes.io/canary=

2. Define and create the Canary MachineConfigPool manifest:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: canary
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker, canary]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/canary: ""

Save as canary-mcp.yaml and apply:

oc apply -f canary-mcp.yaml

Step 2: Pause the Primary Worker Pool

Before starting the cluster-wide upgrade, pause the default worker pool so its nodes remain on the current version while the control plane and your canary pool update.

1. Pause the worker MCP:

oc patch mcp/worker --type merge --patch '{"spec":{"paused":true}}'

2. Verify MCP status:

oc get mcp

Output:

NAME     CONFIG                                UPDATED  UPDATING  DEGRADED  PAUSED
master   rendered-master-11111                 True     False     False     False
worker   rendered-worker-11111                 True     False     False     True
canary   rendered-worker-11111                 True     False     False     False

Step 3: Trigger the OpenShift Upgrade

Initiate the cluster upgrade as normal:

oc adm upgrade --to=<target_version>

What happens next:

1. The Cluster Version Operator (CVO) updates the Control Plane (master nodes).
2. The Machine Config Operator (MCO) updates the canary MCP (re-imaging and rebooting worker-canary-0).
3. The worker MCP is skipped entirely because spec.paused is set to true.

Step 4: Validate the Canary Node

Check that the canary pool reaches an UPDATED=True state and run application validation:

# Verify the Canary node has completed its rollout
oc get mcp/canary
# Verify the node is running the target release kernel and core components
oc describe node worker-canary-0.example.com

• Leave the cluster running in this state for your designated soak period (e.g., 24–48 hours) to verify application performance and stability.

Step 5: Unpause and Complete Cluster Rollout

Once you confirm the canary update is stable and safe:

1. Unpause the worker pool:

oc patch mcp/worker --type merge --patch '{"spec":{"paused":false}}'

2. Monitor the full rollout:
   The MCO will immediately begin cordoning, draining, re-imaging, and rebooting the remaining worker nodes one by one:

oc get mcp -w

OCP Cluster Upgrade — Best Practices Guide

The Golden Rule

Never upgrade a sick cluster. Every check below exists because upgrading on top of an existing problem turns a recoverable issue into a potential outage.

Phase 1 — Pre-Upgrade Preparation (Days Before)

1.1 Know Your Upgrade Path

OCP does not support arbitrary version jumps. Upgrades must follow a validated graph:

4.12 → 4.13 → 4.14       ✅ valid sequential path
4.12 → 4.14              ❌ may not be supported (check the graph)
4.14.5 → 4.14.12         ✅ z-stream (patch) always OK
4.14.12 → 4.15.0         ✅ minor version (must check graph)

Always verify the upgrade path first:

# Check available updates from the official graph
oc adm upgrade
# Output example:
# Recommended updates:
#   VERSION   IMAGE
#   4.14.13   quay.io/openshift-release-dev/ocp-release@sha256:...
#   4.14.14   quay.io/openshift-release-dev/ocp-release@sha256:...

Use the official upgrade path tool:

https://access.redhat.com/labs/ocpupgradegraph/update_path

1.2 Read the Release Notes

For every version you are jumping through:

https://docs.openshift.com/container-platform/<version>/release_notes/ocp-<version>-release-notes.html

Look specifically for:

• Deprecated APIs — if your workloads use removed APIs, they break post-upgrade
• Known issues — bugs that affect your environment
• Operator compatibility — OLM-managed operators may need separate upgrades
• Behavior changes — network policy, SCC, authentication changes

1.3 Check Operator Compatibility (OLM)

Operators installed via OLM have their own upgrade lifecycle, separate from OCP:

# List all installed operators and their current versions
oc get csv -A
# Check operator subscription channels
oc get subscription -A -o custom-columns=\
'NAME:.metadata.name,CHANNEL:.spec.channel,SOURCE:.spec.source'

Check the OperatorHub or vendor docs to confirm your operators support the target OCP version before upgrading the platform.

1.4 Check for Deprecated API Usage

OCP removes old API versions between minor releases. If your workloads use removed APIs they will fail silently after upgrade:

# Check for deprecated API usage in the cluster
oc get apirequestcounts -o json | jq -r \
  '.items[] |
   select(.status.removedInRelease != null) |
   select(.status.requestCount > 0) |
   "\(.metadata.name) → removed in \(.status.removedInRelease) | used \(.status.requestCount)x"'

Any resource showing removedInRelease with a non-zero requestCount must be migrated before upgrading.

1.5 Validate etcd Health

etcd is the most critical component. A degraded etcd during upgrade can corrupt cluster state:

# Check etcd cluster operator
oc get co etcd
# Check etcd member status
oc get etcd -o jsonpath=\
  '{.items[0].status.conditions[?(@.type=="EtcdMembersAvailable")].status}'
# Expected: True
# Check etcd pods directly
oc get pods -n openshift-etcd -l app=etcd
# Check etcd endpoint health
oc rsh -n openshift-etcd $(oc get pods -n openshift-etcd -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}') \
  etcdctl endpoint health \
  --cluster \
  --cacert /etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt \
  --cert /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-$(hostname).crt \
  --key /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-$(hostname).key

Expected output:

https://192.168.1.10:2379 is healthy: committed revision: 294858, took: 3ms
https://192.168.1.11:2379 is healthy: committed revision: 294858, took: 4ms
https://192.168.1.12:2379 is healthy: committed revision: 294858, took: 3ms

1.6 Verify All Nodes Are Ready

# All nodes must be Ready with no scheduling issues
oc get nodes
# Check for any taints blocking scheduling
oc get nodes -o custom-columns=\
'NAME:.metadata.name,STATUS:.status.conditions[-1].type,TAINTS:.spec.taints'
# Check node resource pressure
oc adm top nodes

1.7 Check All Cluster Operators

# All must be: AVAILABLE=True  PROGRESSING=False  DEGRADED=False
oc get co
# Quick one-liner to find any not healthy:
oc get co -o json | jq -r \
  '.items[] | select(
     (.status.conditions[] | select(.type=="Degraded" and .status=="True"))
     or
     (.status.conditions[] | select(.type=="Available" and .status=="False"))
   ) | .metadata.name'
# Expected: empty output

1.8 Check MachineConfigPools

# All pools must be: UPDATED=True  UPDATING=False  DEGRADED=False
oc get mcp
# Check for paused pools (they won't upgrade automatically)
oc get mcp -o jsonpath='{.items[?(@.spec.paused==true)].metadata.name}'

1.9 Review PodDisruptionBudgets

# Find PDBs that currently block all disruptions
oc get pdb -A -o json | jq -r \
  '.items[] |
   select(.status.disruptionsAllowed == 0) |
   "\(.metadata.namespace)/\(.metadata.name) — minAvailable:\(.spec.minAvailable)"'

For each blocking PDB, verify whether it will open up during drain (more replicas than minAvailable) or whether it is misconfigured and will permanently block node drains.

1.10 Check Active Alerts

# Via AlertManager route
ALERTMANAGER=$(oc get route alertmanager-main \
  -n openshift-monitoring \
  -o jsonpath='{.spec.host}')
TOKEN=$(oc create token prometheus-k8s -n openshift-monitoring)
curl -sk -H "Authorization: Bearer $TOKEN" \
  "https://${ALERTMANAGER}/api/v2/alerts" | \
  jq -r '.[] | select(.labels.severity=="critical") | .labels.alertname'

No critical alerts should be firing before upgrade begins.

Phase 2 — The Day Before

2.1 Take an etcd Backup

This is non-negotiable. If the upgrade corrupts cluster state, this is your only recovery path:

# SSH to a master node
ssh core@master-0
# Run backup
sudo /usr/local/bin/cluster-backup.sh /home/core/backup
# Verify both files created
ls -lh /home/core/backup/
# snapshot_<timestamp>.db
# static_kuberesources_<timestamp>.tar.gz
# Copy off-cluster immediately (S3, NFS, bastion host)
scp core@master-0:/home/core/backup/* bastion:/backups/pre-upgrade/

The backup must be from the same z-stream as the running cluster. A 4.14.12 cluster backup cannot restore a 4.15.x cluster.

2.2 Notify Stakeholders

Upgrades cause rolling node reboots. Even with zero downtime on properly configured workloads, communicate:

• Upgrade window start/end time
• Which node groups reboot and in what order (masters first, then workers)
• Expected impact on any workloads that are not highly available

2.3 Scale Down Non-Critical Workloads (Optional)

For resource-constrained clusters, reducing workload during upgrade gives the upgrade controller more room to drain nodes:

# Note current replicas first, then scale down
oc get deployments -A --no-headers | \
  awk '{print $1, $2, $3}' > /tmp/pre-upgrade-replicas.txt
# Scale down non-critical namespaces
oc scale deployment --all -n non-critical-namespace --replicas=0

Phase 3 — Initiating the Upgrade

3.1 Choose the Right Channel

OCP upgrade channels control which versions are offered:

stable-4.14    → production-recommended, fully validated
fast-4.14      → validated but released sooner than stable
candidate-4.14 → pre-release, NOT for production
eus-4.14       → Extended Update Support (long-term maintenance)

# Check current channel
oc get clusterversion -o jsonpath='{.items[0].spec.channel}'
# Set channel if needed
oc patch clusterversion version \
  --type merge \
  --patch '{"spec":{"channel":"stable-4.14"}}'

3.2 Trigger the Upgrade

# Upgrade to the latest available in channel
oc adm upgrade --to-latest=true
# Upgrade to a specific version (recommended — explicit and auditable)
oc adm upgrade --to=4.14.13
# For disconnected clusters (specify the full image digest)
oc adm upgrade \
  --to-image=quay.io/openshift-release-dev/ocp-release@sha256:<digest> \
  --allow-explicit-upgrade

3.3 Monitor the Upgrade Progress

# Watch CVO progress — the top-level view
watch oc get clusterversion
# More detail
oc describe clusterversion version | grep -A 30 "Conditions:"
# Watch all cluster operators rolling through upgrade
watch oc get co
# Watch nodes rebooting (masters first, then workers)
watch oc get nodes
# Tail CVO operator logs for real-time detail
oc logs -n openshift-cluster-version \
  $(oc get pods -n openshift-cluster-version \
    -o jsonpath='{.items[0].metadata.name}') -f

3.4 Understand the Upgrade Sequence

OCP upgrades in a strict order — never all at once:

1. CVO downloads new release image
        │
        ▼
2. Control plane operators update
   (etcd, kube-apiserver, controller-manager, scheduler)
   One at a time, with health checks between each
        │
        ▼
3. Master nodes drain → reboot → rejoin
   One at a time (never all 3 simultaneously)
        │
        ▼
4. Cluster operators update
   (authentication, console, ingress, monitoring, etc.)
        │
        ▼
5. Worker nodes drain → reboot → rejoin
   Controlled by MachineConfigPool maxUnavailable setting
        │
        ▼
6. CVO marks upgrade complete
   clusterversion: Progressing=False, Available=True

3.5 Control Worker Upgrade Speed

By default only 1 worker upgrades at a time. For large clusters you can safely increase this:

# Check current maxUnavailable (default is 1)
oc get mcp worker -o jsonpath='{.spec.maxUnavailable}'
# Allow up to 20% of workers to upgrade simultaneously
oc patch mcp worker \
  --type merge \
  --patch '{"spec":{"maxUnavailable":"20%"}}'

⚠️ Never set maxUnavailable higher than your workloads can tolerate. If minAvailable in your PDBs is 2 and you only have 3 replicas, draining 2 workers simultaneously will block.

Phase 4 — During the Upgrade

Watch for Stuck Upgrades

# If upgrade appears frozen, check:
# 1. Any operator stuck progressing?
oc get co | grep -v "True.*False.*False"
# 2. Any node stuck draining?
oc get nodes | grep SchedulingDisabled
# 3. Any pods blocking drain (PDB)?
oc get pdb -A | grep -v "^NAMESPACE"
# 4. Pending CSRs (workers trying to re-join after reboot)
oc get csr | grep Pending
oc get csr -o name | xargs oc adm certificate approve
# 5. Any pods stuck Terminating (blocking node drain)?
oc get pods -A | grep Terminating
# Force delete if stuck:
oc delete pod <pod> -n <ns> --grace-period=0 --force

Approve Pending CSRs

After each node reboots, it issues a CSR to re-register with the cluster. In some configurations these need manual approval:

# Watch for and approve all pending CSRs
watch -n 10 "oc get csr | grep Pending"
# Approve all at once
oc get csr -o name | xargs oc adm certificate approve

Phase 5 — Post-Upgrade Validation

5.1 Confirm Upgrade Completed

# Check cluster version
oc get clusterversion
# AVAILABLE=True  PROGRESSING=False  VERSION=<new-version>
# Confirm exact version
oc get clusterversion -o jsonpath='{.items[0].status.desired.version}'

5.2 Validate All Operators

# All must return to healthy state
oc get co
# All: AVAILABLE=True  PROGRESSING=False  DEGRADED=False

5.3 Validate All Nodes

# All nodes Ready, running new kubelet version
oc get nodes
# All: STATUS=Ready, VERSION=v1.27.x (new version)

5.4 Validate Workloads

# Check for any crashlooping or failed pods
oc get pods -A | grep -vE "Running|Completed|Succeeded"
# Check deployments fully available
oc get deployments -A | grep -v "^\(NAMESPACE\)" | \
  awk '$3 != $4 {print}'   # READY != DESIRED
# Check routes are responding
oc get routes -A

5.5 Take a Post-Upgrade etcd Backup

# The pre-upgrade backup is now stale — take a fresh one
sudo /usr/local/bin/cluster-backup.sh /home/core/backup-post-upgrade
scp core@master-0:/home/core/backup-post-upgrade/* bastion:/backups/post-upgrade/

Phase 6 — OLM Operator Upgrades

Platform upgrade does not upgrade OLM-managed operators. Do this separately:

# Check which operators have updates available
oc get csv -A | grep -v Succeeded
# Check subscription installPlanApproval settings
oc get subscription -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,APPROVAL:.spec.installPlanApproval'
# For Manual approval — approve pending InstallPlans
oc get installplan -A | grep RequiresApproval
oc patch installplan <name> -n <namespace> \
  --type merge \
  --patch '{"spec":{"approved":true}}'

Complete Upgrade Checklist

PRE-UPGRADE
  □ Verify upgrade path on Red Hat graph
  □ Read release notes for target version
  □ Check deprecated API usage (apirequestcounts)
  □ Verify etcd health (all 3 members healthy)
  □ All cluster operators: Available=True, Degraded=False
  □ All MachineConfigPools: Updated=True, Degraded=False
  □ All nodes: Ready
  □ No PDBs blocking all disruptions (disruptionsAllowed=0 on critical apps)
  □ No critical alerts firing
  □ Operator compatibility confirmed (OLM operators)
  □ Stakeholders notified
  □ etcd backup taken and stored off-cluster
  □ Upgrade channel set correctly (stable/fast/eus)
DURING UPGRADE
  □ Monitor CVO progress (watch oc get clusterversion)
  □ Monitor cluster operator rollout (watch oc get co)
  □ Monitor node reboots (watch oc get nodes)
  □ Approve pending CSRs as nodes rejoin
  □ Watch for stuck Terminating pods
  □ Watch for PDB-blocked drains
POST-UPGRADE
  □ clusterversion: Available=True, Progressing=False
  □ All cluster operators healthy
  □ All nodes Ready on new version
  □ All workload pods Running/Completed
  □ All routes responding
  □ Upgrade OLM operators to compatible versions
  □ Take fresh etcd backup
  □ Update documentation / CMDB with new version

Key Timings to Expect

Planning an OpenShift Container Platform (OCP) upgrade requires structured validation across 5 distinct phases. Because OpenShift performs rolling updates of both the control plane and the host OS (RHCOS), executing pre-upgrade checks prevents node-drain locks, degraded operators, or broken API dependencies.

Phase 1: Pre-Upgrade Review & Path Mapping

1. Verify Your Upgrade Path:
   • OpenShift does not support skipping Y-stream releases (e.g., jumping from 4.14 to 4.16 directly). You must upgrade sequentially (4.14 → 4.15 → 4.16).
   • Check your configured update channel (stable-4.x, fast-4.x, or eus-4.x).For enterprise production environments, stick to stable-4.x or eus-4.x (Extended Update Support).
   • Query the OpenShift Update Service (OSUS) via CLI:
     adm upgrade
2. Audit Deprecated & Removed Kubernetes APIs:
   • Before a minor version jump (e.g., 4.15 → 4.16), check if your workloads use removed K8s API versions.
   • Review cluster alerts for API deprecations and migrate manifests accordingly. OCP will block minor version upgrades if administrator acknowledgment is missing for API removals.
3. Check Operator & OLM Compatibility:
   • Non-core platform operators (installed via OperatorHub/OLM) must be upgraded to versions compatible with the target OCP release before updating the cluster core.

Phase 2: Cluster Health & Capacity Assessment

Run a complete health scan. Never trigger an upgrade on a degraded cluster.

1. Verify ClusterOperator Status:Ensure all core operators report AVAILABLE=True, PROGRESSING=False, and DEGRADED=False:
   oc get clusteroperator
2. Verify MachineConfigPool (MCP) Health:Ensure all node pools are updated and none are paused or degraded:
   oc get mcp (If an MCP is paused for canary testing, verify it is intentional before proceeding.)
3. Check Node Capacity & PodDisruptionBudgets (PDBs):
   • Node Capacity: During node-draining, remaining workers must have enough CPU/RAM capacity to host evacuated workloads.
   • PDB Deadlocks: Inspect custom PodDisruptionBudgets (oc get pdb -A). If minAvailable is set equal to the total replica count (or maxUnavailable: 0), node drains will stall, hanging the upgrade loop.
4. Clear Active Alerts:In the console, go to Observe → Alerting and resolve all critical alerts.

Phase 3: Mandatory Backups (Non-Negotiable)

Because OpenShift updates are fail-forward (rolling back to an older minor version is not supported), taking backups prior to execution is mandatory:

1. Backup etcd: Take a manual snapshot of the etcd database from a control plane node:
   oc debug node/<master-node-name> -- /usr/local/bin/cluster-backup.sh /host/var/usr/temp/etcd-backup
2. Export Cluster Custom Resources (CRDs): Export your custom deployment manifests, routes, and secret configurations using a GitOps tool (ArgoCD/ACM) or CLI backups.
3. Storage Snapshots: Create CSI volume snapshots for critical stateful application data.

Phase 4: Execution & Monitoring

1. Trigger the Upgrade:
   • Via Web Console:Navigate to Administration → Cluster Settings → Details, select your desired target version, and confirm.
   • Via CLI:
     oc adm upgrade --to=<target_version>
2. Monitor the Phased Rollout:
   • Control Plane Phase: Watch the CVO reconcile cluster operators:
     oc get clusterversion -w
   • Worker Pool Phase: Watch the MCO cordon, drain, re-image (rpm-ostree), reboot, and uncordon nodes pool-by-pool:
     oc get nodes
   • oc get mcp

Phase 5: Post-Upgrade Validation

Once the CVO reports Cluster version is <target_version>:

• [ ] Verify all nodes report Ready and run the new RHCOS kernel version.
• [ ] Confirm all ClusterOperators return to an AVAILABLE=True state.
• [ ] Verify routes, ingress controllers, and critical business applications are accepting traffic.
• [ ] Re-enable any paused MachineConfigPools or automated maintenance windows.

In OpenShift (OCP) and Kubernetes, a taint is a core scheduling mechanism applied to a node that allows the node to repel pods.

Think of a taint as a “Keep Out” sign placed on a server.By default, the OpenShift scheduler will not place any application pod on a tainted node unless that pod explicitly carries a matching toleration (a “key pass”).

While Node Selectors and Node Affinity are used to attract pods to specific nodes, Taints and Tolerations are used to repel unwanted pods away from nodes.

1. Anatomy of a Taint

A taint consists of three components: a Key, an optional Value, and a Taint Effect:

Taint} = key=value:Effect

The Three Taint Effects:

• NoSchedule (Hard Restriction):New pods without a matching toleration will never be scheduled onto this node.However, any pods already running on the node before the taint was applied are left alone.
• PreferNoSchedule (Soft Restriction):The scheduler tries to avoid placing pods on this node, but if no other compute resources are available in the cluster, it will place the pod here anyway.
• NoExecute (Eviction Restriction):The strongest effect.Any pod currently running on the node that does not tolerate the taint is immediately evicted (killed and rescheduled elsewhere).

2. Common Real-World Use Cases in OpenShift

• Dedicated Infrastructure Nodes:Isolating OpenShift Infra components (Ingress Routers, Monitoring, Image Registry) onto dedicated worker nodes so user applications cannot drain their resources.
• Specialized Hardware (GPUs / High-RAM):Tainting nodes that have expensive NVIDIA GPUs (gpu=true:NoSchedule) so standard web apps don’t accidentally get scheduled on them, reserving the hardware strictly for AI/ML workloads.
• Master Node Isolation:OpenShift automatically places a taint on Control Plane (master) nodes (node-role.kubernetes.io/master=:NoSchedule) so user workloads only run on worker nodes.
• Node Maintenance / Failure (Taint-based Eviction):When a node loses network connection or runs out of disk, OpenShift automatically applies temporary system taints like node.kubernetes.io/unreachable:NoExecute or node.kubernetes.io/disk-pressure:NoSchedule.

3. Declarative Setup Example

Step A: Applying a Taint to an OCP Node

Using the CLI, an administrator marks a node designated for GPU processing:

oc adm taint nodes worker-gpu-0 gpu=nvidia:NoSchedule

Step B: Adding a Toleration to a Pod Deployment

To allow a machine learning pod to run on that node, you add a tolerations block to its PodSpec:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-trainer
spec:
  template:
    spec:
      containers:
        - name: cuda-runner
          image: quay.io/ai-lab/cuda-runner:latest
      # The Pod MUST have this matching toleration to land on worker-gpu-0
      tolerations:
        - key: "gpu"
          operator: "Equal"
          value: "nvidia"
          effect: "NoSchedule"

Note: A toleration allows a pod to run on a tainted node, but it does not force it to go there.If you want to force the pod onto that specific GPU node, combine the Toleration with a nodeSelector or NodeAffinity.

OpenShift and AI is one of the fastest-growing areas in enterprise IT. Red Hat has built an entire AI platform around OpenShift, making it possible to run Generative AI, Machine Learning, MLOps, and AI inference on the same Kubernetes platform that hosts enterprise applications.

For an OpenShift Architect interview, you should understand not only AI concepts but also how OpenShift integrates with AI services.

OpenShift AI Ecosystem

                           Users
                             │
                   Web / Mobile Apps
                             │
                    OpenShift Routes
                             │
                ------------------------
                │                      │
          Business Apps         AI Applications
                │                      │
                └──────────────┬───────┘
                               │
                       OpenShift Platform
                               │
      ------------------------------------------------------
      │            │            │             │
   RHOAI       OpenShift     GPU Nodes    OpenShift GitOps
               Pipelines
      │
      ▼
 Kubeflow / Jupyter / Model Serving
      │
      ▼
 LLM (Llama, Granite, Mistral, GPT, etc.)
      │
      ▼
 Vector Database / AI Search
      │
      ▼
 Enterprise Data Sources

What is Red Hat OpenShift AI (RHOAI)?

Red Hat OpenShift AI (formerly Red Hat OpenShift Data Science) is Red Hat’s enterprise AI/ML platform built on OpenShift.

It provides:

• Jupyter Notebooks
• Model training
• Model serving
• Pipelines
• Experiment tracking
• GPU scheduling
• AI model deployment
• MLOps
• LLM serving
• Vector database integration

Think of it as:

OpenShift
        +
AI Platform
        +
MLOps

AI Architecture on OpenShift

              Developers

                 │

           Git Repository

                 │

           OpenShift GitOps

                 │

           AI Pipeline (Tekton)

                 │

        ------------------------

        │                      │

  Train Model           Build Container

        │                      │

        ------------------------

                 │

          Model Registry

                 │

          Model Serving

                 │

            REST API

                 │

             Applications

Major AI Components

1. Jupyter Notebooks

Used for:

• Data science
• Python
• TensorFlow
• PyTorch
• Hugging Face
• Experiments

Example:

Data Scientist
↓
Notebook
↓
Python
↓
Train Model

2. Model Serving

Once trained:

Model
↓
Model Server
↓
REST API
↓
Application

Supported technologies include:

• KServe
• vLLM (commonly used for LLM inference)
• NVIDIA Triton Inference Server
• Caikit (used in some Red Hat AI scenarios)

3. Pipelines

Uses Kubeflow Pipelines (or Tekton depending on workflow).

Example:

Dataset
↓
Preprocessing
↓
Training
↓
Evaluation
↓
Deploy
↓
Production

Everything becomes repeatable.

GPU Support

OpenShift schedules GPUs like any other resource.

Architecture:

GPU Node
↓
NVIDIA Driver
↓
NVIDIA Device Plugin
↓
Pod Requests GPU
↓
GPU Allocated

Example Pod:

resources:
  limits:
    nvidia.com/gpu: 1

AI Model Lifecycle

Collect Data
↓
Train Model
↓
Validate
↓
Containerize
↓
Deploy
↓
Monitor
↓
Retrain

This is called MLOps.

OpenShift AI with LLMs

You can deploy models like:

• IBM Granite
• Llama 3.x
• Mistral
• DeepSeek (where licensing and hardware permit)
• Gemma
• Phi

Architecture:

User
↓
Chat Application
↓
OpenShift Route
↓
Model Server
↓
LLM
↓
Response

RAG (Retrieval-Augmented Generation)

This is one of the most common enterprise AI architectures.

User Question
↓
Embedding Model
↓
Vector Database
↓
Relevant Documents
↓
LLM
↓
Answer

Instead of relying only on the model’s knowledge, the LLM searches company documents.

Enterprise Banking AI Example

Customer
↓
Chatbot
↓
OpenShift Route
↓
Authentication
↓
API Gateway
↓
LLM Gateway
↓
RAG
↓
Vector Database
↓
Bank Policies
↓
Azure AI Search /
OpenSearch /
Milvus /
PgVector
↓
Response

This keeps answers grounded in enterprise data rather than only the model’s pretraining.

AI Security

Security is critical.

Authentication

• OAuth
• OpenID Connect
• LDAP
• SSO

Authorization

RBAC

Data Scientist
↓
Namespace
↓
Notebook
↓
GPU

Network

NetworkPolicies isolate:

• Model servers
• Databases
• Pipelines
• Notebooks

Secrets

Store:

• API Keys
• OpenAI keys
• Hugging Face tokens
• Database credentials

using Kubernetes Secrets or an external secret manager such as HashiCorp Vault or cloud-native secret services.

AI Observability

Monitor:

• GPU utilization
• CPU
• Memory
• Inference latency
• Token generation rate
• Request throughput
• Error rates

Using:

Prometheus
↓
Grafana
↓
Alertmanager

Logging:

Vector
↓
Loki
or
Splunk
or
Elasticsearch

AI Storage

Training:

• S3
• Ceph
• OpenShift Data Foundation
• NFS

Model storage:

Model Registry
↓
Object Storage

AI Networking

Inference traffic:

Client
↓
Route
↓
Model Service
↓
LLM

Training traffic:

Notebook
↓
Object Storage
↓
GPU Worker

AI Scaling

Model serving uses Kubernetes autoscaling.

Traffic
↓
HPA
↓
2 Pods
↓
5 Pods
↓
20 Pods

For LLMs, scaling decisions often also consider GPU availability and model loading time.

GitOps for AI

Everything is stored in Git.

Git
↓
ArgoCD
↓
Notebook
↓
Pipeline
↓
Model
↓
Serving

AI CI/CD

Git Push
↓
Tekton
↓
Train
↓
Test
↓
Build
↓
Deploy

AI Monitoring

Monitor:

GPU
Inference
Latency
Memory
Model Accuracy
Token Usage
Failures

In production, you should also monitor:

• Model drift
• Data drift
• Hallucination rates (where measurable)
• Business KPIs

AI Governance

Enterprise AI requires:

• Model versioning
• Approval workflows
• Audit logging
• Data lineage
• Dataset versioning
• Explainability where required
• Compliance controls

AI Operators

Common Operators include:

• NVIDIA GPU Operator
• Red Hat OpenShift AI Operator
• OpenShift Pipelines Operator
• OpenShift GitOps Operator
• Service Mesh Operator (optional)
• OpenTelemetry Operator

AI + OpenShift Architecture

                        Users
                           │
                     Web / Mobile
                           │
                     OpenShift Route
                           │
                     API Gateway
                           │
                  Authentication (OAuth)
                           │
                 -----------------------
                 │                     │
                 ▼                     ▼
          Business APIs         AI Inference API
                                       │
                                 Model Serving
                                       │
                           ------------------------
                           │                      │
                     LLM (Granite/Llama)    Embedding Model
                           │                      │
                           └──────────┬───────────┘
                                      ▼
                               Vector Database
                                      │
                             Enterprise Documents
                                      │
                             S3 / ODF / Database

OpenShift AI vs Azure OpenAI

Many enterprises use both:

• Azure OpenAI for managed GPT models.
• OpenShift AI for on-premises inference, data sovereignty, or running open models.

Interview Questions

What is OpenShift AI?

Enterprise AI platform built on OpenShift for model development, training, deployment, and lifecycle management.

Why OpenShift for AI?

• Kubernetes-native
• GPU orchestration
• MLOps
• Security
• Multi-cloud
• GitOps
• Scalability
• Enterprise support

How do you deploy an LLM?

1. Deploy GPU Operator.
2. Configure GPU nodes.
3. Deploy the model server (e.g., vLLM).
4. Download or mount the model.
5. Expose via a Service and Route.
6. Monitor latency and resource usage.

How would you secure AI?

• RBAC
• NetworkPolicies
• OAuth/OIDC
• Secrets management
• Image signing
• Signed models where supported
• Audit logging
• TLS everywhere

Enterprise Banking AI Architecture

                 Customer
                    │
               Mobile App
                    │
             OpenShift Route
                    │
              API Gateway
                    │
        Authentication (OAuth/OIDC)
                    │
              AI Gateway Service
                    │
          ------------------------
          │                      │
          ▼                      ▼
      LLM Inference         Embedding Model
          │                      │
          └──────────┬───────────┘
                     ▼
              Vector Database
                     │
        ----------------------------
        │            │             │
        ▼            ▼             ▼
  Banking Docs   Policies     Knowledge Base

Interview Answer (2 Minutes)

“OpenShift provides an enterprise platform for building, deploying, and operating AI applications through Red Hat OpenShift AI. It supports the full MLOps lifecycle, including Jupyter notebooks for development, pipelines for model training, model registries, and scalable model serving. For generative AI, I typically design a Retrieval-Augmented Generation architecture where applications authenticate through OpenShift, call an API or model-serving layer, retrieve relevant enterprise documents from a vector database, and then pass that context to an LLM such as IBM Granite or Llama. OpenShift handles GPU scheduling through the NVIDIA GPU Operator, while GitOps, RBAC, NetworkPolicies, Secrets, and the monitoring stack provide secure and repeatable operations. For regulated industries like banking, I recommend combining RAG, centralized audit logging, strong identity controls, and model governance so that AI responses are based on approved enterprise knowledge and meet compliance requirements.”