OCP Cluster Upgrade — Best Practices Guide

The Golden Rule

Never upgrade a sick cluster. Every check below exists because upgrading on top of an existing problem turns a recoverable issue into a potential outage.

Phase 1 — Pre-Upgrade Preparation (Days Before)

1.1 Know Your Upgrade Path

OCP does not support arbitrary version jumps. Upgrades must follow a validated graph:

4.12 → 4.13 → 4.14       ✅ valid sequential path
4.12 → 4.14              ❌ may not be supported (check the graph)
4.14.5 → 4.14.12         ✅ z-stream (patch) always OK
4.14.12 → 4.15.0         ✅ minor version (must check graph)

Always verify the upgrade path first:

# Check available updates from the official graph
oc adm upgrade
# Output example:
# Recommended updates:
#   VERSION   IMAGE
#   4.14.13   quay.io/openshift-release-dev/ocp-release@sha256:...
#   4.14.14   quay.io/openshift-release-dev/ocp-release@sha256:...

Use the official upgrade path tool:

https://access.redhat.com/labs/ocpupgradegraph/update_path

1.2 Read the Release Notes

For every version you are jumping through:

https://docs.openshift.com/container-platform/<version>/release_notes/ocp-<version>-release-notes.html

Look specifically for:

• Deprecated APIs — if your workloads use removed APIs, they break post-upgrade
• Known issues — bugs that affect your environment
• Operator compatibility — OLM-managed operators may need separate upgrades
• Behavior changes — network policy, SCC, authentication changes

1.3 Check Operator Compatibility (OLM)

Operators installed via OLM have their own upgrade lifecycle, separate from OCP:

# List all installed operators and their current versions
oc get csv -A
# Check operator subscription channels
oc get subscription -A -o custom-columns=\
'NAME:.metadata.name,CHANNEL:.spec.channel,SOURCE:.spec.source'

Check the OperatorHub or vendor docs to confirm your operators support the target OCP version before upgrading the platform.

1.4 Check for Deprecated API Usage

OCP removes old API versions between minor releases. If your workloads use removed APIs they will fail silently after upgrade:

# Check for deprecated API usage in the cluster
oc get apirequestcounts -o json | jq -r \
  '.items[] |
   select(.status.removedInRelease != null) |
   select(.status.requestCount > 0) |
   "\(.metadata.name) → removed in \(.status.removedInRelease) | used \(.status.requestCount)x"'

Any resource showing removedInRelease with a non-zero requestCount must be migrated before upgrading.

1.5 Validate etcd Health

etcd is the most critical component. A degraded etcd during upgrade can corrupt cluster state:

# Check etcd cluster operator
oc get co etcd
# Check etcd member status
oc get etcd -o jsonpath=\
  '{.items[0].status.conditions[?(@.type=="EtcdMembersAvailable")].status}'
# Expected: True
# Check etcd pods directly
oc get pods -n openshift-etcd -l app=etcd
# Check etcd endpoint health
oc rsh -n openshift-etcd $(oc get pods -n openshift-etcd -l app=etcd \
  -o jsonpath='{.items[0].metadata.name}') \
  etcdctl endpoint health \
  --cluster \
  --cacert /etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt \
  --cert /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-$(hostname).crt \
  --key /etc/kubernetes/static-pod-certs/secrets/etcd-all-certs/etcd-peer-$(hostname).key

Expected output:

https://192.168.1.10:2379 is healthy: committed revision: 294858, took: 3ms
https://192.168.1.11:2379 is healthy: committed revision: 294858, took: 4ms
https://192.168.1.12:2379 is healthy: committed revision: 294858, took: 3ms

1.6 Verify All Nodes Are Ready

# All nodes must be Ready with no scheduling issues
oc get nodes
# Check for any taints blocking scheduling
oc get nodes -o custom-columns=\
'NAME:.metadata.name,STATUS:.status.conditions[-1].type,TAINTS:.spec.taints'
# Check node resource pressure
oc adm top nodes

1.7 Check All Cluster Operators

# All must be: AVAILABLE=True  PROGRESSING=False  DEGRADED=False
oc get co
# Quick one-liner to find any not healthy:
oc get co -o json | jq -r \
  '.items[] | select(
     (.status.conditions[] | select(.type=="Degraded" and .status=="True"))
     or
     (.status.conditions[] | select(.type=="Available" and .status=="False"))
   ) | .metadata.name'
# Expected: empty output

1.8 Check MachineConfigPools

# All pools must be: UPDATED=True  UPDATING=False  DEGRADED=False
oc get mcp
# Check for paused pools (they won't upgrade automatically)
oc get mcp -o jsonpath='{.items[?(@.spec.paused==true)].metadata.name}'

1.9 Review PodDisruptionBudgets

# Find PDBs that currently block all disruptions
oc get pdb -A -o json | jq -r \
  '.items[] |
   select(.status.disruptionsAllowed == 0) |
   "\(.metadata.namespace)/\(.metadata.name) — minAvailable:\(.spec.minAvailable)"'

For each blocking PDB, verify whether it will open up during drain (more replicas than minAvailable) or whether it is misconfigured and will permanently block node drains.

1.10 Check Active Alerts

# Via AlertManager route
ALERTMANAGER=$(oc get route alertmanager-main \
  -n openshift-monitoring \
  -o jsonpath='{.spec.host}')
TOKEN=$(oc create token prometheus-k8s -n openshift-monitoring)
curl -sk -H "Authorization: Bearer $TOKEN" \
  "https://${ALERTMANAGER}/api/v2/alerts" | \
  jq -r '.[] | select(.labels.severity=="critical") | .labels.alertname'

No critical alerts should be firing before upgrade begins.

Phase 2 — The Day Before

2.1 Take an etcd Backup

This is non-negotiable. If the upgrade corrupts cluster state, this is your only recovery path:

# SSH to a master node
ssh core@master-0
# Run backup
sudo /usr/local/bin/cluster-backup.sh /home/core/backup
# Verify both files created
ls -lh /home/core/backup/
# snapshot_<timestamp>.db
# static_kuberesources_<timestamp>.tar.gz
# Copy off-cluster immediately (S3, NFS, bastion host)
scp core@master-0:/home/core/backup/* bastion:/backups/pre-upgrade/

The backup must be from the same z-stream as the running cluster. A 4.14.12 cluster backup cannot restore a 4.15.x cluster.

2.2 Notify Stakeholders

Upgrades cause rolling node reboots. Even with zero downtime on properly configured workloads, communicate:

• Upgrade window start/end time
• Which node groups reboot and in what order (masters first, then workers)
• Expected impact on any workloads that are not highly available

2.3 Scale Down Non-Critical Workloads (Optional)

For resource-constrained clusters, reducing workload during upgrade gives the upgrade controller more room to drain nodes:

# Note current replicas first, then scale down
oc get deployments -A --no-headers | \
  awk '{print $1, $2, $3}' > /tmp/pre-upgrade-replicas.txt
# Scale down non-critical namespaces
oc scale deployment --all -n non-critical-namespace --replicas=0

Phase 3 — Initiating the Upgrade

3.1 Choose the Right Channel

OCP upgrade channels control which versions are offered:

stable-4.14    → production-recommended, fully validated
fast-4.14      → validated but released sooner than stable
candidate-4.14 → pre-release, NOT for production
eus-4.14       → Extended Update Support (long-term maintenance)

# Check current channel
oc get clusterversion -o jsonpath='{.items[0].spec.channel}'
# Set channel if needed
oc patch clusterversion version \
  --type merge \
  --patch '{"spec":{"channel":"stable-4.14"}}'

3.2 Trigger the Upgrade

# Upgrade to the latest available in channel
oc adm upgrade --to-latest=true
# Upgrade to a specific version (recommended — explicit and auditable)
oc adm upgrade --to=4.14.13
# For disconnected clusters (specify the full image digest)
oc adm upgrade \
  --to-image=quay.io/openshift-release-dev/ocp-release@sha256:<digest> \
  --allow-explicit-upgrade

3.3 Monitor the Upgrade Progress

# Watch CVO progress — the top-level view
watch oc get clusterversion
# More detail
oc describe clusterversion version | grep -A 30 "Conditions:"
# Watch all cluster operators rolling through upgrade
watch oc get co
# Watch nodes rebooting (masters first, then workers)
watch oc get nodes
# Tail CVO operator logs for real-time detail
oc logs -n openshift-cluster-version \
  $(oc get pods -n openshift-cluster-version \
    -o jsonpath='{.items[0].metadata.name}') -f

3.4 Understand the Upgrade Sequence

OCP upgrades in a strict order — never all at once:

1. CVO downloads new release image
        │
        ▼
2. Control plane operators update
   (etcd, kube-apiserver, controller-manager, scheduler)
   One at a time, with health checks between each
        │
        ▼
3. Master nodes drain → reboot → rejoin
   One at a time (never all 3 simultaneously)
        │
        ▼
4. Cluster operators update
   (authentication, console, ingress, monitoring, etc.)
        │
        ▼
5. Worker nodes drain → reboot → rejoin
   Controlled by MachineConfigPool maxUnavailable setting
        │
        ▼
6. CVO marks upgrade complete
   clusterversion: Progressing=False, Available=True

3.5 Control Worker Upgrade Speed

By default only 1 worker upgrades at a time. For large clusters you can safely increase this:

# Check current maxUnavailable (default is 1)
oc get mcp worker -o jsonpath='{.spec.maxUnavailable}'
# Allow up to 20% of workers to upgrade simultaneously
oc patch mcp worker \
  --type merge \
  --patch '{"spec":{"maxUnavailable":"20%"}}'

⚠️ Never set maxUnavailable higher than your workloads can tolerate. If minAvailable in your PDBs is 2 and you only have 3 replicas, draining 2 workers simultaneously will block.

Phase 4 — During the Upgrade

Watch for Stuck Upgrades

# If upgrade appears frozen, check:
# 1. Any operator stuck progressing?
oc get co | grep -v "True.*False.*False"
# 2. Any node stuck draining?
oc get nodes | grep SchedulingDisabled
# 3. Any pods blocking drain (PDB)?
oc get pdb -A | grep -v "^NAMESPACE"
# 4. Pending CSRs (workers trying to re-join after reboot)
oc get csr | grep Pending
oc get csr -o name | xargs oc adm certificate approve
# 5. Any pods stuck Terminating (blocking node drain)?
oc get pods -A | grep Terminating
# Force delete if stuck:
oc delete pod <pod> -n <ns> --grace-period=0 --force

Approve Pending CSRs

After each node reboots, it issues a CSR to re-register with the cluster. In some configurations these need manual approval:

# Watch for and approve all pending CSRs
watch -n 10 "oc get csr | grep Pending"
# Approve all at once
oc get csr -o name | xargs oc adm certificate approve

Phase 5 — Post-Upgrade Validation

5.1 Confirm Upgrade Completed

# Check cluster version
oc get clusterversion
# AVAILABLE=True  PROGRESSING=False  VERSION=<new-version>
# Confirm exact version
oc get clusterversion -o jsonpath='{.items[0].status.desired.version}'

5.2 Validate All Operators

# All must return to healthy state
oc get co
# All: AVAILABLE=True  PROGRESSING=False  DEGRADED=False

5.3 Validate All Nodes

# All nodes Ready, running new kubelet version
oc get nodes
# All: STATUS=Ready, VERSION=v1.27.x (new version)

5.4 Validate Workloads

# Check for any crashlooping or failed pods
oc get pods -A | grep -vE "Running|Completed|Succeeded"
# Check deployments fully available
oc get deployments -A | grep -v "^\(NAMESPACE\)" | \
  awk '$3 != $4 {print}'   # READY != DESIRED
# Check routes are responding
oc get routes -A

5.5 Take a Post-Upgrade etcd Backup

# The pre-upgrade backup is now stale — take a fresh one
sudo /usr/local/bin/cluster-backup.sh /home/core/backup-post-upgrade
scp core@master-0:/home/core/backup-post-upgrade/* bastion:/backups/post-upgrade/

Phase 6 — OLM Operator Upgrades

Platform upgrade does not upgrade OLM-managed operators. Do this separately:

# Check which operators have updates available
oc get csv -A | grep -v Succeeded
# Check subscription installPlanApproval settings
oc get subscription -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,APPROVAL:.spec.installPlanApproval'
# For Manual approval — approve pending InstallPlans
oc get installplan -A | grep RequiresApproval
oc patch installplan <name> -n <namespace> \
  --type merge \
  --patch '{"spec":{"approved":true}}'

Complete Upgrade Checklist

PRE-UPGRADE
  □ Verify upgrade path on Red Hat graph
  □ Read release notes for target version
  □ Check deprecated API usage (apirequestcounts)
  □ Verify etcd health (all 3 members healthy)
  □ All cluster operators: Available=True, Degraded=False
  □ All MachineConfigPools: Updated=True, Degraded=False
  □ All nodes: Ready
  □ No PDBs blocking all disruptions (disruptionsAllowed=0 on critical apps)
  □ No critical alerts firing
  □ Operator compatibility confirmed (OLM operators)
  □ Stakeholders notified
  □ etcd backup taken and stored off-cluster
  □ Upgrade channel set correctly (stable/fast/eus)
DURING UPGRADE
  □ Monitor CVO progress (watch oc get clusterversion)
  □ Monitor cluster operator rollout (watch oc get co)
  □ Monitor node reboots (watch oc get nodes)
  □ Approve pending CSRs as nodes rejoin
  □ Watch for stuck Terminating pods
  □ Watch for PDB-blocked drains
POST-UPGRADE
  □ clusterversion: Available=True, Progressing=False
  □ All cluster operators healthy
  □ All nodes Ready on new version
  □ All workload pods Running/Completed
  □ All routes responding
  □ Upgrade OLM operators to compatible versions
  □ Take fresh etcd backup
  □ Update documentation / CMDB with new version

Key Timings to Expect

Planning an OpenShift Container Platform (OCP) upgrade requires structured validation across 5 distinct phases. Because OpenShift performs rolling updates of both the control plane and the host OS (RHCOS), executing pre-upgrade checks prevents node-drain locks, degraded operators, or broken API dependencies.

Phase 1: Pre-Upgrade Review & Path Mapping

1. Verify Your Upgrade Path:
   • OpenShift does not support skipping Y-stream releases (e.g., jumping from 4.14 to 4.16 directly). You must upgrade sequentially (4.14 → 4.15 → 4.16).
   • Check your configured update channel (stable-4.x, fast-4.x, or eus-4.x).For enterprise production environments, stick to stable-4.x or eus-4.x (Extended Update Support).
   • Query the OpenShift Update Service (OSUS) via CLI:
     adm upgrade
2. Audit Deprecated & Removed Kubernetes APIs:
   • Before a minor version jump (e.g., 4.15 → 4.16), check if your workloads use removed K8s API versions.
   • Review cluster alerts for API deprecations and migrate manifests accordingly. OCP will block minor version upgrades if administrator acknowledgment is missing for API removals.
3. Check Operator & OLM Compatibility:
   • Non-core platform operators (installed via OperatorHub/OLM) must be upgraded to versions compatible with the target OCP release before updating the cluster core.

Phase 2: Cluster Health & Capacity Assessment

Run a complete health scan. Never trigger an upgrade on a degraded cluster.

1. Verify ClusterOperator Status:Ensure all core operators report AVAILABLE=True, PROGRESSING=False, and DEGRADED=False:
   oc get clusteroperator
2. Verify MachineConfigPool (MCP) Health:Ensure all node pools are updated and none are paused or degraded:
   oc get mcp (If an MCP is paused for canary testing, verify it is intentional before proceeding.)
3. Check Node Capacity & PodDisruptionBudgets (PDBs):
   • Node Capacity: During node-draining, remaining workers must have enough CPU/RAM capacity to host evacuated workloads.
   • PDB Deadlocks: Inspect custom PodDisruptionBudgets (oc get pdb -A). If minAvailable is set equal to the total replica count (or maxUnavailable: 0), node drains will stall, hanging the upgrade loop.
4. Clear Active Alerts:In the console, go to Observe → Alerting and resolve all critical alerts.

Phase 3: Mandatory Backups (Non-Negotiable)

Because OpenShift updates are fail-forward (rolling back to an older minor version is not supported), taking backups prior to execution is mandatory:

1. Backup etcd: Take a manual snapshot of the etcd database from a control plane node:
   oc debug node/<master-node-name> -- /usr/local/bin/cluster-backup.sh /host/var/usr/temp/etcd-backup
2. Export Cluster Custom Resources (CRDs): Export your custom deployment manifests, routes, and secret configurations using a GitOps tool (ArgoCD/ACM) or CLI backups.
3. Storage Snapshots: Create CSI volume snapshots for critical stateful application data.

Phase 4: Execution & Monitoring

1. Trigger the Upgrade:
   • Via Web Console:Navigate to Administration → Cluster Settings → Details, select your desired target version, and confirm.
   • Via CLI:
     oc adm upgrade --to=<target_version>
2. Monitor the Phased Rollout:
   • Control Plane Phase: Watch the CVO reconcile cluster operators:
     oc get clusterversion -w
   • Worker Pool Phase: Watch the MCO cordon, drain, re-image (rpm-ostree), reboot, and uncordon nodes pool-by-pool:
     oc get nodes
   • oc get mcp

Phase 5: Post-Upgrade Validation

Once the CVO reports Cluster version is <target_version>:

• [ ] Verify all nodes report Ready and run the new RHCOS kernel version.
• [ ] Confirm all ClusterOperators return to an AVAILABLE=True state.
• [ ] Verify routes, ingress controllers, and critical business applications are accepting traffic.
• [ ] Re-enable any paused MachineConfigPools or automated maintenance windows.

In OpenShift (OCP) and Kubernetes, a taint is a core scheduling mechanism applied to a node that allows the node to repel pods.

Think of a taint as a “Keep Out” sign placed on a server.By default, the OpenShift scheduler will not place any application pod on a tainted node unless that pod explicitly carries a matching toleration (a “key pass”).

While Node Selectors and Node Affinity are used to attract pods to specific nodes, Taints and Tolerations are used to repel unwanted pods away from nodes.

1. Anatomy of a Taint

A taint consists of three components: a Key, an optional Value, and a Taint Effect:

Taint} = key=value:Effect

The Three Taint Effects:

• NoSchedule (Hard Restriction):New pods without a matching toleration will never be scheduled onto this node.However, any pods already running on the node before the taint was applied are left alone.
• PreferNoSchedule (Soft Restriction):The scheduler tries to avoid placing pods on this node, but if no other compute resources are available in the cluster, it will place the pod here anyway.
• NoExecute (Eviction Restriction):The strongest effect.Any pod currently running on the node that does not tolerate the taint is immediately evicted (killed and rescheduled elsewhere).

2. Common Real-World Use Cases in OpenShift

• Dedicated Infrastructure Nodes:Isolating OpenShift Infra components (Ingress Routers, Monitoring, Image Registry) onto dedicated worker nodes so user applications cannot drain their resources.
• Specialized Hardware (GPUs / High-RAM):Tainting nodes that have expensive NVIDIA GPUs (gpu=true:NoSchedule) so standard web apps don’t accidentally get scheduled on them, reserving the hardware strictly for AI/ML workloads.
• Master Node Isolation:OpenShift automatically places a taint on Control Plane (master) nodes (node-role.kubernetes.io/master=:NoSchedule) so user workloads only run on worker nodes.
• Node Maintenance / Failure (Taint-based Eviction):When a node loses network connection or runs out of disk, OpenShift automatically applies temporary system taints like node.kubernetes.io/unreachable:NoExecute or node.kubernetes.io/disk-pressure:NoSchedule.

3. Declarative Setup Example

Step A: Applying a Taint to an OCP Node

Using the CLI, an administrator marks a node designated for GPU processing:

oc adm taint nodes worker-gpu-0 gpu=nvidia:NoSchedule

Step B: Adding a Toleration to a Pod Deployment

To allow a machine learning pod to run on that node, you add a tolerations block to its PodSpec:

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-trainer
spec:
  template:
    spec:
      containers:
        - name: cuda-runner
          image: quay.io/ai-lab/cuda-runner:latest
      # The Pod MUST have this matching toleration to land on worker-gpu-0
      tolerations:
        - key: "gpu"
          operator: "Equal"
          value: "nvidia"
          effect: "NoSchedule"

Note: A toleration allows a pod to run on a tainted node, but it does not force it to go there.If you want to force the pod onto that specific GPU node, combine the Toleration with a nodeSelector or NodeAffinity.

OpenShift and AI is one of the fastest-growing areas in enterprise IT. Red Hat has built an entire AI platform around OpenShift, making it possible to run Generative AI, Machine Learning, MLOps, and AI inference on the same Kubernetes platform that hosts enterprise applications.

For an OpenShift Architect interview, you should understand not only AI concepts but also how OpenShift integrates with AI services.

OpenShift AI Ecosystem

                           Users
                             │
                   Web / Mobile Apps
                             │
                    OpenShift Routes
                             │
                ------------------------
                │                      │
          Business Apps         AI Applications
                │                      │
                └──────────────┬───────┘
                               │
                       OpenShift Platform
                               │
      ------------------------------------------------------
      │            │            │             │
   RHOAI       OpenShift     GPU Nodes    OpenShift GitOps
               Pipelines
      │
      ▼
 Kubeflow / Jupyter / Model Serving
      │
      ▼
 LLM (Llama, Granite, Mistral, GPT, etc.)
      │
      ▼
 Vector Database / AI Search
      │
      ▼
 Enterprise Data Sources

What is Red Hat OpenShift AI (RHOAI)?

Red Hat OpenShift AI (formerly Red Hat OpenShift Data Science) is Red Hat’s enterprise AI/ML platform built on OpenShift.

It provides:

• Jupyter Notebooks
• Model training
• Model serving
• Pipelines
• Experiment tracking
• GPU scheduling
• AI model deployment
• MLOps
• LLM serving
• Vector database integration

Think of it as:

OpenShift
        +
AI Platform
        +
MLOps

AI Architecture on OpenShift

              Developers

                 │

           Git Repository

                 │

           OpenShift GitOps

                 │

           AI Pipeline (Tekton)

                 │

        ------------------------

        │                      │

  Train Model           Build Container

        │                      │

        ------------------------

                 │

          Model Registry

                 │

          Model Serving

                 │

            REST API

                 │

             Applications

Major AI Components

1. Jupyter Notebooks

Used for:

• Data science
• Python
• TensorFlow
• PyTorch
• Hugging Face
• Experiments

Example:

Data Scientist
↓
Notebook
↓
Python
↓
Train Model

2. Model Serving

Once trained:

Model
↓
Model Server
↓
REST API
↓
Application

Supported technologies include:

• KServe
• vLLM (commonly used for LLM inference)
• NVIDIA Triton Inference Server
• Caikit (used in some Red Hat AI scenarios)

3. Pipelines

Uses Kubeflow Pipelines (or Tekton depending on workflow).

Example:

Dataset
↓
Preprocessing
↓
Training
↓
Evaluation
↓
Deploy
↓
Production

Everything becomes repeatable.

GPU Support

OpenShift schedules GPUs like any other resource.

Architecture:

GPU Node
↓
NVIDIA Driver
↓
NVIDIA Device Plugin
↓
Pod Requests GPU
↓
GPU Allocated

Example Pod:

resources:
  limits:
    nvidia.com/gpu: 1

AI Model Lifecycle

Collect Data
↓
Train Model
↓
Validate
↓
Containerize
↓
Deploy
↓
Monitor
↓
Retrain

This is called MLOps.

OpenShift AI with LLMs

You can deploy models like:

• IBM Granite
• Llama 3.x
• Mistral
• DeepSeek (where licensing and hardware permit)
• Gemma
• Phi

Architecture:

User
↓
Chat Application
↓
OpenShift Route
↓
Model Server
↓
LLM
↓
Response

RAG (Retrieval-Augmented Generation)

This is one of the most common enterprise AI architectures.

User Question
↓
Embedding Model
↓
Vector Database
↓
Relevant Documents
↓
LLM
↓
Answer

Instead of relying only on the model’s knowledge, the LLM searches company documents.

Enterprise Banking AI Example

Customer
↓
Chatbot
↓
OpenShift Route
↓
Authentication
↓
API Gateway
↓
LLM Gateway
↓
RAG
↓
Vector Database
↓
Bank Policies
↓
Azure AI Search /
OpenSearch /
Milvus /
PgVector
↓
Response

This keeps answers grounded in enterprise data rather than only the model’s pretraining.

AI Security

Security is critical.

Authentication

• OAuth
• OpenID Connect
• LDAP
• SSO

Authorization

RBAC

Data Scientist
↓
Namespace
↓
Notebook
↓
GPU

Network

NetworkPolicies isolate:

• Model servers
• Databases
• Pipelines
• Notebooks

Secrets

Store:

• API Keys
• OpenAI keys
• Hugging Face tokens
• Database credentials

using Kubernetes Secrets or an external secret manager such as HashiCorp Vault or cloud-native secret services.

AI Observability

Monitor:

• GPU utilization
• CPU
• Memory
• Inference latency
• Token generation rate
• Request throughput
• Error rates

Using:

Prometheus
↓
Grafana
↓
Alertmanager

Logging:

Vector
↓
Loki
or
Splunk
or
Elasticsearch

AI Storage

Training:

• S3
• Ceph
• OpenShift Data Foundation
• NFS

Model storage:

Model Registry
↓
Object Storage

AI Networking

Inference traffic:

Client
↓
Route
↓
Model Service
↓
LLM

Training traffic:

Notebook
↓
Object Storage
↓
GPU Worker

AI Scaling

Model serving uses Kubernetes autoscaling.

Traffic
↓
HPA
↓
2 Pods
↓
5 Pods
↓
20 Pods

For LLMs, scaling decisions often also consider GPU availability and model loading time.

GitOps for AI

Everything is stored in Git.

Git
↓
ArgoCD
↓
Notebook
↓
Pipeline
↓
Model
↓
Serving

AI CI/CD

Git Push
↓
Tekton
↓
Train
↓
Test
↓
Build
↓
Deploy

AI Monitoring

Monitor:

GPU
Inference
Latency
Memory
Model Accuracy
Token Usage
Failures

In production, you should also monitor:

• Model drift
• Data drift
• Hallucination rates (where measurable)
• Business KPIs

AI Governance

Enterprise AI requires:

• Model versioning
• Approval workflows
• Audit logging
• Data lineage
• Dataset versioning
• Explainability where required
• Compliance controls

AI Operators

Common Operators include:

• NVIDIA GPU Operator
• Red Hat OpenShift AI Operator
• OpenShift Pipelines Operator
• OpenShift GitOps Operator
• Service Mesh Operator (optional)
• OpenTelemetry Operator

AI + OpenShift Architecture

                        Users
                           │
                     Web / Mobile
                           │
                     OpenShift Route
                           │
                     API Gateway
                           │
                  Authentication (OAuth)
                           │
                 -----------------------
                 │                     │
                 ▼                     ▼
          Business APIs         AI Inference API
                                       │
                                 Model Serving
                                       │
                           ------------------------
                           │                      │
                     LLM (Granite/Llama)    Embedding Model
                           │                      │
                           └──────────┬───────────┘
                                      ▼
                               Vector Database
                                      │
                             Enterprise Documents
                                      │
                             S3 / ODF / Database

OpenShift AI vs Azure OpenAI

Many enterprises use both:

• Azure OpenAI for managed GPT models.
• OpenShift AI for on-premises inference, data sovereignty, or running open models.

Interview Questions

What is OpenShift AI?

Enterprise AI platform built on OpenShift for model development, training, deployment, and lifecycle management.

Why OpenShift for AI?

• Kubernetes-native
• GPU orchestration
• MLOps
• Security
• Multi-cloud
• GitOps
• Scalability
• Enterprise support

How do you deploy an LLM?

1. Deploy GPU Operator.
2. Configure GPU nodes.
3. Deploy the model server (e.g., vLLM).
4. Download or mount the model.
5. Expose via a Service and Route.
6. Monitor latency and resource usage.

How would you secure AI?

• RBAC
• NetworkPolicies
• OAuth/OIDC
• Secrets management
• Image signing
• Signed models where supported
• Audit logging
• TLS everywhere

Enterprise Banking AI Architecture

                 Customer
                    │
               Mobile App
                    │
             OpenShift Route
                    │
              API Gateway
                    │
        Authentication (OAuth/OIDC)
                    │
              AI Gateway Service
                    │
          ------------------------
          │                      │
          ▼                      ▼
      LLM Inference         Embedding Model
          │                      │
          └──────────┬───────────┘
                     ▼
              Vector Database
                     │
        ----------------------------
        │            │             │
        ▼            ▼             ▼
  Banking Docs   Policies     Knowledge Base

Interview Answer (2 Minutes)

“OpenShift provides an enterprise platform for building, deploying, and operating AI applications through Red Hat OpenShift AI. It supports the full MLOps lifecycle, including Jupyter notebooks for development, pipelines for model training, model registries, and scalable model serving. For generative AI, I typically design a Retrieval-Augmented Generation architecture where applications authenticate through OpenShift, call an API or model-serving layer, retrieve relevant enterprise documents from a vector database, and then pass that context to an LLM such as IBM Granite or Llama. OpenShift handles GPU scheduling through the NVIDIA GPU Operator, while GitOps, RBAC, NetworkPolicies, Secrets, and the monitoring stack provide secure and repeatable operations. For regulated industries like banking, I recommend combining RAG, centralized audit logging, strong identity controls, and model governance so that AI responses are based on approved enterprise knowledge and meet compliance requirements.”

DNS Operator in OpenShift

The DNS Operator deploys, configures, and continuously manages CoreDNS, which provides DNS resolution and Kubernetes Service discovery for Pods inside an OpenShift cluster.

A simple way to remember it:

The DNS Operator manages the DNS platform; CoreDNS answers the DNS queries.

DNS Operator
     │
     ▼
CoreDNS DaemonSet
     │
     ▼
DNS Service IP
     │
     ▼
Pods resolve Services and external names

The DNS Operator is installed automatically during OpenShift installation. It runs as a Deployment, while CoreDNS runs as a DaemonSet managed by that Operator. (Red Hat Documentation)

DNS Operator vs CoreDNS

These are separate components:

DNS Operator
      │
      ▼
Creates and manages
      │
      ├── CoreDNS DaemonSet
      ├── DNS Service
      ├── CoreDNS ConfigMap
      └── Node resolver DaemonSet

The Operator implements the cluster-scoped DNS API in the operator.openshift.io API group. (Red Hat Documentation)

Where the Components Run

The DNS Operator normally runs in:

openshift-dns-operator

CoreDNS and node-resolver components normally run in:

openshift-dns

Check the Operator:

oc get deployment -n openshift-dns-operator

Check DNS Pods:

oc get pods -n openshift-dns -o wide

Check DaemonSets:

oc get daemonset -n openshift-dns

Typical output includes:

dns-default
node-resolver

Main DNS Architecture

Application Pod
      │
      │ DNS query
      ▼
Cluster DNS Service IP
      │
      ▼
CoreDNS Pod
      │
      ├── Internal cluster name?
      │       └── Answer from Kubernetes API data
      │
      └── External name?
              └── Forward to upstream DNS resolver

The default internal cluster domain is:

cluster.local

CoreDNS provides resolution for internal names such as:

service.namespace.svc.cluster.local

The default DNS resource is named default. (Red Hat Documentation)

Kubernetes Service Discovery

Suppose you create this Service:

apiVersion: v1
kind: Service
metadata:
  name: payments-api
  namespace: banking
spec:
  selector:
    app: payments-api
  ports:
    - port: 8443

CoreDNS creates DNS-based service discovery for it.

A Pod in the same namespace can use:

payments-api

A Pod in another namespace can use:

payments-api.banking

The complete DNS name is:

payments-api.banking.svc.cluster.local

Flow:

Client Pod
    │
    ▼
payments-api.banking.svc.cluster.local
    │
    ▼
CoreDNS
    │
    ▼
Service ClusterIP
    │
    ▼
Ready application Pods

Pod DNS Configuration

The kubelet configures each normal Pod to use the cluster DNS Service.

Inside a Pod:

cat /etc/resolv.conf

Typical content:

search banking.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.30.0.10
options ndots:5

The exact DNS Service IP depends on the cluster Service network. OpenShift commonly assigns the DNS Service a stable IP from the Service CIDR. (Red Hat Documentation)

You can check it with:

oc get dns.operator/default -o yaml

Look under:

status:
  clusterDomain: cluster.local
  clusterIP: 172.30.0.10

Why CoreDNS Runs as a DaemonSet

CoreDNS normally runs through the dns-default DaemonSet.

Node 1          Node 2          Node 3
  │               │               │
CoreDNS         CoreDNS         CoreDNS

Benefits:

• DNS capacity scales as nodes are added.
• DNS is distributed across the cluster.
• A single DNS Pod failure does not stop name resolution.
• Requests can be handled close to workloads.
• DNS remains available if individual nodes fail.

The DNS Operator creates the CoreDNS DaemonSet and exposes it through a Service with a stable IP. (Red Hat Documentation)

Internal Name Resolution

CoreDNS watches the Kubernetes API for Services, EndpointSlices, namespaces, and other relevant resources.

For a Service query:

Pod asks for:
payments-api.banking.svc.cluster.local
                │
                ▼
CoreDNS queries its Kubernetes data
                │
                ▼
Returns the Service ClusterIP

For a headless Service:

spec:
  clusterIP: None

CoreDNS can return Pod or endpoint IPs instead of one Service ClusterIP.

This is useful for:

• StatefulSets
• Databases
• Cluster members
• Direct endpoint discovery

External DNS Resolution

If a Pod asks for an external name:

api.example.com

CoreDNS normally forwards the query to an upstream resolver.

Application Pod
      │
      ▼
CoreDNS
      │
      ▼
Corporate or cloud DNS
      │
      ▼
External DNS result

Upstream resolvers normally come from node resolver configuration or from explicit DNS Operator settings.

DNS Forwarding

The DNS Operator supports forwarding particular DNS zones to specific resolvers.

Example requirement:

*.corp.bank.local
        │
        ▼
Corporate DNS servers

A simplified configuration is:

apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  servers:
    - name: corporate-dns
      zones:
        - corp.bank.local
      forwardPlugin:
        upstreams:
          - 10.20.30.10
          - 10.20.30.11

Flow:

Query: database.corp.bank.local
              │
              ▼
CoreDNS matches corp.bank.local
              │
              ▼
Forwards to 10.20.30.10 / 10.20.30.11

Queries that do not match a configured zone fall back to the configured upstream resolvers. (Red Hat Documentation)

Edit DNS configuration with:

oc edit dns.operator/default

Do not manually edit the Operator-generated CoreDNS ConfigMap as the primary configuration method.

DNS Caching

CoreDNS caches successful and unsuccessful DNS responses.

This reduces:

• Query latency
• Load on upstream DNS servers
• Repeated external lookups
• Network traffic

OpenShift supports configuring positive and negative cache TTLs through the DNS Operator resource. (Red Hat Documentation)

Example:

apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  cache:
    positiveTTL: 1h
    negativeTTL: 30s

Be careful with TTL tuning:

• Very low TTLs increase DNS load.
• Very high TTLs can retain stale results longer.
• Negative caching can make a recently created record appear unavailable until the negative TTL expires.

Node Resolver

The DNS Operator also manages a node-resolver DaemonSet.

Check it with:

oc get daemonset node-resolver -n openshift-dns

The node-resolver component helps maintain node-level name resolution information, including managed entries in the node’s /etc/hosts where required.

DNS Operator
      │
      ├── dns-default DaemonSet
      │       └── Cluster DNS
      │
      └── node-resolver DaemonSet
              └── Node hostname resolution support

Reconciliation Loop

The DNS Operator continuously compares desired DNS configuration with the actual resources.

DNS resource: default
        │
        ▼
DNS Operator reads desired state
        │
        ▼
Checks DaemonSets, Service and ConfigMap
        │
        ▼
Difference found?
     ┌──┴──┐
     │     │
    No    Yes
     │     │
     ▼     ▼
  Wait   Recreate or update resources
               │
               ▼
         Validate DNS availability
               │
               ▼
        Update Operator status

Examples that trigger reconciliation:

• CoreDNS Pod fails.
• DNS forwarding configuration changes.
• Cache settings change.
• OpenShift is upgraded.
• A managed ConfigMap changes.
• Node placement is modified.
• A DaemonSet does not match the desired configuration.

DNS Operator Health

Check the ClusterOperator:

oc get clusteroperator dns

Healthy output:

NAME   AVAILABLE   PROGRESSING   DEGRADED
dns    True        False         False

Interpretation:

Detailed information:

oc describe clusteroperator dns

The Operator considers DNS available when the DNS Service has a ClusterIP and at least one CoreDNS Pod is available. (Red Hat Documentation)

Useful Commands

Check the Operator

oc get deployment dns-operator \
  -n openshift-dns-operator

oc logs -n openshift-dns-operator \
  deployment/dns-operator

Check DNS configuration

oc get dns.operator/default -o yaml

oc describe dns.operator/default

Check CoreDNS Pods

oc get pods -n openshift-dns -o wide

Check DaemonSets

oc get daemonset -n openshift-dns

Check DNS Service

oc get service dns-default -n openshift-dns

Check generated configuration

oc get configmap dns-default \
  -n openshift-dns \
  -o yaml

Check events

oc get events -n openshift-dns \
  --sort-by='.lastTimestamp'

Testing DNS from a Pod

Create a temporary troubleshooting Pod:

oc run dns-test \
  --image=registry.access.redhat.com/ubi9/ubi-minimal \
  --restart=Never \
  -- sleep 3600

Enter it:

oc rsh dns-test

Check its resolver configuration:

cat /etc/resolv.conf

Test an internal Service:

getent hosts kubernetes.default.svc.cluster.local

Test your application:

getent hosts payments-api.banking.svc.cluster.local

Test an external address:

getent hosts example.com

Delete the Pod afterward:

oc delete pod dns-test

Troubleshooting Flow

Use this sequence:

Application DNS error
        │
        ▼
Check Pod /etc/resolv.conf
        │
        ▼
Test short and full service names
        │
        ▼
Check Service and EndpointSlices
        │
        ▼
Check DNS Service ClusterIP
        │
        ▼
Check CoreDNS Pods
        │
        ▼
Check DNS Operator status
        │
        ▼
Check CoreDNS and Operator logs
        │
        ▼
Check upstream DNS and network policies

Scenario 1: Internal Service Does Not Resolve

Example:

payments-api.banking.svc.cluster.local

Check the Service:

oc get svc payments-api -n banking

Check namespace and spelling:

oc get namespace banking

Test the full name:

getent hosts payments-api.banking.svc.cluster.local

Then check:

oc get pods -n openshift-dns
oc get svc dns-default -n openshift-dns

Remember:

• DNS can resolve a Service even if its backend Pods are unhealthy.
• DNS resolution does not prove the application itself is reachable.
• EndpointSlices affect application traffic, not necessarily the existence of the Service DNS record.

Scenario 2: External Names Fail but Internal Names Work

For example:

payments-api.banking.svc.cluster.local → works
example.com                            → fails

This usually indicates an upstream-forwarding issue.

Check:

• Upstream DNS server availability
• DNS Operator forwarding configuration
• Firewall access to UDP/TCP port 53
• Node /etc/resolv.conf
• Corporate DNS reachability
• Egress restrictions
• Forwarding loops

Review CoreDNS logs:

oc logs -n openshift-dns <dns-default-pod> \
  -c dns

Scenario 3: One Node Has DNS Problems

If Pods on one node fail DNS while other nodes work:

oc get pods -n openshift-dns -o wide

Check whether the affected node has a healthy CoreDNS Pod.

Then examine:

• Node networking
• OVN connectivity
• DNS DaemonSet Pod
• kubelet configuration
• Service routing
• Firewall rules
• MTU problems
• Node resource pressure

Test from Pods scheduled on both a healthy and affected node.

Scenario 4: DNS Query Is Slow

Possible causes:

• Slow upstream resolver
• DNS forwarding loop
• Packet loss
• Excessive query volume
• Too-low cache TTL
• CoreDNS CPU throttling
• Node network problems
• Search-domain expansion caused by ndots
• External queries being tried as multiple internal names first

Measure lookup time:

time getent hosts external.example.com

Compare internal and external queries separately.

Scenario 5: CoreDNS Pods Are Pending

Check:

oc describe pod <dns-pod> -n openshift-dns

Possible causes:

• Node selector mismatch
• Missing toleration
• Insufficient CPU or memory
• Node taints
• Scheduling restrictions
• Image pull problem

CoreDNS and node-resolver placement can be controlled using node selectors and tolerations in the DNS Operator configuration. (Red Hat Documentation)

Common DNS Errors

Could not resolve host

Possible causes:

• CoreDNS unavailable
• Wrong Pod resolver configuration
• Upstream DNS failure
• NetworkPolicy or firewall blocking DNS
• Typographical error

SERVFAIL

Possible causes:

• Upstream resolver failure
• DNS forwarding loop
• Invalid zone configuration
• DNSSEC or upstream issue

NXDOMAIN

Means the requested name does not exist according to the resolver.

Check:

• Service name
• Namespace
• DNS zone
• Negative cache
• External record creation

DNS works but connection fails

DNS only returned an IP address. Check:

• Service port
• EndpointSlices
• Pod readiness
• NetworkPolicy
• Application process
• TLS configuration

DNS Operator vs DNS Operator Configuration

Avoid directly editing:

CoreDNS DaemonSet
dns-default Service
generated dns-default ConfigMap

These are Operator-managed and changes may be reverted.

Configure DNS using:

oc edit dns.operator/default

The Operator then generates the appropriate CoreDNS configuration and performs reconciliation.

Relationship with Other Operators

Cluster Version Operator
        │
        ▼
DNS Operator
        │
        ├── CoreDNS DaemonSet
        ├── Node Resolver DaemonSet
        ├── DNS Service
        └── DNS ConfigMap

Dependencies include:

Interview Answer

The OpenShift DNS Operator deploys and manages CoreDNS to provide internal name resolution and Kubernetes Service discovery. The Operator runs as a Deployment in openshift-dns-operator, while it manages the dns-default CoreDNS DaemonSet, the node-resolver DaemonSet, a DNS Service with a stable ClusterIP, and the generated CoreDNS configuration in openshift-dns.

Pods send DNS queries to the DNS Service IP configured in their /etc/resolv.conf. CoreDNS resolves internal names such as service.namespace.svc.cluster.local using Kubernetes API information and forwards external or configured private-zone queries to upstream resolvers. The Operator continuously reconciles these resources and supports configuration through the cluster-scoped dns.operator/default object, including forwarding, caching, and node placement.

For troubleshooting, I start with oc get co dns, inspect the DNS Operator and CoreDNS Pods, check the dns-default Service and ConfigMap, test DNS from a Pod, and determine whether the failure affects internal names, external names, or only Pods on a particular node. I then check upstream resolvers, OVN connectivity, NetworkPolicies, firewall rules and CoreDNS logs.

kubelet on OCP Master Nodes

Why kubelet Runs on Masters

In OCP, the control plane components are not system services (systemd units). They run as static pods — and kubelet is the only thing that can run static pods.

Without kubelet on masters:
  ❌ etcd cannot start
  ❌ kube-apiserver cannot start
  ❌ kube-controller-manager cannot start
  ❌ kube-scheduler cannot start

kubelet IS the bootstrap mechanism for the entire control plane.

Static Pods — The Core Concept

kubelet watches one directory constantly:

/etc/kubernetes/manifests/
  ├── etcd-pod.yaml
  ├── kube-apiserver-pod.yaml
  ├── kube-controller-manager-pod.yaml
  └── kube-scheduler-pod.yaml

Any .yaml dropped here → kubelet starts it as a pod. Any .yaml removed → kubelet stops and removes it.

No API server needed. kubelet reads these files directly from disk. This is exactly why cluster-restore.sh works by moving manifests in and out of this directory — it’s controlling the control plane via kubelet’s static pod mechanism.

cluster-restore.sh
  └── mv etcd-pod.yaml → /etc/kubernetes/manifests/
              │
              ▼
           kubelet notices file change (inotify watch)
              │
              ▼
           kubelet tells CRI-O to start the etcd containers
              │
              ▼
           etcd is running

What kubelet Does on a Master (vs Worker)

Master Node Taint (workload isolation)

By default, OCP master nodes carry a taint that prevents regular workloads from landing on them:

oc describe node master-0 | grep Taint
# Taints: node-role.kubernetes.io/master:NoSchedule

So while kubelet on masters CAN run any pod, the scheduler won’t place regular workloads there unless you explicitly tolerate the taint. The taint does NOT affect static pods — kubelet runs those directly, bypassing the scheduler entirely.

kubelet Startup Sequence on a Master

This is what happens when a master node boots:

1. RHCOS boots
        │
        ▼
2. systemd starts kubelet.service
        │
        ▼
3. kubelet reads /etc/kubernetes/manifests/
        │
        ├── etcd-pod.yaml          → starts etcd containers
        ├── kube-apiserver-pod.yaml → starts API server containers
        ├── kube-controller-manager-pod.yaml
        └── kube-scheduler-pod.yaml
        │
        ▼
4. etcd becomes healthy (has quorum)
        │
        ▼
5. kube-apiserver connects to etcd, starts serving
        │
        ▼
6. kubelet registers THIS master node with the API server
        │
        ▼
7. Cluster operators come up (CVO, etcd-operator, etc.)

Notice: kubelet starts etcd, and etcd enables the API, and the API is what kubelet later registers with. kubelet bootstraps its own control plane and then registers with it.

Verify kubelet on a Master

# SSH into a master node
ssh core@master-0
# kubelet is a systemd service
systemctl status kubelet
# kubelet logs
journalctl -u kubelet -f
# kubelet process
ps aux | grep kubelet
# kubelet managing static pods
crictl pods | grep -E "etcd|apiserver|scheduler|controller"

What Happens if kubelet Dies on a Master

kubelet dies on master-0
        │
        ├── etcd container keeps running (CRI-O manages it independently)
        ├── But if etcd crashes → kubelet not there to restart it
        ├── kube-apiserver keeps running (same)
        │
        └── After 5 min: node shows NotReady (no heartbeat to API)
                │
                └── etcd operator detects degraded member
                        └── alerts fire, but cluster may still function
                            if other 2 masters are healthy

Key insight: CRI-O keeps containers alive after kubelet dies — but kubelet is the only thing that will restart a crashed container. Without kubelet, a crashed etcd stays dead.

The Dependency Chain in One View

RHCOS
  └── systemd
        └── kubelet.service          ← runs on master as a systemd unit
              └── reads /etc/kubernetes/manifests/
                    ├── etcd-pod.yaml
                    │     └── CRI-O runs etcd containers
                    │           └── etcd cluster (the database)
                    │
                    ├── kube-apiserver-pod.yaml
                    │     └── CRI-O runs API server containers
                    │           └── Kubernetes/OCP API
                    │
                    ├── kube-controller-manager-pod.yaml
                    │     └── CRI-O runs controller-manager
                    │
                    └── kube-scheduler-pod.yaml
                          └── CRI-O runs scheduler

The entire OCP control plane is just kubelet reading files from a directory and telling CRI-O what to run. This simplicity is what makes the restore procedure work — you control the control plane by controlling what kubelet sees in /etc/kubernetes/manifests/.