openshift

Mastering OpenShift on VMware and Bare Metal: Key Insights

techhadoop OCP , , , ,

Administering OpenShift on VMware vSphere or Bare Metal is significantly more complex than cloud environments because you are responsible for the “underlay” (the physical or virtual infrastructure) as well as the “overlay” (OpenShift).

In a 2026 interview, expect a focus on automation, connectivity in restricted environments, and hardware lifecycle.

1. Installation & Provisioning (The Foundation)

Q1: Compare IPI vs. UPI in the context of VMware vSphere.
  • IPI (Installer-Provisioned Infrastructure): The installer has the vCenter credentials. It automatically creates the Folder, Virtual Machines, and Resource Pools. It also handles the VIP (Virtual IPs) for the API and Ingress via Keepalived.
  • UPI (User-Provisioned Infrastructure): You manually create the VMs, set up the Load Balancers (F5, HAProxy), and configure DNS.
  • Interview Tip: Mention that IPI is preferred for speed and “automated scaling,” but UPI is often mandatory in “Brownfield” environments where the networking team won’t give the installer full control over the VLANs.
Q2: How does OpenShift interact with physical hardware for Bare Metal?

Answer: It uses the Metal3 project and the Bare Metal Operator (BMO).

  • The admin provides the BMC (Baseboard Management Controller) details—like IPMI, iDRAC (Dell), or iLO (HP)—to OpenShift.
  • OpenShift uses these to remotely power on the server, PXE boot it, and install RHCOS (Red Hat Enterprise Linux CoreOS).

2. Infrastructure Operations

Q3: What is a “Disconnected” (Air-Gapped) Installation?

Answer: Common in on-prem data centers with high security.

  • The Problem: OpenShift usually pulls images from quay.io.
  • The Solution: You must set up a Local Mirror Registry (like Red Hat Quay or Sonatype Nexus).
  • Process: You use the oc mirror plugin to download all required images to a portable disk, move it inside the secure zone, and push them to your local registry. You then configure the cluster to use an ImageContentSourcePolicy to redirect all image pulls to your local IP.
Q4: How do you handle storage on VMware vs. Bare Metal?
  • VMware: Use the vSphere CSI Driver. This allows OpenShift to talk to vCenter and dynamically provision .vmdk files as Persistent Volumes (PVs).
  • Bare Metal: You typically use LVM (Local Storage Operator) for fast local SSDs or OpenShift Data Foundation (ODF) (based on Ceph). ODF is the industry standard for on-prem because it provides S3-compatible, Block, and File storage within the cluster itself.

3. High Availability & Networking

Q5: On Bare Metal, how do you handle Load Balancing for the API and Ingress?

Answer: Since there is no “AWS ELB” on-prem, you have two choices:

  1. External: Use a physical appliance like an F5 Big-IP or a pair of HAProxy nodes managed by your team.
  2. Internal (MetalLB): Use the MetalLB Operator. It allows you to assign a range of IPs from your corporate network to the OpenShift Router so it can act like a cloud load balancer.
Q6: What happens if a Master (Control Plane) node dies in a Bare Metal cluster?

Answer: * Quorum: You must have 3 Masters to maintain an etcd quorum. If one dies, the cluster survives. If two die, the API becomes read-only or crashes.

  • Recovery: On Bare Metal, recovery is manual. You must reinstall the OS, use the kube-etcd-operator to remove the old member, and then use the cluster-bootstrap process to add the new node back into the etcd ring.

4. Advanced Troubleshooting

Q7: A worker node is “NotReady” on VMware. What is your first check?

Answer: Beyond the logs, I check the VMware Tools status and Time Sync.

  • If the ESXi host and the VM have a clock drift (common if NTP is misconfigured), the certificates for the Kubelet will fail to validate, and the node will go NotReady.
  • I would also check the MachineConfigPool (MCP). If the node is stuck in “Updating,” it might be failing to pull an OS image from the internal registry.
Q8: What is “Assisted Installer”?

Answer: It’s the modern way to install OpenShift on-prem. It provides a web-based GUI that generates a “Discovery ISO.” You boot your physical servers with this ISO; they “check in” to the portal, and you can then click “Install” to deploy the whole cluster without writing complex YAML files.

Technical “Buzzwords” for 2026:

  • OVN-Kubernetes: The default network plugin (replaces OpenShift SDN).
  • LVM Storage: Used for high-performance databases on bare metal.
  • Red Hat Advanced Cluster Management (RHACM): If the company has multiple on-prem clusters, they will use this to manage them all from one dashboard.

Debugging etcd is the highest level of OpenShift administration. If etcd is healthy, the cluster is healthy; if etcd is failing, the API will be sluggish or completely unresponsive.

Here is the technical deep-dive on how to diagnose and fix etcd on-premise.

1. Checking the High-Level Status

Before diving into logs, check if the Etcd Operator is happy. If the operator is degraded, it usually means it’s struggling to manage the quorum.

# Check the status of the etcd cluster operator
oc get clusteroperator etcd
# Check the status of the individual etcd pods
oc get pods -n openshift-etcd -l app=etcd

2. Testing Quorum and Health (The etcdctl way)

In OpenShift 4.x, etcd runs as Static Pods on the master nodes. To run diagnostic commands, you must use a helper script or exec into the container.

The “Is it alive?” check:

# Get a list of etcd members and their health
oc rsh -n openshift-etcd etcd-master-0 etcdctl endpoint health --cluster -w table
The Performance check (Disk Latency):

On-premise (especially VMware), Disk I/O latency is the #1 killer of etcd. If your storage is slow, etcd will lose quorum.

# Check the sync duration
oc rsh -n openshift-etcd etcd-master-0 etcdctl check perf

Interview Pro-Tip: Mention that etcd requires fsync latency of less than 10ms. If it’s higher, your underlying VMware datastore or Bare Metal disks are too slow for an enterprise cluster.

3. Investigating Logs

If a pod is crashing, check the logs specifically for “leader” issues or “wal” (Write Ahead Log) errors.

# View the last 100 lines of logs from a specific member
oc logs -n openshift-etcd etcd-master-0 -c etcd --tail=100

What to look for:

  • "lost leader": Indicates network instability between master nodes.
  • "apply entries took too long": Indicates slow disk or high CPU pressure on the master node.
  • "database space exceeded": The 8GB quota has been reached (requires a defrag).

4. Critical Recovery: The “Master Node Replacement”

If a master node (e.g., master-1) hardware fails permanently on Bare Metal, you must follow these steps to restore the cluster health:

  1. Remove the ghost member:Tell etcd to stop looking for the dead node.Bashoc rsh -n openshift-etcd etcd-master-0 etcdctl member list oc rsh -n openshift-etcd etcd-master-0 etcdctl member remove <dead-member-id>
  2. Clean up the Node object:oc delete node master-1
  3. Re-provision: Boot the new hardware with the RHCOS ISO. If using IPI, the Machine API might do this for you. If UPI, you must manually trigger the CSR (Certificate Signing Request) approval.
  4. Approve CSRs:The new master won’t join until you approve its certificates:oc get csr | grep Pending | awk '{print $1}' | xargs oc adm certificate approve

5. Compaction and Defragmentation

Over time, etcd keeps versions of objects. If the database grows too large, the cluster will stop accepting writes (Error: mvcc: database space exceeded).

The Fix:

OpenShift normally handles this automatically, but as an admin, you might need to force it:

# Defragment the endpoint
oc rsh -n openshift-etcd etcd-master-0 etcdctl defrag --cluster

The “Final Boss” Interview Question:

“We lost 2 out of 3 master nodes. The API is down. How do you recover?”

The Answer:

  1. Since quorum is lost (needs $n/2 + 1$ nodes), you must perform a Single Master Recovery.
  2. Stop the etcd service on the remaining healthy master.
  3. Run the etcd-snapshot-restore.sh script (shipped with OpenShift) using a previous backup.
  4. This forces the remaining master to become a “New Cluster” of one.
  5. Once the API is back up, you re-join the other two nodes as brand-new members.

Since OpenShift 4.12+, OVN-Kubernetes has become the default network provider, replacing the older OpenShift SDN. For an on-premise administrator, understanding this is vital because it changes how traffic flows from your physical switches into your pods.

1. OVN-Kubernetes Architecture

Unlike the old SDN which used Open vSwitch (OVS) in a basic way, OVN (Open Virtual Network) brings a distributed logical router and switch to every node.

  • Geneve Encap: OVN uses Geneve (Generic Network Virtualization Encapsulation) instead of VXLAN to tunnel traffic between nodes. It’s more flexible and allows for more metadata.
  • The Gateway: Every node has a “Gateway” that handles traffic entering and exiting the cluster. On-premise, this is where your physical network interface (e.g., eno1 or ens192) meets the virtual world.

2. On-Premise Networking Challenges

Q1: How does OpenShift handle “External” IPs on-prem?

In the cloud, you have a LoadBalancer service. On-prem, you don’t.

The Admin Solution: MetalLB.

As an admin, you configure a MetalLB Operator with an IP address pool from your actual data center VLAN. When a developer creates a Service of type LoadBalancer, MetalLB uses ARP (Layer 2) or BGP (Layer 3) to announce that IP address to your physical routers.

Q2: What is the “Ingress VIP” and “API VIP”?

During a VMware/Bare Metal IPI install, you are asked for two IPs:

  1. API VIP: The floating IP used to talk to the control plane (Port 6443).
  2. Ingress VIP: The floating IP for all application traffic (Ports 80/443).Mechanism: OpenShift uses Keepalived and HAProxy internally to float these IPs between the master nodes (for API) or worker nodes (for Ingress). If the node holding the IP fails, it “floats” to another node in seconds.

3. Troubleshooting the Network

If pods can’t talk to each other, follow this “inside-out” path:

Step 1: Check the Cluster Network Operator (CNO)

If the CNO is degraded, the entire network is unstable.

oc get clusteroperator network
Step 2: Trace the Flow with oc adm network

OpenShift provides a built-in tool to verify if two pods can actually talk to each other across nodes:

Bash

oc adm pod-network diagnostic
Step 3: Inspect the OVN Database

Since OVN stores the network state in a database (Northbound and Southbound DBs), you can check if the logical flows are actually created.

# Get the logs of the ovnkube-master
oc logs -n openshift-ovn-kubernetes -l app=ovnkube-master
4. Key Concepts for Interview Scenarios
Scenario: “Applications are slow only when talking to external databases.”
  • Likely Culprit: MTU Mismatch. * Explanation: Geneve encapsulation adds 100 bytes of overhead to every packet. If your physical network is set to standard MTU (1500), but OpenShift is also sending 1500, the packets get fragmented, causing a massive performance hit.
  • The Fix: Ensure the cluster MTU is set to 1400 (1500 – 100) or enable Jumbo Frames (9000) on your physical switches.
Scenario: “How do you isolate traffic between two departments on the same cluster?”
  • The Answer: NetworkPolicies. * OVN-Kubernetes supports standard Kubernetes NetworkPolicy objects. By default, all pods can talk to all pods. I would implement a “Deny-All” default policy and then explicitly allow traffic only between required microservices.

Summary for Administrator Interview

FeatureOpenShift SDN (Old)OVN-Kubernetes (New/Standard)
EncapsulationVXLANGeneve
Network PolicyLimitedFully Featured (Egress/Ingress)
Hybrid CloudHard to implementDesigned for it (IPsec support)
Windows SupportNoYes

OCP troubleshooting

techhadoop Uncategorized , , , ,

In an interview, the ability to walk through a logical “drilling down” process is more important than knowing the exact answer immediately. Here is a classic scenario for an OpenShift Admin role.

The Scenario: “The Disappearing Images”

The Symptom: You are paged because developers cannot push or pull images to the internal OpenShift registry. You run oc get co and see that the image-registry operator is Degraded.

Your Task: Walk me through how you find the root cause and fix it.

Your Mock Troubleshooting Response

1. The High-Level Check

“First, I’ll check the high-level error message provided by the ClusterOperator resource. This usually gives a hint if it’s a configuration issue or a backend failure.”

oc describe clusteroperator image-registry

Interview Result: The message says: “Progressing: Unable to apply resources: storage backend not configured” or “Degraded: error creating registry pod: persistentvolumeclaim “image-registry-storage” not found.”

2. Investigate the Operator Configuration

“Since the error mentions storage, I need to look at the Image Registry’s custom configuration to see where it’s trying to store data.”

oc get configs.imageregistry.operator.openshift.io cluster -o yaml

What you are looking for: Check the spec.storage section. Is it set to pvc, s3, azure, or emptyDir?

3. Deep Dive into the Namespace

“I’ll jump into the openshift-image-registry namespace to check the health of the actual registry pods and the status of the PVC.”

oc get pods,pvc -n openshift-image-registry

Case A (PVC is Pending): “If the PVC is Pending, I’ll run oc describe pvc <pvc-name>. Usually, this reveals that the requested StorageClass doesn’t exist or there is no capacity left in the storage provider.”

Case B (Pod is CrashLoopBackOff): “If the pod is crashing, I’ll check the logs: oc logs <pod_name>. Often, this is a permission issue where the registry container can’t write to the mounted volume due to UID mismatches.”

4. The Fix

“Depending on the find, I would:”

  • If storage was missing: Update the configs.imageregistry to point to a valid StorageClass.
  • If it’s a bare-metal install: Patch the registry to use emptyDir (for non-prod) or configure a manual PV.
  • If it’s Cloud (AWS/Azure): Check if the Operator has the right IAM permissions to create the S3 bucket or Blob storage.

Bonus “Pro” Answer: The Authentication Operator

If you want to impress the interviewer, mention the Authentication Operator and Certificates.

The Scenario: Authentication is degraded because of expired certificates.

The Pro Tip: “I would check the v4-0-config-system-router-certs secret in the openshift-authentication namespace. If the Ingress wildcard cert was manually replaced but the Auth operator wasn’t updated, it will go Degraded because it can no longer validate the OAuth callback URL. I’d fix this by ensuring the router-ca is correctly synced.”

Interviewer Follow-up:

“What if you fix the storage, but the Operator is still showing Degraded after 10 minutes?”

Your Answer: “Sometimes the Operator’s ‘Sync’ loop gets stuck. I would try a graceful restart of the operator pod itself by running oc delete pod -l name=cluster-image-registry-operator -n openshift-image-registry-operator. Since it’s a deployment, a new pod will spin up, re-scan the environment, and should clear the Degraded status if the underlying issue is resolved.”

A “Pending” pod is one of the most common issues you’ll face. In an interview, the key is to show you understand that Pending = A Scheduling Problem, whereas CrashLoopBackOff = An Application Problem.

Here is how to handle this scenario like a seasoned admin.

1. The Core Diagnostic: oc describe

The first thing you must say is: “I check the Events section.” The scheduler is very vocal about why it can’t place a pod.

oc describe pod <pod-name>

Look at the very bottom under “Events”. You will usually see a FailedScheduling warning with a specific reason.

2. Common Reasons (The “Big Four”)

A. Insufficient Resources (CPU/Memory)

  • The Message: 0/6 nodes are available: 3 Insufficient cpu, 3 Insufficient memory.
  • The Reality: Kubernetes schedules based on Requests, not actual usage. Even if a node looks idle, if other pods have “reserved” that space via high requests, the scheduler won’t touch it.
  • The Fix: Scale up the cluster (Autoscaler), add nodes, or ask the developer to lower their resources.requests.

B. Mismatched NodeSelectors / Affinity

  • The Message: 0/6 nodes are available: 6 node(s) didn't match node selector.
  • The Reality: The pod is looking for a label like disktype=ssd, but no nodes have that label.
  • The Fix: Label the nodes or fix the typo in the Deployment YAML.

C. Taints and Tolerations

  • The Message: 0/6 nodes are available: 6 node(s) had taints that the pod didn't tolerate.
  • The Reality: You might have “Infra” nodes or “GPU” nodes that are tainted to keep regular apps off them. If the pod doesn’t have a matching “Toleration,” it’s banned from those nodes.
  • The Fix: Add the correct tolerations to the pod spec.

D. Unbound PersistentVolumeClaims (PVC)

  • The Message: pod has unbound immediate PersistentVolumeClaims.
  • The Reality: The pod is waiting for a disk. Maybe the StorageClass is wrong, or the disk is in US-East-1a while the nodes are in US-East-1b.
  • The Fix: Check the PVC status with oc get pvc.

3. Advanced Troubleshooting: “Resource Quotas”

If oc describe doesn’t show a scheduling error, check the Namespace Quota.

oc get quota

The Scenario: If a project has a limit of 10 CPUs and existing pods are already using 9.5, a new pod requesting 1 CPU will stay Pending because it would violate the project’s “budget,” even if the physical nodes have plenty of room.

4. Summary for the Interviewer

“To summarize, if I see a Pending pod, I follow this hierarchy:”

  1. Check Events: Use oc describe to see the scheduler’s ‘FailedScheduling’ message.
  2. Check Resources: Compare pod requests against node allocatable capacity.
  3. Check Constraints: Verify nodeSelectors, Taints, and Affinity rules.
  4. Check Storage: Ensure the PVC is bound and in the correct zone.
  5. Check Quotas: Ensure the namespace hasn’t hit its hard limit.

In an OpenShift (OCP) admin interview, “Networking” is the area where theory meets reality. By 2026, the focus has shifted entirely to OVN-Kubernetes (the default network provider) and complex traffic patterns like Egress Control.

Here are the most common networking scenarios and questions you’ll encounter.

1. OVN-Kubernetes: The Modern Standard

OpenShift transitioned from the legacy “OpenShift SDN” to OVN-Kubernetes. Interviewers will expect you to know why.

  • Question: Why did OpenShift move to OVN-Kubernetes?
    • Answer: OVN-K is built on Open vSwitch (OVS) and provides better scalability for large clusters, native support for IPv6, and advanced features like Egress IPs and IPsec encryption for pod-to-pod traffic.
  • Troubleshooting Tip: If networking feels “sluggish,” check the OVN Northbound and Southbound databases. These are the “brain” of the network. If they get out of sync, pods might have IPs but can’t talk to each other.
    • Command: oc get pods -n openshift-ovn-kubernetes (Check for failing ovnkube-node or ovnkube-control-plane pods).

2. Egress Traffic: “How do we leave the cluster?”

In enterprise environments, security teams often demand that traffic leaving the cluster has a predictable, static IP for firewall whitelisting.

  • Question: How do you give a specific Project a dedicated external IP?
    • Answer: By using an Egress IP. You assign an IP to a Namespace, and any traffic leaving that namespace to the outside world will appear to come from that specific IP, rather than the node’s IP.
  • The “Egress Firewall” (EgressNetworkPolicy):
    • This is used to prevent pods from reaching specific external destinations (e.g., “Allow pods to talk to the corporate DB, but block all other internet access”).
    • Limit: You can only have one EgressNetworkPolicy per project.

3. Service vs. Route vs. Ingress

This is a classic “bread and butter” question.

  • The Problem: A developer says their application is unreachable from the internet.
  • The Admin Drill:
    1. Check the Route: Does it exist? Is it “Admitted” by the Ingress Controller? (oc get route)
    2. Check the Service: Does the Route point to a valid Service? Does that Service have Endpoints? (oc get endpoints)
    3. Check the Pod: Are the pods running? Are they passing their Readiness Probes? If a probe fails, the endpoint is removed, and the Route will return a 503 Service Unavailable.

4. Common Failure: MTU Mismatches

If you can ping a service but large data transfers (like file uploads) hang or fail, it is almost always an MTU (Maximum Transmission Unit) mismatch.

  • Scenario: You are running OCP on a platform (like Azure or a specific VPC) that uses encapsulation (VXLAN/GENEVE).
  • The Fix: The cluster network MTU must be smaller than the physical network MTU to account for the “header overhead.” If the physical network is 1500, your OVN-K network should usually be 1400.

5. Network Observability (The 2026 Edge)

In 2026, admins don’t just guess; they use the Network Observability Operator.

  • Question: How do you find out which pod is hogging all the bandwidth?
    • Answer: I use the Network Observability Operator (based on Loki). It provides a flow-collector that visualizes traffic in the OCP Console. I can see a “Top Talkers” graph to identify which pod or namespace is causing network congestion.

The “Pro” Interview Summary

If you want to sound like an expert, use these keywords:

  • East-West Traffic: Communication between pods (secured by NetworkPolicies).
  • North-South Traffic: Communication into or out of the cluster (managed by Routes/EgressIP).
  • Hairpinning: When a pod tries to reach itself via the external Route (can cause loops if not configured correctly).

In an OpenShift (OCP) interview, storage is a “Day 2” topic. By 2026, the discussion has moved from simply “how to attach a disk” to software-defined storage and data resilience.

Administrators are expected to understand the abstraction layers between the physical disk and the application.

1. The Core Abstraction (PV, PVC, and StorageClass)

Interviewers will start with the basics to ensure you know the “Kubernetes way” of handling state.

  • StorageClass (SC): The “template” for storage. It defines the provider (AWS EBS, VMware vSphere, Azure Disk) and parameters like reclaimPolicy (Delete vs. Retain).
  • PersistentVolumeClaim (PVC): The developer’s request. “I need 10GB of RWO storage.”
  • PersistentVolume (PV): The actual slice of storage that gets bound to the PVC.

2. OpenShift Data Foundation (ODF)

This is the “Enterprise” way to do storage in OCP. It is based on Ceph and Rook.

  • Question: Why use ODF instead of just direct cloud-native CSI drivers?
    • Answer: ODF provides a unified layer. It gives you Block (RWO), File (RWX), and Object (S3) storage regardless of where the cluster is running. It also enables advanced features like data replication, snapshots, and disaster recovery (DR) across clusters.
  • Key Component (NooBaa): Mention “Multicloud Object Gateway” (NooBaa). It allows you to store data across different cloud providers (e.g., AWS S3 and Azure Blob) while presenting a single S3 endpoint to the app.

3. Access Modes: RWO vs. RWX

This is a frequent “trap” question in interviews.

  • ReadWriteOnce (RWO): Can be mounted by a single node. Best for databases (PostgreSQL, MongoDB).
  • ReadWriteMany (RWX): Can be mounted by many nodes simultaneously. Essential for shared file systems or web servers serving the same static content.
    • Note: Cloud block storage (EBS/Azure Disk) is almost always RWO. To get RWX, you usually need ODF (CephFS) or a managed service like AWS EFS.

4. Critical Admin Tasks & Commands

An interviewer might ask: “A developer says their database is out of space. Walk me through the fix.”

  1. Check Capability: oc get sc <storage-class-name> -o yaml. Look for allowVolumeExpansion: true.
  2. The Fix: Edit the PVC directly: oc edit pvc <pvc-name>.
  3. The Result: If the CSI driver supports it, the PV will expand automatically, and the file system inside the pod will grow without a restart (usually).

5. Advanced: LVM Storage vs. Local Storage Operator

For bare metal or Single Node OpenShift (SNO):

  • LVM Storage Operator (LVMS): The modern (2025/2026) choice. It takes local disks and turns them into a Volume Group, allowing dynamic provisioning of small chunks of local storage.
  • Local Storage Operator (LSO): The “old” way. It binds a whole raw disk to a single PV. It’s less flexible than LVMS because it lacks dynamic resizing.

6. Storage Troubleshooting Checklist

  • PVC stuck in “Pending”:
    • Check oc describe pvc.
    • Cause: No PV available that matches the request, or the StorageClass doesn’t support “Wait For First Consumer” (scheduling issues).
  • Volume stuck in “Terminating”:
    • Cause: A pod is still using the volume. You must find the pod (oc get pods -A | grep <pvc-name>) and delete it before the storage can be released.
  • Multi-Zone Issues:
    • Cause: In AWS/Azure, a volume created in Zone A cannot be mounted by a node in Zone B. This is why “topology-aware” scheduling is critical.