Role of the Scheduler in OpenShift

The scheduler decides which node should run a newly created Pod.

It does not start containers itself. Its job is to select the best eligible node and assign the Pod to it.

User creates Pod
      │
      ▼
API Server stores Pod
      │
      ▼
Pod has no nodeName
      │
      ▼
Scheduler evaluates nodes
      │
      ▼
Scheduler selects one node
      │
      ▼
kubelet starts the Pod

Where the Scheduler Runs

In a standard OpenShift cluster, the Kubernetes scheduler runs on the control-plane nodes as a static Pod.

Check it with:

oc get pods -n openshift-kube-scheduler -o wide

The scheduler is managed by the Kubernetes Scheduler Operator.

Check its status:

oc get co kube-scheduler

Healthy status:

AVAILABLE=True
PROGRESSING=False
DEGRADED=False

Scheduler vs Scheduler Operator

These are different components.

Scheduler Operator
        │
        ▼
Manages kube-scheduler
        │
        ▼
kube-scheduler assigns Pods

Scheduling Flow

Suppose a Pod is created:

apiVersion: v1
kind: Pod
metadata:
  name: payments-api
spec:
  containers:
    - name: app
      image: registry.example.com/payments-api:1.0

Initially, the Pod has no node assignment:

spec:
  nodeName: ""

The scheduler watches the API server for these unscheduled Pods.

Step 1: Watch for Pending Pods

The scheduler detects:

Pod: payments-api
Status: Pending
Node: none

It adds the Pod to its scheduling queue.

Step 2: Filter Nodes

The scheduler eliminates nodes that cannot run the Pod.

This is called the filtering phase.

Example:

Available nodes:
worker-1
worker-2
worker-3
worker-4

The scheduler checks:

• CPU availability
• Memory availability
• Node readiness
• Taints and tolerations
• Node selectors
• Node affinity
• Pod affinity and anti-affinity
• Persistent volume topology
• Host ports
• Pod count limits
• Resource constraints

After filtering:

worker-1 → eligible
worker-2 → insufficient memory
worker-3 → taint not tolerated
worker-4 → nodeSelector mismatch

Only worker-1 remains.

Step 3: Score Nodes

If several nodes are eligible, the scheduler scores them.

Example:

worker-1 → score 85
worker-2 → score 72
worker-3 → score 91

The scheduler selects:

worker-3

Scoring can consider:

• Resource balance
• Node affinity preferences
• Pod spreading
• Image locality
• Existing workload distribution
• Topology preferences

Step 4: Bind the Pod

The scheduler updates the Pod through the API server:

spec:
  nodeName: worker-3

This is called binding.

Step 5: kubelet Starts the Pod

The kubelet on worker-3 sees the assignment.

kubelet
   │
   ▼
CRI-O
   │
   ▼
Pull image
   │
   ▼
Configure networking
   │
   ▼
Mount volumes
   │
   ▼
Start container

The scheduler is no longer involved after the node assignment unless the Pod is recreated.

Important Point: Scheduler Does Not Move Running Pods

The scheduler normally handles only Pods that do not yet have a node assignment.

It does not automatically move a running Pod from one node to another just because another node becomes less busy.

Running Pod on worker-1
          │
          ▼
Scheduler does not rebalance it automatically

To redistribute workloads, you may use:

• Descheduler
• Node drain
• Pod eviction
• Deployment rollout
• Cluster autoscaling
• Manual rescheduling

Resource Requests

The scheduler uses resource requests, not actual current consumption, when deciding placement.

Example:

resources:
  requests:
    cpu: "2"
    memory: 4Gi
  limits:
    cpu: "4"
    memory: 8Gi

The scheduler reserves:

2 CPU
4 GiB memory

It does not schedule based on the Pod’s current real-time usage.

This is why incorrect requests can cause poor scheduling.

Requests too high

Pod remains Pending
Insufficient CPU or memory

Requests too low

Node becomes overloaded
Pods compete for resources

Node Selectors

A node selector forces a Pod onto nodes with matching labels.

Example:

spec:
  nodeSelector:
    workload-type: payments

Only nodes labeled:

oc label node worker-3 workload-type=payments

are eligible.

Node Affinity

Node affinity provides more expressive placement rules.

Example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - zone-a
                - zone-b

This means the Pod must run in zone-a or zone-b.

Pod Affinity

Pod affinity places Pods near other Pods.

Example use case:

Application Pod
      near
Cache Pod

This can reduce latency but may reduce fault isolation.

Pod Anti-Affinity

Pod anti-affinity spreads Pods apart.

Example:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: payments-api
        topologyKey: kubernetes.io/hostname

This prevents two payments-api replicas from running on the same node.

worker-1 → payments-api-1
worker-2 → payments-api-2
worker-3 → payments-api-3

This improves high availability.

Topology Spread Constraints

Topology spread constraints distribute Pods across zones or nodes.

Example:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payments-api

This helps distribute replicas evenly across availability zones.

Zone A → 2 Pods
Zone B → 2 Pods
Zone C → 2 Pods

Taints and Tolerations

A taint prevents Pods from being scheduled unless they tolerate it.

Example taint:

oc adm taint nodes worker-3 dedicated=payments:NoSchedule

Only Pods with this toleration can run there:

tolerations:
  - key: dedicated
    operator: Equal
    value: payments
    effect: NoSchedule

Typical OpenShift uses include:

• Control-plane protection
• Infrastructure nodes
• GPU nodes
• Storage nodes
• Dedicated application nodes

Control-Plane Nodes

Control-plane nodes normally have taints that prevent ordinary application workloads.

node-role.kubernetes.io/master:NoSchedule

or:

node-role.kubernetes.io/control-plane:NoSchedule

Platform Pods have the required tolerations, but regular application Pods do not.

Storage-Aware Scheduling

For Pods using persistent storage, the scheduler must consider volume topology.

Example:

PVC is available only in zone-a
         │
         ▼
Pod must be scheduled to node in zone-a

The scheduler evaluates:

• StorageClass
• PersistentVolume
• Volume node affinity
• Availability zone
• CSI driver constraints

A Pod may remain Pending if no eligible node can access the volume.

Scheduler Profiles in OpenShift

OpenShift supports scheduler profiles that influence how workloads are placed.

Common profiles include:

• LowNodeUtilization
• HighNodeUtilization
• NoScoring

A simplified interpretation:

Check scheduler configuration:

oc get scheduler cluster -o yaml

Example:

apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  name: cluster
spec:
  profile: LowNodeUtilization

Use supported configuration through the cluster Scheduler resource rather than editing static Pod manifests.

High Availability

The scheduler runs on control-plane nodes, but only one instance is active as leader at a time.

scheduler-master-0 → leader
scheduler-master-1 → standby
scheduler-master-2 → standby

If the leader fails:

Leader unavailable
       │
       ▼
Leader election
       │
       ▼
Another scheduler becomes active

Existing Pods continue running during a scheduler outage, but new Pods cannot be assigned until scheduling resumes.

What Happens if the Scheduler Is Down?

Existing workloads generally continue running.

However:

• New Pods stay Pending.
• Failed Pods cannot be placed on another node.
• Deployments cannot scale successfully.
• New Jobs remain unscheduled.
• Node drain replacements may remain Pending.
• Cluster upgrades can be affected.

Example:

Deployment replicas desired: 5
Running: 3
Pending: 2

Pending Pod Troubleshooting

Start with:

oc get pod <pod-name> -n <namespace>

Then:

oc describe pod <pod-name> -n <namespace>

Look at Events.

Common messages:

0/10 nodes are available:
3 insufficient memory
2 node(s) had untolerated taint
4 node(s) didn't match node selector
1 node(s) had volume node affinity conflict

This message is often the fastest way to identify the scheduling problem.

Common Scheduling Failures

Insufficient CPU

0/5 nodes are available:
5 Insufficient cpu

Check:

oc adm top nodes
oc describe node <node>

Remember that scheduling uses requested CPU, not actual CPU usage.

Insufficient memory

0/5 nodes are available:
5 Insufficient memory

Check allocated requests:

oc describe node <node>

Look under:

Allocated resources

Untolerated taint

node(s) had untolerated taint

Check:

oc describe node <node> | grep -i taint

Then verify the Pod tolerations.

Node selector mismatch

node(s) didn't match Pod's node affinity/selector

Check:

oc get nodes --show-labels

Compare with:

oc get pod <pod> -o yaml

Pod anti-affinity conflict

The placement rules may be too strict for the available number of nodes.

Example:

3 replicas
Only 2 eligible nodes
Required anti-affinity

The third Pod remains Pending.

Volume node affinity conflict

The Pod and volume are tied to different zones.

Check:

oc describe pod <pod>
oc get pv <pv-name> -o yaml
oc get pvc -n <namespace>

Too many Pods on a node

The node may have reached its Pod capacity.

Check:

oc describe node <node>

Look for:

pods: 250

and the number currently allocated.

Scheduler Operator Troubleshooting

Check the ClusterOperator:

oc get co kube-scheduler

Describe it:

oc describe co kube-scheduler

Check scheduler Pods:

oc get pods -n openshift-kube-scheduler -o wide

Check logs:

oc logs -n openshift-kube-scheduler \
  <scheduler-pod> \
  -c kube-scheduler

Check the Operator:

oc get pods -n openshift-kube-scheduler-operator

oc logs -n openshift-kube-scheduler-operator \
  deployment/openshift-kube-scheduler-operator

Check API readiness:

oc get --raw='/readyz?verbose'

The scheduler depends on a healthy API server and etcd.

Scheduling Troubleshooting Flow

Pod Pending
    │
    ▼
oc describe pod
    │
    ▼
Read FailedScheduling event
    │
    ├── Insufficient resources
    ├── Taints/tolerations
    ├── Node selectors
    ├── Affinity rules
    ├── Storage topology
    └── Pod capacity
           │
           ▼
Check scheduler Operator only if many unrelated Pods are affected

A single Pending Pod usually indicates a workload placement issue.

Many unrelated Pending Pods across the cluster may indicate:

• Scheduler failure
• API server problem
• etcd problem
• Cluster-wide capacity shortage
• Broken scheduler configuration

Scheduler vs Autoscaler

The scheduler and autoscaler have different responsibilities.

Flow:

Pod cannot fit
      │
      ▼
Scheduler marks it unschedulable
      │
      ▼
Cluster Autoscaler detects it
      │
      ▼
New node created
      │
      ▼
Scheduler places Pod

Scheduler vs Descheduler

The descheduler does not directly move a Pod. It evicts it, and then the scheduler assigns the replacement.

Interview Answer

The OpenShift scheduler is responsible for assigning unscheduled Pods to suitable nodes. It watches the API server for Pods that do not have a nodeName, places them in a scheduling queue, filters out nodes that cannot satisfy the Pod’s requirements, scores the remaining eligible nodes, and binds the Pod to the best node.

During filtering, it evaluates resource requests, node readiness, taints and tolerations, node selectors, affinity and anti-affinity, topology spread constraints, host ports, and persistent-volume topology. After the scheduler writes the selected node into the Pod specification, the kubelet on that node uses CRI-O to start the containers.

The scheduler does not run containers and normally does not rebalance already running Pods. It is managed by the Kubernetes Scheduler Operator and runs highly available on the control-plane nodes using leader election. For troubleshooting a Pending Pod, I first use oc describe pod and inspect the FailedScheduling event before checking node capacity, labels, taints, affinity rules, storage topology, and finally the scheduler Operator if the problem affects many workloads.