The Importance of Kubelet in OpenShift (OCP) Master Nodes

techhadoop OCP

kubelet on OCP Master Nodes

Why kubelet Runs on Masters

In OCP, the control plane components are not system services (systemd units). They run as static pods — and kubelet is the only thing that can run static pods.

Without kubelet on masters:
  ❌ etcd cannot start
  ❌ kube-apiserver cannot start
  ❌ kube-controller-manager cannot start
  ❌ kube-scheduler cannot start

kubelet IS the bootstrap mechanism for the entire control plane.

Static Pods — The Core Concept

kubelet watches one directory constantly:

/etc/kubernetes/manifests/
  ├── etcd-pod.yaml
  ├── kube-apiserver-pod.yaml
  ├── kube-controller-manager-pod.yaml
  └── kube-scheduler-pod.yaml

Any .yaml dropped here → kubelet starts it as a pod. Any .yaml removed → kubelet stops and removes it.

No API server needed. kubelet reads these files directly from disk. This is exactly why cluster-restore.sh works by moving manifests in and out of this directory — it’s controlling the control plane via kubelet’s static pod mechanism.

cluster-restore.sh
  └── mv etcd-pod.yaml → /etc/kubernetes/manifests/
              │
              ▼
           kubelet notices file change (inotify watch)
              │
              ▼
           kubelet tells CRI-O to start the etcd containers
              │
              ▼
           etcd is running
What kubelet Does on a Master (vs Worker)
ResponsibilityMaster kubeletWorker kubelet
Runs static pod manifests✅ (etcd, apiserver, scheduler, controller-manager)❌ (no static pods by default)
Runs regular workload pods✅ (if not tainted)✅
Reports node status to API✅✅
Handles PVs / volume mounts✅✅
Manages CRI-O container runtime✅✅
Applies MachineConfig changes✅✅
Handles CSR bootstrap✅✅
Master Node Taint (workload isolation)

By default, OCP master nodes carry a taint that prevents regular workloads from landing on them:

oc describe node master-0 | grep Taint
# Taints: node-role.kubernetes.io/master:NoSchedule

So while kubelet on masters CAN run any pod, the scheduler won’t place regular workloads there unless you explicitly tolerate the taint. The taint does NOT affect static pods — kubelet runs those directly, bypassing the scheduler entirely.

kubelet Startup Sequence on a Master

This is what happens when a master node boots:

1. RHCOS boots
        │
        ▼
2. systemd starts kubelet.service
        │
        ▼
3. kubelet reads /etc/kubernetes/manifests/
        │
        ├── etcd-pod.yaml          → starts etcd containers
        ├── kube-apiserver-pod.yaml → starts API server containers
        ├── kube-controller-manager-pod.yaml
        └── kube-scheduler-pod.yaml
        │
        ▼
4. etcd becomes healthy (has quorum)
        │
        ▼
5. kube-apiserver connects to etcd, starts serving
        │
        ▼
6. kubelet registers THIS master node with the API server
        │
        ▼
7. Cluster operators come up (CVO, etcd-operator, etc.)

Notice: kubelet starts etcd, and etcd enables the API, and the API is what kubelet later registers with. kubelet bootstraps its own control plane and then registers with it.

Verify kubelet on a Master
# SSH into a master node
ssh core@master-0
# kubelet is a systemd service
systemctl status kubelet
# kubelet logs
journalctl -u kubelet -f
# kubelet process
ps aux | grep kubelet
# kubelet managing static pods
crictl pods | grep -E "etcd|apiserver|scheduler|controller"
What Happens if kubelet Dies on a Master
kubelet dies on master-0
        │
        ├── etcd container keeps running (CRI-O manages it independently)
        ├── But if etcd crashes → kubelet not there to restart it
        ├── kube-apiserver keeps running (same)
        │
        └── After 5 min: node shows NotReady (no heartbeat to API)
                │
                └── etcd operator detects degraded member
                        └── alerts fire, but cluster may still function
                            if other 2 masters are healthy

Key insight: CRI-O keeps containers alive after kubelet dies — but kubelet is the only thing that will restart a crashed container. Without kubelet, a crashed etcd stays dead.

The Dependency Chain in One View
RHCOS
  └── systemd
        └── kubelet.service          ← runs on master as a systemd unit
              └── reads /etc/kubernetes/manifests/
                    ├── etcd-pod.yaml
                    │     └── CRI-O runs etcd containers
                    │           └── etcd cluster (the database)
                    │
                    ├── kube-apiserver-pod.yaml
                    │     └── CRI-O runs API server containers
                    │           └── Kubernetes/OCP API
                    │
                    ├── kube-controller-manager-pod.yaml
                    │     └── CRI-O runs controller-manager
                    │
                    └── kube-scheduler-pod.yaml
                          └── CRI-O runs scheduler

The entire OCP control plane is just kubelet reading files from a directory and telling CRI-O what to run. This simplicity is what makes the restore procedure work — you control the control plane by controlling what kubelet sees in /etc/kubernetes/manifests/.

Leave a Reply