To truly master Kubernetes and Red Hat OpenShift, you have to look past the YAML manifests and understand the underlying mechanics of the control plane, the data plane, and how Red Hat overlays enterprise-grade security and automation onto upstream Kubernetes.

Here is a deep dive into the internal machinery that powers these platforms.

1. Upstream Kubernetes Control Plane Internals

The Kubernetes control plane is a distributed system that manages the global state of your cluster. It operates on a continuous reconciliation loop (Current State vs. Desired State).

etcd: The Distributed Truth Engine

• The Mechanics: etcd is a strongly consistent, distributed key-value store using the Raft Consensus Algorithm. Every single configuration, pod status, and secret lives here.
• The Reality: The API server is the only component allowed to talk directly to etcd. All other components must query the API server.
• Performance Trap: etcd writes sequentially to disk and relies heavily on fast disk fsync times. If your underlying storage IOPS drop, Raft leader elections will fail, causing the entire cluster control plane to crash.

kube-apiserver: The Gatekeeper

The API server is a stateless REST engine that processes cluster requests. When you run kubectl apply -f, the API server processes the request through three distinct internal phases:

1. Authentication & Authorization: Checks who you are (via Client Certificates, Webhooks, or OIDC tokens) and what you can do (RBAC roles).
2. Mutating Admission Controllers: Modifies the request object on the fly (e.g., injecting default storage classes or sidecar proxies like Istio).
3. Validating Admission Controllers: Evaluates the finalized object against structural safety schemas. If it passes, the API server serializes the data and commits it to etcd.

kube-scheduler: The Placement Engine

The scheduler’s sole job is to watch for newly created Pods that have no nodeName assigned, select the optimal node for them, and bind them. It evaluates nodes using a two-stage process:

• Filtering (Predicates): Knocks out nodes that don’t match the pod’s requirements (e.g., insufficient CPU/RAM, unmatched nodeSelector, or disk/port conflicts).
• Scoring (Priorities): Ranks the remaining nodes based on optimization rules (e.g., balancing resource utilization, keeping pods from the same deployment spread across different availability zones via anti-affinity rules).

kube-controller-manager: The Orchestrator

This is a collection of background continuous loops wrapped into a single binary. It runs the DeploymentController, StatefulSetController, NodeController, and others. Each controller watches the API server for changes to its assigned resource type, detects when reality deviates from your desired YAML declaration, and fires commands to fix it.

2. Worker Node Data Plane Internals

The worker nodes execute your actual containers. Three core components handle the heavy lifting:

Kubelet: The Node Captain

The kubelet is an agent that runs directly on the bare-metal or virtual machine operating system. It watches the API server for PodSpecs assigned to its specific node.

• It interacts with the local runtime via the Container Runtime Interface (CRI) to start or stop containers.
• It performs continuous health monitoring (liveness and readiness probes) on running pods and reports node status back to the API server.

Container Runtime (CRI-O / containerd)

Modern Kubernetes does not use Docker directly. It relies on lightweight runtimes compliant with the Open Container Initiative (OCI). OpenShift standardizes on CRI-O, while upstream Kubernetes frequently uses containerd. The runtime pulls images, configures cgroups (resource limits), and configures namespaces (isolation boundaries) at the Linux kernel level.

Kube-Proxy: The Traffic Director

kube-proxy manages network routing rules on each node to fulfill the Kubernetes Service abstraction. Depending on your configuration, it manipulates packet routing in one of two ways:

• IPTables Mode (Legacy): Appends sequential firewall rules to the node’s Linux network stack. While highly reliable, it suffers from scaling bottlenecks when thousands of services exist because every packet must traverse long, sequential rule lists.
• IPVS Mode / eBPF: Utilizes Linux Virtual Server hashing algorithms or high-performance eBPF (Extended Berkeley Packet Filter) kernel hooks to route packets at near-instant, constant speed ($O(1)$ complexity) regardless of cluster size.

3. The OpenShift Structural Transformation

Red Hat OpenShift is not a fork of Kubernetes; it is a highly opinionated packaging of upstream Kubernetes, hard-coded to run exclusively on top of Red Hat Enterprise Linux CoreOS (RHCOS).

The Operator Framework: Automated Day-2 Operations

In native Kubernetes, upgrading a cluster requires manually upgrading etcd, updating API server binaries, and migrating network plugins. OpenShift automates this via the Cluster Version Operator (CVO) and specific component operators.

• Every foundational component of OpenShift (DNS, Ingress, Monitoring, Storage) is controlled by an internal Operator.
• To upgrade an OpenShift cluster, the CVO changes a single target image tag. The internal operators see this, orchestrate their own database migrations, gracefully drain worker nodes, update the host OS under the hood, and verify component health completely automatically.

Enterprise Identity & Integration

Upstream Kubernetes contains no built-in user database; it relies entirely on administrators setting up complex external authentication proxies. OpenShift includes a native, built-in OAuth Server. Out of the box, it provides a unified login mechanism that bridges cluster CLI access (oc login) and the visual Web Console dashboard directly into enterprise directories via OIDC, LDAP, or Keystone.

4. Deep-Dive Networking Comparison: Upstream vs. OCP

Kubernetes mandates that every pod must receive a unique IP address and be able to communicate with any other pod across the cluster without NAT. How this is executed depends entirely on the Container Network Interface (CNI) plugin.

Upstream Kubernetes (Calico / Flannel / Cilium)

Upstream allows you to select your own networking provider. Historically, implementations relied on simple Overlay Networks using VXLAN or Geneve encapsulation, which wraps pod-to-pod packets inside standard UDP host packets. While highly portable, this introduces a performance tax due to the packet encapsulation/decapsulation overhead on the node CPU.

OpenShift: OVNKubernetes CNI

OpenShift standardizes on OVNKubernetes, an enterprise-grade CNI built on top of Open Virtual Network (OVN) and Open vSwitch (OVS).

• Native Network Policies: It enforces highly performant network access control lists (ACLs) directly inside the OVS kernel space, preventing rogue pods from executing lateral network discovery attacks.
• Egress IPs: It provides native configuration to assign static, public egress IPs to specific namespaces. This allows legacy corporate firewalls outside the cluster to whitelist traffic originating from specific container microservices, which is historically difficult with standard, volatile Kubernetes pod routing.
• Hybrid Cloud Connectivity via Submariner: Through OVN integrations, OpenShift can leverage Submariner to securely map and route traffic across completely different physical clouds, allowing a pod in an AWS cluster to communicate natively with a pod on an on-premises VMware cluster using encrypted, private IP tunnels.