Kubernetes gives you:          OpenShift adds on top:
─────────────────────          ──────────────────────
Container orchestration   →    Everything Kubernetes + opinionated defaults
Manual TLS setup          →    Automatic TLS everywhere (service mesh ready)
No built-in registry      →    Integrated image registry
No built-in CI/CD         →    OpenShift Pipelines (Tekton) built in
No web console            →    Full-featured web console
Manual RBAC setup         →    SCCs (Security Context Constraints)
No OS management          →    RHCOS + MachineConfig operator
No upgrade automation     →    Cluster Version Operator (CVO)

Control Plane Components:
│
├── etcd
│     └── Distributed key-value store — the cluster database
│         Stores ALL cluster state (every object ever created)
│         Requires 3 or 5 members for quorum (majority must agree)
│
├── kube-apiserver
│     └── The front door — all oc/kubectl commands hit this
│         Validates and persists objects to etcd
│         Enforces authentication + authorization (RBAC)
│
├── kube-controller-manager
│     └── The reconciliation engine
│         Runs control loops: deployment controller, node controller,
│         endpoint controller, namespace controller, etc.
│         Constantly compares desired state vs actual state
│
└── kube-scheduler
      └── Decides WHICH node a pod runs on
          Evaluates: resource requests, affinity, taints/tolerations,
          topology spread, PriorityClass

/etc/kubernetes/manifests/
  ├── etcd-pod.yaml
  ├── kube-apiserver-pod.yaml
  ├── kube-controller-manager-pod.yaml
  └── kube-scheduler-pod.yaml

Problem: API server needs etcd to start.
         etcd needs to be scheduled somehow.
         Scheduler needs the API server.
         → Classic chicken-and-egg problem.
Solution: kubelet reads manifests directly from disk.
          No API server needed to start the control plane.
          kubelet bootstraps the entire control plane itself.

3-member cluster:   quorum = 2   (can lose 1 member)
5-member cluster:   quorum = 3   (can lose 2 members)
2-member cluster:   quorum = 2   (can lose 0 — never do this)
Formula: quorum = floor(N/2) + 1

etcd stops accepting writes
  → API server returns 503
  → No new pods scheduled
  → No config changes possible
  → Cluster is read-only at best, completely down at worst

DeploymentConfig (DC)          Deployment
─────────────────────          ──────────
OCP-specific resource          Native Kubernetes resource
Managed by OCP DC controller   Managed by kube-controller-manager
Supports image change triggers  No built-in image triggers
Supports config change triggers Use Kyverno/ArgoCD for policy-driven rollouts
Lifecycle hooks (pre/mid/post)  No lifecycle hooks
Being deprecated in OCP 4.x    Recommended going forward

Route (OCP-native)                   Ingress (Kubernetes-native)
──────────────────                   ───────────────────────────
OCP-specific resource                Standard Kubernetes resource
Handled by HAProxy router            Requires Ingress controller
Supports edge/passthrough/reencrypt  Only HTTP/HTTPS
TLS termination built-in             TLS via annotations (controller-specific)
Wildcard routes supported            Wildcard support varies
Fine-grained timeout/LB control      Limited, annotation-based

SCC controls:
  - Can the pod run as root?
  - Which UID range is allowed?
  - Can it use privileged containers?
  - Which Linux capabilities are allowed?
  - Can it mount hostPath volumes?
  - Can it use the host network/PID/IPC namespace?
  - Which SELinux labels are allowed?

restricted-v2   → default, no root, random UID, most secure
restricted      → legacy default
baseline        → some relaxations
nonroot         → any non-root UID
anyuid          → any UID including root (use carefully)
hostmount-anyuid→ + host volume mounts
hostnetwork-v2  → + host network
hostaccess      → host network + host filesystem
privileged      → everything allowed (only for system pods)

1. Watches clusterversion object for desired version
2. Downloads the release image from the update graph
3. Extracts manifests for every cluster operator
4. Applies them in dependency order
5. Waits for each operator to become Available before proceeding
6. Manages the upgrade of ALL cluster operators
7. Reports overall cluster version status

# The CVO's source of truth:
oc get clusterversion -o yaml
# Key fields:
spec.channel          → which update channel (stable-4.14)
spec.desiredUpdate    → target version
status.history        → all past versions
status.conditions     → Available / Progressing / Degraded

RHCOS characteristics:
  - Immutable root filesystem (no yum install, no manual changes)
  - All changes via MachineConfig operator
  - Atomic upgrades (entire OS image, not package-by-package)
  - Built-in CRI-O (no Docker)
  - Ignition for first-boot configuration
  - Automatic updates coordinated by OCP upgrade
  - SELinux enforcing by default
  - Transactional updates via rpm-ostree

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: worker-custom-chrony
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/chrony.conf
          contents:
            source: data:,server%20ntp.example.com%20iburst

You apply MachineConfig
        │
        ▼
MCO renders a new MachineConfig (merged composite)
        │
        ▼
MachineConfigPool detects rendered config changed
        │
        ▼
MachineConfigDaemon on each node:
  - Downloads new config
  - Drains the node
  - Writes files / systemd units to disk
  - Updates kernel arguments via rpm-ostree
  - Reboots node
        │
        ▼
Node comes back with new config applied

oc get mcp       # master pool, worker pool, custom pools
oc get mc        # all MachineConfig objects
oc get mcd -n openshift-machine-config-operator  # daemon pods

New Pod Created
      │
      ▼
FILTERING PHASE (eliminate ineligible nodes)
  ├── Node has enough CPU/Memory? (resource requests)
  ├── Node matches nodeSelector?
  ├── Node satisfies all node affinity rules?
  ├── Node is not tainted (or pod tolerates it)?
  ├── Pod fits within PodTopologySpread constraints?
  └── Ports available? Volume can be attached?
      │
      ▼
SCORING PHASE (rank remaining nodes 0-100)
  ├── LeastAllocated → prefer nodes with most free resources
  ├── BalancedAllocation → prefer balanced CPU/memory ratio
  ├── NodeAffinity → preferred affinity rules get points
  ├── PodTopologySpread → prefer spreading across zones
  └── ImageLocality → prefer nodes that already have the image
      │
      ▼
Node with highest score wins → Pod bound to that node
      │
      ▼
kubelet on winning node starts the pod

# Taint a node — no pods without toleration can schedule here
oc adm taint node worker-3 dedicated=gpu:NoSchedule
# Pod must have this toleration to schedule on tainted node
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"

NoSchedule        → new pods without toleration won't schedule here
PreferNoSchedule  → scheduler tries to avoid, but not hard block
NoExecute         → existing pods without toleration are EVICTED

node-role.kubernetes.io/master:NoSchedule  → keeps workloads off masters
node.kubernetes.io/not-ready:NoExecute     → evicts pods from NotReady nodes
node.kubernetes.io/unreachable:NoExecute   → evicts pods from unreachable nodes

ConfigMap                          Secret
─────────                          ──────
Non-sensitive configuration        Sensitive data (passwords, tokens, certs)
Stored as plaintext in etcd        Stored base64-encoded in etcd (NOT encrypted by default)
No size limit (practical: <1MB)    1MB size limit
Visible in oc describe             Values hidden in oc describe
Can be mounted as files or env     Same
No automatic rotation              Can integrate with Vault/external secrets

# Enable etcd encryption
oc patch apiserver cluster \
  --type merge \
  --patch '{"spec":{"encryption":{"type":"aescbc"}}}'
# Verify encryption status
oc get apiserver -o jsonpath='{.items[0].status.encryption}'

StorageClass (SC)
  └── Defines HOW storage is provisioned
      (which provisioner, IOPS tier, reclaim policy)
      Example: aws-ebs-gp3, nfs-client, ceph-rbd
PersistentVolume (PV)
  └── A piece of actual storage
      Can be pre-provisioned (static) or
      auto-created by SC (dynamic provisioning)
PersistentVolumeClaim (PVC)
  └── A pod's REQUEST for storage
      Specifies: size, accessMode, storageClassName
      Binds to a matching PV
Pod
  └── Mounts the PVC as a volume

ReadWriteOnce (RWO)   → one node can mount read-write (block storage: EBS, iSCSI)
ReadOnlyMany  (ROX)   → many nodes can mount read-only
ReadWriteMany (RWX)   → many nodes can mount read-write (NFS, CephFS, Azure File)
ReadWriteOncePod      → only ONE pod can mount read-write (strongest isolation)

OVN-Kubernetes architecture:
  ├── OVN (Open Virtual Network) — control plane
  │     Maintains a logical network topology database
  │
  ├── OVS (Open vSwitch) — data plane
  │     Fast kernel-level packet switching on each node
  │
  ├── ovnkube-master (on control plane)
  │     Watches Kubernetes API for pods, services, network policies
  │     Translates into OVN logical flows
  │
  └── ovnkube-node (on each node)
        Programs OVS with flows for local pod traffic
        Handles VXLAN/Geneve encapsulation for cross-node traffic

Pod A (node-1) → OVS on node-1 → Geneve tunnel → OVS on node-2 → Pod B (node-2)

ClusterIP (default)
  └── Internal cluster IP only
      Accessible only from inside the cluster
      Used for service-to-service communication
      Example: my-service.my-namespace.svc.cluster.local
NodePort
  └── Opens a port (30000-32767) on EVERY node
      External traffic → NodeIP:NodePort → Service → Pod
      Not recommended for production (exposes all nodes)
LoadBalancer
  └── Provisions an external load balancer (cloud provider)
      Gets a public IP automatically (AWS ELB, Azure LB, etc.)
      Best for cloud deployments needing external access
Route (OCP-specific)
  └── Goes through the HAProxy Ingress Controller
      DNS-based routing (hostname → service)
      TLS termination built in
      Preferred method in OCP

Without ImageStream:
  Deployment → image: myapp:latest
  → "latest" can change anytime → unpredictable
With ImageStream:
  Deployment → image: myapp:prod (ImageStreamTag)
  → ImageStreamTag points to sha256:abc123 (immutable digest)
  → When new image pushed: tag updates, can trigger auto-redeploy

# Create ImageStream
oc create imagestream myapp
# Import an image into the stream
oc import-image myapp:latest \
  --from=registry.example.com/myapp:latest \
  --confirm
# See what digest the tag points to
oc get istag myapp:latest -o jsonpath='{.image.dockerImageReference}'

Standard Kubernetes:
  You: create Deployment + Service + ConfigMap manually
  You: handle upgrades, backups, failover manually
With an Operator:
  You: create a single CR  e.g. kind: PostgreSQLCluster
  Operator: creates Deployment + Service + ConfigMap + PVC
            handles upgrades, backups, failover automatically

Level 1 — Basic Install          → installs the application
Level 2 — Seamless Upgrades      → handles version upgrades
Level 3 — Full Lifecycle         → backup, restore, failure recovery
Level 4 — Deep Insights          → metrics, alerts, log integration
Level 5 — Auto Pilot             → horizontal/vertical autoscaling

RBAC building blocks:
Role / ClusterRole
  └── WHAT can be done
      List of rules: [verbs] on [resources]
      Role = namespace-scoped
      ClusterRole = cluster-wide
RoleBinding / ClusterRoleBinding
  └── WHO gets to do it
      Binds a Role to a Subject (User / Group / ServiceAccount)
      RoleBinding = in one namespace
      ClusterRoleBinding = across all namespaces

# Create a role that allows reading pods
oc create role pod-reader \
  --verb=get,list,watch \
  --resource=pods \
  -n my-namespace
# Bind it to a user
oc create rolebinding pod-reader-binding \
  --role=pod-reader \
  --user=john \
  -n my-namespace
# Check what a user can do
oc auth can-i --list --as=john -n my-namespace
# Check specific permission
oc auth can-i delete pods --as=john -n my-namespace

Node stops sending heartbeats to API server
        │
        ▼ (after 40 seconds default)
Node Controller marks node as Unknown/NotReady
        │
        ▼ (after node-monitor-grace-period: 40s)
Node gets taint:
  node.kubernetes.io/not-ready:NoExecute
  node.kubernetes.io/unreachable:NoExecute
        │
        ▼ (after pod-eviction-timeout: 5 minutes)
Pods on the node get eviction tolerations expire
  → Pods marked for deletion
  → Scheduler reschedules them on healthy nodes
        │
        ▼
ReplicaSets/Deployments detect pod count is low
  → New pods created on other nodes
        │
        ▼
If node comes back:
  → kubelet re-registers
  → CSR issued and approved
  → Node returns to Ready
  → Orphaned pods cleaned up

# Step 1 — describe the pod, read Events section
oc describe pod <pod-name> -n <namespace>
# Look for: "0/3 nodes are available: ..."
# Common causes and fixes:
# Insufficient resources
"0/3 nodes available: 3 Insufficient cpu"
→ oc adm top nodes        # check actual usage
→ reduce pod resource requests
→ add more nodes
# Node selector mismatch
"0/3 nodes available: 3 node(s) didn't match node selector"
→ oc get nodes --show-labels
→ fix nodeSelector in pod spec
# Taint not tolerated
"0/3 nodes available: 3 node(s) had taint that pod didn't tolerate"
→ oc describe node <node> | grep Taint
→ add toleration to pod spec
# PVC not bound
"0/3 nodes available: 3 pod has unbound immediate PersistentVolumeClaims"
→ oc get pvc -n <namespace>
→ check StorageClass provisioner
→ check available PVs
# Topology constraints
"0/3 nodes: 3 node(s) didn't match pod topology spread constraints"
→ review topologySpreadConstraints
→ may need more nodes in different zones

# Current state: blue is live
oc get svc my-app   # selector: version=blue
# 1. Deploy green alongside blue
oc apply -f green-deployment.yaml
# Both blue and green pods running
# 2. Verify green is healthy
oc get pods -l version=green
oc rollout status deployment/my-app-green
# 3. Run smoke tests against green directly
oc expose deployment my-app-green \
  --name=my-app-green-test \
  --port=8080
# Test via the test route
# 4. Switch traffic: patch the service selector
oc patch service my-app \
  --type merge \
  --patch '{"spec":{"selector":{"version":"green"}}}'
# 100% traffic now on green — instant cutover
# 5. Monitor for issues
# If problems: instant rollback
oc patch service my-app \
  --type merge \
  --patch '{"spec":{"selector":{"version":"blue"}}}'
# 6. Once stable, remove blue
oc delete deployment my-app-blue

# BACKUP — run on one master node
sudo /usr/local/bin/cluster-backup.sh /home/core/backup
# Produces:
#   snapshot_<timestamp>.db           — full etcd database dump
#   static_kuberesources_<ts>.tar.gz  — control plane certs + manifests
# RESTORE — run when all control plane is lost
sudo /usr/local/bin/cluster-restore.sh /home/core/backup
# Process:
#   1. Validates snapshot integrity
#   2. Stops etcd (moves manifest out of /etc/kubernetes/manifests/)
#   3. Backs up old /var/lib/etcd/member
#   4. Extracts static pod resources (certs + manifests)
#   5. Starts restore-etcd pod (special one-time pod)
#   6. restore-etcd pod: runs etcdutl snapshot restore
#                        bumps revision (prevents Raft confusion)
#                        swaps in normal etcd-pod.yaml
#   7. etcd comes up as single member
#   8. API server becomes available
#   9. etcd operator re-adds master-1 and master-2 as members

oc debug node/worker-0
chroot /host
systemctl status kubelet

oc adm must-gather
# Collects: logs, events, resource configs, network info
# Creates: must-gather.tar.gz

oc get csr -o name | xargs oc adm certificate approve