This is one of the most important OpenShift Architect interview questions because Operators are the foundation of the platform. Interviewers want to know whether you understand how Kubernetes continuously drives the actual cluster state toward the desired state.

A complete answer should cover:

• Control loop theory
• Watches (Events)
• Work Queue
• Reconcile() function
• Desired vs Current State
• Idempotency
• Status updates
• Error handling and retries

What is the Reconciliation Loop?

An Operator is essentially a control loop.

Instead of performing an action only once, it continuously watches the cluster and reconciles differences between the desired state and the actual state.

Think of it as:

Desired State → Compare → Fix → Verify → Repeat

Unlike a shell script that runs once and exits, an Operator never stops reconciling while it is running.

High-Level Architecture

                    Kubernetes API Server
                           │
      ┌────────────────────┼────────────────────┐
      │                    │                    │
   Custom Resource      Deployments          Services
      │                    │                    │
      └────────────────────┼────────────────────┘
                           │
                    Watch Events
                           │
                           ▼
                     Operator Controller
                           │
                    Work Queue/Event Queue
                           │
                           ▼
                     Reconcile()
                           │
        Compare Desired vs Actual State
                           │
        ┌──────────────────┴──────────────────┐
        │                                     │
   Already Matches                     Drift Detected
        │                                     │
        ▼                                     ▼
     Do Nothing                     Create/Update/Delete
                           │
                           ▼
                    Update Status
                           │
                           ▼
                    Wait for Next Event

Example: PostgreSQL Operator

Suppose a developer creates:

apiVersion: database.example.com/v1
kind: PostgreSQL
metadata:
  name: prod-db
spec:
  replicas: 3
  version: 16
  storage: 500Gi

This YAML represents the desired state.

Nothing exists yet.

Step 1 – API Server Stores the CR

The Kubernetes API Server stores the Custom Resource in etcd.

The Operator has registered a watch for PostgreSQL resources.

Immediately after creation:

Event:
PostgreSQL Created

Step 2 – Watch Event

The controller-runtime library detects:

ADD Event
↓
Queue Request
↓
Reconcile()

No polling is required; the Operator reacts to events.

Step 3 – Work Queue

The request enters the controller’s work queue.

Queue
↓
prod-db
↓
Reconcile(prod-db)

The queue prevents race conditions and allows retries if reconciliation fails.

Step 4 – Read Desired State

Inside Reconcile(), the Operator retrieves the Custom Resource:

client.Get(ctx, req.NamespacedName, postgres)

Desired state:

Replicas = 3
Version = 16
Storage = 500Gi

Step 5 – Read Actual State

The Operator queries the cluster:

Deployment
PVC
Service
Secrets
ConfigMaps
StatefulSet

Suppose it finds:

Deployment
Replicas = 0

Step 6 – Compare

The Operator compares:

Desired
3 replicas
↓
Current
0 replicas

A difference exists.

Step 7 – Reconcile

The Operator creates the missing resources.

Example:

Deployment
PVC
Service
Secrets
ConfigMap

The Kubernetes API Server stores them in etcd.

Step 8 – Kubernetes Does Its Job

The Deployment controller notices:

Deployment
↓
ReplicaSet
↓
Pods

Pods start running.

The Operator doesn’t directly start containers; it creates the desired Kubernetes objects, and the native controllers take over.

Step 9 – Verify

The Operator checks:

Deployment Ready?
Pods Running?
PVC Bound?
Database Initialized?
Replication Healthy?

If everything is healthy, it updates the status.

Step 10 – Update Status

Example:

status:
  phase: Ready
  readyReplicas: 3
  version: 16

The spec represents the desired state, while the status reflects the observed state.

Continuous Monitoring

Suppose an administrator accidentally deletes a pod.

kubectl delete pod postgres-1

The sequence becomes:

Pod Deleted
↓
Watch Event
↓
Queue
↓
Reconcile()
↓
Current State = 2
Desired State = 3
↓
Create Missing Pod

The Operator restores the desired state automatically.

Scaling Example

A user updates:

spec:
  replicas: 5

The flow is:

Update Event
↓
Queue
↓
Reconcile()
↓
Desired = 5
Current = 3
↓
Scale Deployment
↓
Ready = 5

Upgrade Example

The Custom Resource changes to:

version: 17

The Operator might:

1. Create a backup.
2. Validate compatibility.
3. Perform the database upgrade.
4. Restart pods in sequence.
5. Verify health.
6. Update status.

This illustrates why Operators are more powerful than Helm charts—they can encode operational knowledge.

Self-Healing Example

Imagine someone manually changes a Deployment:

kubectl scale deployment postgres --replicas=1

The next reconciliation detects:

Desired = 3
Current = 1

The Operator scales it back to 3 replicas.

Error Handling

Suppose the PVC cannot be created because the StorageClass doesn’t exist.

Create PVC
↓
Failed
↓
Return Error
↓
Controller Runtime
↓
Retry Later

The work queue applies backoff before retrying, instead of failing permanently.

Idempotency

A well-designed Reconcile() function is idempotent.

This means calling it multiple times produces the same final state once the cluster already matches the desired configuration.

For example:

Desired = 3
Current = 3
↓
Do Nothing
↓
Success

Repeated reconciliation should not create duplicate resources or introduce unintended changes.

Why OpenShift Uses Operators Everywhere

Nearly every OpenShift platform component is managed by an Operator, including:

• Authentication
• DNS
• Ingress
• Monitoring
• Image Registry
• Machine Config
• Networking
• Cluster Version

Each Operator continuously reconciles its component to maintain the desired platform state.

Reconciliation Lifecycle

User Creates/Updates CR
          │
          ▼
API Server stores CR in etcd
          │
          ▼
Watch Event Generated
          │
          ▼
Controller Work Queue
          │
          ▼
Reconcile()
          │
          ▼
Read Desired State (spec)
          │
          ▼
Read Actual Cluster State
          │
          ▼
Compare States
          │
    ┌─────┴─────┐
    │           │
 Match      Difference
    │           │
    ▼           ▼
 No Action  Create/Update/Delete Resources
                 │
                 ▼
         Verify Health
                 │
                 ▼
         Update Status
                 │
                 ▼
       Wait for Next Event

Interview Answer (2-Minute Version)

“The reconciliation loop is the core of every Kubernetes Operator. The Operator watches the API server for changes to its Custom Resources and places events into a work queue. When an event is processed, the Reconcile() function reads the desired state from the resource’s spec and compares it with the actual state of Deployments, StatefulSets, Services, PVCs, and other managed objects. If differences exist, the Operator creates, updates, or deletes resources to bring the cluster back to the desired state. After making changes, it verifies the outcome, updates the resource’s status, and waits for the next event. The reconciliation process is continuous, event-driven, idempotent, and includes automatic retries on failure. This continuous reconciliation enables self-healing, automated scaling, rolling upgrades, backups, and lifecycle management, which is why Operators are central to OpenShift.”