Senior SRE Scenario 1: The Cascading “OOM / Throttling” Outage & Traffic Drop

Scenario

During a high-traffic promo event, an e-commerce platform’s core checkout microservice starts dropping connections. The SRE on call sees pods repeatedly restarting with Exit Code 137 while remaining pods exhibit severe response latency spikes (P99 > 8s).

The cluster autoscaler (HPA) fails to scale the Deployment quickly enough. Upon investigation, you notice:

1. limits.memory is set to 2Gi, and requests.memory is 512Mi.
2. limits.cpu is set to 500m, and requests.cpu is 500m.
3. Java 17 service with default JVM flags running inside the container.
4. Downstream database connections are spiking to maximum pool capacity.

Questions for the Candidate:

• Explain the exact Linux kernel/cgroup mechanisms triggering the latency and restarts.
• Why did the HPA fail to prevent this outage, and how does CPU limit enforcement impact metrics?
• Design a comprehensive remediation plan addressing container runtime configuration, application runtime, and Kubernetes scaling policy.

Expected Solution

1. Technical Root Cause Analysis

• Memory (Exit Code 137 / OOMKilled): The large gap between requests (512Mi) and limits (2Gi) creates a Burstable QoS class. During the spike, off-heap memory plus JVM heap expands past 2Gi. The kernel Linux OOM Killer terminates the container process (SIGKILL, exit code $128 + 9 = 137$).
• Latency Spikes (CFS Throttling): Setting limits.cpu = 500m configures the Linux Completely Fair Scheduler (CFS) quota mechanism (cpu.cfs_quota_us). When the multi-threaded Java process exhausts its quota within a 100ms period, CFS throttles the thread execution completely for the rest of the period—even if the underlying node has 80% free CPU. This produces extreme P99 tail latency.
• HPA Failure: If the HPA target is set to targetCPUUtilizationPercentage: 80, it calculates utilization as $\frac{\text{Actual Usage}}{\text{CPU Request}}$. Since requests are capped tightly and pods are dying from OOM before HPA can actuate new pods, the new pods take 45+ seconds to pass cold-start JVM initializations, aggravating the load on surviving pods.

2. Architecture & Remediation Plan

• QoS Alignment: Align requests and limits to achieve Guaranteed QoS or remove CPU limits entirely (relying on CPU requests for scheduling and node-level protection).
  YAMLresources: requests: cpu: "2" memory: "4Gi" limits: # Omit cpu limit or match request high enough to prevent CFS throttling memory: "4Gi"
• JVM Tuning: Tune the JVM heap inside the container using percentage flags:
  Bash-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0
• Scaling Strategy Enhancement:
  • Configure HPA to scale on custom metrics (e.g., HTTP request rate via Prometheus metric http_requests_per_second or active connection pool count) rather than solely CPU usage.
  • Utilize KEDA (Kubernetes Event-driven Autoscaling) or configure behavior policy in HPA for rapid scale-up without stabilization windows delaying the actuation.

Senior SRE Scenario 2: Silent Ingress Failure During Cross-AZ Node Migration

Scenario

A team is migrating a stateful processing cluster to new worker nodes across three Availability Zones (us-east-1a, us-east-1b, us-east-1c). They perform a kubectl drain on old nodes.

While kubectl get pods shows all application pods in the Running state on new nodes and readiness probes pass, 15% of public HTTP traffic via NGINX Ingress Controller returns 502 Bad Gateway intermittently for over 10 minutes.

Cluster state reveals:

• Pods are using AWS VPC CNI with direct IP routable networking.
• Pod shutdown grace period is terminationGracePeriodSeconds: 30.
• Application takes 5 seconds to gracefully stop processing active connections.

Questions for the Candidate:

• What race condition causes the 502 Bad Gateway errors during pod eviction and rescheduling?
• How does the Kubernetes Control Plane handle endpoint deregistration relative to pod termination signals (SIGTERM)?
• What architectural changes guarantee true zero-downtime traffic routing during node drains?

Expected Solution

1. Root Cause Analysis (Control Plane Race Condition)

When kubectl drain evicts a pod:

1. Parallel Execution: The API Server simultaneously marks the Pod state as Terminating AND sends an async endpoint update to the Endpoint/EndpointSlice Controller.
2. Kubelet vs. Ingress Sync Delay:
   • Kubelet immediately sends SIGTERM to the container process.
   • Concurrently, kube-proxy or the Ingress Controller receives the updated EndpointSlice list, regenerates its upstream configuration, and reloads its routing table.
3. The Race: Regenerating and propagating Ingress routing tables across nodes takes 2–8 seconds. If the container process stops accepting connections or exits immediately upon receiving SIGTERM, the Ingress Controller continues sending incoming public traffic to the dying Pod IP for several seconds, resulting in 502 Bad Gateway / Connection Refused.

[Image of Kubernetes Pod Lifecycle and PreStop Hook Execution Flow]

2. Architecture & Remediation Plan

YAML

spec:
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          # Delay SIGTERM handling to allow Ingress/Kube-proxy propagation
          command: ["/bin/sh", "-c", "sleep 15"]

• preStop Lifecycle Hook: Add a preStop sleep hook (e.g., sleep 15). When termination starts, the container stays active while Kubernetes removes its IP from EndpointSlices. Traffic stops arriving before the application process receives SIGTERM.
• PodDisruptionBudgets (PDB): Enforce strict maintenance budgets so drains cannot take down all replicas in a zone simultaneously:
  YAMLapiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: app-pdb spec: minAvailable: "80%"
• Topology Spread Constraints: Ensure application replicas are spread evenly across AZs so single-zone node maintenance doesn’t collapse capacity.

Senior SRE Scenario 3: Persistent CoreDNS Resolution Timeouts Under Burst Load

Scenario

A high-throughput distributed microservice environment running on Kubernetes (v1.28+) encounters sudden, intermittent DNS lookup timeouts (i/o timeout after 5 seconds) across hundreds of worker nodes during peak load.

Observations:

• CoreDNS CPU and Memory utilization remain well below limits (CPU at 20%).
• Increasing CoreDNS replica count from 5 to 30 does not resolve the intermittent 5-second timeouts.
• Nodes run standard Linux kernel configurations with iptables / netfilter modes for kube-proxy.

Questions for the Candidate:

• What underlying Linux networking issue causes exact 5-second DNS delays in Kubernetes clusters using UDP/iptables?
• Why did scaling CoreDNS horizontally fail to solve the issue?
• What are the industry-standard solutions to permanently fix this problem at scale?

Expected Solution

1. Root Cause Analysis

• Netfilter Conntrack Race Condition (conntrack flaw): UDP is connectionless. When multiple threads in a pod send simultaneous DNS requests over UDP to the CoreDNS ClusterIP, two identical socket connections attempt to create conntrack entries in the Linux kernel netfilter table concurrently.
• SNAT/DNAT Race:iptables performs Source NAT and Destination NAT. When two UDP packets race through netfilter simultaneously, one packet fails the __nf_conntrack_confirm lookup due to a lock race and gets dropped silently by the kernel.
• 5-Second Timeout: The client GLIBC DNS resolver waits exactly 5 seconds before retrying the UDP query.
• Why Scaling CoreDNS Failed: The race condition happens inside the client node’s kernel netfilter stack, not within CoreDNS itself. Adding more CoreDNS pods does not change the kernel netfilter locking behavior on the client nodes.

2. Remediation Strategy

YAML

# Pod spec fix via dnsConfig
spec:
  dnsConfig:
    options:
      - name: single-request-reopen
      - name: ndots
        value: "2" # Optimizes search domain expansion overhead

Kubernetes interview questions have shifted significantly. Modern engineering loops care less about basic definitions and focus heavily on architectural trade-offs, production debugging, and scenario-based problem solving.

The core Kubernetes interview questions are categorized by level and topic.

Category 1: Foundational Architecture

1. What happens under the hood when you run kubectl apply -f deployment.yaml?

How to answer: Walk through the Control Plane workflow sequentially:

1. API Server (kube-apiserver):Receives the HTTP request, authenticates/authorizes the user via RBAC, and runs Mutating/Validating Admission Webhooks.
2. etcd: The API Server persists the updated Deployment manifest to etcd.
3. Deployment Controller:Detects the new/updated Deployment via a watch loop and creates or updates a ReplicaSet.
4. ReplicaSet Controller:Sees the required replica count and creates the desired Pod objects in Unscheduled state.
5. kube-scheduler:Notices unscheduled Pods, evaluates nodes (Filtering & Scoring based on taints, affinity, resource requests), and binds the Pod to a node by updating its nodeName field in the API Server.
6. kubelet:The node-agent watches for Pods assigned to its node, calls the Container Runtime Interface (CRI, e.g., containerd) to pull images and run containers, and configures storage (CSI) and networking (CNI).

Category 2: Practical Scenario & Debugging

2. A Pod is stuck in CrashLoopBackOff. Walk me through your step-by-step triage.

How to answer: Frame your response as a structured diagnostic methodology:

1. Get state details  ──► kubectl describe pod <pod-name> (Check Exit Code, Restart Count, Events)
2. Read application   ──► kubectl logs <pod-name> --previous (Check stdout/stderr prior to crash)
3. Check resources    ──► Check if OOMKilled (Exit Code 137 = Memory Limit Exceeded)
4. Verify config/envs ──► Ensure secret/configmap keys are not missing or improperly mounted

• Exit Code 137: Out of Memory (OOMKilled) — increase memory limits or fix memory leaks.
• Exit Code 1/128/139: Application/Runtime error, missing environment variable, broken database connection string, or dynamic linking error.

3. A Pod is stuck in Pending. What are the most likely causes?

How to answer: Mention that Pending means the Pod hasn’t been scheduled onto a node yet. Check using kubectl describe pod <pod-name>:

• Insufficient Resources: No nodes have enough unreserved CPU/memory to fulfill the Pod’s requests.
• Taints and Tolerations: The nodes have taints that the Pod lacks tolerations for.
• Unsatisfied Affinity Rules:Node affinity, pod anti-affinity, or topology spread constraints cannot be met.
• Unbound PVC:The Pod relies on a PersistentVolumeClaim that hasn’t bound yet (e.g., waiting for dynamic provisioning or zone matching).

Category 3: Advanced Concepts & Workload Engineering

4. What is the operational difference between Requests vs. Limits, and how do QoS classes work?

How to answer:

• Resource Requests:Used by kube-scheduler to find a node with enough capacity. It acts as a baseline guarantee.
• Resource Limits: Enforced at runtime by the Linux kernel via cgroups.
  • Exceeding CPU limit causes Throttling (CPU time-slicing).
  • Exceeding Memory limit triggers OOMKill (container dies with Exit Code 137).

Kubernetes assigns a Quality of Service (QoS) class based on how these are set:

1. Guaranteed: Every container in the Pod has both CPU and Memory requests equal to limits. (Highest priority; last to be evicted under node memory pressure).
2. Burstable: At least one container has a request set, but requests $\neq$ limits.
3. BestEffort: No requests or limits set. (Lowest priority; first to be evicted under pressure).

5. Compare Liveness, Readiness, and Startup Probes. What happens when each fails?

6. Deployment vs. StatefulSet vs. DaemonSet: When do you use which?

• Deployment: Best for stateless microservices. Replicas are completely interchangeable, pods get random hash suffixes, and can be created/destroyed in any order.
• StatefulSet:Best for stateful apps like databases (PostgreSQL, Kafka, Elasticsearch).Provides persistent pod identities (pod-0, pod-1), ordered startup/teardown, and dedicated PVC mapping per ordinal.
• DaemonSet: Runs exactly one copy of a Pod on every eligible node. Used for node-level infrastructure like log collectors (Fluentbit), CNI plugins (Calico), or monitoring agents (Datadog).

Category 4: Real-World Architecture & Production

7. How do you implement Zero-Downtime Deployments in Kubernetes?

To guarantee seamless updates:

1. Rolling Updates Strategy: Set maxSurge (how many extra pods can be created) and maxUnavailable (how many pods can be down during the rollout).
2. Proper Readiness Probes: Ensure new pods don’t receive traffic until they can process it.
3. Graceful Termination (preStop hooks): Give active connections time to finish when stopping a pod:
   YAML lifecycle: preStop: exec: command: [“/bin/sh”, “-c”, “sleep 15”] # Gives ingress time to update endpoint routing
4. PodDisruptionBudget (PDB): Enforces that a minimum number of replicas remain online during node drains or voluntary cluster maintenance.