Senior SRE Scenario 1: The Cascading “OOM / Throttling” Outage & Traffic Drop

Scenario

During a high-traffic promo event, an e-commerce platform’s core checkout microservice starts dropping connections. The SRE on call sees pods repeatedly restarting with Exit Code 137 while remaining pods exhibit severe response latency spikes (P99 > 8s).

The cluster autoscaler (HPA) fails to scale the Deployment quickly enough. Upon investigation, you notice:

1. limits.memory is set to 2Gi, and requests.memory is 512Mi.
2. limits.cpu is set to 500m, and requests.cpu is 500m.
3. Java 17 service with default JVM flags running inside the container.
4. Downstream database connections are spiking to maximum pool capacity.

Questions for the Candidate:

• Explain the exact Linux kernel/cgroup mechanisms triggering the latency and restarts.
• Why did the HPA fail to prevent this outage, and how does CPU limit enforcement impact metrics?
• Design a comprehensive remediation plan addressing container runtime configuration, application runtime, and Kubernetes scaling policy.

Expected Solution

1. Technical Root Cause Analysis

• Memory (Exit Code 137 / OOMKilled): The large gap between requests (512Mi) and limits (2Gi) creates a Burstable QoS class. During the spike, off-heap memory plus JVM heap expands past 2Gi. The kernel Linux OOM Killer terminates the container process (SIGKILL, exit code $128 + 9 = 137$).
• Latency Spikes (CFS Throttling): Setting limits.cpu = 500m configures the Linux Completely Fair Scheduler (CFS) quota mechanism (cpu.cfs_quota_us). When the multi-threaded Java process exhausts its quota within a 100ms period, CFS throttles the thread execution completely for the rest of the period—even if the underlying node has 80% free CPU. This produces extreme P99 tail latency.
• HPA Failure: If the HPA target is set to targetCPUUtilizationPercentage: 80, it calculates utilization as $\frac{\text{Actual Usage}}{\text{CPU Request}}$. Since requests are capped tightly and pods are dying from OOM before HPA can actuate new pods, the new pods take 45+ seconds to pass cold-start JVM initializations, aggravating the load on surviving pods.

2. Architecture & Remediation Plan

• QoS Alignment: Align requests and limits to achieve Guaranteed QoS or remove CPU limits entirely (relying on CPU requests for scheduling and node-level protection).
  YAMLresources: requests: cpu: "2" memory: "4Gi" limits: # Omit cpu limit or match request high enough to prevent CFS throttling memory: "4Gi"
• JVM Tuning: Tune the JVM heap inside the container using percentage flags:
  Bash-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0
• Scaling Strategy Enhancement:
  • Configure HPA to scale on custom metrics (e.g., HTTP request rate via Prometheus metric http_requests_per_second or active connection pool count) rather than solely CPU usage.
  • Utilize KEDA (Kubernetes Event-driven Autoscaling) or configure behavior policy in HPA for rapid scale-up without stabilization windows delaying the actuation.

Senior SRE Scenario 2: Silent Ingress Failure During Cross-AZ Node Migration

Scenario

A team is migrating a stateful processing cluster to new worker nodes across three Availability Zones (us-east-1a, us-east-1b, us-east-1c). They perform a kubectl drain on old nodes.

While kubectl get pods shows all application pods in the Running state on new nodes and readiness probes pass, 15% of public HTTP traffic via NGINX Ingress Controller returns 502 Bad Gateway intermittently for over 10 minutes.

Cluster state reveals:

• Pods are using AWS VPC CNI with direct IP routable networking.
• Pod shutdown grace period is terminationGracePeriodSeconds: 30.
• Application takes 5 seconds to gracefully stop processing active connections.

Questions for the Candidate:

• What race condition causes the 502 Bad Gateway errors during pod eviction and rescheduling?
• How does the Kubernetes Control Plane handle endpoint deregistration relative to pod termination signals (SIGTERM)?
• What architectural changes guarantee true zero-downtime traffic routing during node drains?

Expected Solution

1. Root Cause Analysis (Control Plane Race Condition)

When kubectl drain evicts a pod:

1. Parallel Execution: The API Server simultaneously marks the Pod state as Terminating AND sends an async endpoint update to the Endpoint/EndpointSlice Controller.
2. Kubelet vs. Ingress Sync Delay:
   • Kubelet immediately sends SIGTERM to the container process.
   • Concurrently, kube-proxy or the Ingress Controller receives the updated EndpointSlice list, regenerates its upstream configuration, and reloads its routing table.
3. The Race: Regenerating and propagating Ingress routing tables across nodes takes 2–8 seconds. If the container process stops accepting connections or exits immediately upon receiving SIGTERM, the Ingress Controller continues sending incoming public traffic to the dying Pod IP for several seconds, resulting in 502 Bad Gateway / Connection Refused.

[Image of Kubernetes Pod Lifecycle and PreStop Hook Execution Flow]

2. Architecture & Remediation Plan

YAML

spec:
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          # Delay SIGTERM handling to allow Ingress/Kube-proxy propagation
          command: ["/bin/sh", "-c", "sleep 15"]

• preStop Lifecycle Hook: Add a preStop sleep hook (e.g., sleep 15). When termination starts, the container stays active while Kubernetes removes its IP from EndpointSlices. Traffic stops arriving before the application process receives SIGTERM.
• PodDisruptionBudgets (PDB): Enforce strict maintenance budgets so drains cannot take down all replicas in a zone simultaneously:
  YAMLapiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: app-pdb spec: minAvailable: "80%"
• Topology Spread Constraints: Ensure application replicas are spread evenly across AZs so single-zone node maintenance doesn’t collapse capacity.

Senior SRE Scenario 3: Persistent CoreDNS Resolution Timeouts Under Burst Load

Scenario

A high-throughput distributed microservice environment running on Kubernetes (v1.28+) encounters sudden, intermittent DNS lookup timeouts (i/o timeout after 5 seconds) across hundreds of worker nodes during peak load.

Observations:

• CoreDNS CPU and Memory utilization remain well below limits (CPU at 20%).
• Increasing CoreDNS replica count from 5 to 30 does not resolve the intermittent 5-second timeouts.
• Nodes run standard Linux kernel configurations with iptables / netfilter modes for kube-proxy.

Questions for the Candidate:

• What underlying Linux networking issue causes exact 5-second DNS delays in Kubernetes clusters using UDP/iptables?
• Why did scaling CoreDNS horizontally fail to solve the issue?
• What are the industry-standard solutions to permanently fix this problem at scale?

Expected Solution

1. Root Cause Analysis

• Netfilter Conntrack Race Condition (conntrack flaw): UDP is connectionless. When multiple threads in a pod send simultaneous DNS requests over UDP to the CoreDNS ClusterIP, two identical socket connections attempt to create conntrack entries in the Linux kernel netfilter table concurrently.
• SNAT/DNAT Race:iptables performs Source NAT and Destination NAT. When two UDP packets race through netfilter simultaneously, one packet fails the __nf_conntrack_confirm lookup due to a lock race and gets dropped silently by the kernel.
• 5-Second Timeout: The client GLIBC DNS resolver waits exactly 5 seconds before retrying the UDP query.
• Why Scaling CoreDNS Failed: The race condition happens inside the client node’s kernel netfilter stack, not within CoreDNS itself. Adding more CoreDNS pods does not change the kernel netfilter locking behavior on the client nodes.

2. Remediation Strategy

YAML

# Pod spec fix via dnsConfig
spec:
  dnsConfig:
    options:
      - name: single-request-reopen
      - name: ndots
        value: "2" # Optimizes search domain expansion overhead