Service Mesh Interview Questions

Beginner Level

Q1. What is a service mesh and why do we need it?

A: A service mesh is a dedicated infrastructure layer that manages service-to-service communication in a microservices architecture. Without it, every service must implement its own retry logic, timeouts, mTLS, and observability — leading to duplicated code across every language/framework.

A service mesh moves all of that into a sidecar proxy (or kernel via eBPF) so the application stays focused on business logic.

Problems it solves:

• No encryption between services by default
• Hard to debug distributed failures
• Retry/timeout logic duplicated in every service
• No visibility into which service is causing latency
• No way to do canary releases at the network level

Q2. What is the difference between a sidecar proxy and a service mesh control plane?

A:

Control Plane (istiod / Linkerd Controller)
  - Manages configuration
  - Issues and rotates mTLS certificates
  - Pushes routing rules to all proxies
  - Collects telemetry
Data Plane (Envoy / linkerd-proxy sidecars)
  - Actually intercepts and forwards traffic
  - Enforces routing rules
  - Enforces mTLS
  - Emits metrics and traces

The control plane tells proxies what to do. The data plane does it.

Q3. What is mTLS and how does a service mesh implement it?

A: mTLS (mutual TLS) means both sides of a connection authenticate each other — unlike regular TLS where only the server presents a certificate.

How a mesh implements it:

1. Control plane acts as a Certificate Authority (CA)
2. Issues a unique certificate to every pod (bound to its service account)
3. Sidecar intercepts all outbound/inbound traffic
4. Automatically performs TLS handshake — app sees plain HTTP
5. Certificates are rotated automatically (e.g. every 24 hours)

Pod A (app) → sidecar (presents cert A) 
            ──────── mTLS ────────────▶
                          sidecar (presents cert B) → Pod B (app)

Application code is completely unaware of TLS.

Q4. What is the difference between PeerAuthentication and AuthorizationPolicy in Istio?

A:

# PeerAuthentication — enforce mTLS
spec:
  mtls:
    mode: STRICT
# AuthorizationPolicy — control who can call what
spec:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/prod/sa/frontend"]

Q5. What is the difference between a VirtualService and a DestinationRule in Istio?

A:

Think of it as:

• VirtualService = traffic cop (route 10% here, 90% there)
• DestinationRule = rules for the destination (max 100 connections, eject unhealthy pods)

Q6. What are the PERMISSIVE and STRICT modes in Istio mTLS?

A:

Best practice: Start with PERMISSIVE during rollout, migrate all services, then switch to STRICT.

Q7. How does sidecar injection work in Kubernetes?

A: Istio uses a Kubernetes Mutating Admission Webhook:

kubectl apply pod.yaml
        ↓
Kubernetes API Server
        ↓
Mutating Admission Webhook (istio-sidecar-injector)
        ↓
Modifies pod spec — adds:
  - initContainer: istio-init (sets iptables rules)
  - container: istio-proxy (Envoy sidecar)
        ↓
Pod runs with sidecar automatically

The istio-init container sets iptables rules to intercept all traffic through the Envoy proxy on ports 15001 (outbound) and 15006 (inbound) — without changing the app.

Intermediate Level

Q8. How does Istio handle certificate rotation and what happens if istiod goes down?

A:

• Istio’s CA (inside istiod) issues SVID certificates (SPIFFE format) to each workload
• Default cert lifetime: 24 hours, rotated at ~75% of lifetime (around 18 hours)
• Rotation happens via SDS (Secret Discovery Service) — Envoy requests new certs automatically

If istiod goes down:

• Existing connections continue (Envoy already has certs)
• New pods cannot get certificates — they’ll fail to start mesh enrollment
• Routing rule updates stop propagating
• Existing routing rules remain cached in Envoy

This is why istiod HA (multiple replicas) is critical in production:

kubectl scale deployment istiod -n istio-system --replicas=3

Q9. Explain circuit breaking in Istio. How is it configured?

A: Circuit breaking stops sending traffic to an unhealthy service to prevent cascading failures — similar to an electrical circuit breaker.

Three mechanisms in Istio:

1. Connection Pool Limits — limits concurrent connections

connectionPool:
  tcp:
    maxConnections: 100
  http:
    http1MaxPendingRequests: 50
    http2MaxRequests: 200

2. Outlier Detection — ejects unhealthy pods from load balancer

outlierDetection:
  consecutive5xxErrors: 5      # 5 errors in a row
  interval: 10s                # check every 10s
  baseEjectionTime: 30s        # eject for 30s minimum
  maxEjectionPercent: 50       # never eject more than 50% of pods

3. Pending Request Limits — rejects requests when queue is full

When circuit opens → caller gets 503 UF (Upstream overflow) immediately instead of waiting for timeout — fails fast.

Q10. What is the difference between Istio’s ingress gateway and a standard Kubernetes Ingress?

A:

# Istio Gateway — defines the listener
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: api-tls-cert
    hosts:
    - api.acme.com
---
# VirtualService — defines routing behind the gateway
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-vs
spec:
  hosts:
  - api.acme.com
  gateways:
  - api-gateway
  http:
  - route:
    - destination:
        host: api-service
        subset: v1

Q11. How do you implement a canary deployment with Istio?

A:

Step 1: Deploy v2 alongside v1 (both running)
Step 2: Create DestinationRule with subsets
Step 3: Use VirtualService to split traffic
Step 4: Gradually shift weight
Step 5: Monitor error rates and latency
Step 6: Cut over to 100% v2 or rollback

# DestinationRule — define v1 and v2 subsets
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api-dr
spec:
  host: api-service
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---
# VirtualService — start at 5% canary
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-canary
spec:
  hosts:
  - api-service
  http:
  - route:
    - destination:
        host: api-service
        subset: v1
      weight: 95
    - destination:
        host: api-service
        subset: v2
      weight: 5    # ← increase this gradually

Progressive delivery tools like Flagger or Argo Rollouts automate this — they watch error rates and auto-promote or rollback.

Q12. What is Envoy’s xDS API and why does it matter?

A: xDS is the configuration API that istiod uses to push config to all Envoy proxies dynamically — without restarting them.

istiod
  ├── LDS → "listen on port 15001"
  ├── RDS → "route /api/* to cluster api-service-v1"
  ├── CDS → "cluster api-service-v1 exists"
  ├── EDS → "api-service-v1 has pods 10.0.1.5, 10.0.1.6"
  └── SDS → "here's your mTLS cert, rotate in 18h"

Why it matters: config changes propagate to thousands of pods in seconds with zero restarts.

Q13. How does Linkerd differ from Istio architecturally?

A:

Key insight: Linkerd’s proxy is written in Rust specifically for this use case — it’s faster and lighter than Envoy. Istio uses Envoy which is a general-purpose proxy with far more knobs.

Q14. What is SPIFFE and how does Istio use it?

A: SPIFFE (Secure Production Identity Framework For Everyone) is a standard for workload identity in cloud-native environments.

Each workload gets a SPIFFE ID in URI format:

spiffe://cluster.local/ns/production/sa/api-service
         └─ trust domain ─┘ └─ namespace ─┘ └─ service account ─┘

Istio issues SVID (SPIFFE Verifiable Identity Document) certificates containing this URI as a SAN (Subject Alternative Name).

AuthorizationPolicy uses SPIFFE IDs:
principals:
  - "cluster.local/ns/production/sa/frontend"
This means: "only the pod with this exact
             service account identity can connect"

This is much stronger than IP-based rules (IPs change when pods restart).

Advanced Level

Q15. How would you debug a 503 error in an Istio mesh?

A: Systematic approach:

# Step 1 — Check if sidecar is injected
kubectl get pod failing-pod -o jsonpath='{.spec.containers[*].name}'
# Step 2 — Check Istio proxy status
istioctl proxy-status   # Are all proxies in sync with istiod?
# Step 3 — Analyze the namespace for misconfigs
istioctl analyze -n production
# Step 4 — Check Envoy access logs for error flags
kubectl logs failing-pod -c istio-proxy | grep -v '"200"'
# Look for: UF=upstream overflow (circuit break)
#           UO=upstream overflow
#           NR=no route found
#           URX=upstream retry exceeded
# Step 5 — Check routing config on the calling pod
istioctl proxy-config routes calling-pod.production
# Step 6 — Check endpoints are healthy
istioctl proxy-config endpoints calling-pod.production \
  | grep api-service
# Step 7 — Check AuthorizationPolicy
istioctl x authz check failing-pod.production
# Step 8 — Check mTLS status
istioctl x describe pod failing-pod.production

Common causes:

• NR — no route: VirtualService misconfigured or missing
• UF — circuit breaker open: too many errors or connections
• RBAC denied — AuthorizationPolicy blocking the call
• Cert mismatch — one service in STRICT, caller not meshed

Q16. How do you handle service mesh in a multi-cluster setup? What are the challenges?

A: Three models:

1. Replicated control planes (most common)

Cluster A                    Cluster B
istiod-A                     istiod-B
  ↓                            ↓
East-West Gateway ←──mTLS──▶ East-West Gateway
(cross-cluster traffic tunneled through gateways)

2. Primary-Remote (single control plane)

Cluster A (Primary)          Cluster B (Remote)
istiod ──────────────────────▶ no istiod
       manages both clusters

3. External control plane

Management Cluster           Workload Clusters
istiod ──────────────────────▶ Cluster A
       ──────────────────────▶ Cluster B

Key challenges:

Q17. What is Ambient Mesh in Istio and why was it introduced?

A: Ambient Mesh is Istio’s sidecar-free architecture introduced to address the main criticisms of the sidecar model:

Problems with sidecars:

• Every pod gets an Envoy sidecar — high memory/CPU overhead
• Sidecar must be restarted to update — causes pod restarts
• Sidecar injection is all-or-nothing per pod
• Complex bootstrapping and lifecycle management

Ambient Mesh approach:

Traditional:
Pod → Envoy sidecar → Network
Ambient:
Pod → ztunnel (node-level, L4) → waypoint proxy (L7, per service)

Two layers:

• ztunnel — a per-node DaemonSet handling L4 mTLS for all pods on the node (zero pod overhead)
• Waypoint proxy — an optional per-service Envoy for L7 features (routing, retries, auth policies)

Benefits:

• No sidecar injection needed
• Update mesh without restarting pods
• Pay for L7 features only where needed
• Dramatically lower resource footprint

Q18. How does a service mesh interact with Kubernetes Network Policies? Are they redundant?

A: They operate at different layers and are complementary, not redundant:

Best practice — use both:

NetworkPolicy  → "frontend namespace can reach backend namespace on port 8080"
                 (coarse-grained, IP-level firewall)
AuthorizationPolicy → "only pods with SA 'frontend' can call GET /api/*"
                      (fine-grained, cryptographic identity)

Network Policy is the outer wall. AuthorizationPolicy is the inner door lock.

Q19. How would you implement progressive delivery (automated canary) with Istio and Flagger?

A:

Flagger watches a Deployment
  → Creates primary + canary deployments automatically
  → Shifts traffic incrementally
  → Queries Prometheus for error rate + latency
  → Auto-promotes if healthy, auto-rollbacks if not

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: api-service
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  progressDeadlineSeconds: 120
  service:
    port: 80
    targetPort: 8080
    gateways:
    - istio-system/ingressgateway
    hosts:
    - api.acme.com
  analysis:
    interval: 1m           # Check every minute
    threshold: 5           # Max 5 failed checks before rollback
    maxWeight: 50          # Max 50% traffic to canary
    stepWeight: 10         # Increase by 10% each step
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99            # Must maintain 99% success
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500           # P99 must be < 500ms
      interval: 1m
    webhooks:
    - name: load-test
      url: http://loadtester/
      metadata:
        cmd: "hey -z 1m -q 10 http://api-service/"

Traffic flow during canary:

0%  → deploy canary
10% → check metrics (1 min)
20% → check metrics (1 min)
...
50% → check metrics (1 min)
     ✅ promote to 100% OR ❌ rollback to 0%

Q20. What are the performance overheads of a service mesh and how do you mitigate them?

A:

Latency overhead (per hop):

Istio/Envoy:   ~1-3ms added latency per request (P99)
Linkerd:       ~0.5-1ms added latency per request
Cilium eBPF:   ~0.1-0.3ms (kernel-level, minimal overhead)

Resource overhead (per pod):

Envoy sidecar:         50-100MB RAM, 0.1-0.5 CPU cores
linkerd-proxy:         10-20MB RAM, minimal CPU
Cilium (no sidecar):   ~0MB per pod (shared ztunnel on node)

Mitigation strategies:

# Reduce tracing overhead — sample 1% in prod
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-tracing
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: jaeger
    randomSamplingPercentage: 1.0    # ← not 100%

Quick-Fire Questions

Q: What port does Envoy listen on for inbound/outbound? Inbound: 15006, Outbound: 15001, Admin: 15000, Health: 15021

Q: What is the difference between east-west and north-south traffic? North-south = traffic entering/leaving the cluster (ingress/egress). East-west = service-to-service inside the cluster.

Q: Can you use a service mesh without Kubernetes? Yes — Consul Connect supports VMs. Istio has VM mesh support. But Kubernetes is the primary target.

Q: What is SMI (Service Mesh Interface)? A standard set of Kubernetes CRDs that provide a common API across mesh implementations (TrafficSplit, TrafficPolicy, etc.) so tools like Flagger work with any mesh.

Q: What happens to traffic if the sidecar crashes? Traffic is intercepted by iptables rules pointing to the sidecar. If the sidecar is down, traffic is dropped — not bypassed. This is by design for security.

Q: How do you exclude a namespace from the mesh? Remove the istio-injection=enabled label and restart pods. Or annotate pods with sidecar.istio.io/inject: "false".