Understanding Service Mesh in Kubernetes

techhadoop kubernetes

Service Mesh in Kubernetes

What is a Service Mesh?

A service mesh is a dedicated infrastructure layer that handles all service-to-service communication inside a Kubernetes cluster — without changing application code.

WITHOUT Service Mesh:

 
WITH Service Mesh:

Core Features

FeatureWhat it gives you
mTLSEncrypted + authenticated service-to-service traffic
Traffic ManagementCanary, blue/green, weighted routing, mirroring
ObservabilityAutomatic metrics, traces, logs per service
Retries & TimeoutsResilience without code changes
Circuit BreakingStop cascading failures
Rate LimitingProtect services from overload
Access PolicyAllow/deny between specific services

Service Mesh Landscape

MeshBest forComplexity
IstioFull-featured, enterprise🔴 High
LinkerdSimple, lightweight, fast🟢 Low
Consul ConnectMulti-cluster, multi-cloud🟠 Medium
CiliumeBPF-based, no sidecar🟠 Medium
Open Service MeshAKS native (deprecated)🟢 Low

Option 1 — Istio (Most Feature-Rich)

Architecture

┌─────────────────────────────────────────────────────┐
│                   CONTROL PLANE                      │
│                    istiod                            │
│   ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│   │  Pilot   │  │  Citadel │  │     Galley        │  │
│   │(traffic) │  │  (certs) │  │  (config valid.)  │  │
│   └──────────┘  └──────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────┘
                        ↓ xDS API
┌─────────────────────────────────────────────────────┐
│                   DATA PLANE                         │
│   Pod A                          Pod B               │
│  ┌─────────┬───────────┐        ┌──────────┬──────┐  │
│  │  App    │  Envoy    │──mTLS─▶│  Envoy   │ App  │  │
│  │Container│  Sidecar  │        │  Sidecar │      │  │
│  └─────────┴───────────┘        └──────────┴──────┘  │
└─────────────────────────────────────────────────────┘

Install Istio

# Download istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-1.20.0/bin:$PATH
# Install with default profile
istioctl install --set profile=default -y
# Verify
istioctl verify-install
kubectl get pods -n istio-system
# Enable sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled
# Check injection is working
kubectl get namespace production --show-labels

Sidecar Injection

# Namespace-level (all pods in namespace)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    istio-injection: enabled    # ← auto-inject Envoy sidecar
---
# Pod-level override (opt out specific pod)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"    # ← skip this pod

Traffic Management

VirtualService — routing rules

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-routing
  namespace: production
spec:
  hosts:
  - api-service
  http:
  # Canary — 10% to v2, 90% to v1
  - route:
    - destination:
        host: api-service
        subset: v1
      weight: 90
    - destination:
        host: api-service
        subset: v2
      weight: 10
  # Timeout & retry
  timeout: 10s
  retries:
    attempts: 3
    perTryTimeout: 3s
    retryOn: "5xx,reset,connect-failure"

DestinationRule — load balancing & circuit breaking

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api-destination
  namespace: production
spec:
  host: api-service
  trafficPolicy:
    # Load balancing
    loadBalancer:
      simple: LEAST_CONN
    # Connection pool (circuit breaking)
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
    # Outlier detection (eject unhealthy pods)
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
  # Subsets for canary
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

Header-based routing

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-header-routing
spec:
  hosts:
  - api-service
  http:
  # Route beta users to v2
  - match:
    - headers:
        x-user-group:
          exact: beta
    route:
    - destination:
        host: api-service
        subset: v2
  # Everyone else → v1
  - route:
    - destination:
        host: api-service
        subset: v1

Fault Injection (Chaos Testing)

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: fault-injection-test
spec:
  hosts:
  - api-service
  http:
  - fault:
      # Inject 500ms delay for 10% of requests
      delay:
        percentage:
          value: 10
        fixedDelay: 500ms
      # Inject 503 error for 5% of requests
      abort:
        percentage:
          value: 5
        httpStatus: 503
    route:
    - destination:
        host: api-service

Security (mTLS + Authorization)

Enforce strict mTLS

# Namespace-wide strict mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT    # All traffic must be mTLS
---
# Mesh-wide strict mTLS (all namespaces)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system    # ← applies mesh-wide
spec:
  mtls:
    mode: STRICT

AuthorizationPolicy — zero trust between services

# Only allow frontend to call api-service
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: api-service-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-service
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
          - "cluster.local/ns/production/sa/frontend"    # service account
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/*"]
---
# Deny all by default (then allow explicitly)
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: production
spec:
  {}    # Empty spec = deny everything

JWT Authentication

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-service
  jwtRules:
  - issuer: "https://login.microsoftonline.com/tenant-id/v2.0"
    jwksUri: "https://login.microsoftonline.com/tenant-id/discovery/v2.0/keys"
    audiences:
    - "api://your-app-id"

Observability

Install Kiali + Jaeger + Prometheus + Grafana

# Install addons
kubectl apply -f istio-1.20.0/samples/addons/prometheus.yaml
kubectl apply -f istio-1.20.0/samples/addons/grafana.yaml
kubectl apply -f istio-1.20.0/samples/addons/jaeger.yaml
kubectl apply -f istio-1.20.0/samples/addons/kiali.yaml
# Open dashboards
istioctl dashboard kiali      # Service graph + health
istioctl dashboard grafana    # Metrics dashboards
istioctl dashboard jaeger     # Distributed tracing

Key Metrics (auto-generated by Envoy)

istio_requests_total                    # Request count
istio_request_duration_milliseconds     # Latency histogram
istio_request_bytes                     # Request size
istio_response_bytes                    # Response size
istio_tcp_connections_opened_total      # TCP connections

Option 2 — Linkerd (Lightweight & Simple)

Install Linkerd

# Install CLI
curl --proto '=https' --tlsv1.2 -sSfL \
  https://run.linkerd.io/install | sh
export PATH=$PATH:$HOME/.linkerd2/bin
# Pre-install check
linkerd check --pre
# Install CRDs then control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
# Verify
linkerd check
# Install observability (Viz)
linkerd viz install | kubectl apply -f -
linkerd viz dashboard &    # Open dashboard

Inject Linkerd Sidecar

# Namespace annotation
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    linkerd.io/inject: enabled    # ← auto-inject
---
# Or annotate deployment directly
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  annotations:
    linkerd.io/inject: enabled

Traffic Splitting (Canary)

apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: api-canary
  namespace: production
spec:
  service: api-service
  backends:
  - service: api-service-v1
    weight: 90
  - service: api-service-v2
    weight: 10

Retries & Timeouts

apiVersion: policy.linkerd.io/v1beta1
kind: HTTPRoute
metadata:
  name: api-retry
  namespace: production
spec:
  parentRefs:
  - name: api-service
    kind: Service
  rules:
  - backendRefs:
    - name: api-service
      port: 80
    timeouts:
      request: 10s
    retry:
      limit: 3
      timeout: 3s
      conditions:
      - gateway-error
      - http-response-500

Option 3 — Cilium Service Mesh (No Sidecar)

Traditional sidecar mesh:         Cilium (eBPF):
Pod → Sidecar → Network           Pod → eBPF kernel hooks → Network
(extra hop, extra resource)       (in-kernel, no sidecar overhead)
# Install Cilium with service mesh features
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set gatewayAPI.enabled=true \
  --set envoy.enabled=true
# Verify
cilium status
cilium connectivity test
# mTLS via CiliumNetworkPolicy
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-mtls-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-service
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/.*"

Istio vs Linkerd vs Cilium

IstioLinkerdCilium
SidecarEnvoy proxylinkerd-proxy (Rust)No sidecar (eBPF)
ComplexityHighLowMedium
Resource overheadHighLowVery low
mTLS✅✅✅
Traffic splitting✅ Full✅ Basic✅
Circuit breaking✅⚠️ Limited✅
Fault injection✅❌❌
Observability✅ Rich✅ Good✅ Good
Multi-cluster✅✅✅
AKS native support✅✅✅ (BYOCNI)
Best forEnterprise, full controlSimplicity, low overheadPerformance, no sidecar

Multi-Cluster Service Mesh

Cluster A (East US)              Cluster B (West US)
┌─────────────────────┐          ┌─────────────────────┐
│  istiod             │◄────────▶│  istiod             │
│  ┌───────────────┐  │  shared  │  ┌───────────────┐  │
│  │ frontend pod  │  │  control │  │ api-service   │  │
│  │  + sidecar    │──┼──plane───┼─▶│  + sidecar    │  │
│  └───────────────┘  │          │  └───────────────┘  │
│  East Gateway       │◄─mTLS───▶│  West Gateway       │
└─────────────────────┘          └─────────────────────┘
# Istio multi-cluster setup (primary-remote model)
# On primary cluster
istioctl install --set profile=default \
  --set values.pilot.env.EXTERNAL_ISTIOD=true
# Create remote secret (allow primary to manage remote)
istioctl create-remote-secret \
  --name=remote-cluster \
  --context=remote-cluster-context | \
  kubectl apply -f - --context=primary-cluster-context

Observability Stack

Envoy Sidecars (metrics + traces)
        ↓
Prometheus (scrape metrics)
        ↓              ↓
  Grafana          Jaeger / Zipkin
(dashboards)    (distributed tracing)
        ↓
  Kiali (service graph + health)
        ↓
  Alertmanager → PagerDuty / Slack

Golden Signal Dashboards (auto from mesh)

Latency   → P50 / P95 / P99 per service
Traffic   → Requests per second per service
Errors    → 4xx / 5xx rate per service
Saturation → Connection pool usage, CPU

Best Practices

PracticeWhy
Start with Linkerd if new to meshesLower complexity, easier to debug
Enable mTLS mesh-wide from day 1Harder to enforce later
Use PERMISSIVE mode first, then STRICTGradual rollout without breaking traffic
Set resource requests on sidecarsPrevent sidecar starvation
Use AuthorizationPolicy deny-all baselineZero-trust foundation
Monitor sidecar resource usageEnvoy adds ~50-100MB RAM per pod
Use Kiali for topology visibilityCatch unexpected service dependencies
Test with fault injection before prodValidate resilience patterns work
Avoid mesh for batch/job workloadsSidecar overhead not worth it
Use Cilium on performance-critical clusterseBPF = kernel-level, no overhead

Troubleshooting

# Check sidecar injection
kubectl get pod api-pod -n production -o jsonpath='{.spec.containers[*].name}'
# Should show: api envoy (two containers)
# Check mTLS status
istioctl x describe pod api-pod-xxx -n production
# Check Envoy config
istioctl proxy-config cluster api-pod-xxx.production
istioctl proxy-config listener api-pod-xxx.production
istioctl proxy-config route api-pod-xxx.production
# Check authorization policies
istioctl x authz check api-pod-xxx.production
# View Envoy access logs
kubectl logs api-pod-xxx -n production -c istio-proxy
# Debug traffic issues
istioctl analyze -n production    # ← catch misconfigs
# Linkerd debugging
linkerd viz stat deployments -n production
linkerd viz routes deployment/api -n production
linkerd viz edges deployment -n production

A service mesh is the foundation of zero-trust networking inside Kubernetes — mTLS by default, explicit authorization policies, and full observability without touching application code. Start with Linkerd for simplicity or Istio if you need the full feature set.

Leave a Reply