kubernetes

Understanding Kubernetes Taints and Their Effects

techhadoop kubernetes

Kubernetes Taints

Taints are a mechanism that allow a node to repel certain pods from being scheduled on it. They work together with tolerations (defined on pods) to control which pods can run on which nodes.

The core idea

A taint marks a node as “special” or “restricted.” Only pods that explicitly tolerate that taint will be scheduled there. Everything else is kept away.

Taint structure

A taint has three parts:

key=value:effect
  • key — a label-like identifier (e.g. dedicated, gpu)
  • value — optional qualifier (e.g. true, high-memory)
  • effect — what happens to pods that don’t tolerate it
The three effects
EffectBehavior
NoScheduleNew pods won’t be scheduled on the node. Existing pods stay.
PreferNoScheduleKubernetes tries to avoid scheduling pods here, but will if necessary.
NoExecuteNew pods won’t be scheduled AND existing non-tolerating pods are evicted.
Adding and removing taints
# Add a taint
kubectl taint nodes node1 dedicated=gpu:NoSchedule
# Remove a taint (note the trailing -)
kubectl taint nodes node1 dedicated=gpu:NoSchedule-
Tolerations (the pod side)

A pod opts in to a tainted node by declaring a toleration in its spec:

spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"

A toleration with operator: Exists matches any value for that key:

tolerations:
  - key: "dedicated"
    operator: "Exists"
    effect: "NoSchedule"
Common real-world use cases

Dedicated nodes — Reserve a node exclusively for a team or workload (e.g. GPU nodes for ML jobs). Taint the node; only ML pods carry the toleration.

Node issues — Kubernetes itself auto-taints nodes when they’re unhealthy (e.g. node.kubernetes.io/not-ready:NoExecute), causing pods to be evicted.

Control plane isolation — Master/control-plane nodes are tainted by default (node-role.kubernetes.io/control-plane:NoSchedule) so regular workloads don’t land there.

Spot/preemptible nodes — Taint spot instances so only fault-tolerant workloads with the matching toleration run there.

Taints vs. Node Affinity

These are related but different tools:

  • Taints/Tolerations → node repels pods (node-driven, opt-in)
  • Node Affinity → pod seeks nodes (pod-driven, attraction-based)

They’re often used together: taint a node to keep most pods off, and use node affinity on the right pods to actively pull them toward it.

GKE Best Practices for Optimal Performance

techhadoop GCP, GKE, kubernetes

GKE Best Practices

What is GKE?

Google Kubernetes Engine is Google Cloud’s managed Kubernetes service — Google manages the control plane, you manage the worker nodes (or let Autopilot manage everything).

GKE Modes:
┌─────────────────────────────────────────────────────────────┐
│  Standard Mode              │  Autopilot Mode               │
│  ─────────────              │  ───────────────              │
│  You manage node pools      │  Google manages everything    │
│  You choose machine types   │  Pay per pod not node         │
│  Full node customization    │  No node management           │
│  More control               │  More managed/serverless      │
│  Best for: complex workloads│  Best for: simplicity         │
└─────────────────────────────────────────────────────────────┘

1. Cluster Architecture Best Practices

Use Regional Clusters (Not Zonal)
# ❌ Zonal — single point of failure
gcloud container clusters create my-cluster \
  --zone us-central1-a
# ✅ Regional — control plane + nodes across 3 zones
gcloud container clusters create my-cluster \
  --region us-central1 \
  --num-nodes 2              # 2 per zone = 6 total nodes
Zonal Cluster:              Regional Cluster:
us-central1-a               us-central1-a  us-central1-b  us-central1-c
  control plane               control        control        control
  node node node              plane          plane          plane
                              node node      node node      node node
  Zone fails = cluster down   Zone fails = cluster healthy
Separate Node Pools by Workload Type
# System node pool — for cluster components
gcloud container node-pools create system-pool \
  --cluster my-cluster \
  --region us-central1 \
  --machine-type n2-standard-2 \
  --num-nodes 1 \
  --node-taints CriticalAddonsOnly=true:NoSchedule \
  --node-labels pool=system
# Application node pool — for your apps
gcloud container node-pools create app-pool \
  --cluster my-cluster \
  --region us-central1 \
  --machine-type n2-standard-4 \
  --num-nodes 2 \
  --enable-autoscaling \
  --min-nodes 1 \
  --max-nodes 10 \
  --node-labels pool=application
# GPU node pool — for ML workloads
gcloud container node-pools create gpu-pool \
  --cluster my-cluster \
  --region us-central1 \
  --machine-type n1-standard-4 \
  --accelerator type=nvidia-tesla-t4,count=1 \
  --num-nodes 0 \
  --enable-autoscaling \
  --min-nodes 0 \
  --max-nodes 5 \
  --node-taints nvidia.com/gpu=present:NoSchedule
# Spot node pool — for batch / fault-tolerant workloads
gcloud container node-pools create spot-pool \
  --cluster my-cluster \
  --region us-central1 \
  --machine-type n2-standard-4 \
  --spot \
  --enable-autoscaling \
  --min-nodes 0 \
  --max-nodes 20
Terraform Cluster Setup
# main.tf
resource "google_container_cluster" "primary" {
  name     = "prod-cluster"
  location = "us-central1"      # regional
  # Remove default node pool — use custom ones
  remove_default_node_pool = true
  initial_node_count       = 1
  # Networking
  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.subnet.name
  networking_config {
    enable_intra_node_visibility = true
  }
  ip_allocation_policy {
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }
  # Private cluster — no public node IPs
  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false
    master_ipv4_cidr_block  = "172.16.0.0/28"
  }
  # Authorized networks for control plane access
  master_authorized_networks_config {
    cidr_blocks {
      cidr_block   = "10.0.0.0/8"
      display_name = "internal"
    }
    cidr_blocks {
      cidr_block   = var.office_ip
      display_name = "office"
    }
  }
  # Security
  workload_identity_config {
    workload_pool = "${var.project_id}.svc.id.goog"
  }
  # Enable addons
  addons_config {
    horizontal_pod_autoscaling { disabled = false }
    http_load_balancing        { disabled = false }
    network_policy_addon       { disabled = false }
    gce_persistent_disk_csi_driver_config { enabled = true }
    gcs_fuse_csi_driver_config            { enabled = true }
  }
  # Enable network policy
  network_policy {
    enabled  = true
    provider = "CALICO"
  }
  # Cluster autoscaling
  cluster_autoscaling {
    enabled = true
    resource_limits {
      resource_type = "cpu"
      minimum       = 4
      maximum       = 100
    }
    resource_limits {
      resource_type = "memory"
      minimum       = 16
      maximum       = 400
    }
    auto_provisioning_defaults {
      service_account = google_service_account.nodes.email
      oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]
    }
  }
  # Maintenance window
  maintenance_policy {
    recurring_window {
      start_time = "2024-01-01T02:00:00Z"
      end_time   = "2024-01-01T06:00:00Z"
      recurrence = "FREQ=WEEKLY;BYDAY=SA,SU"
    }
  }
  # Logging and monitoring
  logging_config {
    enable_components = [
      "SYSTEM_COMPONENTS",
      "WORKLOADS"
    ]
  }
  monitoring_config {
    enable_components = [
      "SYSTEM_COMPONENTS",
      "WORKLOADS"
    ]
    managed_prometheus {
      enabled = true
    }
  }
  # Release channel — get automatic updates
  release_channel {
    channel = "REGULAR"
  }
}
# System node pool
resource "google_container_node_pool" "system" {
  name       = "system-pool"
  cluster    = google_container_cluster.primary.name
  location   = "us-central1"
  node_count = 1
  node_config {
    machine_type    = "n2-standard-2"
    service_account = google_service_account.nodes.email
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]
    workload_metadata_config {
      mode = "GKE_METADATA"       # Workload Identity
    }
    shielded_instance_config {
      enable_secure_boot          = true
      enable_integrity_monitoring = true
    }
    taint {
      key    = "CriticalAddonsOnly"
      value  = "true"
      effect = "NO_SCHEDULE"
    }
    labels = {
      pool = "system"
    }
  }
  management {
    auto_repair  = true
    auto_upgrade = true
  }
}
# Application node pool with autoscaling
resource "google_container_node_pool" "application" {
  name     = "app-pool"
  cluster  = google_container_cluster.primary.name
  location = "us-central1"
  autoscaling {
    min_node_count = 1
    max_node_count = 10
  }
  node_config {
    machine_type    = "n2-standard-4"
    disk_size_gb    = 100
    disk_type       = "pd-ssd"
    service_account = google_service_account.nodes.email
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]
    workload_metadata_config {
      mode = "GKE_METADATA"
    }
    shielded_instance_config {
      enable_secure_boot          = true
      enable_integrity_monitoring = true
    }
    labels = {
      pool = "application"
      env  = "production"
    }
  }
  management {
    auto_repair  = true
    auto_upgrade = true
  }
  upgrade_settings {
    max_surge       = 1
    max_unavailable = 0
  }
}

2. Security Best Practices

Workload Identity (No Service Account Keys)
# Create GCP service account
gcloud iam service-accounts create api-sa \
  --display-name="API Service Account"
# Grant permissions to GCP SA
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:api-sa@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"
# Create Kubernetes service account
kubectl create serviceaccount api-ksa -n production
# Bind K8s SA to GCP SA
gcloud iam service-accounts add-iam-policy-binding \
  api-sa@$PROJECT_ID.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:$PROJECT_ID.svc.id.goog[production/api-ksa]"
# Annotate K8s SA
kubectl annotate serviceaccount api-ksa \
  -n production \
  iam.gke.io/gcp-service-account=api-sa@$PROJECT_ID.iam.gserviceaccount.com
# Pod uses Workload Identity — no key files needed
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
spec:
  template:
    spec:
      serviceAccountName: api-ksa     # ← K8s SA with WI annotation
      containers:
      - name: api
        image: gcr.io/myproject/api:latest
        # GCP SDK auto-detects credentials via metadata server
        # No GOOGLE_APPLICATION_CREDENTIALS needed
Pod Security Standards
# Enforce restricted security for namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
# Pod that meets restricted standards
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: api
        image: gcr.io/myproject/api:latest
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        volumeMounts:
        - name: tmp
          mountPath: /tmp              # writable tmp dir
        - name: cache
          mountPath: /app/cache
      volumes:
      - name: tmp
        emptyDir: {}
      - name: cache
        emptyDir: {}
Binary Authorization
# Enable Binary Authorization
gcloud services enable binaryauthorization.googleapis.com
# Create attestor — only signed images can deploy
gcloud container binauthz attestors create production-attestor \
  --attestation-authority-note=projects/$PROJECT_ID/notes/production-note \
  --attestation-authority-note-project=$PROJECT_ID
# Set policy — require attestation
cat > /tmp/policy.yaml << EOF
defaultAdmissionRule:
  evaluationMode: REQUIRE_ATTESTATION
  requireAttestationsBy:
  - projects/$PROJECT_ID/attestors/production-attestor
  enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
EOF
gcloud container binauthz policy import /tmp/policy.yaml
Network Policies
# Default deny all
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
# Allow api to reach database only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - port: 5432
---
# Allow egress to Google APIs
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-google-apis
  namespace: production
spec:
  podSelector: {}
  egress:
  - to:
    - ipBlock:
        cidr: 199.36.153.8/30    # restricted.googleapis.com
    ports:
    - port: 443
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - port: 53
      protocol: UDP              # DNS
Secret Management with Secret Manager
# Use External Secrets Operator to sync GCP secrets → K8s secrets
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secret-store
  namespace: production
spec:
  provider:
    gcpsm:
      projectID: my-project-id
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-store
    kind: SecretStore
  target:
    name: api-secrets           # creates K8s secret
    creationPolicy: Owner
  data:
  - secretKey: db-password
    remoteRef:
      key: prod/api/db-password
  - secretKey: api-key
    remoteRef:
      key: prod/api/external-api-key

3. Resource Management Best Practices

Always Set Resource Requests and Limits
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      containers:
      - name: api
        image: gcr.io/myproject/api:latest
        resources:
          requests:
            cpu: "250m"          # guaranteed CPU
            memory: "256Mi"      # guaranteed memory
          limits:
            cpu: "500m"          # max CPU (throttled if exceeded)
            memory: "512Mi"      # max memory (OOM killed if exceeded)
LimitRange — Default Limits per Namespace
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:                     # default limit if not set
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:              # default request if not set
      cpu: "100m"
      memory: "128Mi"
    max:                         # hard max per container
      cpu: "4"
      memory: "8Gi"
    min:                         # minimum per container
      cpu: "50m"
      memory: "64Mi"
  - type: Pod
    max:
      cpu: "8"
      memory: "16Gi"
  - type: PersistentVolumeClaim
    max:
      storage: "100Gi"
ResourceQuota per Namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    # Compute
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    # Objects
    pods: "100"
    services: "20"
    persistentvolumeclaims: "20"
    secrets: "50"
    configmaps: "50"
    # Service types
    services.loadbalancers: "3"
    services.nodeports: "0"
Priority Classes
# Define priority classes
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical
value: 1000000
globalDefault: false
description: "Critical production services"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high
value: 100000
description: "Important production services"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low
value: 1000
description: "Batch and background jobs"
---
# Use in deployment
spec:
  template:
    spec:
      priorityClassName: critical   # ← won't be evicted for lower priority

4. Autoscaling Best Practices

Horizontal Pod Autoscaler
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60    # scale at 60% CPU
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  - type: External
    external:
      metric:
        name: pubsub.googleapis.com|subscription|num_undelivered_messages
        selector:
          matchLabels:
            resource.labels.subscription_id: my-subscription
      target:
        type: AverageValue
        averageValue: "100"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100              # double pods in one step
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5min before scale down
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60
Vertical Pod Autoscaler
# Install VPA first
# kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"          # Recommend only — don't auto-update
    # Options: Off | Initial | Recreate | Auto
  resourcePolicy:
    containerPolicies:
    - containerName: api
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: "2"
        memory: 2Gi
      controlledResources:
      - cpu
      - memory
---
# Check VPA recommendations
# kubectl describe vpa api-vpa -n production
# Look for: Status.Recommendation.ContainerRecommendations
Cluster Autoscaler Best Practices
# Pod Disruption Budget — prevent CA from evicting too many pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: production
spec:
  minAvailable: 2             # keep at least 2 pods running
  # OR
  # maxUnavailable: 1         # allow at most 1 pod down
  selector:
    matchLabels:
      app: api
# Configure cluster autoscaler behavior
gcloud container clusters update my-cluster \
  --region us-central1 \
  --autoscaling-profile optimize-utilization  # or balanced
# Set scale-down delay
gcloud container node-pools update app-pool \
  --cluster my-cluster \
  --region us-central1 \
  --autoscaling-profile optimize-utilization

5. Networking Best Practices

Use Private Cluster with VPC-Native Networking
# Create VPC and subnets
gcloud compute networks create prod-vpc \
  --subnet-mode custom
gcloud compute networks subnets create prod-subnet \
  --network prod-vpc \
  --region us-central1 \
  --range 10.0.0.0/20 \
  --secondary-range pods=10.4.0.0/14,services=10.0.16.0/20
# Create private cluster
gcloud container clusters create prod-cluster \
  --region us-central1 \
  --network prod-vpc \
  --subnetwork prod-subnet \
  --cluster-secondary-range-name pods \
  --services-secondary-range-name services \
  --enable-private-nodes \
  --master-ipv4-cidr 172.16.0.0/28 \
  --enable-ip-alias
Cloud Armor WAF for Ingress
# BackendConfig — attach Cloud Armor policy
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: api-backend-config
  namespace: production
spec:
  securityPolicy:
    name: prod-waf-policy     # Cloud Armor policy name
  connectionDraining:
    drainingTimeoutSec: 60
  healthCheck:
    checkIntervalSec: 15
    timeoutSec: 15
    healthyThreshold: 1
    unhealthyThreshold: 2
    type: HTTP
    requestPath: /health
    port: 8080
---
# Service references BackendConfig
apiVersion: v1
kind: Service
metadata:
  name: api-service
  namespace: production
  annotations:
    cloud.google.com/backend-config: '{"default":"api-backend-config"}'
    cloud.google.com/neg: '{"ingress": true}'    # Container-native LB
spec:
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP
---
# GKE Ingress with HTTPS and managed cert
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.global-static-ip-name: prod-ip
    networking.gke.io/managed-certificates: api-cert
    kubernetes.io/ingress.allow-http: "false"
spec:
  rules:
  - host: api.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
# Create Cloud Armor WAF policy
gcloud compute security-policies create prod-waf-policy \
  --description "Production WAF policy"
# Enable OWASP rules
gcloud compute security-policies rules create 1000 \
  --security-policy prod-waf-policy \
  --expression "evaluatePreconfiguredExpr('xss-v33-stable')" \
  --action deny-403
# Rate limiting
gcloud compute security-policies rules create 2000 \
  --security-policy prod-waf-policy \
  --expression "true" \
  --action throttle \
  --rate-limit-threshold-count 1000 \
  --rate-limit-threshold-interval-sec 60 \
  --conform-action allow \
  --exceed-action deny-429

6. Reliability Best Practices

Pod Anti-Affinity — Spread Across Zones
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
spec:
  replicas: 6
  template:
    spec:
      # Spread across zones
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: api
      # Don't put two api pods on same node
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: api
            topologyKey: kubernetes.io/hostname
Readiness and Liveness Probes
containers:
- name: api
  image: gcr.io/myproject/api:latest
  ports:
  - containerPort: 8080
  # Liveness — restart pod if unhealthy
  livenessProbe:
    httpGet:
      path: /healthz
      port: 8080
    initialDelaySeconds: 30    # wait before first check
    periodSeconds: 10
    failureThreshold: 3        # fail 3 times = restart
    timeoutSeconds: 5
  # Readiness — remove from LB if not ready
  readinessProbe:
    httpGet:
      path: /ready
      port: 8080
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 3
    successThreshold: 1
  # Startup — for slow-starting apps
  startupProbe:
    httpGet:
      path: /healthz
      port: 8080
    failureThreshold: 30       # allow 5 min to start
    periodSeconds: 10
Graceful Shutdown
containers:
- name: api
  lifecycle:
    preStop:
      exec:
        command:
        - /bin/sh
        - -c
        - sleep 15             # allow LB to drain connections
  # Allow time for preStop + app shutdown
  terminationGracePeriodSeconds: 60

7. Cost Optimization Best Practices

Use Spot VMs for Non-Critical Workloads
# Schedule batch jobs on spot nodes
apiVersion: batch/v1
kind: Job
metadata:
  name: data-processing
spec:
  template:
    spec:
      # Target spot node pool
      nodeSelector:
        cloud.google.com/gke-spot: "true"
      tolerations:
      - key: "cloud.google.com/gke-spot"
        operator: Equal
        value: "true"
        effect: NoSchedule
      # Handle spot preemption gracefully
      terminationGracePeriodSeconds: 25   # spot gives 30s warning
      restartPolicy: OnFailure
      containers:
      - name: processor
        image: gcr.io/myproject/processor:latest
Committed Use Discounts
# Purchase committed use for baseline workloads
gcloud compute commitments create prod-commitment \
  --plan 1-year \
  --region us-central1 \
  --resources vcpu=20,memory=80GB
# Savings: ~37% for 1-year, ~55% for 3-year
Node Auto-Provisioning with Resource Limits
# Set cluster-level resource limits for NAP
gcloud container clusters update prod-cluster \
  --region us-central1 \
  --enable-autoprovisioning \
  --max-cpu 100 \
  --max-memory 400 \
  --min-cpu 4 \
  --min-memory 16 \
  --autoprovisioning-scopes=https://www.googleapis.com/auth/cloud-platform

8. Observability Best Practices

Google Cloud Managed Prometheus

# Enable managed Prometheus (built into GKE)
gcloud container clusters update prod-cluster \
  --region us-central1 \
  --enable-managed-prometheus
# Deploy PodMonitoring to scrape your apps
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: api-monitoring
  namespace: production
spec:
  selector:
    matchLabels:
      app: api
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
Structured Logging
# Always log in JSON format for Cloud Logging
import json
import logging
class JsonFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "severity":  record.levelname,
            "message":   record.getMessage(),
            "timestamp": self.formatTime(record),
            "component": record.name,
            "httpRequest": getattr(record, "httpRequest", None),
            "labels": {
                "service": "api",
                "version": "v2",
                "env":     "production"
            }
        })

Cloud Trace Integration

# Auto-instrument with OpenTelemetry
from opentelemetry import trace
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(CloudTraceSpanExporter())
)
trace.set_tracer_provider(provider)

9. CI/CD Best Practices

Cloud Build + Artifact Registry
# cloudbuild.yaml
steps:
# Build image
- name: gcr.io/cloud-builders/docker
  args:
  - build
  - -t
  - us-central1-docker.pkg.dev/$PROJECT_ID/prod/api:$SHORT_SHA
  - -t
  - us-central1-docker.pkg.dev/$PROJECT_ID/prod/api:latest
  - .
# Scan for vulnerabilities
- name: gcr.io/cloud-builders/gcloud
  args:
  - artifacts
  - docker
  - images
  - scan
  - us-central1-docker.pkg.dev/$PROJECT_ID/prod/api:$SHORT_SHA
  - --format=json
# Push to Artifact Registry
- name: gcr.io/cloud-builders/docker
  args:
  - push
  - --all-tags
  - us-central1-docker.pkg.dev/$PROJECT_ID/prod/api
# Deploy to GKE
- name: gcr.io/cloud-builders/kubectl
  args:
  - set
  - image
  - deployment/api
  - api=us-central1-docker.pkg.dev/$PROJECT_ID/prod/api:$SHORT_SHA
  - -n
  - production
  env:
  - CLOUDSDK_COMPUTE_REGION=us-central1
  - CLOUDSDK_CONTAINER_CLUSTER=prod-cluster
options:
  machineType: E2_HIGHCPU_8
  logging: CLOUD_LOGGING_ONLY

10. GKE Best Practices Checklist

Cluster Setup
  ✅ Regional cluster (not zonal)
  ✅ Private cluster (no public node IPs)
  ✅ Separate node pools by workload type
  ✅ Release channel enabled (auto-updates)
  ✅ Maintenance window set
  ✅ VPC-native networking
Security
  ✅ Workload Identity (no SA keys)
  ✅ Binary Authorization
  ✅ Pod Security Standards (restricted)
  ✅ Network Policies (default deny)
  ✅ Secrets in Secret Manager
  ✅ Shielded nodes enabled
  ✅ Container image scanning
  ✅ Cloud Armor WAF on ingress
Resource Management
  ✅ Requests and limits on every container
  ✅ LimitRange per namespace
  ✅ ResourceQuota per namespace
  ✅ Priority classes defined
  ✅ PodDisruptionBudgets set
Reliability
  ✅ Minimum 3 replicas for prod services
  ✅ Pod anti-affinity across zones
  ✅ HPA configured
  ✅ Liveness + readiness + startup probes
  ✅ Graceful shutdown (preStop + terminationGrace)
  ✅ PodDisruptionBudget (minAvailable ≥ 1)
Cost
  ✅ Spot VMs for batch/non-critical
  ✅ Committed use discounts for baseline
  ✅ Cluster autoscaler enabled
  ✅ VPA recommendations reviewed
  ✅ Node auto-provisioning for mixed workloads
Observability
  ✅ Managed Prometheus enabled
  ✅ Cloud Logging with structured JSON
  ✅ Cloud Trace instrumented
  ✅ Dashboards for golden signals
  ✅ Alerts on SLO breaches

GKE best practices come down to three pillars — security by default (private cluster, Workload Identity, least privilege), reliability by design (regional cluster, anti-affinity, autoscaling, probes), and cost efficiency (spot VMs, committed use, right-sizing with VPA). Get these right from day one and you avoid the most painful production incidents.

Top OpenShift Interview Questions for Beginners

techhadoop kubernetes

OpenShift (OCP) Interview Questions

Beginner Level

Q1. What is OpenShift and how does it differ from Kubernetes?

A: OpenShift is Red Hat’s enterprise Kubernetes platform — Kubernetes is the engine, OpenShift is the car built around it.

Kubernetes:                        OpenShift:
─────────────────────────          ─────────────────────────
Container orchestration            Kubernetes + enterprise layer
You bring your own:                Built-in:
  - CI/CD                            - CI/CD (Tekton Pipelines)
  - Image registry                   - Internal registry
  - Ingress controller               - HAProxy router
  - Auth                             - OAuth server
  - Monitoring                       - Prometheus + Grafana
  - Developer tools                  - Developer console
  - Security policies                - SCCs (stricter than PSP)

Key differences:

FeatureKubernetesOpenShift
SecurityPod Security AdmissionSecurity Context Constraints (SCC)
RoutingIngress (install separately)Routes (built-in HAProxy)
RegistryExternalBuilt-in image registry
CI/CDExternalTekton + ArgoCD built-in
AuthExternal OIDCBuilt-in OAuth + Entra ID / LDAP
CLIkubectloc (superset of kubectl)
ProjectsNamespacesProjects (namespaces + annotations)
ConsoleBasic dashboardRich developer + admin console

Q2. What is a Project in OpenShift vs a Namespace in Kubernetes?

A: A Project is OpenShift’s wrapper around a Kubernetes Namespace — it adds metadata, annotations, and access control.

# When you create a Project:
oc new-project my-app \
  --display-name="My Application" \
  --description="Production app for team A"
# OCP automatically creates:
# 1. Namespace: my-app
# 2. RoleBinding: admin role for creator
# 3. NetworkPolicy: default isolation
# 4. LimitRange: default resource limits
# 5. ResourceQuota: (if configured by admin)

Key differences:

NamespaceProject
Creationkubectl create namespaceoc new-project
Access controlManual RBACAuto-assigns creator as admin
AnnotationsManualDisplay name, description built-in
TemplatesNoneProject templates supported
Self-serviceAdmin onlyCan be enabled for developers

Q3. What is a Security Context Constraint (SCC) in OpenShift?

A: SCC is OpenShift’s mechanism to control what a pod is allowed to do at the OS level — more powerful than Kubernetes Pod Security Admission.

SCC controls:
├── Which user IDs a pod can run as
├── Which Linux capabilities it can use
├── Whether it can run as root
├── Whether it can mount host paths
├── Which SELinux labels it can use
└── Whether it can use privileged mode

Built-in SCCs (ordered most to least restrictive):

SCCWhat it allows
restricted-v2Default — no root, random UID, no host access
restrictedLegacy default
baselineSome relaxed restrictions
nonrootAny non-root UID
nonroot-v2Updated nonroot
hostmount-anyuidCan mount host paths
hostnetworkCan use host network
hostnetwork-v2Updated hostnetwork
privilegedUnrestricted — cluster admin only
anyuidAny UID including root
# Check which SCC a pod is using
oc get pod api-pod -o jsonpath='{.metadata.annotations.openshift\.io/scc}'
# Check what SCC a service account can use
oc adm policy who-can use scc restricted
# Add SCC to service account
oc adm policy add-scc-to-user anyuid \
  -z my-service-account \
  -n my-namespace
# View all SCCs
oc get scc
# Describe an SCC
oc describe scc restricted-v2

Q4. What is the difference between oc and kubectl?

A: oc is a superset of kubectl — every kubectl command works with oc, plus OCP-specific commands.

# Everything kubectl does:
oc get pods
oc apply -f deployment.yaml
oc describe node worker-1
# OCP-specific additions:
oc new-project myapp              # create project
oc new-app --image=nginx          # deploy from image
oc expose service api             # create Route
oc rollout latest dc/api          # rollout DeploymentConfig
oc adm policy add-scc-to-user    # manage SCCs
oc adm top nodes                  # node resource usage
oc adm must-gather                # collect diagnostics
oc login https://api.cluster.com  # authenticate to cluster
oc whoami                         # current user
oc projects                       # list projects
oc status                         # project overview
oc debug node/worker-1            # debug a node
oc rsh pod/api-pod                # remote shell into pod
oc cp file.txt api-pod:/tmp/      # copy files to pod
oc port-forward pod/api 8080:8080 # port forward

Q5. What is a DeploymentConfig vs a Deployment in OpenShift?

A: DeploymentConfig (DC) is OpenShift’s original deployment resource — predates Kubernetes Deployment. OCP 4.x supports both but Deployment is now recommended.

DeploymentConfigDeployment
OriginOpenShift nativeKubernetes native
TriggersImage change, config changeManual or external
Lifecycle hooksPre/mid/post hooksInit containers
Rolling strategyCustom strategiesRollingUpdate / Recreate
RecommendedLegacy — avoid for new apps✅ Use this
APIapps.openshift.io/v1apps/v1
# OLD way — DeploymentConfig (avoid for new apps)
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  name: api
spec:
  replicas: 3
  triggers:
  - type: ImageChange          # ← OCP-specific trigger
    imageChangeParams:
      automatic: true
      containerNames:
      - api
      from:
        kind: ImageStreamTag
        name: api:latest
  - type: ConfigChange
  template:
    spec:
      containers:
      - name: api
        image: api:latest
---
# NEW way — Deployment (recommended)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: image-registry.openshift-image-registry.svc:5000/production/api:latest

Q6. What is an ImageStream in OpenShift?

A: An ImageStream is an OpenShift abstraction that tracks container images and their tags — like a pointer to images that can trigger deployments automatically.

External Registry          ImageStream              Deployment
─────────────────          ──────────────           ──────────
quay.io/myapp:1.0  ───▶   myapp:1.0  ─────────▶   Pod runs
quay.io/myapp:1.1  ───▶   myapp:1.1  ─────────▶   Auto-redeploy!
quay.io/myapp:latest───▶   myapp:latest
# ImageStream definition
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  name: api
  namespace: production
spec:
  lookupPolicy:
    local: true              # allow pods to reference by ImageStream name
---
# ImageStreamTag — points to specific image
apiVersion: image.openshift.io/v1
kind: ImageStreamTag
metadata:
  name: api:latest
  namespace: production
tag:
  from:
    kind: DockerImage
    name: quay.io/mycompany/api:latest
  importPolicy:
    scheduled: true          # periodically re-import
    importMode: Legacy
# Import image into ImageStream
oc import-image api:latest \
  --from=quay.io/mycompany/api:latest \
  --confirm \
  -n production
# List ImageStreams
oc get imagestreams -n production
# View tags
oc get imagestreamtag -n production
# Check image digest
oc describe imagestreamtag api:latest -n production

Q7. What is the OpenShift internal image registry?

A: OCP ships with a built-in container image registry running inside the cluster — no external registry needed.

Registry endpoint:
image-registry.openshift-image-registry.svc:5000   (internal)
default-route-openshift-image-registry.apps.cluster.com  (external)
Push/pull flow:
Developer → oc build / Tekton → Internal Registry → Deployment
# Expose registry externally (if needed)
oc patch configs.imageregistry.operator.openshift.io/cluster \
  --type=merge \
  -p '{"spec":{"defaultRoute":true}}'
# Login to internal registry
oc registry login
# Push image to internal registry
podman push myimage:latest \
  image-registry.openshift-image-registry.svc:5000/myproject/myimage:latest
# Configure registry storage (production — use S3 / Azure Blob)
oc edit configs.imageregistry.operator.openshift.io cluster

Intermediate Level

Q8. How does OpenShift handle multi-tenancy and namespace isolation?

A: OCP uses multiple layers for multi-tenancy:

1. Projects + RBAC

# Each team gets their own project
# RBAC controls who can do what
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-a-admin
  namespace: team-a
subjects:
- kind: Group
  name: team-a-developers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io

2. NetworkPolicy / OVN-Kubernetes

# Default deny all — then allow specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: team-a
spec:
  podSelector: {}         # all pods
  policyTypes:
  - Ingress
  - Egress
---
# Allow only within namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: team-a
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}    # only from same namespace

3. ResourceQuota per Project

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    pods: "20"
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    persistentvolumeclaims: "10"
    services.loadbalancers: "0"   # no LB services — use Routes

4. LimitRange — default resource limits

apiVersion: v1
kind: LimitRange
metadata:
  name: team-a-limits
  namespace: team-a
spec:
  limits:
  - type: Container
    default:
      cpu: 500m
      memory: 256Mi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    max:
      cpu: "2"
      memory: 2Gi

Q9. Explain OpenShift OAuth and authentication mechanisms.

A: OCP has a built-in OAuth server that acts as an identity broker:

User/CLI
   │
   ▼
OCP OAuth Server (oauth-openshift.apps.cluster.com)
   │
   ├──▶ HTPasswd (local users — dev/test)
   ├──▶ LDAP / Active Directory
   ├──▶ OpenID Connect (Azure AD / Okta / Google)
   ├──▶ GitHub / GitLab

OpenShift Routes vs Kubernetes Ingress: A Comprehensive Guide

techhadoop kubernetes, OCP

Ingress in OpenShift (OCP)

OpenShift vs Kubernetes Ingress

OpenShift takes a different approach to ingress than vanilla Kubernetes. It has its own native routing layer that predates Kubernetes Ingress and is more powerful out of the box.

Kubernetes:                         OpenShift:
─────────────────────────           ─────────────────────────
Ingress resource                    Route resource (native OCP)
  + Ingress Controller                + HAProxy Router (built-in)
  (you install separately)            (pre-installed, managed)
Also supports:                      Also supports:
  Gateway API                         Kubernetes Ingress
                                       Gateway API
                                       Ingress Operator

OCP Routing Architecture

┌──────────────────────────────────────────────────────────────┐
│                      OPENSHIFT CLUSTER                        │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │                  INGRESS OPERATOR                       │  │
│  │   Manages and configures IngressController resources   │  │
│  └───────────────────────┬────────────────────────────────┘  │
│                          │ manages                            │
│  ┌───────────────────────▼────────────────────────────────┐  │
│  │              INGRESSCONTROLLER (HAProxy)                │  │
│  │   - Default router in openshift-ingress namespace      │  │
│  │   - Watches Route + Ingress resources                  │  │
│  │   - Handles TLS termination                            │  │
│  │   - Wildcard DNS (*.apps.cluster.domain.com)           │  │
│  └───────┬───────────────┬───────────────┬────────────────┘  │
│          │               │               │                    │
│          ▼               ▼               ▼                    │
│       Route A         Route B        Ingress C               │
│          │               │               │                    │
│       Service A       Service B       Service C              │
│          │               │               │                    │
│        Pods            Pods            Pods                   │
└──────────────────────────────────────────────────────────────┘
          ▲
  Wildcard DNS
  *.apps.cluster.acme.com

OCP Route — The Native Way

The Route is OpenShift’s own resource — more feature-rich than Kubernetes Ingress and available before Ingress existed.

Basic Route

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: api-route
  namespace: production
spec:
  host: api.apps.cluster.acme.com   # ← auto-generated if omitted
  to:
    kind: Service
    name: api-service
    weight: 100
  port:
    targetPort: 8080
  wildcardPolicy: None

Auto-generated hostname

# If you omit spec.host, OCP generates:
# <route-name>-<namespace>.apps.<cluster-domain>
# Example: api-route-production.apps.cluster.acme.com

Route TLS Modes

OCP Routes have three TLS termination modes — more flexible than Kubernetes Ingress:

Edge Termination:
Client ──HTTPS──▶ Router (terminates TLS) ──HTTP──▶ Pod
                  (cert lives on router)
Passthrough:
Client ──HTTPS──▶ Router (passes through) ──HTTPS──▶ Pod
                  (cert lives on pod, router can't inspect)
Re-encrypt:
Client ──HTTPS──▶ Router (terminates TLS) ──HTTPS──▶ Pod
                  (two separate TLS sessions)

Edge TLS (Most Common)

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: api-edge-tls
  namespace: production
spec:
  host: api.apps.cluster.acme.com
  to:
    kind: Service
    name: api-service
  port:
    targetPort: 8080
  tls:
    termination: edge              # ← TLS at router
    insecureEdgeTerminationPolicy: Redirect   # HTTP → HTTPS
    certificate: |                 # ← custom cert (optional)
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
    key: |
      -----BEGIN PRIVATE KEY-----
      ...
      -----END PRIVATE KEY-----
    caCertificate: |               # ← optional CA cert
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

Passthrough TLS

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: api-passthrough
  namespace: production
spec:
  host: api.apps.cluster.acme.com
  to:
    kind: Service
    name: api-service
  port:
    targetPort: 8443               # ← must be HTTPS port
  tls:
    termination: passthrough       # ← router passes raw TLS

Re-encrypt TLS

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: api-reencrypt
  namespace: production
spec:
  host: api.apps.cluster.acme.com
  to:
    kind: Service
    name: api-service
  port:
    targetPort: 8443
  tls:
    termination: reencrypt         # ← two TLS hops
    destinationCACertificate: |    # ← verify backend cert
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
    insecureEdgeTerminationPolicy: Redirect

Route Traffic Splitting (Blue/Green & Canary)

OCP Routes natively support weighted traffic splitting — no annotations needed:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: api-canary
  namespace: production
spec:
  host: api.apps.cluster.acme.com
  to:
    kind: Service
    name: api-v1              # primary
    weight: 90                # ← 90% traffic
  alternateBackends:
  - kind: Service
    name: api-v2              # canary
    weight: 10                # ← 10% traffic
  port:
    targetPort: 8080

Progressive canary:

# Shift traffic gradually using oc CLI
oc set route-backends api-canary \
  api-v1=80 api-v2=20
oc set route-backends api-canary \
  api-v1=50 api-v2=50
oc set route-backends api-canary \
  api-v1=0 api-v2=100
# Full cutover — remove alternate backend
oc patch route api-canary \
  --type=json \
  -p='[{"op":"remove","path":"/spec/alternateBackends"}]'

Route Annotations (HAProxy Tuning)

metadata:
  annotations:
    # Timeouts
    haproxy.router.openshift.io/timeout: 60s
    haproxy.router.openshift.io/timeout-tunnel: 1h
    # Load balancing algorithm
    haproxy.router.openshift.io/balance: leastconn
    # Options: roundrobin | leastconn | source | random
    # Rate limiting
    haproxy.router.openshift.io/rate-limit-connections: "true"
    haproxy.router.openshift.io/rate-limit-connections.concurrent-tcp: "100"
    haproxy.router.openshift.io/rate-limit-connections.rate-tcp: "100"
    # Sticky sessions
    haproxy.router.openshift.io/disable_cookies: "false"
    haproxy.router.openshift.io/cookie-name: "ROUTE_SESSION"
    # IP whitelisting
    haproxy.router.openshift.io/ip_whitelist: "10.0.0.0/8 192.168.1.0/24"
    # Response headers
    haproxy.router.openshift.io/hsts_header: >
      max-age=31536000;includeSubDomains;preload
    # Custom headers to backend
    haproxy.router.openshift.io/set-forwarded-headers: append
    # WebSocket support
    haproxy.router.openshift.io/timeout-tunnel: 1h

IngressController Resource

OCP manages HAProxy routers via IngressController CRDs — the Ingress Operator watches these:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  # Number of router pods
  replicas: 3
  # Which domain this controller handles
  domain: apps.cluster.acme.com
  # Where to expose (LoadBalancer, NodePort, HostNetwork)
  endpointPublishingStrategy:
    type: LoadBalancerService
    loadBalancer:
      scope: External              # External or Internal
  # Which routes this controller handles
  routeSelector:
    matchLabels:
      router: default              # only routes with this label
  # Which namespaces
  namespaceSelector:
    matchLabels:
      network: public
  # TLS security profile
  tlsSecurityProfile:
    type: Custom
    custom:
      ciphers:
      - ECDHE-RSA-AES256-GCM-SHA384
      - ECDHE-RSA-AES128-GCM-SHA256
      minTLSVersion: VersionTLS12
  # Node placement for router pods
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/infra: ""
    tolerations:
    - key: node-role.kubernetes.io/infra
      effect: NoSchedule

Multiple IngressControllers (Multi-tenant)

Run separate routers for different teams or traffic types:

┌─────────────────────────────────────────────────┐
│               OPENSHIFT CLUSTER                  │
│                                                  │
│  ┌──────────────────┐  ┌──────────────────────┐  │
│  │  IngressController│  │  IngressController   │  │
│  │  (default)        │  │  (internal)          │  │
│  │                   │  │                      │  │
│  │  *.apps.acme.com  │  │  *.internal.acme.com │  │
│  │  External LB      │  │  Internal LB only    │  │
│  │  Public routes    │  │  Private routes      │  │
│  └──────────────────┘  └──────────────────────┘  │
└─────────────────────────────────────────────────┘
# Internal router — not exposed externally
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: internal
  namespace: openshift-ingress-operator
spec:
  domain: internal.apps.cluster.acme.com
  replicas: 2
  endpointPublishingStrategy:
    type: LoadBalancerService
    loadBalancer:
      scope: Internal              # ← internal LB only
  routeSelector:
    matchLabels:
      router: internal             # ← only internal routes
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/infra: ""
---
# Route that uses internal router
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: admin-internal
  labels:
    router: internal               # ← picked up by internal router
spec:
  host: admin.internal.apps.cluster.acme.com
  to:
    kind: Service
    name: admin-service

Kubernetes Ingress in OCP

OCP also fully supports standard Kubernetes Ingress resources — the Ingress Operator converts them to Routes automatically:

# Standard Kubernetes Ingress works in OCP
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    # OCP-specific annotation to select router
    route.openshift.io/termination: "edge"
spec:
  ingressClassName: openshift-default
  rules:
  - host: api.apps.cluster.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
  tls:
  - hosts:
    - api.apps.cluster.acme.com
    secretName: api-tls-secret

What happens under the hood:

kubectl apply Ingress
      ↓
Ingress Operator watches it
      ↓
Auto-creates equivalent Route
      ↓
HAProxy Router picks up Route
      ↓
Traffic flows

cert-manager with OCP Routes

# cert-manager supports OCP Routes natively
# Install the OCP route plugin
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-cert
  namespace: production
spec:
  secretName: api-tls-secret
  dnsNames:
  - api.apps.cluster.acme.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
---
# Route uses the cert secret
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: api-route
  annotations:
    cert-manager.io/issuer-name: letsencrypt-prod
    cert-manager.io/issuer-kind: ClusterIssuer
spec:
  host: api.apps.cluster.acme.com
  tls:
    termination: edge
    # cert-manager injects cert automatically
  to:
    kind: Service
    name: api-service

Gateway API in OCP

OCP 4.12+ supports Gateway API — recommended for new deployments:

# Enable Gateway API in OCP
oc apply -f https://github.com/kubernetes-sigs/gateway-api/releases/
         download/v1.1.0/standard-install.yaml
# Install supported controller (e.g. NGINX Gateway Fabric)
helm install ngf oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric \
  --namespace nginx-gateway \
  --create-namespace
# GatewayClass
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: gateway.nginx.org/nginx-gateway-controller
---
# Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main-gateway
  namespace: openshift-ingress
spec:
  gatewayClassName: nginx
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-tls
---
# HTTPRoute — same as vanilla Kubernetes
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
  - name: main-gateway
    namespace: openshift-ingress
  hostnames:
  - api.apps.cluster.acme.com
  rules:
  - backendRefs:
    - name: api-service
      port: 8080

OCP Ingress Security

Network Policies with Routes

# Allow only router pods to reach your service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-router
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-service
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          network.openshift.io/policy-group: ingress

Restrict Route Creation (RBAC)

# Only allow developers to create Routes in their namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: route-creator
  namespace: team-a
rules:
- apiGroups:
  - route.openshift.io
  resources:
  - routes
  verbs:
  - get
  - list
  - create
  - update
  - patch
---
# Restrict to specific hostnames via admission webhook
# or use OCP's built-in route admission policy
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
spec:
  routeAdmission:
    namespaceOwnership: Strict   # ← namespace must own the hostname
    wildcardPolicy: WildcardsDisallowed

Quick Reference — OCP CLI Commands

# Create a route quickly
oc expose service api-service \
  --hostname=api.apps.cluster.acme.com \
  --port=8080
# Create TLS edge route
oc create route edge api-tls \
  --service=api-service \
  --hostname=api.apps.cluster.acme.com \
  --cert=api.crt \
  --key=api.key
# Create passthrough route
oc create route passthrough api-pass \
  --service=api-service \
  --hostname=api.apps.cluster.acme.com
# View all routes
oc get routes -A
# Describe route
oc describe route api-route -n production
# Check router pods
oc get pods -n openshift-ingress
# View router logs
oc logs -n openshift-ingress \
  deployment/router-default -f
# Check IngressController status
oc get ingresscontroller -n openshift-ingress-operator
oc describe ingresscontroller default \
  -n openshift-ingress-operator
# Scale router
oc patch ingresscontroller default \
  -n openshift-ingress-operator \
  --type=merge \
  -p '{"spec":{"replicas":3}}'

Route vs Ingress vs Gateway API in OCP

OCP RouteK8s IngressGateway API
Native to OCP✅⚠️ Converted to Route❌ Needs controller
TLS modesEdge/Passthrough/ReencryptEdge onlyAll modes
Traffic splitting✅ Native weights❌ Annotations✅ Native
HAProxy tuning✅ Annotations⚠️ Limited⚠️ Controller-specific
Role separation❌❌✅
TCP/gRPC❌❌✅
PortabilityOCP only✅✅
Recommended forOCP-native workloadsMigration from K8sNew multi-team setups

Decision Guide for OCP

New OCP cluster, single team?
  └──▶ Use OCP Routes — simplest, most native
Migrating from vanilla Kubernetes?
  └──▶ Use Kubernetes Ingress — auto-converted to Routes
       migrate to Routes over time
Multi-team cluster, need role separation?
  └──▶ Use Gateway API with supported controller
Need passthrough TLS (app owns cert)?
  └──▶ Use OCP Route with termination: passthrough
Need internal-only routing?
  └──▶ Create separate IngressController with Internal LB
       label Routes with router: internal
Need canary / blue-green?
  └──▶ Use OCP Route alternateBackends (native)
       or Argo Rollouts + Route integration

OCP Routes are the most integrated and operationally simple option for OpenShift — they work out of the box with HAProxy, support all TLS modes, and have native traffic splitting. Use Kubernetes Ingress for portability and Gateway API when you need multi-team role separation or advanced L7 features.

Top Kubernetes Ingress Interview Questions

techhadoop kubernetes

Kubernetes Ingress — Interview Questions & Answers

Section 1 — Fundamentals

Q1. What is Kubernetes Ingress and why do we need it?

Ingress is a Kubernetes API object that manages external HTTP/HTTPS access to services inside a cluster. It provides a single entry point that can route traffic to multiple services based on host, path, or headers — without needing a separate LoadBalancer service per application.

Without Ingress:
  Service A → LoadBalancer → public IP 20.x.x.1  ($$$)
  Service B → LoadBalancer → public IP 20.x.x.2  ($$$)
  Service C → LoadBalancer → public IP 20.x.x.3  ($$$)
With Ingress:
  Single LoadBalancer → public IP 20.x.x.1
    /api    → Service A
    /app    → Service B
    /admin  → Service C
  One IP, one LB, one TLS certificate

Q2. What is the difference between Ingress and a Service of type LoadBalancer?

LoadBalancer ServiceIngress
LayerL4 (TCP/UDP)L7 (HTTP/HTTPS)
RoutingIP + port onlyhost, path, headers
TLS termination❌✅
CostOne LB per serviceOne LB for all services
Path-based routing❌✅
Host-based routing❌✅

Q3. What are the components of an Ingress resource?

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  namespace: production
  annotations:                            # controller-specific config
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx                 # which controller handles this
  tls:                                    # TLS configuration
    - hosts:
        - app.contoso.com
      secretName: tls-secret
  rules:                                  # routing rules
    - host: app.contoso.com              # host-based routing
      http:
        paths:
          - path: /api                   # path-based routing
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 8080
          - path: /                      # default backend
            pathType: Prefix
            backend:
              service:
                name: frontend-service
                port:
                  number: 80

Q4. What is an Ingress Controller? Is it included in Kubernetes by default?

An Ingress Controller is the implementation — it reads Ingress resources and configures an actual reverse proxy or load balancer accordingly. Kubernetes defines the Ingress API but ships with no controller by default — you must install one.

Popular controllers:

ControllerBest for
NGINX Ingress ControllerGeneral purpose — most widely used
TraefikDynamic config, microservices
HAProxyHigh performance, enterprise
AWS ALB Ingress ControllerEKS on AWS
Azure Application Gateway Ingress (AGIC)AKS on Azure
GCE Ingress ControllerGKE on GCP
Istio GatewayService mesh environments

Q5. What are the three pathType values and how do they differ?

# Exact — must match exactly
- path: /api/v1
  pathType: Exact
# Matches: /api/v1
# Does NOT match: /api/v1/, /api/v1/users
# Prefix — matches path prefix split by /
- path: /api
  pathType: Prefix
# Matches: /api, /api/, /api/v1, /api/v1/users
# Does NOT match: /apiv1, /apiusers
# ImplementationSpecific — controller decides
- path: /api/*
  pathType: ImplementationSpecific
# Behaviour depends on your ingress controller
# NGINX: regex support
# Traefik: glob support

Q6. What is the difference between host-based and path-based routing?

# Host-based routing — different subdomains → different services
rules:
  - host: api.contoso.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: api-service
              port:
                number: 80
  - host: app.contoso.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: frontend-service
              port:
                number: 80
# Path-based routing — same host, different paths → different services
rules:
  - host: contoso.com
    http:
      paths:
        - path: /api
          pathType: Prefix
          backend:
            service:
              name: api-service
              port:
                number: 80
        - path: /static
          pathType: Prefix
          backend:
            service:
              name: cdn-service
              port:
                number: 80
        - path: /
          pathType: Prefix
          backend:
            service:
              name: frontend-service
              port:
                number: 80

Q7. What is a default backend?

The default backend handles requests that match no rule in any Ingress. It returns a 404 or custom error page:

spec:
  defaultBackend:
    service:
      name: default-404-service
      port:
        number: 80
  rules:
    - host: app.contoso.com
      ...
# Any request not matching app.contoso.com → default-404-service

Section 2 — Ingress Controllers

Q8. How does NGINX Ingress Controller work internally?

1. You apply an Ingress resource to the cluster
2. NGINX Ingress Controller watches Ingress objects via
   Kubernetes API server watch stream
3. Controller translates Ingress rules into nginx.conf:
   server {
     server_name api.contoso.com;
     location /v1 {
       proxy_pass http://api-service.production.svc.cluster.local:8080;
     }
   }
4. Controller hot-reloads NGINX with new config
   (without dropping existing connections)
5. Traffic arrives at NGINX pod → routes to upstream service

Q9. What is IngressClass and why was it introduced?

IngressClass was introduced in Kubernetes 1.18 to allow multiple ingress controllers in the same cluster, each handling different Ingress resources:

# Define an IngressClass
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-internal
  annotations:
    ingressclass.kubernetes.io/is-default-class: "false"
spec:
  controller: k8s.io/ingress-nginx
---
# Reference it in an Ingress
spec:
  ingressClassName: nginx-internal  # only nginx-internal controller handles this

Common multi-controller pattern:

nginx-external → handles public internet traffic
nginx-internal → handles internal VPN-only traffic
agic           → handles Azure Application Gateway traffic

Q10. How do annotations work in Ingress and give examples?

Annotations configure controller-specific behaviour that the Ingress spec itself doesn’t support:

metadata:
  annotations:
    # NGINX — rewrite URL before forwarding
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    # NGINX — rate limiting
    nginx.ingress.kubernetes.io/limit-rps: "10"
    # NGINX — enable CORS
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://contoso.com"
    # NGINX — client body size limit
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    # NGINX — connection timeout
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    # NGINX — sticky sessions
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    # NGINX — whitelist specific IPs
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
    # cert-manager — auto issue TLS certificate
    cert-manager.io/cluster-issuer: "letsencrypt-prod"

Q11. What is the difference between NGINX Ingress Controller and AGIC (Azure Application Gateway Ingress Controller)?

NGINX IngressAGIC
Runs asPod inside clusterAzure resource outside cluster
Load balancerService type LB in frontAzure Application Gateway
WAFManual configAzure WAF built-in
TLScert-manager or manualAzure-managed certs
Health probesInternalAzure LB health probes
AutoscalingHPA on NGINX podsApp Gateway autoscales natively
Best forAny clusterAKS on Azure
ARO support✅Limited

Section 3 — TLS and Security

Q12. How do you configure TLS termination in Ingress?

# Step 1: Create TLS secret
kubectl create secret tls tls-contoso \
  --cert=tls.crt \
  --key=tls.key \
  -n production
# Step 2: Reference in Ingress
spec:
  tls:
    - hosts:
        - app.contoso.com
        - api.contoso.com
      secretName: tls-contoso   # must be in same namespace as Ingress
  rules:
    - host: app.contoso.com
      ...

TLS is terminated at the Ingress controller — traffic between the controller and backend pods travels unencrypted inside the cluster by default (unless you enable backend SSL).

Q13. How does cert-manager automate TLS certificate issuance?

# Install cert-manager, then create a ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@contoso.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: nginx   # cert-manager creates a temporary Ingress for ACME challenge
---
# Reference in Ingress — cert-manager auto-issues and renews
metadata:
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
    - hosts:
        - app.contoso.com
      secretName: tls-app-contoso   # cert-manager creates this Secret automatically

cert-manager watches the Ingress → sees the annotation → calls Let’s Encrypt ACME API → completes HTTP-01 challenge → stores cert in the named Secret → auto-renews before expiry.

Q14. How do you enforce HTTPS redirect in Ingress?

metadata:
  annotations:
    # Force HTTP → HTTPS redirect (301)
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    # Add HSTS header
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

Q15. How do you implement authentication at the Ingress level?

# Basic auth
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth-secret
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
# External OAuth2 (with oauth2-proxy)
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "http://oauth2-proxy.auth.svc.cluster.local/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://auth.contoso.com/oauth2/start"
# mTLS (client certificate authentication)
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-secret: "production/ca-secret"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"

Section 4 — Advanced Routing

Q16. How do you implement canary deployments with Ingress?

# Production Ingress (main traffic)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-production
spec:
  rules:
    - host: app.contoso.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-v1
                port:
                  number: 80
---
# Canary Ingress (10% of traffic)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"   # 10% to v2
spec:
  rules:
    - host: app.contoso.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-v2
                port:
                  number: 80

Other canary strategies:

# Route by header — specific users get canary
nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "true"
# Route by cookie — sticky canary for logged-in users
nginx.ingress.kubernetes.io/canary-by-cookie: "canary_user"

Q17. How do you configure rate limiting in Ingress?

metadata:
  annotations:
    # Limit requests per second per IP
    nginx.ingress.kubernetes.io/limit-rps: "10"
    # Limit connections per IP
    nginx.ingress.kubernetes.io/limit-connections: "5"
    # Limit requests per minute
    nginx.ingress.kubernetes.io/limit-rpm: "100"
    # Whitelist IPs from rate limiting
    nginx.ingress.kubernetes.io/limit-whitelist: "10.0.0.0/8,172.16.0.0/12"
    # Return 429 when rate limit exceeded (default is 503)
    nginx.ingress.kubernetes.io/limit-req-status-code: "429"

Q18. How does rewrite-target work and what is a common gotcha?

# Without capture group — rewrites entire path
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: contoso.com
      http:
        paths:
          - path: /api
            # Request: contoso.com/api/users
            # Forwarded as: backend/         ← loses /users
# With capture group — preserves remainder of path
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: contoso.com
      http:
        paths:
          - path: /api(/|$)(.*)             # capture group $2
            pathType: ImplementationSpecific
            # Request: contoso.com/api/users
            # Forwarded as: backend/users   ✅

Q19. What is the difference between Ingress and Gateway API?

Gateway API is the next generation of Kubernetes ingress — more expressive, role-oriented, and extensible:

FeatureIngressGateway API
API stabilityStable (v1)Stable (v1 for core)
Role separationSingle resourceGatewayClass · Gateway · HTTPRoute
TCP/UDP routing❌✅ TCPRoute · UDPRoute
Header manipulationAnnotation onlyNative spec
Traffic weightingAnnotation onlyNative spec
Multi-tenantDifficultDesigned for it
TLS passthroughAnnotationNative
Future directionMaintenance modeActive development
# Gateway API example
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-route
spec:
  parentRefs:
    - name: prod-gateway
  hostnames:
    - app.contoso.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      backendRefs:
        - name: api-service
          port: 8080
          weight: 90
        - name: api-service-v2
          port: 8080
          weight: 10     # native canary without annotations

Section 5 — Troubleshooting

Q20. A service is returning 404 from Ingress. How do you debug it?

# Step 1: Verify Ingress resource is created and has an address
kubectl get ingress -n production
# NAME         CLASS   HOSTS             ADDRESS        PORTS   AGE
# my-ingress   nginx   app.contoso.com   20.x.x.x       80,443  5m
# If ADDRESS is empty → controller not reconciling → check controller pods
# Step 2: Describe Ingress — look for events
kubectl describe ingress my-ingress -n production
# Events show: backend service not found, port mismatch, etc.
# Step 3: Verify backend service exists and has endpoints
kubectl get svc api-service -n production
kubectl get endpoints api-service -n production
# If endpoints are empty → no pods matching service selector
# Step 4: Check service selector matches pod labels
kubectl get pods -n production --show-labels
kubectl get svc api-service -n production -o yaml | grep selector
# Step 5: Check Ingress controller logs
kubectl logs -n ingress-nginx \
  -l app.kubernetes.io/name=ingress-nginx \
  --tail=100
# Step 6: Verify IngressClass matches controller
kubectl get ingressclass
kubectl get ingress my-ingress -o yaml | grep ingressClassName

Q21. Ingress is returning 502 Bad Gateway. What are the causes?

# 502 means Ingress reached the backend but got an error
# Cause 1: Backend pod is crashing
kubectl get pods -n production
kubectl logs <pod-name> -n production --previous
# Cause 2: Port mismatch — Ingress port ≠ container port
kubectl get ingress -o yaml  # check backend port
kubectl get svc api-service -o yaml  # check targetPort
kubectl get pods -o yaml | grep containerPort
# Cause 3: Backend requires HTTPS but Ingress sending HTTP
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"  # add this
# Cause 4: Upstream timeout
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
# Cause 5: Pod not ready — readiness probe failing
kubectl describe pod <pod-name> -n production
# Check: Readiness probe failed

Q22. How does Ingress handle WebSocket connections?

# WebSockets require long-lived connections — default timeouts too short
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    # NGINX automatically detects Upgrade header and handles WebSocket
    # No additional annotation needed for basic WebSocket support
# For sticky sessions (WebSocket clients must hit same pod)
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "ws-route"
    nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"

Q23. What happens when two Ingress resources define the same host and path?

Two Ingress resources both define:
  host: app.contoso.com
  path: /api
Result: NGINX Ingress uses the OLDER resource (by creation timestamp)
        The newer one is effectively ignored
Best practice:
  - Use a single Ingress per namespace for overlapping hosts
  - Or merge rules into one Ingress resource
  - NGINX logs a warning about duplicate path
Check for conflicts:
  kubectl get ingress -A | grep app.contoso.com

Q24. How do you expose an Ingress controller in different environments?

# Cloud (AKS, EKS, GKE) — LoadBalancer service (most common)
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
spec:
  type: LoadBalancer    # cloud provider provisions LB + public IP
  ports:
    - port: 80
    - port: 443
# On-premises / bare metal — NodePort
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 30080
    - port: 443
      nodePort: 30443
# On-premises with MetalLB — LoadBalancer with IP pool
# MetalLB assigns IP from your on-prem address pool
# ARO / OpenShift — use Route instead of Ingress
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: my-app
spec:
  host: app.cluster.aroapp.io
  to:
    kind: Service
    name: my-service
  tls:
    termination: edge

Quick-Reference Cheat Sheet

Ingress components:
  ingressClassName  → which controller handles this
  tls               → TLS secret reference
  rules             → host + path → service mapping
  defaultBackend    → catch-all for unmatched requests
  annotations       → controller-specific configuration
pathType values:
  Exact             → /api matches only /api
  Prefix            → /api matches /api, /api/v1, /api/users
  ImplementationSpecific → controller decides (regex, glob)
Common debug commands:
  kubectl get ingress -A
  kubectl describe ingress <name>
  kubectl get endpoints <service>
  kubectl logs -n ingress-nginx -l app=ingress-nginx
Common HTTP errors:
  404  → no matching rule / wrong path
  502  → backend unreachable or crashing
  503  → no healthy endpoints / rate limited
  504  → backend timeout (increase proxy-read-timeout)
  SSL certificate error → cert not in same namespace as Ingress

Understanding Ingress Limitations and Gateway API Benefits

techhadoop kubernetes

Ingress vs Gateway API

The Problem Ingress Has

Ingress was designed in the early days of Kubernetes and was never meant to handle the complexity of modern traffic management. Over time its limitations became painful.

Ingress limitations:
├── No native traffic splitting (canary needs annotations)
├── No TCP/UDP routing
├── Annotations are controller-specific (NGINX ≠ Traefik ≠ AGIC)
├── No role separation (devs and ops share same resource)
├── No header manipulation without annotations
└── Everything crammed into one resource

Gateway API was built to fix all of this.

Side-by-Side Architecture

INGRESS                              GATEWAY API
─────────────────────────────────    ─────────────────────────────────
                                     
┌─────────────────────────┐          ┌──────────────┐
│      Ingress            │          │GatewayClass  │ (cluster admin)
│  - host routing         │          │ which        │ defines controller
│  - path routing         │          │ controller   │
│  - TLS                  │          └──────┬───────┘
│  - annotations for      │                 │
│    everything else      │          ┌──────▼───────┐
│                         │          │   Gateway    │ (infra team)
│  Everything in          │          │ - listeners  │ defines ports,
│  ONE resource           │          │ - TLS        │ TLS, addresses
└─────────────────────────┘          └──────┬───────┘
                                            │
                                     ┌──────▼───────┐
                                     │  HTTPRoute   │ (app developer)
                                     │  TCPRoute    │ defines routing
                                     │  GRPCRoute   │ rules per service
                                     └──────────────┘

Role Separation (Biggest Difference)

Gateway API introduces a clear separation of concerns between personas:

INGRESS — everyone edits the same resource:
Cluster Admin ─┐
Infra Team    ─┼──▶ Ingress resource ◀─┬─ App Team A
Platform Team ─┘                        └─ App Team B
                   (conflicts, no boundaries)
GATEWAY API — each role owns their layer:
Cluster Admin ──▶ GatewayClass  (which controller to use)
                        │
Infra / Platform ──▶ Gateway    (ports, TLS, addresses)
                        │
App Team A ──▶ HTTPRoute        (routing rules for service A)
App Team B ──▶ HTTPRoute        (routing rules for service B)
# Cluster admin owns this — once
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: k8s.io/ingress-nginx
---
# Infra team owns this — defines the entry point
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main-gateway
  namespace: infra
spec:
  gatewayClassName: nginx
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-tls
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            gateway-access: "true"   # ← only these namespaces can attach
---
# App team owns this — in their own namespace
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: team-a          # ← app team's namespace
spec:
  parentRefs:
  - name: main-gateway
    namespace: infra         # ← attaches to infra gateway
  hostnames:
  - api.acme.com
  rules:
  - backendRefs:
    - name: api-service
      port: 8080

Traffic Splitting

# INGRESS — needs controller-specific annotations
# NGINX example (won't work on Traefik or AGIC):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
  - host: api.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-v2          # canary target
            port:
              number: 80
# GATEWAY API — native, works on any conformant controller
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-canary
spec:
  parentRefs:
  - name: main-gateway
  hostnames:
  - api.acme.com
  rules:
  - backendRefs:
    - name: api-v1
      port: 8080
      weight: 90          # ← native weight, no annotations
    - name: api-v2
      port: 8080
      weight: 10

Header Manipulation

# INGRESS — annotation-based, controller-specific
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header X-Version "v2";
      proxy_set_header X-Real-IP $remote_addr;
# GATEWAY API — native, portable
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
  rules:
  - filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: X-Real-IP
          value: "client-ip"
        set:
        - name: X-Version
          value: "v2"
        remove:
        - X-Internal-Token
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        add:
        - name: Strict-Transport-Security
          value: "max-age=31536000"
    backendRefs:
    - name: api-service
      port: 8080

URL Rewriting & Redirects

# INGRESS
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: acme.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: api-service
            port:
              number: 80
# GATEWAY API — clean and readable
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
  rules:
  # Rewrite /api/v1/users → /users
  - matches:
    - path:
        type: PathPrefix
        value: /api/v1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: api-service
      port: 8080
  # Redirect HTTP → HTTPS
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301

TCP and gRPC Routing

# INGRESS — cannot do this natively
# GATEWAY API — TCPRoute (L4)
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: postgres-route
spec:
  parentRefs:
  - name: main-gateway
    sectionName: postgres-listener   # port 5432
  rules:
  - backendRefs:
    - name: postgres-service
      port: 5432
---
# GATEWAY API — GRPCRoute
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: grpc-route
spec:
  parentRefs:
  - name: main-gateway
  hostnames:
  - grpc.acme.com
  rules:
  - matches:
    - method:
        service: acme.UserService
        method: GetUser
    backendRefs:
    - name: user-grpc-service
      port: 50051

Advanced Matching

# GATEWAY API — rich matching out of the box
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
spec:
  rules:
  # Match by header
  - matches:
    - headers:
      - name: X-User-Group
        value: beta
    backendRefs:
    - name: api-v2
      port: 8080
  # Match by query param
  - matches:
    - queryParams:
      - name: version
        value: "2"
    backendRefs:
    - name: api-v2
      port: 8080
  # Match by HTTP method
  - matches:
    - method: POST
      path:
        type: PathPrefix
        value: /api/events
    backendRefs:
    - name: events-service
      port: 8080
  # Default route
  - backendRefs:
    - name: api-v1
      port: 8080

TLS Configuration

# INGRESS
spec:
  tls:
  - hosts:
    - api.acme.com
    secretName: api-tls-secret
  rules:
  - host: api.acme.com
    ...
# GATEWAY API — TLS lives in Gateway (infra concern)
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
  listeners:
  # HTTPS with TLS termination
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: wildcard-tls
        namespace: infra
  # TLS passthrough — app handles TLS
  - name: tls-passthrough
    port: 8443
    protocol: TLS
    tls:
      mode: Passthrough     # ← forwards raw TLS to backend
  # mTLS
  - name: mtls
    port: 9443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        gateway.nginx.org/client-cert-verification: "on"

Cross-Namespace Routing

# INGRESS — cannot natively route across namespaces
# GATEWAY API — ReferenceGrant allows cross-namespace routing
# Team B grants permission for Gateway in infra namespace to
# route to their service
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-infra-gateway
  namespace: team-b           # ← in the target namespace
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: infra           # ← allow routes from this namespace
  to:
  - group: ""
    kind: Service              # ← to reach Services in team-b
---
# Now team-b's service can be referenced cross-namespace
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: infra
spec:
  rules:
  - backendRefs:
    - name: team-b-service
      namespace: team-b        # ← cross-namespace reference
      port: 8080

Full Feature Comparison

FeatureIngressGateway API
HTTP routing✅ Basic✅ Advanced
HTTPS / TLS✅✅
TCP routing❌✅ TCPRoute
UDP routing❌✅ UDPRoute
gRPC routing❌✅ GRPCRoute
Traffic splitting⚠️ Annotations✅ Native weights
Header manipulation⚠️ Annotations✅ Native filters
URL rewrite⚠️ Annotations✅ Native
Redirect⚠️ Annotations✅ Native
Role separation❌✅ 3-tier model
Cross-namespace❌✅ ReferenceGrant
TLS passthrough⚠️ Controller-specific✅ Native
mTLS⚠️ Annotations✅ Native
Portability❌ Annotations vary✅ Standardized
Extensibility❌✅ Policy attachment

Policy Attachment (Gateway API Extension)

Gateway API supports attaching policies to any resource — a clean extensibility model:

# Timeout policy on a route
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  rules:
  - backendRefs:
    - name: api-service
      port: 8080
    timeouts:
      request: 10s            # ← native timeout, no annotation
      backendRequest: 5s
---
# Rate limit policy (controller-specific extension)
apiVersion: gateway.nginx.org/v1alpha1
kind: ObservabilityPolicy
metadata:
  name: api-observability
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: api-route
  tracing:
    strategy: ratio
    ratio: 10

Migration Path: Ingress → Gateway API

Step 1: Install Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/
        releases/download/v1.1.0/standard-install.yaml
Step 2: Create GatewayClass (once per cluster)
        Maps to your existing NGINX/Traefik controller
Step 3: Create Gateway (replaces controller Service config)
        Moves TLS config here
Step 4: Convert Ingress rules → HTTPRoutes
        One HTTPRoute per service/team
Step 5: Test HTTPRoutes alongside existing Ingress
        Both can run simultaneously
Step 6: Remove Ingress resources when validated
# Helpful tool — ingress2gateway converter
go install sigs.k8s.io/ingress2gateway@latest
ingress2gateway print \
  --input-file ingress.yaml \
  --providers nginx
# Outputs equivalent Gateway API resources

When to Use Each

Use Ingress if:
✅ Simple HTTP/HTTPS routing
✅ Single team manages everything
✅ Already heavily invested in Ingress annotations
✅ Your controller doesn't support Gateway API yet
✅ K8s version < 1.24
Use Gateway API if:
✅ Multiple teams share the cluster
✅ Need TCP / UDP / gRPC routing
✅ Need native traffic splitting (canary)
✅ Want controller-portable config
✅ Building new clusters from scratch
✅ K8s 1.24+ (stable support)
✅ Need fine-grained TLS control
✅ Enterprise / production-grade setup

Summary

INGRESS                         GATEWAY API
───────────────────────         ───────────────────────
One resource does everything    Three-tier role model
Annotations for everything      Native filters + rules
Controller-specific behaviour   Portable across controllers
HTTP only                       HTTP, TCP, UDP, gRPC
No role separation              GatewayClass → Gateway → Route
Stable, widely supported        Stable from K8s 1.28
Good for simple setups          Built for enterprise scale

Gateway API is the future of Kubernetes traffic management — Ingress is in maintenance mode. For new clusters and teams, Gateway API is the clear choice. For existing Ingress setups, migration is straightforward and the ingress2gateway tool makes it even easier.

Understanding Ingress in Kubernetes

techhadoop kubernetes

Ingress in Kubernetes

What is Ingress?

Ingress is a Kubernetes API object that manages external HTTP/HTTPS access to services inside a cluster. It acts as a smart entry point — routing traffic based on hostnames, paths, and rules.

WITHOUT Ingress:                    WITH Ingress:
                                    
Internet                            Internet
   │                                    │
   ├──▶ LoadBalancer (Service A)        │
   ├──▶ LoadBalancer (Service B)    ┌───▼────────────────┐
   ├──▶ LoadBalancer (Service C)    │   Single Ingress   │
   └──▶ LoadBalancer (Service D)    │   (1 LoadBalancer) │
   (4 cloud LBs = 4x cost)         └───┬────────────────┘
                                        │
                               ┌────────┼────────┐
                               ▼        ▼        ▼
                           Service A  Service B  Service C

One entry point, many services — saves cost and complexity.

Core Components

┌─────────────────────────────────────────────────────────┐
│                    KUBERNETES CLUSTER                    │
│                                                          │
│  ┌─────────────────────────────────────────────────┐    │
│  │              INGRESS CONTROLLER                  │    │
│  │         (NGINX / Traefik / AGIC / HAProxy)       │    │
│  │    - Watches Ingress resources                   │    │
│  │    - Configures the actual proxy/LB              │    │
│  │    - Handles TLS termination                     │    │
│  └──────────────────────┬──────────────────────────┘    │
│                         │ reads                          │
│  ┌──────────────────────▼──────────────────────────┐    │
│  │               INGRESS RESOURCE                   │    │
│  │    - Rules (host + path → service)               │    │
│  │    - TLS config                                  │    │
│  │    - Annotations (controller-specific settings)  │    │
│  └──────────────────────┬──────────────────────────┘    │
│                         │ routes to                      │
│              ┌──────────┼──────────┐                    │
│              ▼          ▼          ▼                    │
│         Service A   Service B   Service C               │
│              │          │          │                    │
│           Pod(s)     Pod(s)     Pod(s)                  │
└─────────────────────────────────────────────────────────┘
         ▲
    LoadBalancer Service
    (single external IP)

Ingress vs Other Networking Types

ClusterIPNodePortLoadBalancerIngress
AccessInternal onlyNode IP + portExternal IPExternal HTTP/HTTPS
L7 routing❌❌❌✅
TLS termination❌❌❌✅
Host/path routing❌❌❌✅
CostFreeFree1 LB per service1 LB total
Use caseService-to-serviceDev/debuggingTCP/UDP servicesHTTP apps

Basic Ingress Resource

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx           # ← which controller handles this
  rules:
  - host: api.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

Routing Patterns

Host-Based Routing

Different domains → different services:

spec:
  rules:
  # api.acme.com → api-service
  - host: api.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
  # app.acme.com → frontend-service
  - host: app.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 3000
  # admin.acme.com → admin-service
  - host: admin.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: admin-service
            port:
              number: 8080

Path-Based Routing

Same domain, different paths → different services:

spec:
  rules:
  - host: acme.com
    http:
      paths:
      # acme.com/api/* → api-service
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
      # acme.com/auth/* → auth-service
      - path: /auth
        pathType: Prefix
        backend:
          service:
            name: auth-service
            port:
              number: 80
      # acme.com/ → frontend
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 3000

Combined Host + Path Routing

spec:
  rules:
  - host: api.acme.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-v1-service
            port:
              number: 80
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-v2-service
            port:
              number: 80

Path Types Explained

PathTypeBehaviorExample ruleMatchesDoes NOT match
ExactExact match only/api/api/api/, /api/users
PrefixMatches prefix/api/api, /api/, /api/users/apiv2
ImplementationSpecificController decidesvariesvariesvaries

TLS / HTTPS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-ingress
  namespace: production
  annotations:
    # Auto-provision cert via cert-manager
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  # TLS config
  tls:
  - hosts:
    - api.acme.com
    - app.acme.com
    secretName: acme-tls-secret     # cert stored here by cert-manager
  rules:
  - host: api.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
  - host: app.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 3000

TLS flow:

Client ──HTTPS──▶ Ingress Controller (terminates TLS)
                         │
                   plain HTTP
                         │
                         ▼
                    Backend Service

Ingress Controllers

The Ingress resource is just config — you need a controller to actually implement it.

Popular Controllers

ControllerBest forMaintained by
NGINX IngressGeneral purpose, most popularKubernetes community
TraefikDynamic config, Let’s Encrypt built-inTraefik Labs
AGIC (App Gateway)AKS / Azure nativeMicrosoft
AWS ALBEKS / AWS nativeAWS
HAProxyHigh performanceHAProxy Tech
KongAPI gateway featuresKong
Istio GatewayService mesh environmentsIstio

Install NGINX Ingress Controller

# Via Helm (recommended)
helm repo add ingress-nginx \
  https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.replicaCount=2 \
  --set controller.nodeSelector."kubernetes\.io/os"=linux
# Verify
kubectl get pods -n ingress-nginx
kubectl get svc -n ingress-nginx
# NAME                      TYPE           EXTERNAL-IP
# ingress-nginx-controller  LoadBalancer   20.10.5.100  ← your entry point

Install Traefik

helm repo add traefik https://traefik.github.io/charts
helm install traefik traefik/traefik \
  --namespace traefik \
  --create-namespace

AGIC on AKS (Azure)

# Enable via AKS addon
az aks enable-addons \
  --resource-group myRG \
  --name myAKSCluster \
  --addons ingress-appgw \
  --appgw-name myAppGateway \
  --appgw-subnet-cidr "10.2.0.0/16"

NGINX Annotations (Most Common)

Annotations let you configure controller-specific behaviour:

metadata:
  annotations:
    # Redirect HTTP → HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # Rewrite path before forwarding
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    # Rate limiting
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/limit-connections: "5"
    # Timeouts
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "10"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
    # Body size limit
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    # CORS
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://app.acme.com"
    # Whitelist IPs
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
    # Auth
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth-secret
    # Custom headers
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header X-Frame-Options "SAMEORIGIN";
      add_header X-Content-Type-Options "nosniff";

Default Backend

Handles requests that don’t match any rule:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-with-default
spec:
  ingressClassName: nginx
  # Catch-all — shown when no rule matches
  defaultBackend:
    service:
      name: custom-404-service
      port:
        number: 80
  rules:
  - host: api.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

Multiple Ingress Controllers

You can run multiple controllers in one cluster using IngressClass:

# Define two ingress classes
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-public
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"  # default
spec:
  controller: k8s.io/ingress-nginx
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-internal
spec:
  controller: k8s.io/ingress-nginx
  parameters:
    apiGroup: k8s.nginx.org
    kind: IngressClassParameters
    name: internal-lb-params
# Public ingress — uses public LB
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-api
spec:
  ingressClassName: nginx-public    # ← public controller
  rules:
  - host: api.acme.com
    ...
---
# Internal ingress — uses internal LB
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: internal-admin
spec:
  ingressClassName: nginx-internal  # ← internal controller
  rules:
  - host: admin.internal.acme.com
    ...

Real-World Production Setup

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: production-ingress
  namespace: production
  annotations:
    # TLS
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # Security headers
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header Strict-Transport-Security "max-age=31536000" always;
      add_header X-Frame-Options "DENY" always;
      add_header X-Content-Type-Options "nosniff" always;
    # Rate limiting
    nginx.ingress.kubernetes.io/limit-rps: "50"
    # Timeouts
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
    # Body size
    nginx.ingress.kubernetes.io/proxy-body-size: "5m"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - acme.com
    - api.acme.com
    - app.acme.com
    secretName: acme-wildcard-tls
  rules:
  - host: acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 3000
  - host: api.acme.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-v1
            port:
              number: 8080
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-v2
            port:
              number: 8080
  - host: app.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: webapp
            port:
              number: 3000

Ingress vs Gateway API

Kubernetes introduced Gateway API as the next generation of Ingress — more expressive and powerful.

IngressGateway API
StabilityStable (v1)Stable (v1 from K8s 1.28)
L7 routingBasicAdvanced
TCP/UDP routing❌✅
Traffic splittingAnnotation-basedNative
Multi-teamLimitedRole-based (Gateway vs Route)
Header manipulationAnnotation-basedNative
FutureMaintenance modeActively developed
# Gateway API equivalent
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  parentRefs:
  - name: main-gateway
  hostnames:
  - api.acme.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v2
    backendRefs:
    - name: api-v2
      port: 8080
      weight: 90
    - name: api-v2-canary
      port: 8080
      weight: 10    # ← native traffic splitting, no annotations

Troubleshooting

# Check ingress is created
kubectl get ingress -n production
# Check ingress details and events
kubectl describe ingress production-ingress -n production
# Check ingress controller logs
kubectl logs -n ingress-nginx \
  deployment/ingress-nginx-controller -f
# Check if backend service exists and has endpoints
kubectl get svc api-service -n production
kubectl get endpoints api-service -n production
# Test routing from inside cluster
kubectl run test --image=curlimages/curl -it --rm -- \
  curl -H "Host: api.acme.com" http://ingress-nginx-controller.ingress-nginx
# Check TLS certificate
kubectl describe certificate acme-tls-secret -n production
# Common issues checklist
# ❌ 404 — path not matching, check pathType
# ❌ 502 — backend pod not running or wrong port
# ❌ 503 — no healthy endpoints
# ❌ SSL error — cert not ready, check cert-manager
# ❌ No address — ingress controller not installed

Summary

User Request
    │
    ▼
DNS (api.acme.com → Ingress LB IP)
    │
    ▼
Ingress Controller (NGINX / Traefik / AGIC)
    │  reads Ingress resources
    │  terminates TLS
    │  matches host + path rules
    │
    ├──▶ /api     → api-service     → pods
    ├──▶ /auth    → auth-service    → pods
    └──▶ /        → frontend        → pods

Ingress is the front door of your Kubernetes cluster — one external IP, intelligent routing, TLS termination, and full control over how traffic reaches your services.

Understanding Sidecars in Kubernetes

techhadoop kubernetes

Sidecar in Kubernetes

What is a Sidecar?

A sidecar is an additional container running inside the same pod as your main application container — sharing the same network, storage, and lifecycle.

┌─────────────────────────────────────────┐
│                  POD                     │
│                                          │
│  ┌─────────────────┐  ┌───────────────┐  │
│  │   Main App      │  │   Sidecar     │  │
│  │  Container      │  │  Container    │  │
│  │                 │  │               │  │
│  │  (your code)    │  │ (helper code) │  │
│  └─────────────────┘  └───────────────┘  │
│                                          │
│  Shared: Network (localhost) │ Volumes   │
└─────────────────────────────────────────┘

The name comes from a motorcycle sidecar — it’s attached to the main vehicle, travels with it, but serves a separate purpose.

Why Use a Sidecar?

The core idea is separation of concerns — your app does business logic, the sidecar handles cross-cutting concerns.

WITHOUT sidecar:                WITH sidecar:
┌────────────────────┐          ┌──────────────┐  ┌──────────────┐
│   App Container    │          │     App      │  │   Sidecar    │
│                    │          │  (just biz   │  │  - logging   │
│  - business logic  │          │   logic)     │  │  - metrics   │
│  - logging         │          │              │  │  - mTLS      │
│  - metrics         │          │              │  │  - tracing   │
│  - mTLS            │          │              │  │              │
│  - tracing         │          └──────────────┘  └──────────────┘
│  - config reload   │
└────────────────────┘

How Containers Share Resources in a Pod

Pod Network Namespace
├── localhost (127.0.0.1) shared by ALL containers
├── Same IP address
└── Same port space (containers can't use same port)
Shared Volumes
├── emptyDir — shared scratch space
├── configMap / secret — shared config
└── PersistentVolume — shared storage
Process isolation
└── Containers have separate filesystems and processes
    (unless shareProcessNamespace: true)

Common Sidecar Patterns

Pattern 1 — Proxy / Service Mesh Sidecar

The most common use case — Envoy or linkerd-proxy intercepts all network traffic.

Incoming request
      ↓
┌─────────────────────────────────────┐
│  POD                                │
│  ┌──────────────┐  ┌─────────────┐  │
│  │  Envoy       │  │   App       │  │
│  │  Sidecar     │──▶  Container  │  │
│  │              │  │             │  │
│  │ - mTLS       │  │ sees plain  │  │
│  │ - retries    │  │ HTTP only   │  │
│  │ - metrics    │  │             │  │
│  │ - tracing    │  └─────────────┘  │
│  └──────────────┘                   │
└─────────────────────────────────────┘
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      containers:
      # Main app
      - name: api
        image: myapp:latest
        ports:
        - containerPort: 8080
      # Proxy sidecar
      - name: envoy-proxy
        image: envoyproxy/envoy:v1.28
        ports:
        - containerPort: 9901    # admin
        - containerPort: 15001   # outbound
        - containerPort: 15006   # inbound
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
      
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config

Pattern 2 — Log Shipper Sidecar

App writes logs to a shared volume, sidecar tails and ships them.

┌──────────────────────────────────────────┐
│  POD                                      │
│                                           │
│  ┌──────────────┐       ┌──────────────┐  │
│  │   App        │       │  Fluentd /   │  │
│  │  Container   │       │  Filebeat    │  │
│  │              │       │  Sidecar     │  │
│  │ writes logs  │       │              │  │
│  │ to /var/log/ │       │ tails logs   │  │
│  │              │       │ ships to     │  │
│  └──────┬───────┘       │ Elasticsearch│  │
│         │               └──────▲───────┘  │
│         │  shared volume        │          │
│         └──────────────────────┘          │
│              /var/log/app/                │
└──────────────────────────────────────────┘
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-with-logging
spec:
  template:
    spec:
      containers:
      # Main app — writes to shared volume
      - name: api
        image: myapp:latest
        volumeMounts:
        - name: log-volume
          mountPath: /var/log/app
      # Log shipper sidecar
      - name: log-shipper
        image: fluent/fluentd:latest
        volumeMounts:
        - name: log-volume
          mountPath: /var/log/app      # same volume
          readOnly: true
        env:
        - name: ELASTICSEARCH_HOST
          value: "elasticsearch.logging.svc"
      volumes:
      - name: log-volume
        emptyDir: {}                   # shared scratch space

Pattern 3 — Config Watcher / Reloader

Sidecar watches for config changes and reloads the app without restarting the pod.

┌──────────────────────────────────────────┐
│  POD                                      │
│                                           │
│  ┌──────────────┐       ┌──────────────┐  │
│  │   Nginx /    │       │  Config      │  │
│  │   App        │◀──────│  Reloader    │  │
│  │              │ SIGHUP│  Sidecar     │  │
│  │ uses config  │       │              │  │
│  │ from /config │       │ watches      │  │
│  │              │       │ ConfigMap    │  │
│  └──────────────┘       │ changes      │  │
└──────────────────────────────────────────┘
      containers:
      # Main app
      - name: nginx
        image: nginx:latest
        volumeMounts:
        - name: config
          mountPath: /etc/nginx

      # Config reloader sidecar
      - name: config-reloader
        image: jimmidyson/configmap-reload:latest
        args:
        - --volume-dir=/config
        - --webhook-url=http://localhost:80/-/reload
        volumeMounts:
        - name: config
          mountPath: /config
          readOnly: true

      volumes:
      - name: config
        configMap:
          name: nginx-config

Pattern 4 — Secret / Cert Sync Sidecar

Fetches secrets from Vault or Key Vault and writes them to a shared volume.

┌──────────────────────────────────────────┐
│  POD                                      │
│                                           │
│  ┌──────────────┐       ┌──────────────┐  │
│  │   App        │       │  Vault       │  │
│  │              │       │  Agent       │  │
│  │ reads creds  │       │  Sidecar     │  │
│  │ from         │       │              │  │
│  │ /vault/      │       │ fetches +    │  │
│  │ secrets/     │       │ renews creds │  │
│  └──────┬───────┘       └──────▲───────┘  │
│         │  shared volume        │ writes   │
│         └──────────────────────┘          │
│              /vault/secrets/              │
└──────────────────────────────────────────┘
      containers:
      # Main app
      - name: api
        image: myapp:latest
        volumeMounts:
        - name: vault-secrets
          mountPath: /vault/secrets
          readOnly: true

      # Vault agent sidecar
      - name: vault-agent
        image: hashicorp/vault:latest
        args:
        - agent
        - -config=/vault/config/agent.hcl
        volumeMounts:
        - name: vault-secrets
          mountPath: /vault/secrets     # writes here
        - name: vault-config
          mountPath: /vault/config

      volumes:
      - name: vault-secrets
        emptyDir:
          medium: Memory               # in-memory — never on disk
      - name: vault-config
        configMap:
          name: vault-agent-config

Pattern 5 — Ambassador Sidecar

Acts as a local proxy to external services — simplifies connection logic in the app.

┌──────────────────────────────────────────┐
│  POD                                      │
│                                           │
│  ┌──────────────┐       ┌──────────────┐  │
│  │   App        │       │  Ambassador  │  │
│  │              │       │  Sidecar     │  │
│  │ connects to  │──────▶│              │  │
│  │ localhost:   │       │ handles:     │  │
│  │ 5432         │       │ - TLS        │  │
│  │ (thinks it's │       │ - retries    │  │
│  │  local DB)   │       │ - failover   │  │
│  └──────────────┘       └──────┬───────┘  │
└──────────────────────────────────────────┘
                                 │
                    ┌────────────▼──────────┐
                    │  Remote DB / Service  │
                    │  (with mTLS, etc.)    │
                    └───────────────────────┘

Init Containers vs Sidecar Containers

These are often confused — they are very different:

Init ContainerSidecar Container
When runsBefore main container startsAlongside main container
LifecycleRuns once, must complete successfullyRuns for entire pod lifetime
PurposeSetup / bootstrap tasksOngoing helper tasks
FailurePod won’t start if it failsPod restarts if it crashes
ExamplesDB migration, wait-for-service, download configLogging, proxy, metrics
spec:
  # Init containers — run first, in order
  initContainers:
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 'until nc -z db-service 5432; do sleep 2; done']
  - name: run-migrations
    image: myapp:latest
    command: ['python', 'manage.py', 'migrate']
  # Main + sidecar — run together after init completes
  containers:
  - name: api
    image: myapp:latest
  - name: log-shipper
    image: fluentd:latest

Native Sidecar Support (Kubernetes 1.29+)

Before K8s 1.29, sidecars were just regular containers with ordering hacks. Now there is first-class sidecar support via initContainers with restartPolicy: Always.

spec:
  initContainers:
  # New native sidecar — starts before main app
  # but keeps running alongside it
  - name: log-shipper
    image: fluentd:latest
    restartPolicy: Always       # ← this makes it a native sidecar
    volumeMounts:
    - name: log-volume
      mountPath: /var/log/app
  containers:
  - name: api
    image: myapp:latest

Benefits of native sidecars:

Issue (old way)Fix (native sidecar)
Sidecar starts same time as app — race conditionSidecar starts before main app
Job pods never complete (sidecar keeps running)Sidecar exits when main container exits
No guaranteed startup orderGuaranteed: init sidecars → main containers
Probe failures affect main containerSidecar lifecycle is independent

Resource Management for Sidecars

Always set resource limits — sidecars can starve your main app.

      containers:
      - name: api
        image: myapp:latest
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

      # Sidecar should be lean
      - name: log-shipper
        image: fluentd:latest
        resources:
          requests:
            memory: "50Mi"     # Keep small
            cpu: "50m"
          limits:
            memory: "100Mi"
            cpu: "100m"

Automatic Sidecar Injection (Service Mesh)

In Istio/Linkerd, sidecars are injected automatically via a Mutating Admission Webhook — you never write the sidecar spec yourself.

You apply:                       Kubernetes actually runs:
┌──────────────┐                 ┌──────────────┬──────────────┐
│  Deployment  │  Webhook        │   Your App   │    Envoy     │
│  (1 container│─────────────▶  │  Container   │   Sidecar    │
│   your app)  │  injects sidecar│              │  (injected)  │
└──────────────┘                 └──────────────┴──────────────┘
# Enable auto-injection for a namespace
kubectl label namespace production istio-injection=enabled
# Now every new pod in that namespace
# automatically gets an Envoy sidecar
kubectl apply -f deployment.yaml   # ← sidecar added automatically
# Verify
kubectl get pod api-pod \
  -o jsonpath='{.spec.containers[*].name}'
# Output: api istio-proxy

Sidecar Anti-Patterns

Anti-PatternProblemFix
No resource limits on sidecarSidecar can OOM-kill the nodeAlways set requests + limits
Sidecar does heavy computeStarves the main appKeep sidecars lightweight
Too many sidecars per podHigh overhead, complex debuggingMax 2-3 sidecars per pod
Sidecar holds critical business logicViolates separation of concernsBusiness logic belongs in main app
No health checks on sidecarPod looks healthy but sidecar is brokenAdd readiness probes to sidecars
Tight coupling between app and sidecarCan’t deploy independentlySidecar should be generic and reusable

When to Use a Sidecar

✅ Use sidecar when:
   - Cross-cutting concern (logging, metrics, tracing)
   - Needs to intercept network traffic (proxy)
   - Works the same regardless of app language
   - Logic is reusable across many services
❌ Don't use sidecar when:
   - Logic is app-specific
   - Simple enough for a library
   - Performance is ultra-critical (adds overhead)
   - You only have one or two services

The sidecar pattern is one of the most powerful patterns in Kubernetes — it’s what makes service meshes, log aggregation pipelines, and secret management work transparently across every service in your cluster without touching application code.

Understanding Service Mesh: Key Interview Questions

techhadoop kubernetes

Service Mesh Interview Questions

Beginner Level

Q1. What is a service mesh and why do we need it?

A: A service mesh is a dedicated infrastructure layer that manages service-to-service communication in a microservices architecture. Without it, every service must implement its own retry logic, timeouts, mTLS, and observability — leading to duplicated code across every language/framework.

A service mesh moves all of that into a sidecar proxy (or kernel via eBPF) so the application stays focused on business logic.

Problems it solves:

  • No encryption between services by default
  • Hard to debug distributed failures
  • Retry/timeout logic duplicated in every service
  • No visibility into which service is causing latency
  • No way to do canary releases at the network level

Q2. What is the difference between a sidecar proxy and a service mesh control plane?

A:

Control Plane (istiod / Linkerd Controller)
  - Manages configuration
  - Issues and rotates mTLS certificates
  - Pushes routing rules to all proxies
  - Collects telemetry
Data Plane (Envoy / linkerd-proxy sidecars)
  - Actually intercepts and forwards traffic
  - Enforces routing rules
  - Enforces mTLS
  - Emits metrics and traces

The control plane tells proxies what to do. The data plane does it.

Q3. What is mTLS and how does a service mesh implement it?

A: mTLS (mutual TLS) means both sides of a connection authenticate each other — unlike regular TLS where only the server presents a certificate.

How a mesh implements it:

  1. Control plane acts as a Certificate Authority (CA)
  2. Issues a unique certificate to every pod (bound to its service account)
  3. Sidecar intercepts all outbound/inbound traffic
  4. Automatically performs TLS handshake — app sees plain HTTP
  5. Certificates are rotated automatically (e.g. every 24 hours)
Pod A (app) → sidecar (presents cert A) 
            ──────── mTLS ────────────▶
                          sidecar (presents cert B) → Pod B (app)

Application code is completely unaware of TLS.

Q4. What is the difference between PeerAuthentication and AuthorizationPolicy in Istio?

A:

PeerAuthenticationAuthorizationPolicy
WhatHow traffic is authenticatedWhat traffic is allowed
ControlsmTLS mode (STRICT / PERMISSIVE / DISABLE)Allow/deny rules between services
Think of it as“Must use mTLS”“Only frontend can call backend”
# PeerAuthentication — enforce mTLS
spec:
  mtls:
    mode: STRICT
# AuthorizationPolicy — control who can call what
spec:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/prod/sa/frontend"]

Q5. What is the difference between a VirtualService and a DestinationRule in Istio?

A:

VirtualServiceDestinationRule
PurposeRouting rules (where traffic goes)Traffic policy (how it gets there)
ControlsWeights, retries, timeouts, headers, fault injectionLoad balancing, circuit breaking, subsets, TLS settings
AppliedOn the client sideOn the destination side

Think of it as:

  • VirtualService = traffic cop (route 10% here, 90% there)
  • DestinationRule = rules for the destination (max 100 connections, eject unhealthy pods)

Q6. What are the PERMISSIVE and STRICT modes in Istio mTLS?

A:

ModeBehaviorUse case
PERMISSIVEAccepts both mTLS and plain HTTPMigration phase — some services not yet meshed
STRICTAccepts mTLS only — rejects plain HTTPProduction zero-trust target state
DISABLEPlain text only, no mTLSLegacy compatibility (avoid)

Best practice: Start with PERMISSIVE during rollout, migrate all services, then switch to STRICT.

Q7. How does sidecar injection work in Kubernetes?

A: Istio uses a Kubernetes Mutating Admission Webhook:

kubectl apply pod.yaml
        ↓
Kubernetes API Server
        ↓
Mutating Admission Webhook (istio-sidecar-injector)
        ↓
Modifies pod spec — adds:
  - initContainer: istio-init (sets iptables rules)
  - container: istio-proxy (Envoy sidecar)
        ↓
Pod runs with sidecar automatically

The istio-init container sets iptables rules to intercept all traffic through the Envoy proxy on ports 15001 (outbound) and 15006 (inbound) — without changing the app.

Intermediate Level

Q8. How does Istio handle certificate rotation and what happens if istiod goes down?

A:

  • Istio’s CA (inside istiod) issues SVID certificates (SPIFFE format) to each workload
  • Default cert lifetime: 24 hours, rotated at ~75% of lifetime (around 18 hours)
  • Rotation happens via SDS (Secret Discovery Service) — Envoy requests new certs automatically

If istiod goes down:

  • Existing connections continue (Envoy already has certs)
  • New pods cannot get certificates — they’ll fail to start mesh enrollment
  • Routing rule updates stop propagating
  • Existing routing rules remain cached in Envoy

This is why istiod HA (multiple replicas) is critical in production:

kubectl scale deployment istiod -n istio-system --replicas=3

Q9. Explain circuit breaking in Istio. How is it configured?

A: Circuit breaking stops sending traffic to an unhealthy service to prevent cascading failures — similar to an electrical circuit breaker.

Three mechanisms in Istio:

1. Connection Pool Limits — limits concurrent connections

connectionPool:
  tcp:
    maxConnections: 100
  http:
    http1MaxPendingRequests: 50
    http2MaxRequests: 200

2. Outlier Detection — ejects unhealthy pods from load balancer

outlierDetection:
  consecutive5xxErrors: 5      # 5 errors in a row
  interval: 10s                # check every 10s
  baseEjectionTime: 30s        # eject for 30s minimum
  maxEjectionPercent: 50       # never eject more than 50% of pods

3. Pending Request Limits — rejects requests when queue is full

When circuit opens → caller gets 503 UF (Upstream overflow) immediately instead of waiting for timeout — fails fast.

Q10. What is the difference between Istio’s ingress gateway and a standard Kubernetes Ingress?

A:

K8s IngressIstio Gateway + VirtualService
L7 routingBasic (host/path)Advanced (headers, weights, regex)
mTLS terminationController-dependentNative
Traffic splittingNot supportedFull canary/weighted support
Fault injection❌✅
Retries/timeouts❌✅
ObservabilityLimitedFull Envoy metrics + traces
# Istio Gateway — defines the listener
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: api-tls-cert
    hosts:
    - api.acme.com
---
# VirtualService — defines routing behind the gateway
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-vs
spec:
  hosts:
  - api.acme.com
  gateways:
  - api-gateway
  http:
  - route:
    - destination:
        host: api-service
        subset: v1

Q11. How do you implement a canary deployment with Istio?

A:

Step 1: Deploy v2 alongside v1 (both running)
Step 2: Create DestinationRule with subsets
Step 3: Use VirtualService to split traffic
Step 4: Gradually shift weight
Step 5: Monitor error rates and latency
Step 6: Cut over to 100% v2 or rollback
# DestinationRule — define v1 and v2 subsets
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api-dr
spec:
  host: api-service
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---
# VirtualService — start at 5% canary
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-canary
spec:
  hosts:
  - api-service
  http:
  - route:
    - destination:
        host: api-service
        subset: v1
      weight: 95
    - destination:
        host: api-service
        subset: v2
      weight: 5    # ← increase this gradually

Progressive delivery tools like Flagger or Argo Rollouts automate this — they watch error rates and auto-promote or rollback.

Q12. What is Envoy’s xDS API and why does it matter?

A: xDS is the configuration API that istiod uses to push config to all Envoy proxies dynamically — without restarting them.

APIControls
LDS (Listener Discovery Service)Ports to listen on
RDS (Route Discovery Service)Routing rules
CDS (Cluster Discovery Service)Upstream service endpoints
EDS (Endpoint Discovery Service)Individual pod IPs per cluster
SDS (Secret Discovery Service)TLS certificates
istiod
  ├── LDS → "listen on port 15001"
  ├── RDS → "route /api/* to cluster api-service-v1"
  ├── CDS → "cluster api-service-v1 exists"
  ├── EDS → "api-service-v1 has pods 10.0.1.5, 10.0.1.6"
  └── SDS → "here's your mTLS cert, rotate in 18h"

Why it matters: config changes propagate to thousands of pods in seconds with zero restarts.

Q13. How does Linkerd differ from Istio architecturally?

A:

IstioLinkerd
ProxyEnvoy (C++, general purpose)linkerd2-proxy (Rust, purpose-built)
Control planeistiod (monolith of Pilot+Citadel+Galley)destination + identity + proxy-injector
Config modelCRDs (VirtualService, DestinationRule, etc.)SMI / HTTPRoute (simpler)
Resource usage~50-100MB RAM per sidecar~10-20MB RAM per sidecar
Feature setExhaustiveFocused (does less, does it well)
Operational complexityHighLow
Protocol detectionManual hints sometimes neededAutomatic

Key insight: Linkerd’s proxy is written in Rust specifically for this use case — it’s faster and lighter than Envoy. Istio uses Envoy which is a general-purpose proxy with far more knobs.

Q14. What is SPIFFE and how does Istio use it?

A: SPIFFE (Secure Production Identity Framework For Everyone) is a standard for workload identity in cloud-native environments.

Each workload gets a SPIFFE ID in URI format:

spiffe://cluster.local/ns/production/sa/api-service
         └─ trust domain ─┘ └─ namespace ─┘ └─ service account ─┘

Istio issues SVID (SPIFFE Verifiable Identity Document) certificates containing this URI as a SAN (Subject Alternative Name).

AuthorizationPolicy uses SPIFFE IDs:
principals:
  - "cluster.local/ns/production/sa/frontend"
This means: "only the pod with this exact
             service account identity can connect"

This is much stronger than IP-based rules (IPs change when pods restart).

Advanced Level

Q15. How would you debug a 503 error in an Istio mesh?

A: Systematic approach:

# Step 1 — Check if sidecar is injected
kubectl get pod failing-pod -o jsonpath='{.spec.containers[*].name}'
# Step 2 — Check Istio proxy status
istioctl proxy-status   # Are all proxies in sync with istiod?
# Step 3 — Analyze the namespace for misconfigs
istioctl analyze -n production
# Step 4 — Check Envoy access logs for error flags
kubectl logs failing-pod -c istio-proxy | grep -v '"200"'
# Look for: UF=upstream overflow (circuit break)
#           UO=upstream overflow
#           NR=no route found
#           URX=upstream retry exceeded
# Step 5 — Check routing config on the calling pod
istioctl proxy-config routes calling-pod.production
# Step 6 — Check endpoints are healthy
istioctl proxy-config endpoints calling-pod.production \
  | grep api-service
# Step 7 — Check AuthorizationPolicy
istioctl x authz check failing-pod.production
# Step 8 — Check mTLS status
istioctl x describe pod failing-pod.production

Common causes:

  • NR — no route: VirtualService misconfigured or missing
  • UF — circuit breaker open: too many errors or connections
  • RBAC denied — AuthorizationPolicy blocking the call
  • Cert mismatch — one service in STRICT, caller not meshed

Q16. How do you handle service mesh in a multi-cluster setup? What are the challenges?

A: Three models:

1. Replicated control planes (most common)

Cluster A                    Cluster B
istiod-A                     istiod-B
  ↓                            ↓
East-West Gateway ←──mTLS──▶ East-West Gateway
(cross-cluster traffic tunneled through gateways)

2. Primary-Remote (single control plane)

Cluster A (Primary)          Cluster B (Remote)
istiod ──────────────────────▶ no istiod
       manages both clusters

3. External control plane

Management Cluster           Workload Clusters
istiod ──────────────────────▶ Cluster A
       ──────────────────────▶ Cluster B

Key challenges:

ChallengeSolution
Cross-cluster service discoveryServiceEntry + DNS federation
Certificate trust across clustersShared root CA or cert federation
Network latency for cross-cluster callsLocality-aware routing (prefer local)
Different Istio versions per clusterVersion skew policy (n-1 support)
Debugging cross-cluster tracesShared Jaeger/Tempo with cluster labels

Q17. What is Ambient Mesh in Istio and why was it introduced?

A: Ambient Mesh is Istio’s sidecar-free architecture introduced to address the main criticisms of the sidecar model:

Problems with sidecars:

  • Every pod gets an Envoy sidecar — high memory/CPU overhead
  • Sidecar must be restarted to update — causes pod restarts
  • Sidecar injection is all-or-nothing per pod
  • Complex bootstrapping and lifecycle management

Ambient Mesh approach:

Traditional:
Pod → Envoy sidecar → Network
Ambient:
Pod → ztunnel (node-level, L4) → waypoint proxy (L7, per service)

Two layers:

  • ztunnel — a per-node DaemonSet handling L4 mTLS for all pods on the node (zero pod overhead)
  • Waypoint proxy — an optional per-service Envoy for L7 features (routing, retries, auth policies)

Benefits:

  • No sidecar injection needed
  • Update mesh without restarting pods
  • Pay for L7 features only where needed
  • Dramatically lower resource footprint

Q18. How does a service mesh interact with Kubernetes Network Policies? Are they redundant?

A: They operate at different layers and are complementary, not redundant:

K8s Network PolicyService Mesh AuthorizationPolicy
LayerL3/L4 (IP + port)L7 (HTTP method, path, identity)
IdentityPod IP / label selectorSPIFFE cryptographic identity
EnforcementCNI plugin (Calico, Cilium)Envoy sidecar
Spoofable?IP can be spoofedCryptographic — cannot be spoofed
Granularity“Allow pod A to reach port 8080”“Allow GET /api/users from service account X”

Best practice — use both:

NetworkPolicy  → "frontend namespace can reach backend namespace on port 8080"
                 (coarse-grained, IP-level firewall)
AuthorizationPolicy → "only pods with SA 'frontend' can call GET /api/*"
                      (fine-grained, cryptographic identity)

Network Policy is the outer wall. AuthorizationPolicy is the inner door lock.

Q19. How would you implement progressive delivery (automated canary) with Istio and Flagger?

A:

Flagger watches a Deployment
  → Creates primary + canary deployments automatically
  → Shifts traffic incrementally
  → Queries Prometheus for error rate + latency
  → Auto-promotes if healthy, auto-rollbacks if not
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: api-service
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  progressDeadlineSeconds: 120
  service:
    port: 80
    targetPort: 8080
    gateways:
    - istio-system/ingressgateway
    hosts:
    - api.acme.com
  analysis:
    interval: 1m           # Check every minute
    threshold: 5           # Max 5 failed checks before rollback
    maxWeight: 50          # Max 50% traffic to canary
    stepWeight: 10         # Increase by 10% each step
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99            # Must maintain 99% success
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500           # P99 must be < 500ms
      interval: 1m
    webhooks:
    - name: load-test
      url: http://loadtester/
      metadata:
        cmd: "hey -z 1m -q 10 http://api-service/"

Traffic flow during canary:

0%  → deploy canary
10% → check metrics (1 min)
20% → check metrics (1 min)
...
50% → check metrics (1 min)
     ✅ promote to 100% OR ❌ rollback to 0%

Q20. What are the performance overheads of a service mesh and how do you mitigate them?

A:

Latency overhead (per hop):

Istio/Envoy:   ~1-3ms added latency per request (P99)
Linkerd:       ~0.5-1ms added latency per request
Cilium eBPF:   ~0.1-0.3ms (kernel-level, minimal overhead)

Resource overhead (per pod):

Envoy sidecar:         50-100MB RAM, 0.1-0.5 CPU cores
linkerd-proxy:         10-20MB RAM, minimal CPU
Cilium (no sidecar):   ~0MB per pod (shared ztunnel on node)

Mitigation strategies:

StrategyImpact
Use Linkerd instead of Istio for latency-sensitive services3-5x lower latency overhead
Istio Ambient Mesh (ztunnel)Eliminates per-pod sidecar cost
Tune Envoy concurrency (--concurrency 2)Reduce CPU usage
Use PERMISSIVE only where neededAvoid double encryption
Disable unused features (tracing sampling %)Lower telemetry overhead
Set sidecar resource limitsPrevent noisy neighbor issues
Exclude non-mesh workloads (batch jobs)No sidecar for jobs that don’t need it
# Reduce tracing overhead — sample 1% in prod
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-tracing
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: jaeger
    randomSamplingPercentage: 1.0    # ← not 100%

Quick-Fire Questions

Q: What port does Envoy listen on for inbound/outbound? Inbound: 15006, Outbound: 15001, Admin: 15000, Health: 15021

Q: What is the difference between east-west and north-south traffic? North-south = traffic entering/leaving the cluster (ingress/egress). East-west = service-to-service inside the cluster.

Q: Can you use a service mesh without Kubernetes? Yes — Consul Connect supports VMs. Istio has VM mesh support. But Kubernetes is the primary target.

Q: What is SMI (Service Mesh Interface)? A standard set of Kubernetes CRDs that provide a common API across mesh implementations (TrafficSplit, TrafficPolicy, etc.) so tools like Flagger work with any mesh.

Q: What happens to traffic if the sidecar crashes? Traffic is intercepted by iptables rules pointing to the sidecar. If the sidecar is down, traffic is dropped — not bypassed. This is by design for security.

Q: How do you exclude a namespace from the mesh? Remove the istio-injection=enabled label and restart pods. Or annotate pods with sidecar.istio.io/inject: "false".

Understanding Service Mesh in Kubernetes

techhadoop kubernetes

Service Mesh in Kubernetes

What is a Service Mesh?

A service mesh is a dedicated infrastructure layer that handles all service-to-service communication inside a Kubernetes cluster — without changing application code.

WITHOUT Service Mesh:

 
WITH Service Mesh:

Core Features

FeatureWhat it gives you
mTLSEncrypted + authenticated service-to-service traffic
Traffic ManagementCanary, blue/green, weighted routing, mirroring
ObservabilityAutomatic metrics, traces, logs per service
Retries & TimeoutsResilience without code changes
Circuit BreakingStop cascading failures
Rate LimitingProtect services from overload
Access PolicyAllow/deny between specific services

Service Mesh Landscape

MeshBest forComplexity
IstioFull-featured, enterprise🔴 High
LinkerdSimple, lightweight, fast🟢 Low
Consul ConnectMulti-cluster, multi-cloud🟠 Medium
CiliumeBPF-based, no sidecar🟠 Medium
Open Service MeshAKS native (deprecated)🟢 Low

Option 1 — Istio (Most Feature-Rich)

Architecture

┌─────────────────────────────────────────────────────┐
│                   CONTROL PLANE                      │
│                    istiod                            │
│   ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│   │  Pilot   │  │  Citadel │  │     Galley        │  │
│   │(traffic) │  │  (certs) │  │  (config valid.)  │  │
│   └──────────┘  └──────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────┘
                        ↓ xDS API
┌─────────────────────────────────────────────────────┐
│                   DATA PLANE                         │
│   Pod A                          Pod B               │
│  ┌─────────┬───────────┐        ┌──────────┬──────┐  │
│  │  App    │  Envoy    │──mTLS─▶│  Envoy   │ App  │  │
│  │Container│  Sidecar  │        │  Sidecar │      │  │
│  └─────────┴───────────┘        └──────────┴──────┘  │
└─────────────────────────────────────────────────────┘

Install Istio

# Download istioctl
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-1.20.0/bin:$PATH
# Install with default profile
istioctl install --set profile=default -y
# Verify
istioctl verify-install
kubectl get pods -n istio-system
# Enable sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled
# Check injection is working
kubectl get namespace production --show-labels

Sidecar Injection

# Namespace-level (all pods in namespace)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    istio-injection: enabled    # ← auto-inject Envoy sidecar
---
# Pod-level override (opt out specific pod)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"    # ← skip this pod

Traffic Management

VirtualService — routing rules

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-routing
  namespace: production
spec:
  hosts:
  - api-service
  http:
  # Canary — 10% to v2, 90% to v1
  - route:
    - destination:
        host: api-service
        subset: v1
      weight: 90
    - destination:
        host: api-service
        subset: v2
      weight: 10
  # Timeout & retry
  timeout: 10s
  retries:
    attempts: 3
    perTryTimeout: 3s
    retryOn: "5xx,reset,connect-failure"

DestinationRule — load balancing & circuit breaking

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api-destination
  namespace: production
spec:
  host: api-service
  trafficPolicy:
    # Load balancing
    loadBalancer:
      simple: LEAST_CONN
    # Connection pool (circuit breaking)
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
    # Outlier detection (eject unhealthy pods)
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
  # Subsets for canary
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

Header-based routing

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-header-routing
spec:
  hosts:
  - api-service
  http:
  # Route beta users to v2
  - match:
    - headers:
        x-user-group:
          exact: beta
    route:
    - destination:
        host: api-service
        subset: v2
  # Everyone else → v1
  - route:
    - destination:
        host: api-service
        subset: v1

Fault Injection (Chaos Testing)

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: fault-injection-test
spec:
  hosts:
  - api-service
  http:
  - fault:
      # Inject 500ms delay for 10% of requests
      delay:
        percentage:
          value: 10
        fixedDelay: 500ms
      # Inject 503 error for 5% of requests
      abort:
        percentage:
          value: 5
        httpStatus: 503
    route:
    - destination:
        host: api-service

Security (mTLS + Authorization)

Enforce strict mTLS

# Namespace-wide strict mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT    # All traffic must be mTLS
---
# Mesh-wide strict mTLS (all namespaces)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system    # ← applies mesh-wide
spec:
  mtls:
    mode: STRICT

AuthorizationPolicy — zero trust between services

# Only allow frontend to call api-service
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: api-service-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-service
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
          - "cluster.local/ns/production/sa/frontend"    # service account
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/*"]
---
# Deny all by default (then allow explicitly)
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: production
spec:
  {}    # Empty spec = deny everything

JWT Authentication

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-service
  jwtRules:
  - issuer: "https://login.microsoftonline.com/tenant-id/v2.0"
    jwksUri: "https://login.microsoftonline.com/tenant-id/discovery/v2.0/keys"
    audiences:
    - "api://your-app-id"

Observability

Install Kiali + Jaeger + Prometheus + Grafana

# Install addons
kubectl apply -f istio-1.20.0/samples/addons/prometheus.yaml
kubectl apply -f istio-1.20.0/samples/addons/grafana.yaml
kubectl apply -f istio-1.20.0/samples/addons/jaeger.yaml
kubectl apply -f istio-1.20.0/samples/addons/kiali.yaml
# Open dashboards
istioctl dashboard kiali      # Service graph + health
istioctl dashboard grafana    # Metrics dashboards
istioctl dashboard jaeger     # Distributed tracing

Key Metrics (auto-generated by Envoy)

istio_requests_total                    # Request count
istio_request_duration_milliseconds     # Latency histogram
istio_request_bytes                     # Request size
istio_response_bytes                    # Response size
istio_tcp_connections_opened_total      # TCP connections

Option 2 — Linkerd (Lightweight & Simple)

Install Linkerd

# Install CLI
curl --proto '=https' --tlsv1.2 -sSfL \
  https://run.linkerd.io/install | sh
export PATH=$PATH:$HOME/.linkerd2/bin
# Pre-install check
linkerd check --pre
# Install CRDs then control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
# Verify
linkerd check
# Install observability (Viz)
linkerd viz install | kubectl apply -f -
linkerd viz dashboard &    # Open dashboard

Inject Linkerd Sidecar

# Namespace annotation
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    linkerd.io/inject: enabled    # ← auto-inject
---
# Or annotate deployment directly
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  annotations:
    linkerd.io/inject: enabled

Traffic Splitting (Canary)

apiVersion: split.smi-spec.io/v1alpha1
kind: TrafficSplit
metadata:
  name: api-canary
  namespace: production
spec:
  service: api-service
  backends:
  - service: api-service-v1
    weight: 90
  - service: api-service-v2
    weight: 10

Retries & Timeouts

apiVersion: policy.linkerd.io/v1beta1
kind: HTTPRoute
metadata:
  name: api-retry
  namespace: production
spec:
  parentRefs:
  - name: api-service
    kind: Service
  rules:
  - backendRefs:
    - name: api-service
      port: 80
    timeouts:
      request: 10s
    retry:
      limit: 3
      timeout: 3s
      conditions:
      - gateway-error
      - http-response-500

Option 3 — Cilium Service Mesh (No Sidecar)

Traditional sidecar mesh:         Cilium (eBPF):
Pod → Sidecar → Network           Pod → eBPF kernel hooks → Network
(extra hop, extra resource)       (in-kernel, no sidecar overhead)
# Install Cilium with service mesh features
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set gatewayAPI.enabled=true \
  --set envoy.enabled=true
# Verify
cilium status
cilium connectivity test
# mTLS via CiliumNetworkPolicy
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-mtls-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-service
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/.*"

Istio vs Linkerd vs Cilium

IstioLinkerdCilium
SidecarEnvoy proxylinkerd-proxy (Rust)No sidecar (eBPF)
ComplexityHighLowMedium
Resource overheadHighLowVery low
mTLS✅✅✅
Traffic splitting✅ Full✅ Basic✅
Circuit breaking✅⚠️ Limited✅
Fault injection✅❌❌
Observability✅ Rich✅ Good✅ Good
Multi-cluster✅✅✅
AKS native support✅✅✅ (BYOCNI)
Best forEnterprise, full controlSimplicity, low overheadPerformance, no sidecar

Multi-Cluster Service Mesh

Cluster A (East US)              Cluster B (West US)
┌─────────────────────┐          ┌─────────────────────┐
│  istiod             │◄────────▶│  istiod             │
│  ┌───────────────┐  │  shared  │  ┌───────────────┐  │
│  │ frontend pod  │  │  control │  │ api-service   │  │
│  │  + sidecar    │──┼──plane───┼─▶│  + sidecar    │  │
│  └───────────────┘  │          │  └───────────────┘  │
│  East Gateway       │◄─mTLS───▶│  West Gateway       │
└─────────────────────┘          └─────────────────────┘
# Istio multi-cluster setup (primary-remote model)
# On primary cluster
istioctl install --set profile=default \
  --set values.pilot.env.EXTERNAL_ISTIOD=true
# Create remote secret (allow primary to manage remote)
istioctl create-remote-secret \
  --name=remote-cluster \
  --context=remote-cluster-context | \
  kubectl apply -f - --context=primary-cluster-context

Observability Stack

Envoy Sidecars (metrics + traces)
        ↓
Prometheus (scrape metrics)
        ↓              ↓
  Grafana          Jaeger / Zipkin
(dashboards)    (distributed tracing)
        ↓
  Kiali (service graph + health)
        ↓
  Alertmanager → PagerDuty / Slack

Golden Signal Dashboards (auto from mesh)

Latency   → P50 / P95 / P99 per service
Traffic   → Requests per second per service
Errors    → 4xx / 5xx rate per service
Saturation → Connection pool usage, CPU

Best Practices

PracticeWhy
Start with Linkerd if new to meshesLower complexity, easier to debug
Enable mTLS mesh-wide from day 1Harder to enforce later
Use PERMISSIVE mode first, then STRICTGradual rollout without breaking traffic
Set resource requests on sidecarsPrevent sidecar starvation
Use AuthorizationPolicy deny-all baselineZero-trust foundation
Monitor sidecar resource usageEnvoy adds ~50-100MB RAM per pod
Use Kiali for topology visibilityCatch unexpected service dependencies
Test with fault injection before prodValidate resilience patterns work
Avoid mesh for batch/job workloadsSidecar overhead not worth it
Use Cilium on performance-critical clusterseBPF = kernel-level, no overhead

Troubleshooting

# Check sidecar injection
kubectl get pod api-pod -n production -o jsonpath='{.spec.containers[*].name}'
# Should show: api envoy (two containers)
# Check mTLS status
istioctl x describe pod api-pod-xxx -n production
# Check Envoy config
istioctl proxy-config cluster api-pod-xxx.production
istioctl proxy-config listener api-pod-xxx.production
istioctl proxy-config route api-pod-xxx.production
# Check authorization policies
istioctl x authz check api-pod-xxx.production
# View Envoy access logs
kubectl logs api-pod-xxx -n production -c istio-proxy
# Debug traffic issues
istioctl analyze -n production    # ← catch misconfigs
# Linkerd debugging
linkerd viz stat deployments -n production
linkerd viz routes deployment/api -n production
linkerd viz edges deployment -n production

A service mesh is the foundation of zero-trust networking inside Kubernetes — mTLS by default, explicit authorization policies, and full observability without touching application code. Start with Linkerd for simplicity or Istio if you need the full feature set.