Best Practices for Kubernetes Namespace Management

May 2, 2026May 3, 2026 techhadoop kubernetes kubernetes

To set up guardrails for a new team, you need a “pincer maneuver.” A LimitRange ensures their containers are sane, and a ResourceQuota ensures the team doesn’t accidentally run up a massive cloud bill or starve other departments of resources.

Here is the blueprint for a “standard” production-ready namespace.

1. Create the Namespace

Always isolate teams into their own namespaces. This is the foundation for all security and resource policies.

Bash

kubectl create namespace team-alpha

2. Apply the LimitRange (Individual Pod Rules)

This prevents “Naked Pods” (pods without resource definitions) and stops developers from requesting a 16-core CPU for a simple web-server.

YAML

			
apiVersion: v1
kind: LimitRange
metadata:
  name: team-alpha-limits
  namespace: team-alpha
spec:
  limits:
  - type: Container
    default:           # Applied if the developer omits 'limits'
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:    # Applied if the developer omits 'requests'
      cpu: "100m"
      memory: "256Mi"
    max:               # Rejects Pods larger than this
      cpu: "2"
      memory: "2Gi"
    min:               # Rejects Pods smaller than this
      cpu: "10m"
      memory: "64Mi"

		

3. Apply the ResourceQuota (Total Team Budget)

This is your safety net for the cluster’s aggregate capacity.

YAML

			
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "4"         # Total CPU requests allowed in namespace
    requests.memory: "8Gi"    # Total RAM requests allowed
    limits.cpu: "10"          # Max burst CPU across all pods
    limits.memory: "16Gi"     # Max burst RAM across all pods
    pods: "20"                # Max number of Pods allowed
    services: "10"            # Max number of Services
    persistentvolumeclaims: "5" # Max number of storage requests

		

4. Best Practices for Onboarding

Enforce PriorityClasses: If this team’s work is non-critical (like a dev environment), you can add a scope to the ResourceQuota so it only counts against “Low Priority” workloads.
Storage Quotas: Don’t forget to limit requests.storage. Without it, a team could provision dozens of expensive 1TB cloud disks.
The “Admission Refused” Talk: Warn your developers! If they try to deploy a Pod that exceeds these limits, Kubernetes will return an error immediately:Error creating: pods "my-pod" is forbidden: maximum cpu usage per Container is 2, but limit is 4.

The “Dry Run” Check

Before handing over the keys, you can test if your guardrails work by trying to run a “greedy” pod:

Bash

kubectl run greedy-pod --image=nginx --namespace=team-alpha --requests='cpu=5'

If set up correctly, Kubernetes should instantly block this command.

Java and Machine Learning (ML) workloads are notoriously “bursty” and memory-heavy. Java’s JVM needs specific heap settings to respect container limits, and ML models can easily OOM (Out of Memory) a node if they aren’t strictly caged.

For these workloads, your guardrails need to be more generous with memory but stricter on CPU to prevent “noisy neighbor” syndrome.

Guardrails for Java & ML

1. The Java-Optimized LimitRange

Java applications often require a larger gap between request and limit because the JVM performs intensive garbage collection and startup routines that spike CPU.

YAML

			
apiVersion: v1
kind: LimitRange
metadata:
  name: java-ml-limits
  namespace: team-alpha
spec:
  limits:
  - type: Container
    default:
      cpu: "1"         # Java needs more than 500m to start efficiently
      memory: "2Gi"    # Base for standard JVM/Spring Boot apps
    defaultRequest:
      cpu: "500m"
      memory: "1Gi"
    max:
      cpu: "4"         # Cap for ML training or high-perf Java
      memory: "16Gi"   # Heavy ML models/large heaps need room
    min:
      cpu: "100m"
      memory: "512Mi"  # Anything lower often causes JVM startup failures

		

2. The ResourceQuota (Aggregated)

Since ML and Java use significantly more RAM than Go or Node.js apps, your total namespace quota needs to reflect that.

YAML

			
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
spec:
  hard:
    requests.cpu: "16"
    requests.memory: "64Gi"
    limits.cpu: "32"
    limits.memory: "128Gi"
    # If using GPUs for ML, you MUST include them here:
    requests.nvidia.com/gpu: "4" 

		

Crucial Tips for Java & ML Teams

The JVM “Cgroup” Awareness

In older versions of Java, the JVM didn’t know it was in a container and would try to claim the memory of the entire Node rather than the Pod, leading to instant crashes.

Ensure the team is using Java 11+ or Java 8u191+.
Advise them to use the flag -XX:+UseContainerSupport (usually on by default now).
They should set -XX:MaxRAMPercentage=75.0 rather than hardcoding heap sizes (Xmx). This ensures the Java Heap scales automatically if you change the LimitRange later.

ML & “Spot” Instances

If the ML team is doing training (not just inference), their Pods will be expensive.

Guardrail: Use PriorityClasses. Create a low-priority class for training jobs.
If the cluster runs out of room for a production Java API, Kubernetes will evict the ML training job first to save the API.

Local Storage for ML

ML workloads often download massive datasets (ImageNet, etc.).

A standard LimitRange doesn’t stop a Pod from filling up the Node’s local disk with emptyDir volumes.
Add to Quota: requests.ephemeral-storage: "50Gi" to prevent a single ML container from crashing the Node by filling the disk.

Understanding ARO’s Kubernetes API Operations

May 1, 2026May 1, 2026 techhadoop AKS, azure, OCP ai, cloud, devops, kubernetes, technology

Kubernetes API Operations Through the ARO Private Endpoint

Every interaction with an ARO cluster — whether from a human, a tool, or an automated controller — flows through a single TCP connection to port 6443 on the API server private endpoint. The API server is the absolute centre of gravity for all cluster operations.

Every Operation Is a REST Call

The Kubernetes API server exposes a RESTful HTTP/2 API over TLS. Every tool — kubectl, oc, operators, kubelet — translates its work into one of five HTTP verbs against a resource path:

			
GET     /api/v1/namespaces/payments/pods          list pods
GET     /api/v1/namespaces/payments/pods/web-1    get single pod
POST    /api/v1/namespaces/payments/pods          create pod
PUT     /api/v1/namespaces/payments/pods/web-1    replace pod
PATCH   /api/v1/namespaces/payments/pods/web-1    partial update
DELETE  /api/v1/namespaces/payments/pods/web-1    delete pod
GET     /api/v1/namespaces/payments/pods?watch=1  watch stream

		

Every one of these travels as TLS-encrypted HTTP/2 to 10.1.0.8:6443.

Category 1 — Human CLI Operations (kubectl + oc)

kubectl — standard Kubernetes operations

			
# Every one of these becomes a REST call through the private endpoint
# LIST pods → GET /api/v1/namespaces/default/pods
kubectl get pods -n payments
# CREATE deployment → POST /apps/v1/namespaces/payments/deployments
kubectl apply -f deployment.yaml
# EXEC into pod → POST + UPGRADE to SPDY/WebSocket
kubectl exec -it web-1 -- /bin/bash
# PORT-FORWARD → POST + WebSocket tunnel
kubectl port-forward svc/my-app 8080:80
# LOGS → GET /api/v1/namespaces/payments/pods/web-1/log
kubectl logs web-1 --follow
# WATCH resources → GET with ?watch=1 (long-lived streaming connection)
kubectl get pods --watch

		

oc CLI — OpenShift-specific additions

oc wraps kubectl completely and adds calls to OpenShift-specific API groups:

			
# OpenShift Route → POST /apis/route.openshift.io/v1/namespaces/.../routes
oc expose svc/my-app
# Project (OpenShift namespace wrapper)
# → POST /apis/project.openshift.io/v1/projectrequests
oc new-project my-team
# ImageStream → GET /apis/image.openshift.io/v1/namespaces/.../imagestreams
oc get imagestreams
# BuildConfig → POST /apis/build.openshift.io/v1/namespaces/.../builds
oc start-build my-app
# DeploymentConfig (legacy OpenShift resource)
# → GET /apis/apps.openshift.io/v1/namespaces/.../deploymentconfigs
oc rollout latest dc/my-app
# SCC inspection → GET /apis/security.openshift.io/v1/securitycontextconstraints
oc get scc

		

Category 2 — Operators and Controllers

Operators are long-running processes inside the cluster that maintain perpetual watch connections to the API server — the busiest category of API consumers by connection count.

The watch loop — how operators work

			
// Every operator runs this pattern against the API server
// Connection: persistent HTTP/2 stream to 10.1.0.8:6443
// 1. LIST — get current state (one-time at startup)
GET /apis/apps/v1/namespaces/payments/deployments
→ Returns: all deployments + resourceVersion: 48291
// 2. WATCH — subscribe to changes (permanent long-poll)
GET /apis/apps/v1/namespaces/payments/deployments?watch=1&resourceVersion=48291
→ Server keeps connection open indefinitely
→ Pushes events as they occur:
   {"type":"MODIFIED","object":{"metadata":{"name":"web"},...}}
   {"type":"ADDED","object":{"metadata":{"name":"worker"},...}}
   {"type":"DELETED","object":{"metadata":{"name":"old"},...}}
// 3. RECONCILE — when event received, fix actual → desired state
PATCH /apis/apps/v1/namespaces/payments/replicasets/web-abc
→ Creates/deletes pods to match desired replicas
// 4. STATUS UPDATE — write observed state back
PATCH /apis/apps/v1/namespaces/payments/deployments/web/status
→ {"observedGeneration": 5, "availableReplicas": 3}

		

Built-in OpenShift operators that run this loop continuously

Operator	What it watches	What it does
`openshift-apiserver-operator`	`apiservers.config.openshift.io`	Manages API server config and certs
`cluster-version-operator`	`clusterversions.config.openshift.io`	Drives cluster upgrades
`machine-config-operator`	`machineconfigs`, `machineconfigpools`	Applies RHCOS config to nodes
`ingress-operator`	`ingresses.config.openshift.io`	Manages router deployments
`dns-operator`	`dnses.config.openshift.io`	Manages CoreDNS config
`network-operator`	`networks.config.openshift.io`	Manages OVN-Kubernetes
`image-registry-operator`	`configs.imageregistry.operator.openshift.io`	Manages internal registry
`authentication-operator`	`authentications.config.openshift.io`	Manages OAuth server

Every one of these has persistent watch connections open to the API server at all times — a healthy ARO cluster typically has 40–80 active watch streams running 24/7.

Category 3 — Kubelet (Node Agent)

Every worker node runs a kubelet process that maintains its own connection to the API server — reporting node health and receiving pod assignments:

			
Worker node kubelet → 10.1.0.8:6443
Outbound (kubelet → API server):
  POST /api/v1/nodes/worker-1/status          every 10 seconds — node heartbeat
  PATCH /api/v1/namespaces/app/pods/web-1/status  when pod state changes
  POST /api/v1/events                         kubelet events (OOM, image pull)
Inbound (API server → kubelet port 10250):
  GET  https://worker-1:10250/exec/...        kubectl exec forwarding
  GET  https://worker-1:10250/log/...         kubectl logs forwarding
  GET  https://worker-1:10250/metrics         Prometheus scraping

		

If the kubelet loses its connection to the API server for more than the node-monitor-grace-period (default 40 seconds), the node is marked NotReady and pods begin eviction.

Category 4 — CI/CD Pipelines

Self-hosted CI/CD runners inside the VNet authenticate to the API server using a service account token:

			
# Service account for CI/CD — scoped to specific namespace
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cicd-deployer
  namespace: payments
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: payments
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods", "services", "configmaps"]
    verbs: ["get", "list", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cicd-deployer-binding
  namespace: payments
roleRef:
  kind: Role
  name: deployer
subjects:
  - kind: ServiceAccount
    name: cicd-deployer
    namespace: payments

		

GitHub Actions pipeline using this service account:

			
- name: Deploy to ARO
  run: |
    # Authenticate with service account token — all traffic to 10.1.0.8:6443
    oc login ${{ secrets.ARO_API_URL }} \
      --token ${{ secrets.CICD_SA_TOKEN }}
    # Each command = REST call through private endpoint
    oc set image deployment/web \
      web=acrprod.azurecr.io/my-app:${{ github.sha }} \
      -n payments
    oc rollout status deployment/web -n payments

		

Category 5 — Admission Webhooks

Admission webhooks add an external hop during the API server request pipeline — the API server calls out to your webhook service before persisting any object:

			
kubectl apply -f pod.yaml
        ↓
API server receives POST /api/v1/namespaces/payments/pods
        ↓
Authn + RBAC pass
        ↓
Mutating admission webhook:
  API server → POST https://gatekeeper-webhook.gatekeeper-system.svc:443/mutate
  Webhook adds labels, sets resource limits, injects sidecars
  → Returns mutated pod spec
        ↓
Validating admission webhook:
  API server → POST https://gatekeeper-webhook.gatekeeper-system.svc:443/validate
  Checks policy: must have resource limits, no root, valid image registry
  → Returns: allowed: true (or denied with reason)
        ↓
Persist to etcd → notify watchers → return 201 Created

		

Common admission webhooks in ARO:

Webhook	Purpose
OPA Gatekeeper	Policy enforcement — block non-compliant resources
Kyverno	Policy as code — mutate, validate, generate
Istio / OpenShift Service Mesh	Inject Envoy sidecar into pods automatically
Red Hat ACM	Multi-cluster governance policies
Cert-manager	Inject TLS certificates into resources

Category 6 — Monitoring and Observability

			
# Prometheus scrapes API server metrics via the API endpoint
GET https://10.1.0.8:6443/metrics
# Returns: apiserver_request_total, apiserver_request_duration_seconds,
#          etcd_request_duration_seconds, workqueue_depth, ...
# Health endpoints checked by Azure ARO service monitor
GET https://10.1.0.8:6443/healthz    → "ok"
GET https://10.1.0.8:6443/readyz     → "ok"
GET https://10.1.0.8:6443/livez      → "ok"
# OpenShift console reads cluster state continuously
GET /apis/config.openshift.io/v1/clusterversions/version
GET /api/v1/namespaces?limit=500
GET /apis/project.openshift.io/v1/projects

		

The Request Pipeline — What Happens Inside

Every request through the private endpoint traverses this exact pipeline inside kube-apiserver:

			
TLS handshake on 10.1.0.8:6443
        ↓
1. AUTHENTICATION — who are you?
   • OIDC token (Entra ID) → extract user + groups
   • x509 client cert → extract CN as username
   • Bearer token → look up service account
   • Failure → 401 Unauthorized
2. AUTHORIZATION (RBAC) — are you allowed?
   • Check: user + groups + verb + resource + namespace
   • ClusterRoleBinding / RoleBinding lookup
   • OpenShift SCC evaluation for pods
   • Failure → 403 Forbidden
3. ADMISSION CONTROL — is this allowed by policy?
   • Mutating webhooks (modify the object)
   • Built-in admission plugins (ResourceQuota, LimitRanger)
   • Validating webhooks (accept or reject)
   • Failure → 400/403 with reason
4. VALIDATION — is the object schema correct?
   • OpenAPI schema validation
   • CRD schema validation
   • Field immutability checks
   • Failure → 422 Unprocessable Entity
5. PERSIST TO etcd
   • Serialise to protobuf
   • Encrypt at rest (AES-GCM, ARO managed)
   • Write to etcd with optimistic concurrency (resourceVersion)
   • Failure → 409 Conflict (resourceVersion mismatch)
6. NOTIFY WATCHERS
   • Push event to all active watch streams matching the resource
   • Controllers, operators, scheduler, kubelet all receive notification
7. RETURN RESPONSE
   • 200 OK (GET)
   • 201 Created (POST)
   • 200 OK with updated object (PATCH/PUT)
   • 404 Not Found
   • Streaming response for watch/exec/logs/port-forward

		

API Groups — Kubernetes vs OpenShift

The API server serves two parallel API surfaces — Kubernetes core APIs and OpenShift extension APIs — all through the same 10.1.0.8:6443 endpoint:

			
Kubernetes core APIs:
  /api/v1/                          pods, services, configmaps, secrets, nodes
  /apis/apps/v1/                    deployments, replicasets, statefulsets, daemonsets
  /apis/batch/v1/                   jobs, cronjobs
  /apis/rbac.authorization.k8s.io/  clusterroles, rolebindings
  /apis/storage.k8s.io/             storageclasses, persistentvolumes
  /apis/networking.k8s.io/          ingresses, networkpolicies
OpenShift extension APIs:
  /apis/route.openshift.io/         routes (OpenShift ingress primitive)
  /apis/project.openshift.io/       projects (namespace + RBAC wrapper)
  /apis/build.openshift.io/         buildconfigs, builds
  /apis/image.openshift.io/         imagestreams, imagestreamtags
  /apis/apps.openshift.io/          deploymentconfigs (legacy)
  /apis/security.openshift.io/      securitycontextconstraints
  /apis/config.openshift.io/        cluster-wide config (DNS, network, auth)
  /apis/operator.openshift.io/      operator configuration resources
  /apis/machine.openshift.io/       machines, machinesets (MachineAPI)

		

Key Takeaway

The ARO API server private endpoint at 10.1.0.8:6443 is not just the entry point for human CLI commands — it is the nervous system of the entire cluster. Every automated process — the 40+ built-in OpenShift operators maintaining cluster state, every kubelet heartbeating from every worker node every 10 seconds, every CI/CD deployment, every admission webhook validation, every Prometheus health check — flows through this single TLS endpoint. Making it private eliminates the internet attack surface entirely, while the seven-stage request pipeline inside the API server ensures every operation is authenticated, authorised, policy-checked, validated, and durably persisted before any response is returned.

Best Practices for OpenShift on Azure: ARO Guide

April 30, 2026April 30, 2026 techhadoop Uncategorized azure, cloud, devops, kubernetes, technology

OpenShift Container Platform on Azure — ARO Best Practices

Azure Red Hat OpenShift (ARO) is a fully managed OpenShift 4 service jointly operated by Microsoft and Red Hat — both companies share responsibility for the control plane, infrastructure, and SLA (99.95%).

1. Networking Best Practices

Always deploy a private cluster

A private ARO cluster hides the Kubernetes API server behind a private endpoint — no public IP, unreachable from the internet:

az aro create \
  --resource-group rg-aro \
  --name aro-prod \
  --vnet aro-spoke-vnet \
  --master-subnet master-subnet \
  --worker-subnet worker-subnet \
  --apiserver-visibility Private \      # ← API server private
  --ingress-visibility Private \        # ← ingress private
  --pull-secret @pull-secret.txt

Access to the private API server is then through Azure Bastion → jump host, or over ExpressRoute/VPN from on-premises.

Subnet sizing — get this right before deployment (cannot resize after)

ARO consumes IP addresses aggressively — every pod gets its own IP from the node’s subnet range:

Subnet	Minimum	Recommended	Notes
Master subnet	/27	/24	Fixed 3 masters — needs room for Azure infra IPs
Worker subnet	/27	/23 or /22	Every pod consumes an IP — size generously
Ingress subnet	/28	/27	For LB / App Gateway front-end IPs
Private endpoints	/28	/27	One IP per private endpoint

Worker subnet sizing example:
  /23 = 512 addresses
  Azure reserves 5
  Available: 507
  Max pods per node: 250 (default OpenShift SDN)
  Nodes supportable: ~2 per node × workers
  Plan for: 3× current need for growth headroom

Egress lockdown via Azure Firewall

ARO requires outbound internet access for Red Hat update servers, telemetry, and pull.registry.redhat.io. Lock this down with Azure Firewall application rules rather than allowing all outbound:

			
Azure Firewall Application Rules for ARO egress:
  ┌─────────────────────────────────────────────────────────┐
  │ Name                    Target FQDN                     │
  ├─────────────────────────────────────────────────────────┤
  │ aro-rh-registry         registry.redhat.io              │
  │                         registry.access.redhat.com      │
  │                         quay.io                         │
  │                         cdn.quay.io                     │
  ├─────────────────────────────────────────────────────────┤
  │ aro-azure-services      *.blob.core.windows.net         │
  │                         *.servicebus.windows.net        │
  │                         *.table.core.windows.net        │
  ├─────────────────────────────────────────────────────────┤
  │ aro-monitoring          *.ods.opinsights.azure.com      │
  │                         *.oms.opinsights.azure.com      │
  ├─────────────────────────────────────────────────────────┤
  │ aro-rh-telemetry        cert-api.access.redhat.com      │
  │                         api.access.redhat.com           │
  └─────────────────────────────────────────────────────────┘

		

Apply a UDR on the master and worker subnets pointing 0.0.0.0/0 to the Azure Firewall private IP — same hub and spoke pattern as any spoke workload.

Use a custom DNS server

Point the ARO VNet DNS to your hub DNS Private Resolver so cluster nodes can resolve private endpoints and internal domains:

az network vnet update \
  --resource-group rg-aro-network \
  --name aro-spoke-vnet \
  --dns-servers 10.0.5.4    # DNS Private Resolver inbound endpoint IP

2. Availability and Resilience Best Practices

Spread across all three Availability Zones

ARO deploys 3 master nodes — one per AZ automatically. Workers must be explicitly spread via MachineSets:

# MachineSet for AZ1 — replicate for AZ2, AZ3
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: aro-prod-worker-eastus-1
  namespace: openshift-machine-api
spec:
  replicas: 3
  template:
    spec:
      providerSpec:
        value:
          zone: "1"                         # AZ1
          vmSize: Standard_D16s_v3
          osDisk:
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS

Create three MachineSets — one per zone — with equal replica counts. This ensures workloads survive a full AZ failure.

Enable cluster autoscaler

apiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default
spec:
  resourceLimits:
    maxNodesTotal: 24
  scaleDown:
    enabled: true
    delayAfterAdd: 10m
    delayAfterDelete: 5m
    delayAfterFailure: 30s
---
apiVersion: autoscaling.openshift.io/v1beta1
kind: MachineAutoscaler
metadata:
  name: aro-prod-worker-eastus-1
  namespace: openshift-machine-api
spec:
  minReplicas: 3
  maxReplicas: 8
  scaleTargetRef:
    kind: MachineSet
    name: aro-prod-worker-eastus-1

Use zone-redundant storage for persistent volumes

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-premium-zrs
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_ZRS       # Zone-redundant storage — survives AZ failure
  cachingMode: ReadOnly
reclaimPolicy: Retain        # Retain on PVC delete — prevents data loss
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Use Premium_ZRS instead of Premium_LRS for stateful workloads — ZRS replicates the disk synchronously across three AZs so a pod can reschedule to another zone without losing its data.

3. Security Best Practices

Use Workload Identity (pod-level Azure RBAC)

Never put Azure credentials in pods. Use Workload Identity to give individual pods an Azure AD identity with scoped RBAC permissions:

# Enable workload identity on ARO cluster
az aro update \
  --resource-group rg-aro \
  --name aro-prod \
  --enable-managed-identity

# Create a managed identity for a specific workload
az identity create \
  --resource-group rg-aro-workloads \
  --name id-payment-service

# Grant it only what it needs
az role assignment create \
  --assignee &lt;identity-client-id&gt; \
  --role "Key Vault Secrets User" \
  --scope /subscriptions/.../vaults/kv-prod

# Annotate the service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service-sa
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "&lt;managed-identity-client-id&gt;"

Integrate Azure Key Vault for secrets via CSI driver

Never store secrets in OpenShift Secrets (base64 is not encryption). Use the Secrets Store CSI driver to mount Key Vault secrets directly into pods:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-secrets
  namespace: payments
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "&lt;managed-identity-client-id&gt;"
    keyvaultName: kv-prod
    tenantID: "&lt;tenant-id&gt;"
    objects: |
      array:
        - |
          objectName: db-connection-string
          objectType: secret
        - |
          objectName: api-key
          objectType: secret

Integrate with Azure Container Registry via private endpoint

# Create ACR with private endpoint — no public access
az acr create \
  --resource-group rg-aro \
  --name acrprodaro \
  --sku Premium \
  --public-network-enabled false

# Private endpoint in ARO spoke
az network private-endpoint create \
  --name pe-acr-prod \
  --resource-group rg-aro-network \
  --vnet-name aro-spoke-vnet \
  --subnet private-endpoint-subnet \
  --private-connection-resource-id $(az acr show --name acrprodaro --query id -o tsv) \
  --group-id registry \
  --connection-name pe-acr-conn

# Grant ARO pull access
az role assignment create \
  --assignee &lt;aro-kubelet-identity&gt; \
  --role AcrPull \
  --scope $(az acr show --name acrprodaro --query id -o tsv)

Apply OpenShift Security Context Constraints (SCC)

Never run pods as root. Use the restricted-v2 SCC (default in OpenShift 4.11+):

apiVersion: v1
kind: Pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        readOnlyRootFilesystem: true

Enable Microsoft Defender for Containers

az security pricing create \
  --name Containers \
  --tier Standard

Defender for Containers provides runtime threat detection, vulnerability scanning for images in ACR, and Kubernetes audit log analysis — all surfaced in Microsoft Defender for Cloud.

4. Observability Best Practices

Forward logs to Azure Monitor / Log Analytics

# Enable container insights on ARO
az aro update \
  --resource-group rg-aro \
  --name aro-prod \
  --enable-managed-identity

# Deploy the monitoring add-on via Helm
helm repo add microsoft https://microsoft.github.io/charts/repo
helm install azuremonitor-containers \
  microsoft/azuremonitor-containers \
  --set omsagent.secret.wsid=&lt;workspace-id&gt; \
  --set omsagent.secret.key=&lt;workspace-key&gt; \
  --namespace kube-system

Use Azure Monitor alerts for cluster health

Alert	Metric	Threshold
Node CPU pressure	`cpuUsageNanoCores`	> 85% for 5 min
Node memory pressure	`memoryWorkingSetBytes`	> 80% of capacity
Pod restart loop	`restartCount`	> 5 in 10 min
PVC near full	`pvUsedBytes`	> 85% of capacity
Node not ready	`nodeCondition`	NotReady > 2 min

5. Day-2 Operations Best Practices

Cluster upgrade strategy

ARO manages the control plane upgrade automatically — you control timing for worker nodes:

# Check available upgrade versions
az aro get-upgrade-versions \
  --resource-group rg-aro \
  --name aro-prod

# Schedule upgrade in maintenance window
az aro update \
  --resource-group rg-aro \
  --name aro-prod \
  --version 4.14.12

Use the EUS (Extended Update Support) channel for production clusters — it allows staying on a minor version for up to 18 months while still receiving security patches, avoiding the churn of mandatory minor version upgrades every 45 days.

Worker node upgrade — use surge capacity

# MachineConfigPool surge upgrade strategy
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker
spec:
  maxUnavailable: 1          # Upgrade one node at a time

Upgrade workers one node at a time to maintain application availability — pods are gracefully drained before each node reboots into the new RHCOS version.

Summary — ARO Best Practice Checklist

Category	Practice
Network	Private cluster — no public API or ingress
Network	Egress via Azure Firewall with FQDN allow-list
Network	DNS Private Resolver for private endpoint resolution
Network	Worker subnet `/22` or larger — never resize after
Availability	Workers spread across AZs via 3 MachineSets
Availability	Cluster autoscaler min 3 per zone
Availability	Premium_ZRS disks for stateful workloads
Availability	Zone-redundant Azure Load Balancer
Security	Workload Identity — no credentials in pods
Security	Key Vault + CSI driver — no base64 secrets
Security	ACR via private endpoint — no public pull
Security	SCC restricted-v2 — no root containers
Security	Defender for Containers enabled
Observability	Container Insights → Log Analytics
Observability	Azure Monitor alerts on node and pod health
Operations	EUS channel for production stability
Operations	maxUnavailable: 1 for worker upgrades

MCP Operations Server: AI-Enabled Managed Ops Explained

April 25, 2026April 25, 2026 techhadoop ai, AKS, azure ai, cloud, devops, kubernetes, technology

To bridge your local Python code to a production-ready AKS environment, you need a Dockerfile that doesn’t just run the code, but does so securely and efficiently.

By 2026, the standard for MCP servers in production is to move away from STDIO (local command line) and use SSE (Server-Sent Events) over HTTP. This allows your AI agents to talk to the server over a network.

1. The Production Dockerfile

This Dockerfile uses a “non-root” user (security best practice) and installs the necessary drivers to talk to the Docker socket or Kubernetes API.

Dockerfile

			
# Use a lightweight Python 2026-ready base image
FROM python:3.12-slim
# Install system dependencies (curl for health checks)
RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*
# Set working directory
WORKDIR /app
# Create a non-root user for security
RUN groupadd -r mcpuser && useradd -r -g mcpuser mcpuser
# Copy requirements and install
# Note: includes 'mcp[cli]' for server capabilities
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy server code
COPY server.py .
# Give our non-root user access to the app folder
RUN chown -r mcpuser:mcpuser /app
USER mcpuser
# Expose the port for SSE/HTTP transport (Standard for 2026)
EXPOSE 8000
# Start the server using the FastMCP production runner
CMD ["python", "server.py", "--transport", "sse", "--port", "8000"]

		

2. The `requirements.txt`

You’ll need these specific libraries:

Plaintext

			
fastmcp>=1.0.0
docker>=7.0.0
kubernetes>=30.0.0
uvicorn # Required for high-performance HTTP transport

3. Deploying to AKS (The “Support” Strategy)

When you deploy this to your client’s AKS cluster, you’ll use a standard Kubernetes Deployment.

Why this is better for your role:

Scaling: If the dev team grows, you can scale the MCP server to 3 replicas so the AI assistant never lags.
Security: Instead of sharing your personal kubeconfig, the MCP server uses a ServiceAccount with “View Only” permissions. This means the AI can see the logs but can’t accidentally delete the production database.

4. How to Pitch the “AI Operations” Tier

You can now offer a new support tier called “AI-Enabled Managed Ops”:

“I’ve built a custom MCP Operations Server for our cluster. It allows our internal AI agents to perform health checks, retrieve logs, and analyze container stats using natural language. This doesn’t replace me; it allows me to respond to your requests 10x faster because the AI is doing the ‘data gathering’ for me inside our secure perimeter.”

One final piece of the puzzle

To make this work in AKS, the pod needs permission to “see” the other pods.

To finish the MCP server integration on AKS, you need to grant the pod the right permissions to “talk” to the Kubernetes API.

If you don’t do this, the AI will be “blind”—it will try to list pods and get a 403 Forbidden error.

1. The RBAC Strategy

We will use three Kubernetes objects:

ServiceAccount: The identity for your MCP pod.
ClusterRole: A set of rules that allow “Viewing” (reading pods, logs, and events).
ClusterRoleBinding: The “glue” that attaches the Role to the ServiceAccount.

2. The RBAC YAML (`mcp-rbac.yaml`)

YAML

			
# 1. The Identity
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mcp-server-sa
  namespace: default
---
# 2. The Permissions (Read-Only/Viewer)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mcp-pod-viewer
rules:
- apiGroups: [""]
  # Accessing 'pods' for list/get, and 'pods/log' specifically for tracing
  resources: ["pods", "pods/log", "pods/status", "events", "nodes", "services"]
  verbs: ["get", "list", "watch"]
---
# 3. The Connection
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: mcp-server-binding
subjects:
- kind: ServiceAccount
  name: mcp-server-sa
  namespace: default
roleRef:
  kind: ClusterRole
  name: mcp-pod-viewer
  apiGroup: rbac.authorization.k8s.io

		

3. Updating your Deployment

Finally, ensure your MCP server deployment uses this serviceAccountName:

YAML

			
spec:
  template:
    spec:
      serviceAccountName: mcp-server-sa
      containers:
      - name: mcp-server
        image: your-mcp-image:latest
        # ... other config ...

		

4. Why this is “Safe” for your Client

When you explain this to the company, emphasize these three points:

Namespace Scoping: Even though it’s a ClusterRole, you can swap it for a Role if you only want the AI to see specific namespaces (e.g., only production-frontend).
No Secrets Access: Notice that secrets is not in the list of resources. The AI literally cannot see the database passwords, even if it tries.
Auditability: Every time the AI agent asks the MCP server for logs, Kubernetes logs that action under the mcp-server-sa identity. You have a perfect audit trail.

Putting it all together

You now have the Terraform for infrastructure, the Python for the server, the Docker for the container, and the RBAC for security.

You’re ready to pitch this as a “Self-Healing AI Operations Layer.”

Automate Velero AKS Backups with Terraform and Ansible

April 25, 2026April 25, 2026 techhadoop AKS, azure, Uncategorized ai, azure, cloud, kubernetes, technology

Automating Velero AKS Backup with Ansible & Terraform

Option 1: Terraform

Project Structure

			
velero-terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
└── modules/
    ├── storage/
    │   ├── main.tf
    │   └── variables.tf
    ├── identity/
    │   ├── main.tf
    │   └── variables.tf
    └── velero/
        ├── main.tf
        └── variables.tf

		

providers.tf

			
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
}
provider "azurerm" {
  features {}
}
provider "helm" {
  kubernetes {
    host                   = azurerm_kubernetes_cluster.aks.kube_config.0.host
    client_certificate     = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)
    client_key             = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)
    cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)
  }
}
provider "kubernetes" {
  host                   = azurerm_kubernetes_cluster.aks.kube_config.0.host
  client_certificate     = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)
  client_key             = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)
}

		

variables.tf

			
variable "resource_group_name" {
  description = "Resource group name"
  type        = string
  default     = "myResourceGroup"
}
variable "location" {
  description = "Azure region"
  type        = string
  default     = "eastus"
}
variable "aks_cluster_name" {
  description = "AKS cluster name"
  type        = string
  default     = "myAKSCluster"
}
variable "storage_account_name" {
  description = "Storage account for Velero backups"
  type        = string
  default     = "velerobackupstorage"
}
variable "blob_container_name" {
  description = "Blob container for Velero backups"
  type        = string
  default     = "velero-backups"
}
variable "velero_namespace" {
  description = "Kubernetes namespace for Velero"
  type        = string
  default     = "velero"
}
variable "backup_retention_hours" {
  description = "Backup TTL in hours"
  type        = number
  default     = 720  # 30 days
}
variable "backup_schedule" {
  description = "Cron schedule for backups"
  type        = string
  default     = "0 2 * * *"  # Daily at 2am
}

		

main.tf

			
# Resource Group
resource "azurerm_resource_group" "rg" {
  name     = var.resource_group_name
  location = var.location
}
# ── STORAGE ──────────────────────────────────────────────
resource "azurerm_storage_account" "velero" {
  name                     = var.storage_account_name
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  blob_properties {
    delete_retention_policy {
      days = 30
    }
  }
  tags = {
    purpose = "velero-backup"
  }
}
resource "azurerm_storage_container" "velero" {
  name                  = var.blob_container_name
  storage_account_name  = azurerm_storage_account.velero.name
  container_access_type = "private"
}
# ── IDENTITY ─────────────────────────────────────────────
resource "azurerm_user_assigned_identity" "velero" {
  name                = "velero-identity"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
}
# Assign Contributor role to storage account
resource "azurerm_role_assignment" "velero_storage" {
  scope                = azurerm_storage_account.velero.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_user_assigned_identity.velero.principal_id
}
# Assign Contributor role to resource group (for disk snapshots)
resource "azurerm_role_assignment" "velero_rg" {
  scope                = azurerm_resource_group.rg.id
  role_definition_name = "Contributor"
  principal_id         = azurerm_user_assigned_identity.velero.principal_id
}
# ── AKS CLUSTER ──────────────────────────────────────────
resource "azurerm_kubernetes_cluster" "aks" {
  name                = var.aks_cluster_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = var.aks_cluster_name
  default_node_pool {
    name       = "default"
    node_count = 2
    vm_size    = "Standard_D2_v2"
  }
  identity {
    type = "SystemAssigned"
  }
  tags = {
    environment = "production"
  }
}
# ── VELERO NAMESPACE ─────────────────────────────────────
resource "kubernetes_namespace" "velero" {
  metadata {
    name = var.velero_namespace
  }
  depends_on = [azurerm_kubernetes_cluster.aks]
}
# ── VELERO CREDENTIALS SECRET ────────────────────────────
resource "kubernetes_secret" "velero_credentials" {
  metadata {
    name      = "velero-credentials"
    namespace = kubernetes_namespace.velero.metadata[0].name
  }
  data = {
    "cloud" = <<EOF
AZURE_SUBSCRIPTION_ID=${data.azurerm_subscription.current.subscription_id}
AZURE_TENANT_ID=${data.azurerm_subscription.current.tenant_id}
AZURE_CLIENT_ID=${azurerm_user_assigned_identity.velero.client_id}
AZURE_RESOURCE_GROUP=${azurerm_resource_group.rg.name}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF
  }
}
# Current subscription data
data "azurerm_subscription" "current" {}
# ── VELERO HELM RELEASE ──────────────────────────────────
resource "helm_release" "velero" {
  name       = "velero"
  repository = "https://vmware-tanzu.github.io/helm-charts"
  chart      = "velero"
  namespace  = kubernetes_namespace.velero.metadata[0].name
  version    = "5.0.0"
  values = [
    yamlencode({
      configuration = {
        provider = "azure"
        backupStorageLocation = {
          name     = "default"
          provider = "velero.io/azure"
          bucket   = azurerm_storage_container.velero.name
          config = {
            resourceGroup      = azurerm_resource_group.rg.name
            storageAccount     = azurerm_storage_account.velero.name
            subscriptionId     = data.azurerm_subscription.current.subscription_id
          }
        }
        volumeSnapshotLocation = {
          name     = "default"
          provider = "velero.io/azure"
          config = {
            resourceGroup  = azurerm_resource_group.rg.name
            subscriptionId = data.azurerm_subscription.current.subscription_id
          }
        }
      }
      credentials = {
        existingSecret = kubernetes_secret.velero_credentials.metadata[0].name
      }
      initContainers = [
        {
          name            = "velero-plugin-for-azure"
          image           = "velero/velero-plugin-for-microsoft-azure:v1.8.0"
          imagePullPolicy = "IfNotPresent"
          volumeMounts = [
            {
              mountPath = "/target"
              name      = "plugins"
            }
          ]
        }
      ]
      schedules = {
        daily-backup = {
          schedule = var.backup_schedule
          template = {
            ttl                      = "${var.backup_retention_hours}h0m0s"
            includeClusterResources  = true
            excludedNamespaces       = ["kube-system", "velero"]
          }
        }
      }
    })
  ]
  depends_on = [
    kubernetes_secret.velero_credentials,
    azurerm_role_assignment.velero_storage,
    azurerm_role_assignment.velero_rg
  ]
}

		

outputs.tf

			
output "storage_account_name" {
  value = azurerm_storage_account.velero.name
}
output "blob_container_name" {
  value = azurerm_storage_container.velero.name
}
output "velero_identity_client_id" {
  value = azurerm_user_assigned_identity.velero.client_id
}
output "aks_cluster_name" {
  value = azurerm_kubernetes_cluster.aks.name
}
output "kube_config" {
  value     = azurerm_kubernetes_cluster.aks.kube_config_raw
  sensitive = true
}

		

Deploy with Terraform

			
# Initialize
terraform init
# Preview changes
terraform plan -out=tfplan
# Apply
terraform apply tfplan
# Get kubeconfig
terraform output -raw kube_config > ~/.kube/config
# Verify Velero
kubectl get pods -n velero
velero backup-location get

		

Option 2: Ansible

Project Structure

			
velero-ansible/
├── inventory/
│   └── hosts.yml
├── group_vars/
│   └── all.yml
├── roles/
│   ├── azure_storage/
│   │   └── tasks/
│   │       └── main.yml
│   ├── azure_identity/
│   │   └── tasks/
│   │       └── main.yml
│   └── velero/
│       ├── tasks/
│       │   └── main.yml
│       └── templates/
│           ├── credentials.j2
│           └── backup-schedule.yml.j2
└── site.yml

		

group_vars/all.yml

			
# Azure Settings
resource_group: "myResourceGroup"
location: "eastus"
aks_cluster_name: "myAKSCluster"
# Storage Settings
storage_account_name: "velerobackupstorage"
blob_container_name: "velero-backups"
# Velero Settings
velero_namespace: "velero"
velero_version: "v1.12.0"
velero_azure_plugin_version: "v1.8.0"
velero_chart_version: "5.0.0"
# Backup Settings
backup_schedule: "0 2 * * *"
backup_ttl: "720h"
backup_name: "daily-backup"
# Namespaces to exclude
excluded_namespaces:
  - kube-system
  - velero

		

roles/azure_storage/tasks/main.yml

			
---
- name: Create Resource Group
  azure.azcollection.azure_rm_resourcegroup:
    name: "{{ resource_group }}"
    location: "{{ location }}"
    state: present
- name: Create Storage Account
  azure.azcollection.azure_rm_storageaccount:
    resource_group: "{{ resource_group }}"
    name: "{{ storage_account_name }}"
    type: Standard_LRS
    kind: StorageV2
    state: present
  register: storage_account_result
- name: Create Blob Container
  azure.azcollection.azure_rm_storageblob:
    resource_group: "{{ resource_group }}"
    storage_account_name: "{{ storage_account_name }}"
    container: "{{ blob_container_name }}"
    state: present
- name: Get Storage Account Keys
  azure.azcollection.azure_rm_storageaccount_info:
    resource_group: "{{ resource_group }}"
    name: "{{ storage_account_name }}"
  register: storage_info
- name: Set Storage Key Fact
  set_fact:
    storage_account_key: "{{ storage_info.storageaccounts[0].primary_endpoints.key }}"

		

roles/azure_identity/tasks/main.yml

			
---
- name: Get Azure Subscription Info
  azure.azcollection.azure_rm_subscription_info:
  register: subscription_info
- name: Set Subscription Facts
  set_fact:
    subscription_id: "{{ subscription_info.subscriptions[0].subscription_id }}"
    tenant_id: "{{ subscription_info.subscriptions[0].tenant_id }}"
- name: Create Service Principal for Velero
  azure.azcollection.azure_rm_adserviceprincipal:
    app_id: "velero-sp"
    state: present
  register: sp_result
- name: Assign Contributor Role to Service Principal
  azure.azcollection.azure_rm_roleassignment:
    scope: "/subscriptions/{{ subscription_id }}/resourceGroups/{{ resource_group }}"
    assignee_object_id: "{{ sp_result.object_id }}"
    role_definition_name: Contributor
    state: present
- name: Assign Storage Blob Contributor Role
  azure.azcollection.azure_rm_roleassignment:
    scope: "/subscriptions/{{ subscription_id }}/resourceGroups/{{ resource_group }}/providers/Microsoft.Storage/storageAccounts/{{ storage_account_name }}"
    assignee_object_id: "{{ sp_result.object_id }}"
    role_definition_name: "Storage Blob Data Contributor"
    state: present

		

roles/velero/templates/credentials.j2

			
AZURE_SUBSCRIPTION_ID={{ subscription_id }}
AZURE_TENANT_ID={{ tenant_id }}
AZURE_CLIENT_ID={{ client_id }}
AZURE_CLIENT_SECRET={{ client_secret }}
AZURE_RESOURCE_GROUP={{ resource_group }}
AZURE_CLOUD_NAME=AzurePublicCloud

		

roles/velero/templates/backup-schedule.yml.j2

			
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: {{ backup_name }}
  namespace: {{ velero_namespace }}
spec:
  schedule: "{{ backup_schedule }}"
  template:
    ttl: "{{ backup_ttl }}"
    includeClusterResources: true
    excludedNamespaces:
{% for ns in excluded_namespaces %}
      - {{ ns }}
{% endfor %}

		

roles/velero/tasks/main.yml

			
---
- name: Create Velero Namespace
  kubernetes.core.k8s:
    name: "{{ velero_namespace }}"
    api_version: v1
    kind: Namespace
    state: present
- name: Create Velero Credentials File
  template:
    src: credentials.j2
    dest: /tmp/credentials-velero
    mode: '0600'
- name: Create Kubernetes Secret for Velero Credentials
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Secret
      metadata:
        name: velero-credentials
        namespace: "{{ velero_namespace }}"
      stringData:
        cloud: |
          AZURE_SUBSCRIPTION_ID={{ subscription_id }}
          AZURE_TENANT_ID={{ tenant_id }}
          AZURE_CLIENT_ID={{ client_id }}
          AZURE_CLIENT_SECRET={{ client_secret }}
          AZURE_RESOURCE_GROUP={{ resource_group }}
          AZURE_CLOUD_NAME=AzurePublicCloud
- name: Add Velero Helm Repository
  kubernetes.core.helm_repository:
    name: vmware-tanzu
    repo_url: "https://vmware-tanzu.github.io/helm-charts"
    state: present
- name: Install Velero via Helm
  kubernetes.core.helm:
    name: velero
    chart_ref: vmware-tanzu/velero
    chart_version: "{{ velero_chart_version }}"
    namespace: "{{ velero_namespace }}"
    state: present
    values:
      configuration:
        provider: azure
        backupStorageLocation:
          name: default
          provider: velero.io/azure
          bucket: "{{ blob_container_name }}"
          config:
            resourceGroup: "{{ resource_group }}"
            storageAccount: "{{ storage_account_name }}"
            subscriptionId: "{{ subscription_id }}"
        volumeSnapshotLocation:
          name: default
          provider: velero.io/azure
          config:
            resourceGroup: "{{ resource_group }}"
            subscriptionId: "{{ subscription_id }}"
      credentials:
        existingSecret: velero-credentials
      initContainers:
        - name: velero-plugin-for-azure
          image: "velero/velero-plugin-for-microsoft-azure:{{ velero_azure_plugin_version }}"
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /target
              name: plugins
- name: Wait for Velero Pod to be Ready
  kubernetes.core.k8s_info:
    kind: Pod
    namespace: "{{ velero_namespace }}"
    label_selectors:
      - app.kubernetes.io/name=velero
  register: velero_pod
  until: velero_pod.resources[0].status.phase == "Running"
  retries: 10
  delay: 15
- name: Apply Backup Schedule
  kubernetes.core.k8s:
    state: present
    template: backup-schedule.yml.j2
- name: Verify Backup Location
  command: velero backup-location get
  register: backup_location_status
  changed_when: false
- name: Display Backup Location Status
  debug:
    msg: "{{ backup_location_status.stdout }}"
- name: Clean Up Credentials File
  file:
    path: /tmp/credentials-velero
    state: absent

		

site.yml (Main Playbook)

			
---
- name: Setup Velero Backup for AKS
  hosts: localhost
  connection: local
  gather_facts: false
  pre_tasks:
    - name: Verify required tools are installed
      command: "{{ item }} --version"
      loop:
        - az
        - kubectl
        - helm
      register: tool_check
      changed_when: false
    - name: Verify AKS context
      command: kubectl cluster-info
      register: cluster_info
      changed_when: false
    - name: Display cluster info
      debug:
        msg: "{{ cluster_info.stdout_lines[0] }}"
  roles:
    - azure_storage
    - azure_identity
    - velero
  post_tasks:
    - name: Trigger initial manual backup
      command: >
        velero backup create initial-backup
        --include-cluster-resources=true
        --wait
      register: initial_backup
      changed_when: true
    - name: Display backup result
      debug:
        msg: "{{ initial_backup.stdout }}"

		

Run the Ansible Playbook

			
# Install required collections
ansible-galaxy collection install azure.azcollection
ansible-galaxy collection install kubernetes.core
# Install Python dependencies
pip install ansible[azure] kubernetes
# Run the playbook
ansible-playbook site.yml -v
# Run specific role only
ansible-playbook site.yml --tags "velero" -v
# Dry run
ansible-playbook site.yml --check -v

		

Comparison: Terraform vs Ansible

Feature	Terraform	Ansible
Best For	Infrastructure provisioning	Configuration & app deployment
State Management	Yes (tfstate file)	No native state
Idempotency	Built-in	Task-level
Azure Resources	Excellent	Good
Kubernetes Resources	Good	Excellent
Learning Curve	Medium	Low
Rollback	Via state	Manual
Recommended Use	Create AKS + Storage	Install & configure Velero

Best Practice: Combine Both

			
Terraform  →  Creates Azure infrastructure (AKS, Storage, Identity)
    ↓
Ansible    →  Installs and configures Velero on the cluster
    ↓
Velero     →  Runs scheduled backups automatically

		

This gives you the best of both worlds — Terraform for infrastructure and Ansible for application configuration.

How to Backup AKS with Velero: A Step-by-Step Guide

April 25, 2026April 25, 2026 techhadoop AKS, azure, Uncategorized ai, azure, cloud, kubernetes, technology

Backing Up AKS with Velero

Prerequisites

AKS cluster running
Azure CLI installed
kubectl configured
Helm installed

Step 1: Create Azure Storage Account & Blob Container

			
# Set variables
RESOURCE_GROUP="myResourceGroup"
LOCATION="eastus"
STORAGE_ACCOUNT="velerobackupstorage"
BLOB_CONTAINER="velero-backups"
AKS_CLUSTER="myAKSCluster"
# Create resource group (if not exists)
az group create --name $RESOURCE_GROUP --location $LOCATION
# Create storage account
az storage account create \
  --name $STORAGE_ACCOUNT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --sku Standard_LRS \
  --encryption-services blob
# Create blob container
az storage container create \
  --name $BLOB_CONTAINER \
  --account-name $STORAGE_ACCOUNT

		

Step 2: Create Service Principal for Velero

			
# Create service principal
AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
# Create SP and assign contributor role
AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac \
  --name velero-sp \
  --role Contributor \
  --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID \
  --query password \
  -o tsv)
AZURE_CLIENT_ID=$(az ad sp list \
  --display-name velero-sp \
  --query '[0].appId' \
  -o tsv)
echo "Client ID: $AZURE_CLIENT_ID"
echo "Client Secret: $AZURE_CLIENT_SECRET"

		

Step 3: Create Velero Credentials File

			
# Create credentials file
cat << EOF > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

		

Step 4: Install Velero on AKS

			
# Add Velero Helm repo
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update
# Get storage account access key
STORAGE_ACCOUNT_KEY=$(az storage account keys list \
  --account-name $STORAGE_ACCOUNT \
  --resource-group $RESOURCE_GROUP \
  --query '[0].value' \
  -o tsv)
# Install Velero with Azure plugin
velero install \
  --provider azure \
  --plugins velero/velero-plugin-for-microsoft-azure:v1.8.0 \
  --bucket $BLOB_CONTAINER \
  --secret-file ./credentials-velero \
  --backup-location-config \
    resourceGroup=$RESOURCE_GROUP,\
    storageAccount=$STORAGE_ACCOUNT,\
    storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY \
  --snapshot-location-config \
    resourceGroup=$RESOURCE_GROUP,\
    apiTimeout=5m \
  --use-volume-snapshots=true \
  --wait

		

Step 5: Verify Velero Installation

			
# Check Velero pods are running
kubectl get pods -n velero
# Check backup storage location is available
velero backup-location get
# Expected output:
# NAME      PROVIDER   BUCKET/PREFIX     PHASE       LAST VALIDATED
# default   azure      velero-backups    Available   ...

		

Step 6: Create Backups

Manual Backup (Full Cluster)

			
# Backup entire cluster
velero backup create full-cluster-backup \
  --include-cluster-resources=true \
  --wait
# Check backup status
velero backup describe full-cluster-backup
velero backup logs full-cluster-backup

		

Backup Specific Namespace

			
# Backup a single namespace
velero backup create my-app-backup \
  --include-namespaces my-app-namespace \
  --wait
# Backup multiple namespaces
velero backup create multi-ns-backup \
  --include-namespaces namespace1,namespace2 \
  --wait

		

Backup with Labels

			
# Backup resources matching a label
velero backup create label-backup \
  --selector app=my-application \
  --wait

Exclude Specific Resources

			
# Exclude secrets and specific namespaces
velero backup create selective-backup \
  --exclude-namespaces kube-system,velero \
  --exclude-resources secrets \
  --wait

		

Step 7: Schedule Automated Backups

			
# Daily backup at 2am UTC (keep 30 days)
velero schedule create daily-backup \
  --schedule="0 2 * * *" \
  --ttl 720h \
  --include-cluster-resources=true
# Weekly backup every Sunday at midnight (keep 90 days)
velero schedule create weekly-backup \
  --schedule="0 0 * * 0" \
  --ttl 2160h
# Namespace-specific daily backup
velero schedule create daily-app-backup \
  --schedule="0 2 * * *" \
  --include-namespaces production \
  --ttl 168h
# List all schedules
velero schedule get

		

Step 8: Restore from Backup

Full Restore

			
# List available backups
velero backup get
# Restore full cluster backup
velero restore create \
  --from-backup full-cluster-backup \
  --wait
# Check restore status
velero restore get
velero restore describe <restore-name>

		

Restore Specific Namespace

			
# Restore a specific namespace
velero restore create \
  --from-backup my-app-backup \
  --include-namespaces my-app-namespace \
  --wait

		

Restore to Different Namespace

			
# Restore namespace-A into namespace-B
velero restore create \
  --from-backup my-app-backup \
  --namespace-mappings old-namespace:new-namespace \
  --wait

		

Restore Specific Resources Only

			
# Restore only deployments and services
velero restore create \
  --from-backup full-cluster-backup \
  --include-resources deployments,services \
  --wait

		

Step 9: Backup Hooks (Pre/Post Actions)

Useful for database consistency before backup:

			
# deployment-with-backup-hooks.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  annotations:
    pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "mysqldump -u root -p$MYSQL_ROOT_PASSWORD mydb > /backup/dump.sql"]'
    pre.hook.backup.velero.io/timeout: "60s"
    post.hook.backup.velero.io/command: '["/bin/bash", "-c", "rm -f /backup/dump.sql"]'

		

kubectl apply -f deployment-with-backup-hooks.yaml

Step 10: Monitor & Troubleshoot

			
# View backup details
velero backup describe my-backup --details
# View backup logs
velero backup logs my-backup
# View restore logs
velero restore logs my-restore
# Check Velero server logs
kubectl logs deployment/velero -n velero
# List all backups with status
velero backup get
# Delete old backup manually
velero backup delete old-backup-name

		

Summary Cheat Sheet

Task	Command
Full backup	`velero backup create <name> --include-cluster-resources=true`
Namespace backup	`velero backup create <name> --include-namespaces <ns>`
Schedule daily	`velero schedule create <name> --schedule="0 2 * * *"`
List backups	`velero backup get`
Restore backup	`velero restore create --from-backup <name>`
Check status	`velero backup describe <name>`
View logs	`velero backup logs <name>`

Best Practices

Test restores regularly – don’t wait for disaster to find out your backup is broken
Use TTL to automatically expire old backups and save storage costs
Backup before upgrades – always take a snapshot before AKS version upgrades
Store credentials securely – use Azure Key Vault instead of plain credentials files
Use hooks for databases to ensure data consistency
Cross-region storage – store backups in a different Azure region for disaster recovery
Combine with GitOps – use Velero for stateful data + Git for manifests

Top Scenarios to Ace Your AKS Interviews

April 24, 2026May 13, 2026 techhadoop AKS, azure ai, azure, cloud, kubernetes, technology

These are the kinds of enterprise failure scenarios where interviewers test depth, not just commands. A real interview: scenario → pressure → how you respond → what impresses.

Scenario 1: Entire AKS Cluster Becomes Unreachable

Situation

Apps are down
kubectl not responding
API server unreachable

What’s actually happening?

In Azure Kubernetes Service, the control plane is managed by Azure—so this is usually:

Network isolation issue (private cluster)
DNS issue
Azure-side outage (rare)

How to respond (structured)

1. Validate scope

Is it just you or everyone?
Can CI/CD still deploy?

2. Check cluster type

Private cluster?
API server behind Private Endpoint?

3. DNS resolution

Does API server FQDN resolve?

4. Network path

VPN / ExpressRoute working?
NSG blocking?

5. Azure health

Region outage?

Strong answer

“Since AKS control plane is managed, I’d immediately suspect network or DNS issues—especially in private clusters. I’d validate API server resolution, connectivity path, and Azure service health.”

Scenario 2: Production Outage After Deployment

Situation

New release deployed
All pods running
App returning 500 errors

Key Insight

This is NOT infrastructure—it’s application or config.

Approach

1. Rollback immediately

kubectl rollout undo deployment <app>

2. Compare versions

Env vars changed?
Secrets updated?

3. Check logs

App-level errors

4. Validate dependencies

DB reachable?
API endpoints correct?

Strong answer

“If pods are healthy but app fails, I treat it as an application issue. I’d rollback first to restore service, then investigate config drift or dependency failures.”

Scenario 3: Intermittent Failures Across Services

Situation

Random timeouts
Some requests succeed, others fail

Think: networking or scaling

Likely Causes

SNAT port exhaustion
DNS latency
Pod autoscaling delays
Node pressure

What you check

1. Node metrics

CPU/memory spikes

2. Pod distribution

Are pods unevenly spread?

3. Networking limits

Outbound connections?

Strong answer

“Intermittent failures usually point to resource contention or networking limits like SNAT exhaustion. I’d correlate metrics with traffic patterns.”

Scenario 4: AKS Can’t Pull Images from ACR

Situation

Pods stuck in ImagePullBackOff

Root cause area

Identity / permissions

Common causes

AKS not authorized to Azure Container Registry
Managed identity missing role

Fix

Assign AcrPull role to AKS identity

Strong answer

“This is typically a managed identity RBAC issue. I’d verify the cluster identity has AcrPull access to the registry.”

Scenario 5: Traffic Not Routing in Private AKS

Situation

Internal services work
External users can’t reach app

Think enterprise networking

Check:

1. Ingress controller

Running?

2. Azure Load Balancer / App Gateway

3. NSG rules

Ports open?

4. DNS

Internal vs external resolution

Strong answer

“In private AKS, exposure depends on controlled ingress. I’d trace traffic from DNS to ingress to service, checking NSGs and routing.”

Scenario 6: Cluster Autoscaler Not Working

Situation

Pods stuck in Pending
Nodes not scaling

Key checks

Autoscaler enabled?
Max node limit reached?
Subnet IP exhausted?

Strong answer

“Autoscaling failures often tie back to limits—either max node count or subnet capacity in Azure CNI setups.”

Scenario 7: Security Breach Suspicion

Situation

Unexpected outbound traffic
Suspicious container behavior

What matters

Containment + investigation

Actions

1. Isolate

Scale down or cordon node

2. Inspect

Container logs
Image source

3. Check runtime security

Alerts from Microsoft Defender for Cloud

Strong answer

“I’d isolate affected workloads immediately, then investigate logs and image provenance while leveraging Defender for alerts.”

Scenario 8: Multi-Region Failover Fails

Situation

Primary region down
Traffic not failing over

Root causes

DNS not switching
Traffic manager misconfigured
Backend unhealthy

Strong answer

“I’d validate DNS failover mechanism (Traffic Manager/Front Door), then confirm secondary cluster health and readiness.”

ENTERPRISE TROUBLESHOOTING MINDSET

Always think in layers:

1. Application

Logs, config, dependencies

2. Kubernetes

Pods, services, scheduling

3. Node / Compute

VMSS health

4. Networking

VNet, NSG, DNS

5. Azure Platform

Identity, RBAC, outages

What Interviewers REALLY Want

They’re looking for:

Structured thinking
Fast isolation of failure domain
Awareness of Azure-specific constraints
Calm, rollback-first mindset

If you want to go even deeper:

Feature	Kubenet	Azure CNI
IP assignment	NAT	Real VNet IP
Scalability	Better	Limited by subnet
Complexity	Lower	Higher
Use case	Small clusters	Enterprise

Feature	Disk	File
Access	Single pod	Multiple pods
Performance	High	Moderate

Understanding Azure RBAC vs Kubernetes RBAC

April 24, 2026April 24, 2026 techhadoop AKS, azure azure, cloud, devops, kubernetes, technology

When explaining this to a client, it is helpful to describe it as the difference between who can touch the physical server (the building) versus who can edit the files on the computer inside (the office).

In 2026, the industry standard is to use Azure RBAC for both, but they still operate on two distinct “control planes.”

1. The Two Control Planes

In AKS, access is split into two layers:

The Azure Control Plane (Azure RBAC): This governs the “outside” of the cluster. It’s about the Kubernetes resource itself as it exists in your Azure portal.
The Kubernetes Control Plane (Kubernetes RBAC): This governs the “inside” of the cluster. It’s about the pods, namespaces, and deployments running on the nodes.

2. Side-by-Side Comparison

Feature	Azure RBAC	Kubernetes RBAC
Scope	Subscription / Resource Group / AKS Resource	Cluster / Namespace / Specific Pods
Managed Via	Azure Portal, CLI, Terraform	`kubectl`, YAML manifests, Helm
Typical Actions	Scaling nodes, Upgrading K8s version, Deleting the cluster.	Creating a Pod, Editing a Service, Viewing logs in a Namespace.
Identity Source	Microsoft Entra ID (Azure AD)	Service Accounts (or Entra ID via integration)

3. The “Hybrid” Option (Azure RBAC for K8s Authorization)

This is the most confusing part for beginners, but the most important for you to propose to your client.

You can now use Azure RBAC to manage internal Kubernetes permissions. Instead of writing complex RoleBinding YAML files for every user, you assign them a built-in Azure role that Kubernetes understands.

Key Built-in Roles (2026 Standards):

AKS RBAC Viewer: Can see resources in a namespace but can’t see secrets or change anything.
AKS RBAC Writer: Can deploy apps and edit resources.
AKS RBAC Admin: Full control over a namespace.
AKS RBAC Cluster Admin: The “God Mode” for the entire cluster.

4. How to Explain the Workflow to Your Manager

“Think of it like a bank:

Azure RBAC is the security guard at the front door. He checks your ID (Entra ID) and decides if you’re even allowed in the building. He also decides who can add more teller windows (Scale nodes) or renovate the lobby (Upgrade cluster).
Kubernetes RBAC is the permissions on the safe. Once you’re inside, it decides if you can open Drawer A (Namespace ‘Dev’) or Drawer B (Namespace ‘Prod’).”

Pro-Tip: Recommendation

If you want to provide “Gold Standard” support, propose disabling local accounts and moving entirely to Azure RBAC for Kubernetes Authorization. > Why? Because when an employee leaves the company and their Entra ID (Azure AD) is deleted, their access to the Kubernetes cluster is instantly revoked. No orphaned RoleBindings to worry about.

Understanding how a developer goes from their laptop to a running container in a secured AKS environment is the best way to prove the value of your setup.

Here is the step-by-step lifecycle of a developer’s request in a Zero-Trust AKS environment.

The Access Lifecycle (Step-by-Step)

1. Authentication (The Gatekeeper)

The developer doesn’t have a “Kubernetes password.” Instead, they run:

Bash

			
az login
az aks get-credentials --resource-group rg-prod --name aks-01

At this moment, Azure RBAC checks if their Entra ID account has permission to even download the cluster configuration.

2. Authorization (The Office Door)

The developer tries to deploy a new microservice:

Bash

kubectl apply -f my-app.yaml

The AKS API Server intercepts this. Since we are using Azure RBAC for Kubernetes Authorization, it asks Entra ID: “Does this user have the ‘AKS RBAC Writer’ role for the ‘Production’ namespace?” * If Yes: The request proceeds.

If No: The request is blocked with a 403 Forbidden error.

3. Policy Validation (The Safety Inspector)

Before the pod is actually scheduled, Azure Policy (the Admission Controller) scans the my-app.yaml.

It checks: “Is this container trying to run as root? Does it have CPU limits?” * If the YAML is “lazy” (insecure), Azure Policy rejects it immediately, even though the developer has “Writer” permissions.

4. Identity & Secrets (The Secure Handshake)

Once the pod starts, it needs to talk to the database.

The pod presents its Workload Identity (a managed identity) to the Azure Key Vault.
Key Vault verifies the pod’s identity and hands over the database string via the CSI Driver.
The password is never stored in a file or an environment variable where a human could see it.

Summary Table for Your Proposal

To wrap this up for your client, you can present this “Success Path” to show them exactly what they are paying for:

Stage	Security Layer	Purpose
Login	Entra ID	Ensures only active employees can connect.
Action	Azure RBAC	Limits what a developer can do (e.g., Read vs. Write).
Deploy	Azure Policy	Forces best practices (No root, resource limits).
Connect	Workload Identity	Eliminates hardcoded passwords in the code.

Pro-Tip: The “Audit” Hook

Tell your client: “With this setup, we can generate a report at any time showing exactly who accessed the production cluster and what they changed. This makes SOC2 or ISO27001 audits a breeze.”

Understanding Azure RBAC vs Kubernetes RBAC

April 24, 2026April 24, 2026 techhadoop AKS, azure azure, cloud, devops, kubernetes, technology

In 2026, the industry standard is to use Azure RBAC for both, but they still operate on two distinct “control planes.”

1. The Two Control Planes

In AKS, access is split into two layers:

The Azure Control Plane (Azure RBAC): This governs the “outside” of the cluster. It’s about the Kubernetes resource itself as it exists in your Azure portal.
The Kubernetes Control Plane (Kubernetes RBAC): This governs the “inside” of the cluster. It’s about the pods, namespaces, and deployments running on the nodes.

2. Side-by-Side Comparison

Feature	Azure RBAC	Kubernetes RBAC
Scope	Subscription / Resource Group / AKS Resource	Cluster / Namespace / Specific Pods
Managed Via	Azure Portal, CLI, Terraform	`kubectl`, YAML manifests, Helm
Typical Actions	Scaling nodes, Upgrading K8s version, Deleting the cluster.	Creating a Pod, Editing a Service, Viewing logs in a Namespace.
Identity Source	Microsoft Entra ID (Azure AD)	Service Accounts (or Entra ID via integration)

3. The “Hybrid” Option (Azure RBAC for K8s Authorization)

This is the most confusing part for beginners, but the most important for you to propose to your client.

Key Built-in Roles (2026 Standards):

AKS RBAC Viewer: Can see resources in a namespace but can’t see secrets or change anything.
AKS RBAC Writer: Can deploy apps and edit resources.
AKS RBAC Admin: Full control over a namespace.
AKS RBAC Cluster Admin: The “God Mode” for the entire cluster.

4. How to Explain the Workflow to Your Manager

“Think of it like a bank:

Azure RBAC is the security guard at the front door. He checks your ID (Entra ID) and decides if you’re even allowed in the building. He also decides who can add more teller windows (Scale nodes) or renovate the lobby (Upgrade cluster).
Kubernetes RBAC is the permissions on the safe. Once you’re inside, it decides if you can open Drawer A (Namespace ‘Dev’) or Drawer B (Namespace ‘Prod’).”

Pro-Tip: Recommendation

Understanding how a developer goes from their laptop to a running container in a secured AKS environment is the best way to prove the value of your setup.

Here is the step-by-step lifecycle of a developer’s request in a Zero-Trust AKS environment.

The Access Lifecycle (Step-by-Step)

1. Authentication (The Gatekeeper)

The developer doesn’t have a “Kubernetes password.” Instead, they run:

Bash

			
az login
az aks get-credentials --resource-group rg-prod --name aks-01

At this moment, Azure RBAC checks if their Entra ID account has permission to even download the cluster configuration.

2. Authorization (The Office Door)

The developer tries to deploy a new microservice:

Bash

kubectl apply -f my-app.yaml

If No: The request is blocked with a 403 Forbidden error.

3. Policy Validation (The Safety Inspector)

Before the pod is actually scheduled, Azure Policy (the Admission Controller) scans the my-app.yaml.

It checks: “Is this container trying to run as root? Does it have CPU limits?” * If the YAML is “lazy” (insecure), Azure Policy rejects it immediately, even though the developer has “Writer” permissions.

4. Identity & Secrets (The Secure Handshake)

Once the pod starts, it needs to talk to the database.

The pod presents its Workload Identity (a managed identity) to the Azure Key Vault.
Key Vault verifies the pod’s identity and hands over the database string via the CSI Driver.
The password is never stored in a file or an environment variable where a human could see it.

Summary Table for Your Proposal

To wrap this up for your client, you can present this “Success Path” to show them exactly what they are paying for:

Stage	Security Layer	Purpose
Login	Entra ID	Ensures only active employees can connect.
Action	Azure RBAC	Limits what a developer can do (e.g., Read vs. Write).
Deploy	Azure Policy	Forces best practices (No root, resource limits).
Connect	Workload Identity	Eliminates hardcoded passwords in the code.

Pro-Tip: The “Audit” Hook

Tell your client: “With this setup, we can generate a report at any time showing exactly who accessed the production cluster and what they changed. This makes SOC2 or ISO27001 audits a breeze.”

1. Create the Namespace

2. Apply the LimitRange (Individual Pod Rules)

3. Apply the ResourceQuota (Total Team Budget)

4. Best Practices for Onboarding

The “Dry Run” Check

Guardrails for Java & ML

1. The Java-Optimized LimitRange

2. The ResourceQuota (Aggregated)

Crucial Tips for Java & ML Teams

The JVM “Cgroup” Awareness

ML & “Spot” Instances

Local Storage for ML

Kubernetes API Operations Through the ARO Private Endpoint

Every Operation Is a REST Call

Category 1 — Human CLI Operations (kubectl + oc)

kubectl — standard Kubernetes operations

oc CLI — OpenShift-specific additions

Category 2 — Operators and Controllers

The watch loop — how operators work

Built-in OpenShift operators that run this loop continuously

Category 3 — Kubelet (Node Agent)

Category 4 — CI/CD Pipelines

Category 5 — Admission Webhooks

Category 6 — Monitoring and Observability

The Request Pipeline — What Happens Inside

API Groups — Kubernetes vs OpenShift

Key Takeaway

OpenShift Container Platform on Azure — ARO Best Practices

1. Networking Best Practices

Always deploy a private cluster

Subnet sizing — get this right before deployment (cannot resize after)

Egress lockdown via Azure Firewall

Use a custom DNS server

2. Availability and Resilience Best Practices

Spread across all three Availability Zones

Enable cluster autoscaler

Use zone-redundant storage for persistent volumes

3. Security Best Practices

Use Workload Identity (pod-level Azure RBAC)

Integrate Azure Key Vault for secrets via CSI driver

Integrate with Azure Container Registry via private endpoint

Apply OpenShift Security Context Constraints (SCC)

Enable Microsoft Defender for Containers

4. Observability Best Practices

Forward logs to Azure Monitor / Log Analytics

Use Azure Monitor alerts for cluster health

5. Day-2 Operations Best Practices

Cluster upgrade strategy

Worker node upgrade — use surge capacity

Summary — ARO Best Practice Checklist

1. The Production Dockerfile

2. The requirements.txt

3. Deploying to AKS (The “Support” Strategy)

4. How to Pitch the “AI Operations” Tier

One final piece of the puzzle

1. The RBAC Strategy

2. The RBAC YAML (mcp-rbac.yaml)

3. Updating your Deployment

4. Why this is “Safe” for your Client

Putting it all together

Automating Velero AKS Backup with Ansible & Terraform

Option 1: Terraform

Project Structure

providers.tf

variables.tf

main.tf

outputs.tf

Deploy with Terraform

Option 2: Ansible

Project Structure

group_vars/all.yml

roles/azure_storage/tasks/main.yml

roles/azure_identity/tasks/main.yml

roles/velero/templates/credentials.j2

roles/velero/templates/backup-schedule.yml.j2

roles/velero/tasks/main.yml

site.yml (Main Playbook)

Run the Ansible Playbook

Comparison: Terraform vs Ansible

Best Practice: Combine Both

2. The `requirements.txt`

2. The RBAC YAML (`mcp-rbac.yaml`)