Optimizing Azure RAG: Single vs Multi-Spoke Strategies

April 29, 2026 techhadoop Uncategorized ai, artificial-intelligence, azure, cloud, technology

Think of a “Spoke” as a Security Boundary. While Azure allows you to spread services across different spokes, doing so adds complexity to your networking (DNS, Peering, and Firewall rules).

Here is the recommendation based on Azure Landing Zone (ALZ) best practices for 2026:

1. The “Single AI Spoke” Strategy (Recommended)

Most enterprises group Databricks, AI Search, and OpenAI into a single “AI & Data Spoke.”

Why? These services are “chatty.” Databricks needs to push data to AI Search, and OpenAI needs to pull data from AI Search. If they are in the same VNet, this traffic is faster, simpler to secure with Network Security Groups (NSGs), and avoids the “hop” through a central Hub Firewall which can add latency and cost.
Best for: A single business unit or a specific project team building an assistant.

2. The “Multi-Spoke” Strategy (Enterprise Scale)

You would only split them into separate spokes if you are building a Shared AI Platform for the whole company.

How it looks:
- Spoke A (Shared AI): Centralized Azure OpenAI (used by 10 different teams).
- Spoke B (Data Refinery): Your Databricks and ADLS (dedicated to your team’s private data).
- Spoke C (App): Your Frontend Chat UI.
Why? This allows the “Central IT” team to manage OpenAI quotas and costs in one place, while your team manages your own data.
Trade-off: You will need Private DNS Zone peering across all three spokes so the services can find each other’s Private Endpoints.

3. The “Cross-Service” Security Checklist

Regardless of whether they are in one spoke or two, you must handle these three connections:

Connection	Requirement	2026 Standard
Databricks $\rightarrow$ AI Search	Private Endpoint	Use a User-Assigned Managed Identity on the Databricks cluster to push vectors to Search.
OpenAI $\rightarrow$ AI Search	Shared Private Link	This is a special “handshake” in the Azure Portal that lets OpenAI talk to Search without going over the internet.
Frontend $\rightarrow$ OpenAI	VNet Integration	Your Web App/Frontend must be “VNet Integrated” to reach the OpenAI Private Endpoint.

4. Final Recommendation

If you are using Terraform and a Databricks-heavy refinery, I recommend the Single Spoke approach for now.

Create one VNet called vnet-ai-prod.
Create separate subnets for each service:
- snet-databricks-host / snet-databricks-container (for the spark nodes).
- snet-endpoints (for Private Endpoints for OpenAI, AI Search, and ADLS).
Use Private DNS Zones linked to this VNet so that my-openai.openai.azure.com resolves to a local internal IP (e.g., 10.0.1.5).

For a single department, keeping everything in a single spoke is the most efficient, cost-effective, and secure “starter” architecture for 2026. It minimizes networking latency and simplifies the DNS configuration that often trips up Terraform deployments.

Here is my specific recommendation for your single-department networking and security setup:

1. Subnet Segmentation (The “Clean” Spoke)

Don’t put all services in one big subnet. Divide your Spoke VNet into functional zones to apply specific Network Security Groups (NSGs):

Subnet A (Databricks Private): For the worker nodes.
Subnet B (Databricks Public): For the “Secure Cluster Connectivity” (No Public IP) relay.
Subnet C (Private Endpoints): This is the “Safe Zone” where you place the Private Endpoints for OpenAI, AI Search, and ADLS Gen2.
Subnet D (Integration): If you have a Frontend Web App, this is where you’ll use VNet Integration so the app can reach the services in Subnet C.

2. The 2026 Security “Golden Rules”

Identity over Keys: In your Terraform, disable local_auth_enabled for OpenAI and AI Search. This forces the system to use Microsoft Entra ID (RBAC). If a developer accidentally leaks an API key in a notebook, it simply won’t work.
Managed Identities: Use a User-Assigned Managed Identity. Assign it to both the Databricks cluster and the Frontend App. Grant it the Cognitive Services OpenAI User and Search Index Data Reader roles.
Data Exfiltration Protection (DEP): Since it’s a single department, you can easily enable DEP on your Databricks workspace. This prevents users from “exporting” results out to a personal storage account; they can only move data between the approved ADLS and AI Search resources.

3. Shared Private Links (The Handshake)

One of the most common mistakes is forgetting the OpenAI $\rightarrow$ AI Search path. Even if both are in your VNet, the OpenAI service itself needs a “Shared Private Link” to talk to your search index securely.

Action: In the Azure AI Foundry (formerly AI Studio) portal or via Terraform’s azapi provider, create a Shared Private Link from OpenAI to the AI Search resource.

4. Why a “Single Spoke” for one department?

Benefit	Explanation
Lower Cost	You avoid VNet Peering charges (which apply when data moves between spokes).
Simpler DNS	You only need to link your Private DNS Zones to one VNet.
Faster Iteration	Your Terraform script is “self-contained.” You don’t have to wait for a Central IT team to approve peering requests or firewall rules in a different subscription.

5. Final Recommendation Summary

Architecture: One Spoke VNet with VNet-Injected Databricks.
Access: All public access Disabled.
Authentication: 100% Managed Identity (No API keys).
Governance: Unity Catalog enabled for the structured data tables.

Terraform snippet for the Private Endpoint and DNS configuration

I managing both structured (SQL) and unstructured (Docs) data in a single department spoke, the Terraform setup below is designed for a “Zero-Trust” configuration. It ensures that OpenAI and AI Search services are invisible to the public internet and only accessible via your VNet.

1. Networking Strategy

This snippet covers the Private DNS Zones and Private Endpoints—the two areas where 90% of RAG networking issues occur.

Terraform

# 1. Private DNS Zones (The "Phonebook" for your VNet)
resource "azurerm_private_dns_zone" "openai_dns" {
  name                = "privatelink.openai.azure.com"
  resource_group_name = azurerm_resource_group.ai_rg.name
}

resource "azurerm_private_dns_zone" "search_dns" {
  name                = "privatelink.search.windows.net"
  resource_group_name = azurerm_resource_group.ai_rg.name
}

# 2. VNet Links (Telling the DNS zones which VNet to serve)
resource "azurerm_private_dns_zone_virtual_network_link" "openai_link" {
  name                  = "openai-link"
  resource_group_name   = azurerm_resource_group.ai_rg.name
  private_dns_zone_name = azurerm_private_dns_zone.openai_dns.name
  virtual_network_id    = azurerm_virtual_network.ai_vnet.id
}

# 3. Private Endpoint for Azure OpenAI
resource "azurerm_private_endpoint" "openai_endpoint" {
  name                = "pe-openai-department"
  location            = azurerm_resource_group.ai_rg.location
  resource_group_name = azurerm_resource_group.ai_rg.name
  subnet_id           = azurerm_subnet.endpoint_subnet.id

  private_service_connection {
    name                           = "psc-openai"
    private_connection_resource_id = azurerm_cognitive_account.openai.id
    subresource_names              = ["account"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "openai-dns-group"
    private_dns_zone_ids = [azurerm_private_dns_zone.openai_dns.id]
  }
}

2. Key Architectural Components (The “2026 Check”)

Resource	Sub-resource Name	DNS Zone Name
Azure OpenAI	`account`	`privatelink.openai.azure.com`
AI Search	`searchService`	`privatelink.search.windows.net`
ADLS Gen2 (Blob)	`blob`	`privatelink.blob.core.windows.net`
ADLS Gen2 (DFS)	`dfs`	`privatelink.dfs.core.windows.net`

Note: For ADLS Gen2, you need both blob and dfs endpoints if you’re using Databricks, as Spark often uses the DFS endpoint for optimized file operations.

3. Recommendations for your Managed Identities

To make this work for both data types without using API keys, add this to your Terraform:

Terraform

# Grant the AI Assistant (App) permission to use OpenAI
resource "azurerm_role_assignment" "app_openai_user" {
  scope                = azurerm_cognitive_account.openai.id
  role_definition_name = "Cognitive Services OpenAI User"
  principal_id         = azurerm_user_assigned_identity.assistant_id.principal_id
}

# Grant the AI Assistant permission to read from AI Search
resource "azurerm_role_assignment" "app_search_reader" {
  scope                = azurerm_search_service.ai_search.id
  role_definition_name = "Search Index Data Reader"
  principal_id         = azurerm_user_assigned_identity.assistant_id.principal_id
}

4. Final Security Check

Disable Public Access: Ensure public_network_access_enabled = false is set on the Storage Account, AI Search, and OpenAI resources.
Databricks “No Public IP”: In your Databricks workspace resource, set public_network_access_enabled = false and use the no_public_ip parameter for the cluster.
DNS Propagation: Remember that when you apply this Terraform, DNS can take 2–5 minutes to propagate. If your first connection fails, give it a moment to “settle.”

With this setup, your assistant will be able to query Databricks SQL (structured) and AI Search (unstructured) while keeping every single packet of data inside your department’s private network.

Securing AI with Zero-Trust Networking in 2026

April 28, 2026April 28, 2026 techhadoop ai, azure ai, artificial-intelligence, azure, cloud, technology

For an enterprise-grade AI Assistant in 2026, networking and security are the “make-or-break” components. If you are using Terraform and Databricks, you must move away from standard public access and embrace Zero-Trust Networking.

Here is the blueprint for networking and security setup.

1. The Network Backbone: Hub-and-Spoke

To keep your data safe, do not deploy everything into one VNet. Use the Hub-and-Spoke model.

The Hub: Contains shared services like Azure Firewall, VPN Gateway (for on-prem access), and Centralized DNS Zones.
The AI Spoke: This is where your Databricks workspace, AI Search, and OpenAI live.
The Connection: All communication between your spoke and the internet must pass through the Hub’s firewall.

2. Private Link & Managed Identities (No Keys!)

In 2026, API keys are a legacy risk. Your architecture should be “Keyless.”

Private Endpoints: Disable all public network access for ADLS Gen2, AI Search, and OpenAI. Assign each a Private Endpoint within your Spoke VNet. This ensures your data never touches the public internet.
Managed Identities (System-Assigned):
- Give your Databricks Cluster a Managed Identity with Storage Blob Data Contributor on ADLS.
- Give your Azure OpenAI resource a Managed Identity to read from AI Search.
- The Result: No secrets to rotate in your Terraform code or Key Vault.

3. Databricks-Specific Security (The Terraform Focus)

The blog post you mentioned focuses on Terraform for Databricks. For high security, your Terraform must include:

VNet Injection: Do not use the “default” Databricks VNet. Inject Databricks into your own managed VNet with two subnets (public and private).
No Public IP (NPIP): Enable the “Secure Cluster Connectivity” feature. This ensures your Databricks worker nodes have zero public IP addresses, making them invisible to the internet.
Unity Catalog + Private Link: Ensure Unity Catalog is configured to use a Private Access Connector. This allows Databricks to talk to your Metadata store without leaving the Azure backbone.

4. Advanced Protection for RAG

Since this assistant handles sensitive internal data, add these two “2026-standard” layers:

Microsoft Purview Integration: Link your AI Search and OpenAI to Microsoft Purview. This allows you to apply Sensitivity Labels (e.g., “Highly Confidential”). If a document is tagged as such, the AI will refuse to summarize it for a user who doesn’t have that specific clearance.
AI Content Safety: Place an Azure AI Content Safety layer in front of OpenAI. This detects “Prompt Injection” attacks where a user might try to trick the AI into revealing system prompts or unauthorized data.

Summary Checklist for your Terraform Modules

Resource	Security Requirement
ADLS Gen2	Firewall enabled; Allow only “Selected Networks” (your VNet).
Databricks	`enable_no_public_ip = true` and VNet Injection enabled.
AI Search	`public_network_access_enabled = false`; Private Endpoint active.
OpenAI	Managed Identity enabled; `local_auth_enabled = false` (forces Entra ID).
DNS	Private DNS Zones for `privatelink.openai.azure.com` and `privatelink.blob.core.windows.net`.

Pro-Tip: In your Terraform, use the azapi provider if the standard azurerm provider doesn’t yet support the latest 2026 AI Search security features. This allows you to call the Azure Resource Manager API directly for cutting-edge settings.

MCP Operations Server: AI-Enabled Managed Ops Explained

April 25, 2026April 25, 2026 techhadoop ai, AKS, azure ai, cloud, devops, kubernetes, technology

To bridge your local Python code to a production-ready AKS environment, you need a Dockerfile that doesn’t just run the code, but does so securely and efficiently.

By 2026, the standard for MCP servers in production is to move away from STDIO (local command line) and use SSE (Server-Sent Events) over HTTP. This allows your AI agents to talk to the server over a network.

1. The Production Dockerfile

This Dockerfile uses a “non-root” user (security best practice) and installs the necessary drivers to talk to the Docker socket or Kubernetes API.

Dockerfile

			
# Use a lightweight Python 2026-ready base image
FROM python:3.12-slim
# Install system dependencies (curl for health checks)
RUN apt-get update && apt-get install -y \
    curl \
    && rm -rf /var/lib/apt/lists/*
# Set working directory
WORKDIR /app
# Create a non-root user for security
RUN groupadd -r mcpuser && useradd -r -g mcpuser mcpuser
# Copy requirements and install
# Note: includes 'mcp[cli]' for server capabilities
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy server code
COPY server.py .
# Give our non-root user access to the app folder
RUN chown -r mcpuser:mcpuser /app
USER mcpuser
# Expose the port for SSE/HTTP transport (Standard for 2026)
EXPOSE 8000
# Start the server using the FastMCP production runner
CMD ["python", "server.py", "--transport", "sse", "--port", "8000"]

		

2. The `requirements.txt`

You’ll need these specific libraries:

Plaintext

			
fastmcp>=1.0.0
docker>=7.0.0
kubernetes>=30.0.0
uvicorn # Required for high-performance HTTP transport

3. Deploying to AKS (The “Support” Strategy)

When you deploy this to your client’s AKS cluster, you’ll use a standard Kubernetes Deployment.

Why this is better for your role:

Scaling: If the dev team grows, you can scale the MCP server to 3 replicas so the AI assistant never lags.
Security: Instead of sharing your personal kubeconfig, the MCP server uses a ServiceAccount with “View Only” permissions. This means the AI can see the logs but can’t accidentally delete the production database.

4. How to Pitch the “AI Operations” Tier

You can now offer a new support tier called “AI-Enabled Managed Ops”:

“I’ve built a custom MCP Operations Server for our cluster. It allows our internal AI agents to perform health checks, retrieve logs, and analyze container stats using natural language. This doesn’t replace me; it allows me to respond to your requests 10x faster because the AI is doing the ‘data gathering’ for me inside our secure perimeter.”

One final piece of the puzzle

To make this work in AKS, the pod needs permission to “see” the other pods.

To finish the MCP server integration on AKS, you need to grant the pod the right permissions to “talk” to the Kubernetes API.

If you don’t do this, the AI will be “blind”—it will try to list pods and get a 403 Forbidden error.

1. The RBAC Strategy

We will use three Kubernetes objects:

ServiceAccount: The identity for your MCP pod.
ClusterRole: A set of rules that allow “Viewing” (reading pods, logs, and events).
ClusterRoleBinding: The “glue” that attaches the Role to the ServiceAccount.

2. The RBAC YAML (`mcp-rbac.yaml`)

YAML

			
# 1. The Identity
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mcp-server-sa
  namespace: default
---
# 2. The Permissions (Read-Only/Viewer)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mcp-pod-viewer
rules:
- apiGroups: [""]
  # Accessing 'pods' for list/get, and 'pods/log' specifically for tracing
  resources: ["pods", "pods/log", "pods/status", "events", "nodes", "services"]
  verbs: ["get", "list", "watch"]
---
# 3. The Connection
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: mcp-server-binding
subjects:
- kind: ServiceAccount
  name: mcp-server-sa
  namespace: default
roleRef:
  kind: ClusterRole
  name: mcp-pod-viewer
  apiGroup: rbac.authorization.k8s.io

		

3. Updating your Deployment

Finally, ensure your MCP server deployment uses this serviceAccountName:

YAML

			
spec:
  template:
    spec:
      serviceAccountName: mcp-server-sa
      containers:
      - name: mcp-server
        image: your-mcp-image:latest
        # ... other config ...

		

4. Why this is “Safe” for your Client

When you explain this to the company, emphasize these three points:

Namespace Scoping: Even though it’s a ClusterRole, you can swap it for a Role if you only want the AI to see specific namespaces (e.g., only production-frontend).
No Secrets Access: Notice that secrets is not in the list of resources. The AI literally cannot see the database passwords, even if it tries.
Auditability: Every time the AI agent asks the MCP server for logs, Kubernetes logs that action under the mcp-server-sa identity. You have a perfect audit trail.

Putting it all together

You now have the Terraform for infrastructure, the Python for the server, the Docker for the container, and the RBAC for security.

You’re ready to pitch this as a “Self-Healing AI Operations Layer.”

Integrating AI with Azure Kubernetes Service in 2026

April 25, 2026April 29, 2026 techhadoop ai, AKS, azure ai, artificial-intelligence, cloud, llm, technology

Integrating Azure Kubernetes Service (AKS) with AI in 2026 generally falls into two categories: Consuming AI (connecting to models like GPT-4 via API) or Hosting AI (running your own models on GPUs).

Since you are already supporting a microservices environment, adding AI capabilities is a natural “next-tier” service to offer.

1. Consuming AI (The “API” Route)

The most common way to integrate AI is by connecting your Docker microservices to Azure OpenAI.

Service Connector: Use the Azure Service Connector to link your AKS cluster to an Azure OpenAI resource. This handles the networking and credentials for you.
Workload Identity: Avoid using API keys in your code. Grant your pod a User-Assigned Managed Identity and give it the Cognitive Services User role.
Vector Databases: If your microservices need “memory” (Retrieval-Augmented Generation or RAG), you can deploy a vector database like Qdrant or Weaviate directly as a Docker container in AKS to store and search through company data.

2. Hosting AI (The “KAITO” Route)

If your client wants to run their own open-source models (like Llama 3 or Mistral) for privacy or cost reasons, you should use the AI Toolchain Operator (KAITO).

What is KAITO? It’s an AKS-managed operator that simplifies the complex task of running Large Language Models (LLMs).
Auto-Provisioning: KAITO automatically picks the right GPU node size (e.g., Standard_NC) and handles the driver installation so you don’t have to manually configure NVIDIA settings.
Inference Presets: It provides pre-configured images for popular models, making it as easy as deploying a regular Docker microservice.

3. Infrastructure Requirements (GPU Nodes)

AI models are compute-heavy. You cannot run them on standard Linux nodes.

GPU Node Pools: Add a specialized node pool to your cluster using Terraform or CLI.2026 Best Practice: Use Azure Linux 3.0 as the OS for GPU nodes for better performance and reduced overhead.
Scale-to-Zero: Since GPU nodes are expensive ($2-$30+ per hour), configure the Cluster Autoscaler to scale the GPU node pool to zero when no AI jobs are running.

4. Monitoring AI Performance

AI workloads fail differently than web apps. A model might be “up” but providing extremely slow responses.

vLLM Metrics: If you use KAITO, it exposes metrics like Time to First Token (TTFT) and Tokens Per Second.
Managed Grafana: Import the standard “AI Inference Dashboard” into your Grafana instance to track how much GPU memory your models are consuming and whether you need to scale up.

How to Pitch This to Your Client

You can frame AI integration as a “Modernization Initiative”:

“I can upgrade our AKS cluster to support AI Workloads. We can implement the KAITO Operator to host private, cost-effective models for our internal tools, or use Workload Identity to securely connect our microservices to Azure OpenAI without using risky API keys. This ensures our infrastructure is ‘AI-Ready’ for any future features.”

Automate Velero AKS Backups with Terraform and Ansible

April 25, 2026April 25, 2026 techhadoop AKS, azure, Uncategorized ai, azure, cloud, kubernetes, technology

Automating Velero AKS Backup with Ansible & Terraform

Option 1: Terraform

Project Structure

			
velero-terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
└── modules/
    ├── storage/
    │   ├── main.tf
    │   └── variables.tf
    ├── identity/
    │   ├── main.tf
    │   └── variables.tf
    └── velero/
        ├── main.tf
        └── variables.tf

		

providers.tf

			
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
}
provider "azurerm" {
  features {}
}
provider "helm" {
  kubernetes {
    host                   = azurerm_kubernetes_cluster.aks.kube_config.0.host
    client_certificate     = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)
    client_key             = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)
    cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)
  }
}
provider "kubernetes" {
  host                   = azurerm_kubernetes_cluster.aks.kube_config.0.host
  client_certificate     = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)
  client_key             = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)
}

		

variables.tf

			
variable "resource_group_name" {
  description = "Resource group name"
  type        = string
  default     = "myResourceGroup"
}
variable "location" {
  description = "Azure region"
  type        = string
  default     = "eastus"
}
variable "aks_cluster_name" {
  description = "AKS cluster name"
  type        = string
  default     = "myAKSCluster"
}
variable "storage_account_name" {
  description = "Storage account for Velero backups"
  type        = string
  default     = "velerobackupstorage"
}
variable "blob_container_name" {
  description = "Blob container for Velero backups"
  type        = string
  default     = "velero-backups"
}
variable "velero_namespace" {
  description = "Kubernetes namespace for Velero"
  type        = string
  default     = "velero"
}
variable "backup_retention_hours" {
  description = "Backup TTL in hours"
  type        = number
  default     = 720  # 30 days
}
variable "backup_schedule" {
  description = "Cron schedule for backups"
  type        = string
  default     = "0 2 * * *"  # Daily at 2am
}

		

main.tf

			
# Resource Group
resource "azurerm_resource_group" "rg" {
  name     = var.resource_group_name
  location = var.location
}
# ── STORAGE ──────────────────────────────────────────────
resource "azurerm_storage_account" "velero" {
  name                     = var.storage_account_name
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  blob_properties {
    delete_retention_policy {
      days = 30
    }
  }
  tags = {
    purpose = "velero-backup"
  }
}
resource "azurerm_storage_container" "velero" {
  name                  = var.blob_container_name
  storage_account_name  = azurerm_storage_account.velero.name
  container_access_type = "private"
}
# ── IDENTITY ─────────────────────────────────────────────
resource "azurerm_user_assigned_identity" "velero" {
  name                = "velero-identity"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
}
# Assign Contributor role to storage account
resource "azurerm_role_assignment" "velero_storage" {
  scope                = azurerm_storage_account.velero.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_user_assigned_identity.velero.principal_id
}
# Assign Contributor role to resource group (for disk snapshots)
resource "azurerm_role_assignment" "velero_rg" {
  scope                = azurerm_resource_group.rg.id
  role_definition_name = "Contributor"
  principal_id         = azurerm_user_assigned_identity.velero.principal_id
}
# ── AKS CLUSTER ──────────────────────────────────────────
resource "azurerm_kubernetes_cluster" "aks" {
  name                = var.aks_cluster_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = var.aks_cluster_name
  default_node_pool {
    name       = "default"
    node_count = 2
    vm_size    = "Standard_D2_v2"
  }
  identity {
    type = "SystemAssigned"
  }
  tags = {
    environment = "production"
  }
}
# ── VELERO NAMESPACE ─────────────────────────────────────
resource "kubernetes_namespace" "velero" {
  metadata {
    name = var.velero_namespace
  }
  depends_on = [azurerm_kubernetes_cluster.aks]
}
# ── VELERO CREDENTIALS SECRET ────────────────────────────
resource "kubernetes_secret" "velero_credentials" {
  metadata {
    name      = "velero-credentials"
    namespace = kubernetes_namespace.velero.metadata[0].name
  }
  data = {
    "cloud" = <<EOF
AZURE_SUBSCRIPTION_ID=${data.azurerm_subscription.current.subscription_id}
AZURE_TENANT_ID=${data.azurerm_subscription.current.tenant_id}
AZURE_CLIENT_ID=${azurerm_user_assigned_identity.velero.client_id}
AZURE_RESOURCE_GROUP=${azurerm_resource_group.rg.name}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF
  }
}
# Current subscription data
data "azurerm_subscription" "current" {}
# ── VELERO HELM RELEASE ──────────────────────────────────
resource "helm_release" "velero" {
  name       = "velero"
  repository = "https://vmware-tanzu.github.io/helm-charts"
  chart      = "velero"
  namespace  = kubernetes_namespace.velero.metadata[0].name
  version    = "5.0.0"
  values = [
    yamlencode({
      configuration = {
        provider = "azure"
        backupStorageLocation = {
          name     = "default"
          provider = "velero.io/azure"
          bucket   = azurerm_storage_container.velero.name
          config = {
            resourceGroup      = azurerm_resource_group.rg.name
            storageAccount     = azurerm_storage_account.velero.name
            subscriptionId     = data.azurerm_subscription.current.subscription_id
          }
        }
        volumeSnapshotLocation = {
          name     = "default"
          provider = "velero.io/azure"
          config = {
            resourceGroup  = azurerm_resource_group.rg.name
            subscriptionId = data.azurerm_subscription.current.subscription_id
          }
        }
      }
      credentials = {
        existingSecret = kubernetes_secret.velero_credentials.metadata[0].name
      }
      initContainers = [
        {
          name            = "velero-plugin-for-azure"
          image           = "velero/velero-plugin-for-microsoft-azure:v1.8.0"
          imagePullPolicy = "IfNotPresent"
          volumeMounts = [
            {
              mountPath = "/target"
              name      = "plugins"
            }
          ]
        }
      ]
      schedules = {
        daily-backup = {
          schedule = var.backup_schedule
          template = {
            ttl                      = "${var.backup_retention_hours}h0m0s"
            includeClusterResources  = true
            excludedNamespaces       = ["kube-system", "velero"]
          }
        }
      }
    })
  ]
  depends_on = [
    kubernetes_secret.velero_credentials,
    azurerm_role_assignment.velero_storage,
    azurerm_role_assignment.velero_rg
  ]
}

		

outputs.tf

			
output "storage_account_name" {
  value = azurerm_storage_account.velero.name
}
output "blob_container_name" {
  value = azurerm_storage_container.velero.name
}
output "velero_identity_client_id" {
  value = azurerm_user_assigned_identity.velero.client_id
}
output "aks_cluster_name" {
  value = azurerm_kubernetes_cluster.aks.name
}
output "kube_config" {
  value     = azurerm_kubernetes_cluster.aks.kube_config_raw
  sensitive = true
}

		

Deploy with Terraform

			
# Initialize
terraform init
# Preview changes
terraform plan -out=tfplan
# Apply
terraform apply tfplan
# Get kubeconfig
terraform output -raw kube_config > ~/.kube/config
# Verify Velero
kubectl get pods -n velero
velero backup-location get

		

Option 2: Ansible

Project Structure

			
velero-ansible/
├── inventory/
│   └── hosts.yml
├── group_vars/
│   └── all.yml
├── roles/
│   ├── azure_storage/
│   │   └── tasks/
│   │       └── main.yml
│   ├── azure_identity/
│   │   └── tasks/
│   │       └── main.yml
│   └── velero/
│       ├── tasks/
│       │   └── main.yml
│       └── templates/
│           ├── credentials.j2
│           └── backup-schedule.yml.j2
└── site.yml

		

group_vars/all.yml

			
# Azure Settings
resource_group: "myResourceGroup"
location: "eastus"
aks_cluster_name: "myAKSCluster"
# Storage Settings
storage_account_name: "velerobackupstorage"
blob_container_name: "velero-backups"
# Velero Settings
velero_namespace: "velero"
velero_version: "v1.12.0"
velero_azure_plugin_version: "v1.8.0"
velero_chart_version: "5.0.0"
# Backup Settings
backup_schedule: "0 2 * * *"
backup_ttl: "720h"
backup_name: "daily-backup"
# Namespaces to exclude
excluded_namespaces:
  - kube-system
  - velero

		

roles/azure_storage/tasks/main.yml

			
---
- name: Create Resource Group
  azure.azcollection.azure_rm_resourcegroup:
    name: "{{ resource_group }}"
    location: "{{ location }}"
    state: present
- name: Create Storage Account
  azure.azcollection.azure_rm_storageaccount:
    resource_group: "{{ resource_group }}"
    name: "{{ storage_account_name }}"
    type: Standard_LRS
    kind: StorageV2
    state: present
  register: storage_account_result
- name: Create Blob Container
  azure.azcollection.azure_rm_storageblob:
    resource_group: "{{ resource_group }}"
    storage_account_name: "{{ storage_account_name }}"
    container: "{{ blob_container_name }}"
    state: present
- name: Get Storage Account Keys
  azure.azcollection.azure_rm_storageaccount_info:
    resource_group: "{{ resource_group }}"
    name: "{{ storage_account_name }}"
  register: storage_info
- name: Set Storage Key Fact
  set_fact:
    storage_account_key: "{{ storage_info.storageaccounts[0].primary_endpoints.key }}"

		

roles/azure_identity/tasks/main.yml

			
---
- name: Get Azure Subscription Info
  azure.azcollection.azure_rm_subscription_info:
  register: subscription_info
- name: Set Subscription Facts
  set_fact:
    subscription_id: "{{ subscription_info.subscriptions[0].subscription_id }}"
    tenant_id: "{{ subscription_info.subscriptions[0].tenant_id }}"
- name: Create Service Principal for Velero
  azure.azcollection.azure_rm_adserviceprincipal:
    app_id: "velero-sp"
    state: present
  register: sp_result
- name: Assign Contributor Role to Service Principal
  azure.azcollection.azure_rm_roleassignment:
    scope: "/subscriptions/{{ subscription_id }}/resourceGroups/{{ resource_group }}"
    assignee_object_id: "{{ sp_result.object_id }}"
    role_definition_name: Contributor
    state: present
- name: Assign Storage Blob Contributor Role
  azure.azcollection.azure_rm_roleassignment:
    scope: "/subscriptions/{{ subscription_id }}/resourceGroups/{{ resource_group }}/providers/Microsoft.Storage/storageAccounts/{{ storage_account_name }}"
    assignee_object_id: "{{ sp_result.object_id }}"
    role_definition_name: "Storage Blob Data Contributor"
    state: present

		

roles/velero/templates/credentials.j2

			
AZURE_SUBSCRIPTION_ID={{ subscription_id }}
AZURE_TENANT_ID={{ tenant_id }}
AZURE_CLIENT_ID={{ client_id }}
AZURE_CLIENT_SECRET={{ client_secret }}
AZURE_RESOURCE_GROUP={{ resource_group }}
AZURE_CLOUD_NAME=AzurePublicCloud

		

roles/velero/templates/backup-schedule.yml.j2

			
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: {{ backup_name }}
  namespace: {{ velero_namespace }}
spec:
  schedule: "{{ backup_schedule }}"
  template:
    ttl: "{{ backup_ttl }}"
    includeClusterResources: true
    excludedNamespaces:
{% for ns in excluded_namespaces %}
      - {{ ns }}
{% endfor %}

		

roles/velero/tasks/main.yml

			
---
- name: Create Velero Namespace
  kubernetes.core.k8s:
    name: "{{ velero_namespace }}"
    api_version: v1
    kind: Namespace
    state: present
- name: Create Velero Credentials File
  template:
    src: credentials.j2
    dest: /tmp/credentials-velero
    mode: '0600'
- name: Create Kubernetes Secret for Velero Credentials
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Secret
      metadata:
        name: velero-credentials
        namespace: "{{ velero_namespace }}"
      stringData:
        cloud: |
          AZURE_SUBSCRIPTION_ID={{ subscription_id }}
          AZURE_TENANT_ID={{ tenant_id }}
          AZURE_CLIENT_ID={{ client_id }}
          AZURE_CLIENT_SECRET={{ client_secret }}
          AZURE_RESOURCE_GROUP={{ resource_group }}
          AZURE_CLOUD_NAME=AzurePublicCloud
- name: Add Velero Helm Repository
  kubernetes.core.helm_repository:
    name: vmware-tanzu
    repo_url: "https://vmware-tanzu.github.io/helm-charts"
    state: present
- name: Install Velero via Helm
  kubernetes.core.helm:
    name: velero
    chart_ref: vmware-tanzu/velero
    chart_version: "{{ velero_chart_version }}"
    namespace: "{{ velero_namespace }}"
    state: present
    values:
      configuration:
        provider: azure
        backupStorageLocation:
          name: default
          provider: velero.io/azure
          bucket: "{{ blob_container_name }}"
          config:
            resourceGroup: "{{ resource_group }}"
            storageAccount: "{{ storage_account_name }}"
            subscriptionId: "{{ subscription_id }}"
        volumeSnapshotLocation:
          name: default
          provider: velero.io/azure
          config:
            resourceGroup: "{{ resource_group }}"
            subscriptionId: "{{ subscription_id }}"
      credentials:
        existingSecret: velero-credentials
      initContainers:
        - name: velero-plugin-for-azure
          image: "velero/velero-plugin-for-microsoft-azure:{{ velero_azure_plugin_version }}"
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /target
              name: plugins
- name: Wait for Velero Pod to be Ready
  kubernetes.core.k8s_info:
    kind: Pod
    namespace: "{{ velero_namespace }}"
    label_selectors:
      - app.kubernetes.io/name=velero
  register: velero_pod
  until: velero_pod.resources[0].status.phase == "Running"
  retries: 10
  delay: 15
- name: Apply Backup Schedule
  kubernetes.core.k8s:
    state: present
    template: backup-schedule.yml.j2
- name: Verify Backup Location
  command: velero backup-location get
  register: backup_location_status
  changed_when: false
- name: Display Backup Location Status
  debug:
    msg: "{{ backup_location_status.stdout }}"
- name: Clean Up Credentials File
  file:
    path: /tmp/credentials-velero
    state: absent

		

site.yml (Main Playbook)

			
---
- name: Setup Velero Backup for AKS
  hosts: localhost
  connection: local
  gather_facts: false
  pre_tasks:
    - name: Verify required tools are installed
      command: "{{ item }} --version"
      loop:
        - az
        - kubectl
        - helm
      register: tool_check
      changed_when: false
    - name: Verify AKS context
      command: kubectl cluster-info
      register: cluster_info
      changed_when: false
    - name: Display cluster info
      debug:
        msg: "{{ cluster_info.stdout_lines[0] }}"
  roles:
    - azure_storage
    - azure_identity
    - velero
  post_tasks:
    - name: Trigger initial manual backup
      command: >
        velero backup create initial-backup
        --include-cluster-resources=true
        --wait
      register: initial_backup
      changed_when: true
    - name: Display backup result
      debug:
        msg: "{{ initial_backup.stdout }}"

		

Run the Ansible Playbook

			
# Install required collections
ansible-galaxy collection install azure.azcollection
ansible-galaxy collection install kubernetes.core
# Install Python dependencies
pip install ansible[azure] kubernetes
# Run the playbook
ansible-playbook site.yml -v
# Run specific role only
ansible-playbook site.yml --tags "velero" -v
# Dry run
ansible-playbook site.yml --check -v

		

Comparison: Terraform vs Ansible

Feature	Terraform	Ansible
Best For	Infrastructure provisioning	Configuration & app deployment
State Management	Yes (tfstate file)	No native state
Idempotency	Built-in	Task-level
Azure Resources	Excellent	Good
Kubernetes Resources	Good	Excellent
Learning Curve	Medium	Low
Rollback	Via state	Manual
Recommended Use	Create AKS + Storage	Install & configure Velero

Best Practice: Combine Both

			
Terraform  →  Creates Azure infrastructure (AKS, Storage, Identity)
    ↓
Ansible    →  Installs and configures Velero on the cluster
    ↓
Velero     →  Runs scheduled backups automatically

		

This gives you the best of both worlds — Terraform for infrastructure and Ansible for application configuration.

How to Backup AKS with Velero: A Step-by-Step Guide

April 25, 2026April 25, 2026 techhadoop AKS, azure, Uncategorized ai, azure, cloud, kubernetes, technology

Backing Up AKS with Velero

Prerequisites

AKS cluster running
Azure CLI installed
kubectl configured
Helm installed

Step 1: Create Azure Storage Account & Blob Container

			
# Set variables
RESOURCE_GROUP="myResourceGroup"
LOCATION="eastus"
STORAGE_ACCOUNT="velerobackupstorage"
BLOB_CONTAINER="velero-backups"
AKS_CLUSTER="myAKSCluster"
# Create resource group (if not exists)
az group create --name $RESOURCE_GROUP --location $LOCATION
# Create storage account
az storage account create \
  --name $STORAGE_ACCOUNT \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --sku Standard_LRS \
  --encryption-services blob
# Create blob container
az storage container create \
  --name $BLOB_CONTAINER \
  --account-name $STORAGE_ACCOUNT

		

Step 2: Create Service Principal for Velero

			
# Create service principal
AZURE_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
AZURE_TENANT_ID=$(az account show --query tenantId -o tsv)
# Create SP and assign contributor role
AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac \
  --name velero-sp \
  --role Contributor \
  --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID \
  --query password \
  -o tsv)
AZURE_CLIENT_ID=$(az ad sp list \
  --display-name velero-sp \
  --query '[0].appId' \
  -o tsv)
echo "Client ID: $AZURE_CLIENT_ID"
echo "Client Secret: $AZURE_CLIENT_SECRET"

		

Step 3: Create Velero Credentials File

			
# Create credentials file
cat << EOF > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

		

Step 4: Install Velero on AKS

			
# Add Velero Helm repo
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update
# Get storage account access key
STORAGE_ACCOUNT_KEY=$(az storage account keys list \
  --account-name $STORAGE_ACCOUNT \
  --resource-group $RESOURCE_GROUP \
  --query '[0].value' \
  -o tsv)
# Install Velero with Azure plugin
velero install \
  --provider azure \
  --plugins velero/velero-plugin-for-microsoft-azure:v1.8.0 \
  --bucket $BLOB_CONTAINER \
  --secret-file ./credentials-velero \
  --backup-location-config \
    resourceGroup=$RESOURCE_GROUP,\
    storageAccount=$STORAGE_ACCOUNT,\
    storageAccountKeyEnvVar=AZURE_STORAGE_ACCOUNT_ACCESS_KEY \
  --snapshot-location-config \
    resourceGroup=$RESOURCE_GROUP,\
    apiTimeout=5m \
  --use-volume-snapshots=true \
  --wait

		

Step 5: Verify Velero Installation

			
# Check Velero pods are running
kubectl get pods -n velero
# Check backup storage location is available
velero backup-location get
# Expected output:
# NAME      PROVIDER   BUCKET/PREFIX     PHASE       LAST VALIDATED
# default   azure      velero-backups    Available   ...

		

Step 6: Create Backups

Manual Backup (Full Cluster)

			
# Backup entire cluster
velero backup create full-cluster-backup \
  --include-cluster-resources=true \
  --wait
# Check backup status
velero backup describe full-cluster-backup
velero backup logs full-cluster-backup

		

Backup Specific Namespace

			
# Backup a single namespace
velero backup create my-app-backup \
  --include-namespaces my-app-namespace \
  --wait
# Backup multiple namespaces
velero backup create multi-ns-backup \
  --include-namespaces namespace1,namespace2 \
  --wait

		

Backup with Labels

			
# Backup resources matching a label
velero backup create label-backup \
  --selector app=my-application \
  --wait

Exclude Specific Resources

			
# Exclude secrets and specific namespaces
velero backup create selective-backup \
  --exclude-namespaces kube-system,velero \
  --exclude-resources secrets \
  --wait

		

Step 7: Schedule Automated Backups

			
# Daily backup at 2am UTC (keep 30 days)
velero schedule create daily-backup \
  --schedule="0 2 * * *" \
  --ttl 720h \
  --include-cluster-resources=true
# Weekly backup every Sunday at midnight (keep 90 days)
velero schedule create weekly-backup \
  --schedule="0 0 * * 0" \
  --ttl 2160h
# Namespace-specific daily backup
velero schedule create daily-app-backup \
  --schedule="0 2 * * *" \
  --include-namespaces production \
  --ttl 168h
# List all schedules
velero schedule get

		

Step 8: Restore from Backup

Full Restore

			
# List available backups
velero backup get
# Restore full cluster backup
velero restore create \
  --from-backup full-cluster-backup \
  --wait
# Check restore status
velero restore get
velero restore describe <restore-name>

		

Restore Specific Namespace

			
# Restore a specific namespace
velero restore create \
  --from-backup my-app-backup \
  --include-namespaces my-app-namespace \
  --wait

		

Restore to Different Namespace

			
# Restore namespace-A into namespace-B
velero restore create \
  --from-backup my-app-backup \
  --namespace-mappings old-namespace:new-namespace \
  --wait

		

Restore Specific Resources Only

			
# Restore only deployments and services
velero restore create \
  --from-backup full-cluster-backup \
  --include-resources deployments,services \
  --wait

		

Step 9: Backup Hooks (Pre/Post Actions)

Useful for database consistency before backup:

			
# deployment-with-backup-hooks.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  annotations:
    pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "mysqldump -u root -p$MYSQL_ROOT_PASSWORD mydb > /backup/dump.sql"]'
    pre.hook.backup.velero.io/timeout: "60s"
    post.hook.backup.velero.io/command: '["/bin/bash", "-c", "rm -f /backup/dump.sql"]'

		

kubectl apply -f deployment-with-backup-hooks.yaml

Step 10: Monitor & Troubleshoot

			
# View backup details
velero backup describe my-backup --details
# View backup logs
velero backup logs my-backup
# View restore logs
velero restore logs my-restore
# Check Velero server logs
kubectl logs deployment/velero -n velero
# List all backups with status
velero backup get
# Delete old backup manually
velero backup delete old-backup-name

		

Summary Cheat Sheet

Task	Command
Full backup	`velero backup create <name> --include-cluster-resources=true`
Namespace backup	`velero backup create <name> --include-namespaces <ns>`
Schedule daily	`velero schedule create <name> --schedule="0 2 * * *"`
List backups	`velero backup get`
Restore backup	`velero restore create --from-backup <name>`
Check status	`velero backup describe <name>`
View logs	`velero backup logs <name>`

Best Practices

Test restores regularly – don’t wait for disaster to find out your backup is broken
Use TTL to automatically expire old backups and save storage costs
Backup before upgrades – always take a snapshot before AKS version upgrades
Store credentials securely – use Azure Key Vault instead of plain credentials files
Use hooks for databases to ensure data consistency
Cross-region storage – store backups in a different Azure region for disaster recovery
Combine with GitOps – use Velero for stateful data + Git for manifests

Top Scenarios to Ace Your AKS Interviews

April 24, 2026May 13, 2026 techhadoop AKS, azure ai, azure, cloud, kubernetes, technology

These are the kinds of enterprise failure scenarios where interviewers test depth, not just commands. A real interview: scenario → pressure → how you respond → what impresses.

Scenario 1: Entire AKS Cluster Becomes Unreachable

Situation

Apps are down
kubectl not responding
API server unreachable

What’s actually happening?

In Azure Kubernetes Service, the control plane is managed by Azure—so this is usually:

Network isolation issue (private cluster)
DNS issue
Azure-side outage (rare)

How to respond (structured)

1. Validate scope

Is it just you or everyone?
Can CI/CD still deploy?

2. Check cluster type

Private cluster?
API server behind Private Endpoint?

3. DNS resolution

Does API server FQDN resolve?

4. Network path

VPN / ExpressRoute working?
NSG blocking?

5. Azure health

Region outage?

Strong answer

“Since AKS control plane is managed, I’d immediately suspect network or DNS issues—especially in private clusters. I’d validate API server resolution, connectivity path, and Azure service health.”

Scenario 2: Production Outage After Deployment

Situation

New release deployed
All pods running
App returning 500 errors

Key Insight

This is NOT infrastructure—it’s application or config.

Approach

1. Rollback immediately

kubectl rollout undo deployment <app>

2. Compare versions

Env vars changed?
Secrets updated?

3. Check logs

App-level errors

4. Validate dependencies

DB reachable?
API endpoints correct?

Strong answer

“If pods are healthy but app fails, I treat it as an application issue. I’d rollback first to restore service, then investigate config drift or dependency failures.”

Scenario 3: Intermittent Failures Across Services

Situation

Random timeouts
Some requests succeed, others fail

Think: networking or scaling

Likely Causes

SNAT port exhaustion
DNS latency
Pod autoscaling delays
Node pressure

What you check

1. Node metrics

CPU/memory spikes

2. Pod distribution

Are pods unevenly spread?

3. Networking limits

Outbound connections?

Strong answer

“Intermittent failures usually point to resource contention or networking limits like SNAT exhaustion. I’d correlate metrics with traffic patterns.”

Scenario 4: AKS Can’t Pull Images from ACR

Situation

Pods stuck in ImagePullBackOff

Root cause area

Identity / permissions

Common causes

AKS not authorized to Azure Container Registry
Managed identity missing role

Fix

Assign AcrPull role to AKS identity

Strong answer

“This is typically a managed identity RBAC issue. I’d verify the cluster identity has AcrPull access to the registry.”

Scenario 5: Traffic Not Routing in Private AKS

Situation

Internal services work
External users can’t reach app

Think enterprise networking

Check:

1. Ingress controller

Running?

2. Azure Load Balancer / App Gateway

3. NSG rules

Ports open?

4. DNS

Internal vs external resolution

Strong answer

“In private AKS, exposure depends on controlled ingress. I’d trace traffic from DNS to ingress to service, checking NSGs and routing.”

Scenario 6: Cluster Autoscaler Not Working

Situation

Pods stuck in Pending
Nodes not scaling

Key checks

Autoscaler enabled?
Max node limit reached?
Subnet IP exhausted?

Strong answer

“Autoscaling failures often tie back to limits—either max node count or subnet capacity in Azure CNI setups.”

Scenario 7: Security Breach Suspicion

Situation

Unexpected outbound traffic
Suspicious container behavior

What matters

Containment + investigation

Actions

1. Isolate

Scale down or cordon node

2. Inspect

Container logs
Image source

3. Check runtime security

Alerts from Microsoft Defender for Cloud

Strong answer

“I’d isolate affected workloads immediately, then investigate logs and image provenance while leveraging Defender for alerts.”

Scenario 8: Multi-Region Failover Fails

Situation

Primary region down
Traffic not failing over

Root causes

DNS not switching
Traffic manager misconfigured
Backend unhealthy

Strong answer

“I’d validate DNS failover mechanism (Traffic Manager/Front Door), then confirm secondary cluster health and readiness.”

ENTERPRISE TROUBLESHOOTING MINDSET

Always think in layers:

1. Application

Logs, config, dependencies

2. Kubernetes

Pods, services, scheduling

3. Node / Compute

VMSS health

4. Networking

VNet, NSG, DNS

5. Azure Platform

Identity, RBAC, outages

What Interviewers REALLY Want

They’re looking for:

Structured thinking
Fast isolation of failure domain
Awareness of Azure-specific constraints
Calm, rollback-first mindset

If you want to go even deeper:

Feature	Kubenet	Azure CNI
IP assignment	NAT	Real VNet IP
Scalability	Better	Limited by subnet
Complexity	Lower	Higher
Use case	Small clusters	Enterprise

Feature	Disk	File
Access	Single pod	Multiple pods
Performance	High	Moderate

Unified Observability for AKS in 2026

April 24, 2026April 24, 2026 techhadoop AKS, azure ai, artificial-intelligence, azure, cloud, technology

Monitoring AKS in 2026 has moved beyond simple “health checks.” It now centers on a Unified Observability strategy that combines infrastructure metrics, high-cardinality logs, and application traces.

To provide top-tier support, you should propose a two-pronged approach: Azure Native for the platform and Managed Prometheus/Grafana for the microservices.

1. The 2026 Monitoring Stack (The “Big Three”)

Tool	Purpose	What to Propose
Container Insights	Platform Health. Focuses on Inventory, Node CPU/Memory, and K8s Events.	“We’ll use this for the ‘Golden Signals’ of the cluster infrastructure.”
Managed Prometheus	Workload Metrics. High-resolution scraping of your microservices (Pod-level).	“This gives us deep visibility into app-specific metrics like ‘Orders Processed’.”
Managed Grafana	Visualization. The single pane of glass for both Azure and Prometheus data.	“I’ll build executive dashboards to show uptime and developer dashboards for debugging.”

2. Advanced Log Management (Cost & Performance)

Log ingestion is usually the most expensive part of AKS support. As of 2026, Microsoft has introduced “Basic Logs” to help.

Log Analytics (ContainerLogV2): Propose migrating to the ContainerLogV2 schema. It is faster and supports Basic Logs, which can reduce costs by up to 50-70% for high-volume logs you don’t query often.
Diagnostic Settings: Enable these to capture Control Plane logs (API Server, Scheduler). Without these, you are blind to why a cluster upgrade failed or who deleted a namespace.

3. Network Observability (The 2026 Add-on)

Microsoft recently standardized the Network Observability add-on. This is a great “upsell” for security-conscious clients.

What it does: It tracks “East-West” traffic (pod-to-pod).
The Pitch: “If Service A can’t talk to Service B, I can tell you in 30 seconds if it’s a network drop, a DNS failure, or a security policy block.”

4. Proactive Alerting Strategy

Don’t just alert on “CPU > 80%.” That leads to alert fatigue. Instead, propose Service Level Objective (SLO) based alerting:

Latency Alerts: “Alert if 5% of requests take longer than 2 seconds.”
OOMKill Detection: “Alert if any pod in the production namespace is killed due to memory limits.”
Disk Pressure: “Alert if node local storage is at 85% to prevent the ‘DiskPressure’ taint from evicting pods.”

The “SRE” Proposal Snippet

If you want to present this to your company, try this:

“To ensure 99.9% availability for our microservices, I propose implementing the Azure Managed Observability Stack. By moving our logs to the V2 schema and implementing Managed Prometheus, we can reduce our monitoring costs by roughly 30% while gaining the ability to trace transactions across our entire microservice mesh. This transforms our support from ‘fixing breaks’ to ‘preventing downtime’.”

If you are only looking at logs, you are seeing what happened (the “post-mortem”), but you are missing where the bottleneck is and how the services interact.

To take your support to the next level, you should propose moving toward Distributed Tracing. This is the “holy grail” of microservices support.

1. The Missing Piece: Distributed Tracing

When a user says “the app is slow,” logs usually show a bunch of successful 200 OK messages across five different services. You have no way of knowing which of those five services added the 3-second delay.

The Solution: OpenTelemetry (OTel)

By implementing OpenTelemetry (the industry standard in 2026), you give every request a “Trace ID.”

Trace: The entire journey of a request from the user’s click to the database and back.
Span: The time spent inside a single microservice or database call.

2. Azure Application Insights (The Easy Win)

Since you are on AKS, the fastest way to get tracing is Application Insights.

Application Map: This is a live, auto-generated visual of your entire microservice architecture. It shows which services are talking to which, and highlights red links where errors or high latency are occurring.
No-Code Instrumentation: For many Linux/Docker apps (especially .NET, Java, and Python), you can enable tracing without changing a single line of code by using the Azure Monitor OpenTelemetry Distro or a “Sidecar” container.

3. How to Propose “Distributed Tracing” as a Service

This is a major value-add. Frame it as “Reducing Mean Time to Recovery (MTTR).”

The Pitch:

“Currently, when a performance issue occurs, we have to manually comb through logs across multiple containers to find the root cause. I propose implementing Distributed Tracing. This will give us a visual ‘Application Map’ of our microservices, allowing us to pinpoint exactly which service or database query is slowing down the system in seconds rather than hours.”

4. Practical Implementation Plan

If they say “Yes,” here is your 3-step rollout:

Infrastructure: Add the Application Insights resource via your Terraform code.
Instrumentation: Add an OpenTelemetry “Sidecar” to your Docker deployments (or use the Azure Monitor AKS add-on).
Dashboarding: Create a “Latency Heatmap” in Azure Managed Grafana that pulls data from App Insights.

The “SRE” Comparison

To show your client you know your stuff, use this comparison:

Logs: Tell you the “What” (e.g., “Database connection failed”).
Metrics: Tell you the “When” (e.g., “CPU spiked at 3 PM”).
Tracing: Tells you the “Where” (e.g., “The delay is happening specifically in the Authentication Service’s call to the User DB”).

To move from basic logs to full distributed tracing, we need to add Application Insights and a Log Analytics Workspace to your Terraform configuration.

By 2026, the standard is to use Workspace-based Application Insights, which stores all its data in a centralized Log Analytics workspace. This makes it easier to query both your system logs and your application traces in one place.

1. The Terraform Code (Infrastructure)

Add this to your Terraform files. This creates the monitoring “bucket” and the tracing “engine.”

Terraform

			
# 1. Create the Log Analytics Workspace (The storage)
resource "azurerm_log_analytics_workspace" "aks_monitor" {
  name                = "law-aks-prod-01"
  location            = azurerm_resource_group.aks_rg.location
  resource_group_name = azurerm_resource_group.aks_rg.name
  sku                 = "PerGB2018"
  retention_in_days   = 30
}
# 2. Create Application Insights (The tracing engine)
resource "azurerm_application_insights" "aks_app_insights" {
  name                = "ai-microservices-prod"
  location            = azurerm_resource_group.aks_rg.location
  resource_group_name = azurerm_resource_group.aks_rg.name
  workspace_id        = azurerm_log_analytics_workspace.aks_monitor.id
  application_type    = "web"
}
# 3. Output the Connection String (Your apps need this to send data)
output "app_insights_connection_string" {
  value     = azurerm_application_insights.aks_app_insights.connection_string
  sensitive = true
}

		

2. The “Last Mile”: Connecting your Docker Apps

Once the infrastructure is built, your microservices need to know where to send their traces. In 2026, we use Autoinstrumentation so you don’t have to change your code.

Step A: Enable the AKS Monitoring Add-on

In your azurerm_kubernetes_cluster resource, add this block:

Terraform

  oms_agent {
    log_analytics_workspace_id      = azurerm_log_analytics_workspace.aks_monitor.id
    msi_auth_for_monitoring_enabled = true
  }

Step B: Inject the Connection String into your Pods

In your Kubernetes deployment YAML, add the Connection String as an environment variable (best practice: pull this from the Azure Key Vault we set up earlier):

YAML

			
env:
  - name: APPLICATIONINSIGHTS_CONNECTION_STRING
    value: "InstrumentationKey=xxxx-xxxx-xxxx;IngestionEndpoint=..."

3. What this looks like for the Client

Once this is running, you can show the client two powerful views in the Azure Portal:

The Application Map: A live, visual diagram showing how Service A calls Service B. If a service turns red, it means it’s failing. If the line is thick/slow, it’s a bottleneck.
End-to-End Transaction Details: You can click on a single failed request and see exactly where it died—whether it was a code error in the container or a slow query in the database.

The Proposal Pitch

“Currently, we have ‘blind spots’ between our services. By implementing this Terraform-backed tracing, we can move from reactive log-hunting to Visual Troubleshooting. We’ll be able to see exactly how requests flow through our Docker containers, allowing us to fix performance issues before they impact the end-user.”

Understanding Azure RBAC vs Kubernetes RBAC

April 24, 2026April 24, 2026 techhadoop AKS, azure azure, cloud, devops, kubernetes, technology

When explaining this to a client, it is helpful to describe it as the difference between who can touch the physical server (the building) versus who can edit the files on the computer inside (the office).

In 2026, the industry standard is to use Azure RBAC for both, but they still operate on two distinct “control planes.”

1. The Two Control Planes

In AKS, access is split into two layers:

The Azure Control Plane (Azure RBAC): This governs the “outside” of the cluster. It’s about the Kubernetes resource itself as it exists in your Azure portal.
The Kubernetes Control Plane (Kubernetes RBAC): This governs the “inside” of the cluster. It’s about the pods, namespaces, and deployments running on the nodes.

2. Side-by-Side Comparison

Feature	Azure RBAC	Kubernetes RBAC
Scope	Subscription / Resource Group / AKS Resource	Cluster / Namespace / Specific Pods
Managed Via	Azure Portal, CLI, Terraform	`kubectl`, YAML manifests, Helm
Typical Actions	Scaling nodes, Upgrading K8s version, Deleting the cluster.	Creating a Pod, Editing a Service, Viewing logs in a Namespace.
Identity Source	Microsoft Entra ID (Azure AD)	Service Accounts (or Entra ID via integration)

3. The “Hybrid” Option (Azure RBAC for K8s Authorization)

This is the most confusing part for beginners, but the most important for you to propose to your client.

You can now use Azure RBAC to manage internal Kubernetes permissions. Instead of writing complex RoleBinding YAML files for every user, you assign them a built-in Azure role that Kubernetes understands.

Key Built-in Roles (2026 Standards):

AKS RBAC Viewer: Can see resources in a namespace but can’t see secrets or change anything.
AKS RBAC Writer: Can deploy apps and edit resources.
AKS RBAC Admin: Full control over a namespace.
AKS RBAC Cluster Admin: The “God Mode” for the entire cluster.

4. How to Explain the Workflow to Your Manager

“Think of it like a bank:

Azure RBAC is the security guard at the front door. He checks your ID (Entra ID) and decides if you’re even allowed in the building. He also decides who can add more teller windows (Scale nodes) or renovate the lobby (Upgrade cluster).
Kubernetes RBAC is the permissions on the safe. Once you’re inside, it decides if you can open Drawer A (Namespace ‘Dev’) or Drawer B (Namespace ‘Prod’).”

Pro-Tip: Recommendation

If you want to provide “Gold Standard” support, propose disabling local accounts and moving entirely to Azure RBAC for Kubernetes Authorization. > Why? Because when an employee leaves the company and their Entra ID (Azure AD) is deleted, their access to the Kubernetes cluster is instantly revoked. No orphaned RoleBindings to worry about.

Understanding how a developer goes from their laptop to a running container in a secured AKS environment is the best way to prove the value of your setup.

Here is the step-by-step lifecycle of a developer’s request in a Zero-Trust AKS environment.

The Access Lifecycle (Step-by-Step)

1. Authentication (The Gatekeeper)

The developer doesn’t have a “Kubernetes password.” Instead, they run:

Bash

			
az login
az aks get-credentials --resource-group rg-prod --name aks-01

At this moment, Azure RBAC checks if their Entra ID account has permission to even download the cluster configuration.

2. Authorization (The Office Door)

The developer tries to deploy a new microservice:

Bash

kubectl apply -f my-app.yaml

The AKS API Server intercepts this. Since we are using Azure RBAC for Kubernetes Authorization, it asks Entra ID: “Does this user have the ‘AKS RBAC Writer’ role for the ‘Production’ namespace?” * If Yes: The request proceeds.

If No: The request is blocked with a 403 Forbidden error.

3. Policy Validation (The Safety Inspector)

Before the pod is actually scheduled, Azure Policy (the Admission Controller) scans the my-app.yaml.

It checks: “Is this container trying to run as root? Does it have CPU limits?” * If the YAML is “lazy” (insecure), Azure Policy rejects it immediately, even though the developer has “Writer” permissions.

4. Identity & Secrets (The Secure Handshake)

Once the pod starts, it needs to talk to the database.

The pod presents its Workload Identity (a managed identity) to the Azure Key Vault.
Key Vault verifies the pod’s identity and hands over the database string via the CSI Driver.
The password is never stored in a file or an environment variable where a human could see it.

Summary Table for Your Proposal

To wrap this up for your client, you can present this “Success Path” to show them exactly what they are paying for:

Stage	Security Layer	Purpose
Login	Entra ID	Ensures only active employees can connect.
Action	Azure RBAC	Limits what a developer can do (e.g., Read vs. Write).
Deploy	Azure Policy	Forces best practices (No root, resource limits).
Connect	Workload Identity	Eliminates hardcoded passwords in the code.

Pro-Tip: The “Audit” Hook

Tell your client: “With this setup, we can generate a report at any time showing exactly who accessed the production cluster and what they changed. This makes SOC2 or ISO27001 audits a breeze.”

1. The “Single AI Spoke” Strategy (Recommended)

2. The “Multi-Spoke” Strategy (Enterprise Scale)

3. The “Cross-Service” Security Checklist

4. Final Recommendation

1. Subnet Segmentation (The “Clean” Spoke)

2. The 2026 Security “Golden Rules”

3. Shared Private Links (The Handshake)

4. Why a “Single Spoke” for one department?

5. Final Recommendation Summary

1. Networking Strategy

2. Key Architectural Components (The “2026 Check”)

3. Recommendations for your Managed Identities

4. Final Security Check

1. The Network Backbone: Hub-and-Spoke

2. Private Link & Managed Identities (No Keys!)

3. Databricks-Specific Security (The Terraform Focus)

4. Advanced Protection for RAG

Summary Checklist for your Terraform Modules

1. The Production Dockerfile

2. The requirements.txt

3. Deploying to AKS (The “Support” Strategy)

4. How to Pitch the “AI Operations” Tier

One final piece of the puzzle

1. The RBAC Strategy

2. The RBAC YAML (mcp-rbac.yaml)

3. Updating your Deployment

4. Why this is “Safe” for your Client

Putting it all together

1. Consuming AI (The “API” Route)

2. Hosting AI (The “KAITO” Route)

3. Infrastructure Requirements (GPU Nodes)

4. Monitoring AI Performance

How to Pitch This to Your Client

Automating Velero AKS Backup with Ansible & Terraform

Option 1: Terraform

Project Structure

providers.tf

variables.tf

main.tf

outputs.tf

Deploy with Terraform

Option 2: Ansible

Project Structure

group_vars/all.yml

roles/azure_storage/tasks/main.yml

roles/azure_identity/tasks/main.yml

roles/velero/templates/credentials.j2

roles/velero/templates/backup-schedule.yml.j2

roles/velero/tasks/main.yml

site.yml (Main Playbook)

Run the Ansible Playbook

Comparison: Terraform vs Ansible

Best Practice: Combine Both

Backing Up AKS with Velero

Prerequisites

Step 1: Create Azure Storage Account & Blob Container

Step 2: Create Service Principal for Velero

Step 3: Create Velero Credentials File

Step 4: Install Velero on AKS

Step 5: Verify Velero Installation

Step 6: Create Backups

Manual Backup (Full Cluster)

Backup Specific Namespace

Backup with Labels

Exclude Specific Resources

Step 7: Schedule Automated Backups

Step 8: Restore from Backup

Full Restore

Restore Specific Namespace

Restore to Different Namespace

Restore Specific Resources Only

Step 9: Backup Hooks (Pre/Post Actions)

Step 10: Monitor & Troubleshoot

Summary Cheat Sheet

Best Practices

Scenario 1: Entire AKS Cluster Becomes Unreachable

Situation

What’s actually happening?

How to respond (structured)

Strong answer

2. The `requirements.txt`

2. The RBAC YAML (`mcp-rbac.yaml`)