Microsoft Sentinel: Automating Threat Response in Azure

Azure Sentinel (Microsoft Sentinel)

Microsoft Sentinel is Azure’s cloud-native SIEM (Security Information and Event Management) and SOAR (Security Orchestration, Automation and Response) platform — a single service that collects security data from across your entire estate, detects threats using AI and analytics, investigates incidents, and automates responses.


What Sentinel Actually Is — SIEM + SOAR Combined

SIEM (Security Information and Event Management)
→ Collects logs from everything
→ Correlates events across sources
→ Detects threats using rules + AI
→ Surfaces alerts and incidents
SOAR (Security Orchestration, Automation and Response)
→ Automates response to detected threats
→ Runs playbooks (Logic Apps) automatically
→ Integrates with ticketing, ITSM, and remediation tools
→ Reduces mean time to respond (MTTR)
Sentinel = both in one service, built on Log Analytics

The Four Pillars

Pillar 1 — Collect

Data flows into Sentinel through data connectors — pre-built integrations that normalise log formats and write to Log Analytics tables:

Azure native connectors (free ingestion):

  • Microsoft Defender for Cloud
  • Entra ID sign-in and audit logs
  • Azure Activity logs (ARM operations)
  • Azure Firewall logs
  • NSG flow logs
  • Key Vault audit logs
  • Azure Kubernetes Service (AKS/ARO)

Microsoft 365 connectors:

  • Microsoft 365 Defender (XDR)
  • Office 365 (Exchange, SharePoint, Teams)
  • Microsoft Defender for Endpoint
  • Microsoft Defender for Identity
  • Microsoft Defender for Cloud Apps

Third-party connectors:

  • Palo Alto, Fortinet, Check Point firewalls
  • Cisco ASA, Umbrella, Meraki
  • Okta, CrowdStrike, SentinelOne
  • AWS CloudTrail, S3 access logs
  • GCP audit logs

On-premises via agents:

Windows VMs → Log Analytics Agent → SecurityEvent table
Linux VMs → Syslog → Syslog table
Network devices → CEF → AMA agent → CommonSecurityLog table

Pillar 2 — Detect

Sentinel detects threats through five types of analytics rules:

Scheduled rules — KQL queries on a timer
// Detect impossible travel — same user, two countries, <1 hour apart
let threshold_minutes = 60;
SigninLogs
| where TimeGenerated > ago(1d)
| where ResultType == 0 // successful sign-in
| project TimeGenerated, UserPrincipalName,
Location, IPAddress,
Latitude = toreal(LocationDetails.geoCoordinates.latitude),
Longitude = toreal(LocationDetails.geoCoordinates.longitude)
| sort by UserPrincipalName, TimeGenerated asc
| extend PrevLocation = prev(Location, 1),
PrevTime = prev(TimeGenerated, 1),
PrevUser = prev(UserPrincipalName, 1)
| where UserPrincipalName == PrevUser
| extend TimeDiff = datetime_diff('minute', TimeGenerated, PrevTime)
| where TimeDiff < threshold_minutes
| where Location != PrevLocation
| project UserPrincipalName, Location, PrevLocation,
TimeDiff, IPAddress, TimeGenerated
Near Real-Time (NRT) rules — sub-1-minute detection
// Detect Azure Firewall blocking connections to known malicious IPs
AzureDiagnostics
| where Category == "AzureFirewallNetworkRule"
| where msg_s has "Deny"
| parse msg_s with * "from " SourceIP ":" SourcePort
" to " DestIP ":" DestPort ". Action: " Action
| join kind=inner (
ThreatIntelligenceIndicator
| where Active == true
| project NetworkIP, ThreatType, ConfidenceScore
) on $left.DestIP == $right.NetworkIP
| project TimeGenerated, SourceIP, DestIP, ThreatType, ConfidenceScore
Microsoft Security rules — auto-create incidents from Defender alerts

These automatically promote Defender for Cloud, Defender for Endpoint, and Defender for Identity alerts into Sentinel incidents with no KQL needed.

Fusion rules — ML-based multi-stage attack detection

Fusion uses machine learning to correlate low-severity signals across multiple products that individually look benign but together indicate an attack:

Signal 1: Entra ID — suspicious sign-in from anonymising proxy
Signal 2: Office 365 — mass email forwarding rule created
Signal 3: Azure — new service principal with owner role
Individual signals: low severity, easy to miss
Fusion correlation: HIGH severity — likely BEC (Business Email Compromise) attack
Anomaly rules — baseline + deviation detection

Sentinel builds behavioural baselines and alerts on deviations:

  • Unusual volume of data downloaded by a user
  • Login at an unusual time of day for this account
  • Process execution pattern not seen before on this host

Pillar 3 — Investigate

Incidents

Every triggered analytics rule creates an alert. Sentinel groups related alerts into incidents — the unit of work for a SOC analyst:

Incident: Possible BEC attack — john.smith@contoso.com
Severity: High
Status: New
Assigned: SOC Analyst 2
Alerts:
├── Impossible travel detected (Entra ID)
├── Mass forwarding rule created (Office 365)
└── New privileged service principal (Azure Activity)
Entities:
├── User: john.smith@contoso.com
├── IP: 185.220.101.45 (Tor exit node)
└── Host: LAPTOP-JSmith
MITRE ATT&CK:
├── T1078 — Valid accounts
├── T1114 — Email collection
└── T1098 — Account manipulation
Investigation graph

A visual relationship map automatically built from incident entities — shows how a user, IP, host, and mailbox are connected without manual correlation:

185.220.101.45 (Tor IP)
↓ signed in as
john.smith@contoso.com (user)
↓ created
Forward-all-mail rule (Office 365)
↓ same session created
sp-finance-automation (service principal)
↓ granted
Owner role on subscription

Entity pages

Every entity (user, IP, host, app) gets a timeline page showing all activity across all data sources — 90 days of context assembled automatically:

User: john.smith@contoso.com
Last 90 days:
├── Sign-ins: 847 (normal pattern: Mon-Fri 8am-6pm EST)
├── Anomalous sign-ins: 3 (Tor, Russia, Ukraine)
├── Files accessed: 12,847
├── Emails sent: 2,341
├── Azure resource operations: 156
└── Risk score: 94/100 (UEBA)

Pillar 4 — Respond (SOAR)

Playbooks are Azure Logic Apps triggered automatically when an incident is created or updated. They automate the first-response actions that would otherwise require a human:

Playbook 1 — Block compromised user automatically
Trigger: Sentinel incident created
Condition: Severity == High AND Entity type == User
Actions:
1. Get user details from Entra ID
2. Disable user account in Entra ID
3. Revoke all active sessions (MFA re-auth required)
4. Send Teams message to SOC channel:
"User john.smith auto-disabled — incident #1234"
5. Create ServiceNow ticket with incident details
6. Add comment to Sentinel incident:
"User account disabled at 14:32 UTC by playbook"
Playbook 2 — Isolate compromised VM
Trigger: Sentinel incident created
Condition: Severity == High AND Entity type == Host
Actions:
1. Get VM resource ID from entity
2. Apply isolation NSG (deny all inbound + outbound except Bastion)
az network nsg rule create --name ISOLATE --priority 100
--access Deny --direction Inbound --source-address-prefix *
3. Take VM disk snapshot (forensic preservation)
4. Tag VM: {"Status": "Isolated", "IncidentId": "1234"}
5. Notify SOC team via email + Teams
6. Create Jira ticket for IR team
Playbook 3 — Enrich IP with threat intelligence
Trigger: Sentinel alert contains IP entity
Actions:
1. Query VirusTotal API for IP reputation
2. Query Shodan for open ports and services
3. Query AbuseIPDB for abuse reports
4. Add enrichment comment to incident:
"IP 185.220.101.45:
VirusTotal: 47/92 vendors flagged malicious
AbuseIPDB: 847 reports, 100% confidence malicious
Shodan: Tor exit node — AS16276 OVH"
5. If malicious score > 80:
→ Add IP to Azure Firewall deny list automatically

KQL — The Query Language of Sentinel

Everything in Sentinel is queried with KQL (Kusto Query Language):

// Find all failed logins followed by success from same IP
// (credential stuffing pattern)
let failed_logins = SigninLogs
| where TimeGenerated > ago(1h)
| where ResultType != 0 // failed
| summarize FailCount = count() by IPAddress, UserPrincipalName
| where FailCount > 10;
let successful_logins = SigninLogs
| where TimeGenerated > ago(1h)
| where ResultType == 0 // success
| project IPAddress, UserPrincipalName, SuccessTime = TimeGenerated;
successful_logins
| join kind=inner failed_logins on IPAddress
| project IPAddress, UserPrincipalName,
FailCount, SuccessTime
| order by FailCount desc
// Detect Azure privilege escalation — new owner role assignment
AzureActivity
| where TimeGenerated > ago(1d)
| where OperationNameValue == "MICROSOFT.AUTHORIZATION/ROLEASSIGNMENTS/WRITE"
| where ActivityStatusValue == "Success"
| extend RoleDefinitionId = tostring(
parse_json(Properties).requestbody.properties.roleDefinitionId)
| where RoleDefinitionId contains "8e3af657-a8ff-443c-a75c-2fe8c4bcb635" // Owner
| project TimeGenerated, Caller, ResourceGroup,
SubscriptionId, RoleDefinitionId
// Hunt for lateral movement via PsExec or WMI
SecurityEvent
| where TimeGenerated > ago(7d)
| where EventID in (4688, 4624) // process create + logon
| where ProcessName has_any ("psexec", "wmic", "winrm")
or CommandLine has_any ("\\\\", "invoke-wmimethod", "wmiexec")
| summarize count() by Computer, Account, ProcessName, CommandLine
| order by count_ desc

MITRE ATT&CK Integration

Sentinel maps every analytics rule to MITRE ATT&CK tactics and techniques — giving you a visual coverage matrix:

TacticExample TechniqueSentinel Detection
Initial AccessT1078 Valid AccountsImpossible travel rule
PersistenceT1098 Account ManipulationNew owner role assignment
Privilege EscalationT1134 Token ImpersonationService principal abuse
Defence EvasionT1562 Impair DefencesDiagnostic setting deleted
Credential AccessT1110 Brute ForceFailed login threshold
Lateral MovementT1021 Remote ServicesPsExec / WMI detection
ExfiltrationT1048 Exfil over Alt ProtocolLarge blob download
ImpactT1486 Data EncryptedRansomware file extension

Sentinel in Hub and Spoke Context

In an enterprise hub and spoke topology, Sentinel sits at the subscription/tenant level — above the network, collecting from everything:

Microsoft Sentinel (Log Analytics Workspace)
│ data connectors
┌────┴──────────────────────────────────┐
│ │
Hub VNet Spoke VNets
Azure Firewall logs AKS/ARO audit logs
VPN Gateway logs VM security events
Bastion session logs NSG flow logs
DNS resolver logs App Gateway WAF logs
On-premises (via MMA/AMA agent)
Windows Security Events
Linux Syslog
Network device CEF

Sentinel vs Defender for Cloud

Microsoft SentinelDefender for Cloud
TypeSIEM + SOARCSPM + CWPP
FocusThreat detection + responsePosture management + workload protection
ScopeCross-tenant, multi-cloudAzure resources + connected clouds
DataAll log sourcesAzure resource configuration + telemetry
OutputIncidents + playbooksRecommendations + alerts
Use togetherDefender feeds alerts into SentinelSentinel adds SOAR response to Defender alerts

They are designed to work together — Defender for Cloud detects threats at the resource level and feeds high-fidelity alerts into Sentinel, which correlates them with signals from every other source and automates the response.


Pricing Model

Sentinel pricing has two components:

Log Analytics ingestion — pay per GB ingested:

  • Pay-as-you-go: ~$2.76/GB
  • Commitment tiers: 100 GB/day → 500 GB/day → lower per-GB rate

Sentinel capacity reservation — flat daily rate above the free Log Analytics tier:

  • First 10 GB/day per workspace: free
  • Above 10 GB/day: ~$100–$400/day depending on tier

Free data sources — no ingestion charge for:

  • Microsoft Defender alerts
  • Entra ID audit + sign-in logs (Basic SKU)
  • Azure Activity logs
  • Office 365 management activity

Key Takeaway

Microsoft Sentinel is the security brain of your Azure estate — it ingests logs from every corner of your infrastructure (Azure, Microsoft 365, on-premises, third-party), correlates signals using AI and KQL-based rules, groups related alerts into actionable incidents mapped to MITRE ATT&CK, and automates first-response actions through Logic App playbooks. In a hub and spoke network, it sits above the topology collecting from every layer — firewall, gateway, Bastion, ARO, VMs, and on-premises — giving your SOC a single pane of glass across the entire estate.

Understanding ARO’s Kubernetes API Operations

Kubernetes API Operations Through the ARO Private Endpoint

Every interaction with an ARO cluster — whether from a human, a tool, or an automated controller — flows through a single TCP connection to port 6443 on the API server private endpoint. The API server is the absolute centre of gravity for all cluster operations.


Every Operation Is a REST Call

The Kubernetes API server exposes a RESTful HTTP/2 API over TLS. Every tool — kubectl, oc, operators, kubelet — translates its work into one of five HTTP verbs against a resource path:

GET /api/v1/namespaces/payments/pods list pods
GET /api/v1/namespaces/payments/pods/web-1 get single pod
POST /api/v1/namespaces/payments/pods create pod
PUT /api/v1/namespaces/payments/pods/web-1 replace pod
PATCH /api/v1/namespaces/payments/pods/web-1 partial update
DELETE /api/v1/namespaces/payments/pods/web-1 delete pod
GET /api/v1/namespaces/payments/pods?watch=1 watch stream

Every one of these travels as TLS-encrypted HTTP/2 to 10.1.0.8:6443.


Category 1 — Human CLI Operations (kubectl + oc)

kubectl — standard Kubernetes operations

# Every one of these becomes a REST call through the private endpoint
# LIST pods → GET /api/v1/namespaces/default/pods
kubectl get pods -n payments
# CREATE deployment → POST /apps/v1/namespaces/payments/deployments
kubectl apply -f deployment.yaml
# EXEC into pod → POST + UPGRADE to SPDY/WebSocket
kubectl exec -it web-1 -- /bin/bash
# PORT-FORWARD → POST + WebSocket tunnel
kubectl port-forward svc/my-app 8080:80
# LOGS → GET /api/v1/namespaces/payments/pods/web-1/log
kubectl logs web-1 --follow
# WATCH resources → GET with ?watch=1 (long-lived streaming connection)
kubectl get pods --watch

oc CLI — OpenShift-specific additions

oc wraps kubectl completely and adds calls to OpenShift-specific API groups:

# OpenShift Route → POST /apis/route.openshift.io/v1/namespaces/.../routes
oc expose svc/my-app
# Project (OpenShift namespace wrapper)
# → POST /apis/project.openshift.io/v1/projectrequests
oc new-project my-team
# ImageStream → GET /apis/image.openshift.io/v1/namespaces/.../imagestreams
oc get imagestreams
# BuildConfig → POST /apis/build.openshift.io/v1/namespaces/.../builds
oc start-build my-app
# DeploymentConfig (legacy OpenShift resource)
# → GET /apis/apps.openshift.io/v1/namespaces/.../deploymentconfigs
oc rollout latest dc/my-app
# SCC inspection → GET /apis/security.openshift.io/v1/securitycontextconstraints
oc get scc

Category 2 — Operators and Controllers

Operators are long-running processes inside the cluster that maintain perpetual watch connections to the API server — the busiest category of API consumers by connection count.

The watch loop — how operators work

// Every operator runs this pattern against the API server
// Connection: persistent HTTP/2 stream to 10.1.0.8:6443
// 1. LIST — get current state (one-time at startup)
GET /apis/apps/v1/namespaces/payments/deployments
→ Returns: all deployments + resourceVersion: 48291
// 2. WATCH — subscribe to changes (permanent long-poll)
GET /apis/apps/v1/namespaces/payments/deployments?watch=1&resourceVersion=48291
→ Server keeps connection open indefinitely
→ Pushes events as they occur:
{"type":"MODIFIED","object":{"metadata":{"name":"web"},...}}
{"type":"ADDED","object":{"metadata":{"name":"worker"},...}}
{"type":"DELETED","object":{"metadata":{"name":"old"},...}}
// 3. RECONCILE — when event received, fix actual → desired state
PATCH /apis/apps/v1/namespaces/payments/replicasets/web-abc
→ Creates/deletes pods to match desired replicas
// 4. STATUS UPDATE — write observed state back
PATCH /apis/apps/v1/namespaces/payments/deployments/web/status
→ {"observedGeneration": 5, "availableReplicas": 3}

Built-in OpenShift operators that run this loop continuously

OperatorWhat it watchesWhat it does
openshift-apiserver-operatorapiservers.config.openshift.ioManages API server config and certs
cluster-version-operatorclusterversions.config.openshift.ioDrives cluster upgrades
machine-config-operatormachineconfigs, machineconfigpoolsApplies RHCOS config to nodes
ingress-operatoringresses.config.openshift.ioManages router deployments
dns-operatordnses.config.openshift.ioManages CoreDNS config
network-operatornetworks.config.openshift.ioManages OVN-Kubernetes
image-registry-operatorconfigs.imageregistry.operator.openshift.ioManages internal registry
authentication-operatorauthentications.config.openshift.ioManages OAuth server

Every one of these has persistent watch connections open to the API server at all times — a healthy ARO cluster typically has 40–80 active watch streams running 24/7.


Category 3 — Kubelet (Node Agent)

Every worker node runs a kubelet process that maintains its own connection to the API server — reporting node health and receiving pod assignments:

Worker node kubelet → 10.1.0.8:6443
Outbound (kubelet → API server):
POST /api/v1/nodes/worker-1/status every 10 seconds — node heartbeat
PATCH /api/v1/namespaces/app/pods/web-1/status when pod state changes
POST /api/v1/events kubelet events (OOM, image pull)
Inbound (API server → kubelet port 10250):
GET https://worker-1:10250/exec/... kubectl exec forwarding
GET https://worker-1:10250/log/... kubectl logs forwarding
GET https://worker-1:10250/metrics Prometheus scraping

If the kubelet loses its connection to the API server for more than the node-monitor-grace-period (default 40 seconds), the node is marked NotReady and pods begin eviction.


Category 4 — CI/CD Pipelines

Self-hosted CI/CD runners inside the VNet authenticate to the API server using a service account token:

# Service account for CI/CD — scoped to specific namespace
apiVersion: v1
kind: ServiceAccount
metadata:
name: cicd-deployer
namespace: payments
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deployer
namespace: payments
rules:
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "create", "update", "patch"]
- apiGroups: [""]
resources: ["pods", "services", "configmaps"]
verbs: ["get", "list", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cicd-deployer-binding
namespace: payments
roleRef:
kind: Role
name: deployer
subjects:
- kind: ServiceAccount
name: cicd-deployer
namespace: payments

GitHub Actions pipeline using this service account:

- name: Deploy to ARO
run: |
# Authenticate with service account token — all traffic to 10.1.0.8:6443
oc login ${{ secrets.ARO_API_URL }} \
--token ${{ secrets.CICD_SA_TOKEN }}
# Each command = REST call through private endpoint
oc set image deployment/web \
web=acrprod.azurecr.io/my-app:${{ github.sha }} \
-n payments
oc rollout status deployment/web -n payments

Category 5 — Admission Webhooks

Admission webhooks add an external hop during the API server request pipeline — the API server calls out to your webhook service before persisting any object:

kubectl apply -f pod.yaml
API server receives POST /api/v1/namespaces/payments/pods
Authn + RBAC pass
Mutating admission webhook:
API server → POST https://gatekeeper-webhook.gatekeeper-system.svc:443/mutate
Webhook adds labels, sets resource limits, injects sidecars
→ Returns mutated pod spec
Validating admission webhook:
API server → POST https://gatekeeper-webhook.gatekeeper-system.svc:443/validate
Checks policy: must have resource limits, no root, valid image registry
→ Returns: allowed: true (or denied with reason)
Persist to etcd → notify watchers → return 201 Created

Common admission webhooks in ARO:

WebhookPurpose
OPA GatekeeperPolicy enforcement — block non-compliant resources
KyvernoPolicy as code — mutate, validate, generate
Istio / OpenShift Service MeshInject Envoy sidecar into pods automatically
Red Hat ACMMulti-cluster governance policies
Cert-managerInject TLS certificates into resources

Category 6 — Monitoring and Observability

# Prometheus scrapes API server metrics via the API endpoint
GET https://10.1.0.8:6443/metrics
# Returns: apiserver_request_total, apiserver_request_duration_seconds,
# etcd_request_duration_seconds, workqueue_depth, ...
# Health endpoints checked by Azure ARO service monitor
GET https://10.1.0.8:6443/healthz → "ok"
GET https://10.1.0.8:6443/readyz → "ok"
GET https://10.1.0.8:6443/livez → "ok"
# OpenShift console reads cluster state continuously
GET /apis/config.openshift.io/v1/clusterversions/version
GET /api/v1/namespaces?limit=500
GET /apis/project.openshift.io/v1/projects

The Request Pipeline — What Happens Inside

Every request through the private endpoint traverses this exact pipeline inside kube-apiserver:

TLS handshake on 10.1.0.8:6443
1. AUTHENTICATION — who are you?
• OIDC token (Entra ID) → extract user + groups
• x509 client cert → extract CN as username
• Bearer token → look up service account
• Failure → 401 Unauthorized
2. AUTHORIZATION (RBAC) — are you allowed?
• Check: user + groups + verb + resource + namespace
• ClusterRoleBinding / RoleBinding lookup
• OpenShift SCC evaluation for pods
• Failure → 403 Forbidden
3. ADMISSION CONTROL — is this allowed by policy?
• Mutating webhooks (modify the object)
• Built-in admission plugins (ResourceQuota, LimitRanger)
• Validating webhooks (accept or reject)
• Failure → 400/403 with reason
4. VALIDATION — is the object schema correct?
• OpenAPI schema validation
• CRD schema validation
• Field immutability checks
• Failure → 422 Unprocessable Entity
5. PERSIST TO etcd
• Serialise to protobuf
• Encrypt at rest (AES-GCM, ARO managed)
• Write to etcd with optimistic concurrency (resourceVersion)
• Failure → 409 Conflict (resourceVersion mismatch)
6. NOTIFY WATCHERS
• Push event to all active watch streams matching the resource
• Controllers, operators, scheduler, kubelet all receive notification
7. RETURN RESPONSE
• 200 OK (GET)
• 201 Created (POST)
• 200 OK with updated object (PATCH/PUT)
• 404 Not Found
• Streaming response for watch/exec/logs/port-forward

API Groups — Kubernetes vs OpenShift

The API server serves two parallel API surfaces — Kubernetes core APIs and OpenShift extension APIs — all through the same 10.1.0.8:6443 endpoint:

Kubernetes core APIs:
/api/v1/ pods, services, configmaps, secrets, nodes
/apis/apps/v1/ deployments, replicasets, statefulsets, daemonsets
/apis/batch/v1/ jobs, cronjobs
/apis/rbac.authorization.k8s.io/ clusterroles, rolebindings
/apis/storage.k8s.io/ storageclasses, persistentvolumes
/apis/networking.k8s.io/ ingresses, networkpolicies
OpenShift extension APIs:
/apis/route.openshift.io/ routes (OpenShift ingress primitive)
/apis/project.openshift.io/ projects (namespace + RBAC wrapper)
/apis/build.openshift.io/ buildconfigs, builds
/apis/image.openshift.io/ imagestreams, imagestreamtags
/apis/apps.openshift.io/ deploymentconfigs (legacy)
/apis/security.openshift.io/ securitycontextconstraints
/apis/config.openshift.io/ cluster-wide config (DNS, network, auth)
/apis/operator.openshift.io/ operator configuration resources
/apis/machine.openshift.io/ machines, machinesets (MachineAPI)

Key Takeaway

The ARO API server private endpoint at 10.1.0.8:6443 is not just the entry point for human CLI commands — it is the nervous system of the entire cluster. Every automated process — the 40+ built-in OpenShift operators maintaining cluster state, every kubelet heartbeating from every worker node every 10 seconds, every CI/CD deployment, every admission webhook validation, every Prometheus health check — flows through this single TLS endpoint. Making it private eliminates the internet attack surface entirely, while the seven-stage request pipeline inside the API server ensures every operation is authenticated, authorised, policy-checked, validated, and durably persisted before any response is returned.

Understanding ARO’s Private DNS Zones Setup

Private DNS Zones Created by ARO

When you deploy a private ARO cluster, Azure automatically creates two private DNS zones in the ARO-managed resource group — one for the API server and one for application ingress. You own neither; they are managed by the ARO service, but you are responsible for linking them to every VNet that needs to resolve them.


The Two Zones ARO Creates

ARO creates both zones inside the ARO managed resource group — the resource group whose name starts with aro- that Azure creates automatically alongside your cluster. You cannot modify or delete these zones without breaking the cluster.

Zone 1 — API Server

Zone name: cluster.<unique-id>.<region>.aroapp.io
Example: cluster.a1b2c3d4.eastus.aroapp.io
A records:
api → 10.1.0.8 (private endpoint NIC IP)
Full FQDN: api.cluster.a1b2c3d4.eastus.aroapp.io:6443

Zone 2 — Application Ingress

Zone name:   cluster.<unique-id>.<region>.aroapp.io
             (same parent zone, different record)

A records:
  *.apps   →   10.1.1.100   (internal load balancer frontend IP)

Example resolutions:
  my-app.apps.cluster.a1b2c3d4.eastus.aroapp.io  →  10.1.1.100
  console.apps.cluster.a1b2c3d4.eastus.aroapp.io →  10.1.1.100
  grafana.apps.cluster.a1b2c3d4.eastus.aroapp.io →  10.1.1.100


Inspecting the Zones After Deployment

# Get the ARO managed resource group
MANAGED_RG=$(az aro show \
  --resource-group rg-aro \
  --name aro-prod \
  --query clusterProfile.resourceGroupId \
  --output tsv | xargs basename)

# List all private DNS zones ARO created
az network private-dns zone list \
  --resource-group $MANAGED_RG \
  --query "[].{Zone:name, Records:numberOfRecordSets}" \
  --output table

# Output:
# Zone                                          Records
# ─────────────────────────────────────────── ─────────
# cluster.a1b2c3d4.eastus.aroapp.io           4

# List all A records in the zone
az network private-dns record-set a list \
  --resource-group $MANAGED_RG \
  --zone-name cluster.a1b2c3d4.eastus.aroapp.io \
  --output table

# Output:
# Name     TTL    ARecords
# ──────   ────   ──────────────────────
# api      300    [{'ipv4Address': '10.1.0.8'}]
# *.apps   300    [{'ipv4Address': '10.1.1.100'}]

# List VNet links on the zone
az network private-dns link vnet list \
  --resource-group $MANAGED_RG \
  --zone-name cluster.a1b2c3d4.eastus.aroapp.io \
  --output table

# Output:
# Name                    VirtualNetwork          RegistrationEnabled
# ─────────────────────── ──────────────────────  ───────────────────
# aro-spoke-vnet-link     aro-spoke-vnet          false   ← auto-created


The VNet Linking Problem

This is the most common post-deployment mistake. ARO automatically links the private DNS zone to only one VNet — the ARO spoke VNet. Every other VNet that needs to resolve the API server or app routes must be manually linked:

ZONE_NAME="cluster.a1b2c3d4.eastus.aroapp.io"

# Link to hub VNet (required for jump host, Bastion, CI/CD runners)
az network private-dns link vnet create \
  --resource-group $MANAGED_RG \
  --zone-name $ZONE_NAME \
  --name "link-to-hub-vnet" \
  --virtual-network $(az network vnet show \
      --resource-group rg-hub \
      --name hub-vnet \
      --query id -o tsv) \
  --registration-enabled false

# Link to other spoke VNets if they need to call ARO-hosted APIs
az network private-dns link vnet create \
  --resource-group $MANAGED_RG \
  --zone-name $ZONE_NAME \
  --name "link-to-spoke2-vnet" \
  --virtual-network $(az network vnet show \
      --resource-group rg-spoke2 \
      --name spoke2-vnet \
      --query id -o tsv) \
  --registration-enabled false

registration-enabled false is correct here — you are linking for resolution only, not to auto-register VM hostnames into the zone.


On-Premises DNS Conditional Forwarding

On-premises DNS servers cannot be linked to Azure private DNS zones directly — they resolve through the DNS Private Resolver inbound endpoint using conditional forwarding:

On-premises Windows DNS Server:
Conditional Forwarder:
Domain: aroapp.io
Forward to: 10.0.5.4 (DNS Private Resolver inbound endpoint)
On-premises BIND (Linux):
zone "aroapp.io" {
type forward;
forwarders { 10.0.5.4; };
};

With this in place, an on-premises developer running oc login gets the full resolution chain:

1. oc login https://api.cluster.a1b2c3d4.eastus.aroapp.io:6443
2. Laptop DNS → corp DNS server
3. Corp DNS: aroapp.io → forward to 10.0.5.4
4. DNS Private Resolver checks linked private DNS zones
5. Finds: api.cluster.a1b2c3d4.eastus.aroapp.io → 10.1.0.8
6. Returns 10.1.0.8 to laptop
7. oc connects to 10.1.0.8:6443 via ExpressRoute / VPN tunnel
8. Login succeeds ✅

OpenShift Console DNS

The OpenShift web console runs as an application on the cluster, so it resolves through the *.apps wildcard record:

# Get console URL
az aro show \
  --resource-group rg-aro \
  --name aro-prod \
  --query consoleProfile.url \
  --output tsv

# Output:
# https://console-openshift-console.apps.cluster.a1b2c3d4.eastus.aroapp.io

# DNS resolution:
# console-openshift-console.apps.cluster.a1b2c3d4.eastus.aroapp.io
#   → matched by *.apps wildcard A record
#   → returns 10.1.1.100 (internal LB)
#   → browser connects via VPN/ER or Bastion proxy


Custom Domain — Replacing aroapp.io

If you want your own domain (e.g. openshift.contoso.com) instead of aroapp.io, you create a custom private DNS zone and manage the records yourself:

# Create your own private DNS zone
az network private-dns zone create \
  --resource-group rg-aro-network \
  --name openshift.contoso.com

# Add API server A record
az network private-dns record-set a add-record \
  --resource-group rg-aro-network \
  --zone-name openshift.contoso.com \
  --record-set-name api \
  --ipv4-address 10.1.0.8

# Add wildcard apps A record
az network private-dns record-set a add-record \
  --resource-group rg-aro-network \
  --zone-name openshift.contoso.com \
  --record-set-name "*.apps" \
  --ipv4-address 10.1.1.100

# Link to all VNets
az network private-dns link vnet create \
  --resource-group rg-aro-network \
  --zone-name openshift.contoso.com \
  --name link-hub \
  --virtual-network /subscriptions/.../hub-vnet \
  --registration-enabled false

Then update the ARO cluster to use the custom domain during deployment:

az aro create \
  --resource-group rg-aro \
  --name aro-prod \
  --vnet aro-spoke-vnet \
  --master-subnet master-subnet \
  --worker-subnet worker-subnet \
  --apiserver-visibility Private \
  --ingress-visibility Private \
  --domain openshift.contoso.com \    # ← custom domain
  --pull-secret @pull-secret.txt

With a custom domain the API server becomes api.openshift.contoso.com and apps become *.apps.openshift.contoso.com — owned and managed entirely by you, with no dependency on aroapp.io.


Key Takeaway

ARO automatically creates a private DNS zone under aroapp.io with two critical records — api pointing to the API server private endpoint IP and *.apps pointing to the internal load balancer frontend IP. The zone is auto-linked only to the ARO spoke VNet — you must manually link it to the hub VNet, any other spoke VNets, and configure on-premises DNS conditional forwarding to the DNS Private Resolver for the complete name resolution chain to work end-to-end across your hub and spoke estate.

Understanding ARO API Server Private Endpoint IPs

ARO API Server Private Endpoint IP

The ARO API server private endpoint IP is a private IP address automatically assigned by Azure from your master subnet’s address space when the cluster is created — it becomes the sole network entry point for all Kubernetes API traffic in a private cluster.


How the IP Is Assigned — the Full Mechanism

Azure subnet IP allocation order

Every Azure subnet reserves the first 5 IPs unconditionally:

10.1.0.0 — Network address (unusable)
10.1.0.1 — Default gateway
10.1.0.2 — Azure DNS
10.1.0.3 — Azure future use
10.1.0.4 — Broadcast address
─────────────────────────────────
10.1.0.5 → First assignable IP

After cluster provisioning, the master subnet fills up in this order:

10.1.0.5 → Master node 1 VM NIC (AZ1)
10.1.0.6 → Master node 2 VM NIC (AZ2)
10.1.0.7 → Master node 3 VM NIC (AZ3)
10.1.0.8 → API server private endpoint NIC ← auto-assigned
10.1.0.9 → Internal LB health probe IP
10.1.0.10+ → Future ARO platform components

The exact IP depends on provisioning order — Azure assigns the next available IP dynamically. You cannot pre-specify it, but once assigned it is static for the lifetime of the cluster.


The Private Endpoint NIC in Detail

The private endpoint is an Azure resource called a Private Endpoint — distinct from the VM NICs of the master nodes. You can inspect it:

# Find the ARO managed resource group (contains cluster infrastructure)
MANAGED_RG=$(az aro show \
--resource-group rg-aro \
--name aro-prod \
--query clusterProfile.resourceGroupId -o tsv)
# List private endpoints in the managed resource group
az network private-endpoint list \
--resource-group $MANAGED_RG \
--output table
# Output:
# Name ResourceGroup Location
# ───────────────────────────── ──────────────────── ─────────
# aro-prod-pe-apiserver aro-prod-cluster-rg eastus
# Get the private IP
az network private-endpoint show \
--resource-group $MANAGED_RG \
--name aro-prod-pe-apiserver \
--query 'customDnsConfigs[0].ipAddresses[0]' \
--output tsv
# Output: 10.1.0.8

What the Private Endpoint NIC Actually Is

The private endpoint is not a VM — it is a read-only synthetic NIC injected into your subnet by Azure’s network fabric. It has no OS, no compute, no management plane — it is purely a network construct:

Private Endpoint Resource
├── Name: aro-prod-pe-apiserver
├── Type: Microsoft.Network/privateEndpoints
├── NIC IP: 10.1.0.8 (from master subnet)
├── Target: ARO API server internal load balancer
│ (in Microsoft-managed ARO infrastructure)
├── Protocol: TCP
├── Port: 6443
├── Managed by: Microsoft / Red Hat (not customer)
└── Deletable: No — deleting breaks the cluster

Traffic arriving at 10.1.0.8:6443 is forwarded over Azure’s private backbone to the actual API server processes running on the master nodes — the customer never sees or touches the internal path.


How DNS Wires the IP to the FQDN

ARO automatically creates a Private DNS Zone and inserts an A record pointing the API server FQDN to the private endpoint IP:

# Find the private DNS zone
az network private-dns zone list \
--resource-group $MANAGED_RG \
--query "[].name" -o tsv
# Output:
# cluster.eastus.aroapp.io
# Inspect the A records
az network private-dns record-set a list \
--resource-group $MANAGED_RG \
--zone-name cluster.eastus.aroapp.io \
--output table
# Output:
# Name TTL Records
# ───── ──── ──────────
# api 300 10.1.0.8
# *.apps 300 10.1.1.100

The DNS zone is linked to the ARO spoke VNet automatically. You must manually link it to any other VNet (hub VNet, other spokes) that needs to resolve it:

# Link private DNS zone to hub VNet
az network private-dns link vnet create \
--resource-group $MANAGED_RG \
--zone-name cluster.eastus.aroapp.io \
--name link-hub-vnet \
--virtual-network /subscriptions/.../resourceGroups/rg-hub/providers/
Microsoft.Network/virtualNetworks/hub-vnet \
--registration-enabled false
# Verify resolution from jump host
nslookup api.cluster.eastus.aroapp.io 10.0.5.4
# Server: 10.0.5.4 (DNS Private Resolver)
# Address: 10.1.0.8 ← private endpoint IP returned ✅

Getting the API Server URL and IP Programmatically

# Get the full API server URL
API_URL=$(az aro show \
--resource-group rg-aro \
--name aro-prod \
--query apiserverProfile.url \
--output tsv)
echo $API_URL
# https://api.cluster.eastus.aroapp.io:6443
# Extract just the hostname
API_HOST=$(echo $API_URL | sed 's|https://||' | sed 's|:6443||')
echo $API_HOST
# api.cluster.eastus.aroapp.io
# Resolve to private IP (from inside VNet or connected network)
dig +short $API_HOST
# 10.1.0.8
# Verify TCP reachability on port 6443
nc -zv $API_HOST 6443
# Connection to api.cluster.eastus.aroapp.io 6443 port [tcp] succeeded!
# Login using oc
CREDS=$(az aro list-credentials \
--resource-group rg-aro \
--name aro-prod)
oc login $API_URL \
--username $(echo $CREDS | jq -r .kubeadminUsername) \
--password $(echo $CREDS | jq -r .kubeadminPassword) \
--insecure-skip-tls-verify=false

What Happens If You Try to Reach It From the Internet

The private endpoint IP (10.1.0.8) is a RFC 1918 private address — it is not routable on the public internet. From outside Azure, two things happen:

Scenario 1 — Public DNS lookup:
nslookup api.cluster.eastus.aroapp.io (from internet)
→ Returns NXDOMAIN or no answer
→ Public DNS has no record (zone is private only)
Scenario 2 — Direct TCP to port 6443:
curl https://api.cluster.eastus.aroapp.io:6443 (from internet)
→ DNS fails → connection never established
→ Even if DNS were somehow resolved, 10.1.0.8 is unreachable
from internet — packets dropped at Azure network edge

There is no public IP, no public DNS record, and no network path from the internet to the private endpoint — the attack surface is zero.


IP Stability — Does It Ever Change?

Once assigned at cluster creation time, the private endpoint IP is permanent for the cluster lifetime:

Cluster created: 10.1.0.8 assigned to API private endpoint
Cluster running: 10.1.0.8 (unchanged — days, months, years)
Master node reboot: 10.1.0.8 (private endpoint NIC is independent of master VMs)
ARO version upgrade: 10.1.0.8 (control plane upgrades don't change the PE IP)
Cluster deleted: 10.1.0.8 released back to subnet pool

This stability is intentional — UDRs, NSG rules, firewall rules, and DNS records all reference this IP. If it changed, every network policy referencing it would break. Azure guarantees it for the cluster lifetime without any reservation or static IP configuration needed on your part.


Key Takeaway

The ARO API server private endpoint IP is the next available IP after the master node NICs in your master subnet — automatically assigned by Azure during cluster provisioning, registered in a private DNS zone under aroapp.io, and permanently stable for the cluster lifetime. It is a synthetic NIC with no compute behind it — just a network fabric construct that forwards TCP port 6443 traffic over Azure’s private backbone to the actual API server processes on the master nodes. From the public internet it is completely invisible — no DNS record, no routable IP, no open port.

Benefits of a Private ARO Cluster in Azure

Private ARO Cluster

A private ARO cluster removes all public IP addresses from both the Kubernetes API server and the ingress router — making the cluster completely unreachable from the internet. Every connection to the cluster must travel over Azure’s private network backbone via VNet peering, ExpressRoute, or VPN.


Public vs Private ARO — What Changes

ComponentPublic clusterPrivate cluster
API server endpointPublic IP + DNSPrivate endpoint IP only
Ingress routerPublic load balancerInternal load balancer
Worker node IPsPrivate (always)Private (always)
Master node IPsPrivate (always)Private (always)
Access methodAny internet browserVPN / ER / Bastion only
DNS resolutionPublic DNSPrivate DNS zone
Attack surfaceAPI port 6443 exposedZero public exposure

How the API Server Is Hidden — The Mechanism

When you deploy a private ARO cluster, Azure does three things automatically:

1. API Server gets a Private Endpoint NIC

Instead of a public load balancer frontend, the API server is exposed exclusively through a private endpoint — a NIC in your VNet subnet with a private IP:

Public cluster:
  api.cluster.eastus.aroapp.io → 20.x.x.x (public IP)
  Anyone on internet can reach :6443

Private cluster:
  api.cluster.eastus.aroapp.io → 10.1.3.4 (private IP in your VNet)
  Only reachable from within the VNet or peered networks

The private endpoint is deployed into your ARO master subnet automatically during cluster creation. No public IP is allocated.

2. A Private DNS Zone Is Created Automatically

ARO creates a private DNS zone linked to your VNet so the API server FQDN resolves to the private endpoint IP:

Private DNS zone: cluster.eastus.aroapp.io
  A record: api → 10.1.3.4
  A record: *.apps → 10.1.4.8   (ingress internal LB)

Linked to: ARO spoke VNet + hub VNet

This means any VM in a peered VNet can resolve api.cluster.eastus.aroapp.io and get 10.1.3.4 — no public DNS lookup ever occurs.

3. Ingress Router Gets an Internal Load Balancer

The OpenShift ingress router (which handles *.apps.cluster.aroapp.io routes) is fronted by an Azure Internal Load Balancer with a private frontend IP:

Public cluster:   *.apps → Azure Public LB → 20.x.x.x
Private cluster:  *.apps → Azure Internal LB → 10.1.4.8

Applications running on the cluster are only reachable from inside the VNet or connected networks.


Deploying a Private ARO Cluster

# 1. Create resource group and VNet
az group create --name rg-aro --location eastus

az network vnet create \
  --resource-group rg-aro \
  --name aro-spoke-vnet \
  --address-prefixes 10.1.0.0/16

# 2. Create master subnet — disable private endpoint network policies
az network vnet subnet create \
  --resource-group rg-aro \
  --vnet-name aro-spoke-vnet \
  --name master-subnet \
  --address-prefixes 10.1.0.0/24 \
  --disable-private-link-service-network-policies true  # ← required for ARO

# 3. Create worker subnet
az network vnet subnet create \
  --resource-group rg-aro \
  --vnet-name aro-spoke-vnet \
  --name worker-subnet \
  --address-prefixes 10.1.1.0/23

# 4. Deploy private ARO cluster
az aro create \
  --resource-group rg-aro \
  --name aro-prod \
  --vnet aro-spoke-vnet \
  --master-subnet master-subnet \
  --worker-subnet worker-subnet \
  --apiserver-visibility Private \     # ← API server private
  --ingress-visibility Private \       # ← ingress private
  --master-vm-size Standard_D8s_v3 \
  --worker-vm-size Standard_D16s_v3 \
  --worker-count 3 \
  --pull-secret @pull-secret.txt

# Takes ~35 minutes to complete

The Three Access Paths

Path 1 — Azure Bastion + Jump Host (most common)

The simplest pattern — a small Linux VM in the hub VNet with oc and kubectl installed, accessed securely via Bastion:

# 1. Admin opens Azure portal → connects via Bastion to jump-host-vm
# 2. On jump host — get cluster credentials
az aro list-credentials \
  --resource-group rg-aro \
  --name aro-prod

# Output:
# kubeadminPassword: "XXXXX-XXXXX-XXXXX-XXXXX"
# kubeadminUsername: "kubeadmin"

# 3. Get API server URL
API_URL=$(az aro show \
  --resource-group rg-aro \
  --name aro-prod \
  --query apiserverProfile.url -o tsv)

# 4. Login — works because jump host is in peered VNet
oc login $API_URL \
  --username kubeadmin \
  --password XXXXX-XXXXX-XXXXX-XXXXX

# 5. Verify
oc get nodes
oc get clusterversion

Path 2 — ExpressRoute / VPN from on-premises

On-premises developers access the private API server directly from their workstations — but DNS must be configured to resolve the ARO private DNS zone:

On-premises developer workstation
Corporate DNS server: api.cluster.eastus.aroapp.io
↓ conditional forward to Azure DNS Private Resolver (10.0.5.4)
Azure DNS Private Resolver
↓ linked private DNS zone: aroapp.io → 10.1.3.4
Returns: 10.1.3.4
Developer runs: oc login https://api.cluster.eastus.aroapp.io:6443
Traffic travels: workstation → MPLS → ER Gateway → hub VNet peering → ARO spoke → API server

On-premises DNS server conditional forwarder:

Zone: aroapp.io
Forward to: 10.0.5.4 (DNS Private Resolver inbound endpoint)

Path 3 — CI/CD Pipeline (GitHub Actions / Azure DevOps)

For automated deployments, pipelines must also reach the private API server. Use a self-hosted runner inside the VNet:

# GitHub Actions — self-hosted runner in hub VNet
name: Deploy to ARO
on: [push]
jobs:
  deploy:
    runs-on: self-hosted    # ← runner VM inside Azure VNet
    steps:
      - uses: actions/checkout@v4

      - name: Login to ARO
        run: |
          oc login ${{ secrets.ARO_API_URL }} \
            --token ${{ secrets.ARO_SERVICE_ACCOUNT_TOKEN }}

      - name: Deploy application
        run: |
          oc apply -f k8s/
          oc rollout status deployment/my-app

The self-hosted runner is a VM in the hub VNet — it can resolve the private API server DNS and reach port 6443 over VNet peering.


Private DNS — The Critical Detail

After cluster creation, ARO automatically creates a private DNS zone. You must link this zone to every VNet that needs to resolve the API server — including the hub VNet where your jump host and DNS Private Resolver live:

# ARO creates this automatically — linked to ARO spoke VNet
# You must manually link it to the hub VNet

PRIVATE_ZONE=$(az network private-dns zone list \
  --resource-group $(az aro show -g rg-aro -n aro-prod \
    --query clusterProfile.resourceGroupId -o tsv | tr -d '\n') \
  --query "[?contains(name,'aroapp.io')].name" -o tsv)

# Link to hub VNet
az network private-dns link vnet create \
  --resource-group <aro-managed-rg> \
  --zone-name $PRIVATE_ZONE \
  --name link-to-hub-vnet \
  --virtual-network $(az network vnet show \
    --resource-group rg-hub \
    --name hub-vnet --query id -o tsv) \
  --registration-enabled false

Without this link, VMs in the hub VNet cannot resolve api.cluster.eastus.aroapp.io — DNS queries fall through to public DNS which returns NXDOMAIN for a private cluster.


Entra ID (AAD) Integration for Developer Access

Replace the kubeadmin local account with Entra ID authentication — developers log in with their corporate credentials:

# Configure AAD OAuth on ARO
az aro update \
  --resource-group rg-aro \
  --name aro-prod \
  --client-id <app-registration-client-id> \
  --client-secret <app-registration-secret>

# Grant cluster-admin to an AAD group
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: aro-cluster-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: <aad-group-object-id>    # e.g. Platform Engineering team
---
# Grant view-only to a developer group
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-view
  namespace: my-app
roleRef:
  kind: ClusterRole
  name: view
subjects:
  - kind: Group
    name: <dev-aad-group-object-id>

Developers now login via:

oc login $API_URL # redirects to Microsoft login page
# Enter corporate credentials → MFA → issued a token

Egress from a Private Cluster

A private cluster still needs outbound internet access for Red Hat image registries and update servers. Force all egress through Azure Firewall via UDR on both subnets:

# Route table for master and worker subnets
az network route-table create \
  --resource-group rg-aro-network \
  --name rt-aro-subnets

az network route-table route create \
  --resource-group rg-aro-network \
  --route-table-name rt-aro-subnets \
  --name force-to-firewall \
  --address-prefix 0.0.0.0/0 \
  --next-hop-type VirtualAppliance \
  --next-hop-ip-address 10.0.1.4     # Azure Firewall private IP

# Associate with master subnet
az network vnet subnet update \
  --resource-group rg-aro \
  --vnet-name aro-spoke-vnet \
  --name master-subnet \
  --route-table rt-aro-subnets

# Associate with worker subnet
az network vnet subnet update \
  --resource-group rg-aro \
  --vnet-name aro-spoke-vnet \
  --name worker-subnet \
  --route-table rt-aro-subnets

Required Azure Firewall FQDN allow rules for private ARO:

quay.io                          # Red Hat image registry
registry.redhat.io               # Red Hat registry
registry.access.redhat.com       # RHEL content
cdn.quay.io                      # CDN for quay
*.blob.core.windows.net          # Azure storage (etcd backups, images)
*.servicebus.windows.net         # ARO monitoring
management.azure.com             # Azure ARM API
login.microsoftonline.com        # Entra ID auth

Key Takeaway

A private ARO cluster achieves zero public attack surface by replacing the public API server load balancer with a VNet-internal private endpoint, and replacing the public ingress load balancer with an internal one. DNS resolution of both endpoints stays entirely within Azure’s private network. The only access paths are Azure Bastion for interactive access, ExpressRoute or VPN for on-premises connectivity, and self-hosted CI/CD runners for automation — all travelling over encrypted private paths without a single packet touching the public internet.

Best Practices for OpenShift on Azure: ARO Guide

OpenShift Container Platform on Azure — ARO Best Practices

Azure Red Hat OpenShift (ARO) is a fully managed OpenShift 4 service jointly operated by Microsoft and Red Hat — both companies share responsibility for the control plane, infrastructure, and SLA (99.95%).


1. Networking Best Practices

Always deploy a private cluster

A private ARO cluster hides the Kubernetes API server behind a private endpoint — no public IP, unreachable from the internet:

az aro create \
  --resource-group rg-aro \
  --name aro-prod \
  --vnet aro-spoke-vnet \
  --master-subnet master-subnet \
  --worker-subnet worker-subnet \
  --apiserver-visibility Private \      # ← API server private
  --ingress-visibility Private \        # ← ingress private
  --pull-secret @pull-secret.txt

Access to the private API server is then through Azure Bastion → jump host, or over ExpressRoute/VPN from on-premises.


Subnet sizing — get this right before deployment (cannot resize after)

ARO consumes IP addresses aggressively — every pod gets its own IP from the node’s subnet range:

SubnetMinimumRecommendedNotes
Master subnet/27/24Fixed 3 masters — needs room for Azure infra IPs
Worker subnet/27/23 or /22Every pod consumes an IP — size generously
Ingress subnet/28/27For LB / App Gateway front-end IPs
Private endpoints/28/27One IP per private endpoint
Worker subnet sizing example:
  /23 = 512 addresses
  Azure reserves 5
  Available: 507
  Max pods per node: 250 (default OpenShift SDN)
  Nodes supportable: ~2 per node × workers
  Plan for: 3× current need for growth headroom


Egress lockdown via Azure Firewall

ARO requires outbound internet access for Red Hat update servers, telemetry, and pull.registry.redhat.io. Lock this down with Azure Firewall application rules rather than allowing all outbound:

Azure Firewall Application Rules for ARO egress:
┌─────────────────────────────────────────────────────────┐
│ Name Target FQDN │
├─────────────────────────────────────────────────────────┤
│ aro-rh-registry registry.redhat.io │
│ registry.access.redhat.com │
│ quay.io │
│ cdn.quay.io │
├─────────────────────────────────────────────────────────┤
│ aro-azure-services *.blob.core.windows.net │
│ *.servicebus.windows.net │
│ *.table.core.windows.net │
├─────────────────────────────────────────────────────────┤
│ aro-monitoring *.ods.opinsights.azure.com │
│ *.oms.opinsights.azure.com │
├─────────────────────────────────────────────────────────┤
│ aro-rh-telemetry cert-api.access.redhat.com │
│ api.access.redhat.com │
└─────────────────────────────────────────────────────────┘

Apply a UDR on the master and worker subnets pointing 0.0.0.0/0 to the Azure Firewall private IP — same hub and spoke pattern as any spoke workload.


Use a custom DNS server

Point the ARO VNet DNS to your hub DNS Private Resolver so cluster nodes can resolve private endpoints and internal domains:

az network vnet update \
  --resource-group rg-aro-network \
  --name aro-spoke-vnet \
  --dns-servers 10.0.5.4    # DNS Private Resolver inbound endpoint IP


2. Availability and Resilience Best Practices

Spread across all three Availability Zones

ARO deploys 3 master nodes — one per AZ automatically. Workers must be explicitly spread via MachineSets:

# MachineSet for AZ1 — replicate for AZ2, AZ3
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: aro-prod-worker-eastus-1
  namespace: openshift-machine-api
spec:
  replicas: 3
  template:
    spec:
      providerSpec:
        value:
          zone: "1"                         # AZ1
          vmSize: Standard_D16s_v3
          osDisk:
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS

Create three MachineSets — one per zone — with equal replica counts. This ensures workloads survive a full AZ failure.


Enable cluster autoscaler

apiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default
spec:
  resourceLimits:
    maxNodesTotal: 24
  scaleDown:
    enabled: true
    delayAfterAdd: 10m
    delayAfterDelete: 5m
    delayAfterFailure: 30s
---
apiVersion: autoscaling.openshift.io/v1beta1
kind: MachineAutoscaler
metadata:
  name: aro-prod-worker-eastus-1
  namespace: openshift-machine-api
spec:
  minReplicas: 3
  maxReplicas: 8
  scaleTargetRef:
    kind: MachineSet
    name: aro-prod-worker-eastus-1


Use zone-redundant storage for persistent volumes

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-premium-zrs
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_ZRS       # Zone-redundant storage — survives AZ failure
  cachingMode: ReadOnly
reclaimPolicy: Retain        # Retain on PVC delete — prevents data loss
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Use Premium_ZRS instead of Premium_LRS for stateful workloads — ZRS replicates the disk synchronously across three AZs so a pod can reschedule to another zone without losing its data.


3. Security Best Practices

Use Workload Identity (pod-level Azure RBAC)

Never put Azure credentials in pods. Use Workload Identity to give individual pods an Azure AD identity with scoped RBAC permissions:

# Enable workload identity on ARO cluster
az aro update \
  --resource-group rg-aro \
  --name aro-prod \
  --enable-managed-identity

# Create a managed identity for a specific workload
az identity create \
  --resource-group rg-aro-workloads \
  --name id-payment-service

# Grant it only what it needs
az role assignment create \
  --assignee <identity-client-id> \
  --role "Key Vault Secrets User" \
  --scope /subscriptions/.../vaults/kv-prod

# Annotate the service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service-sa
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "<managed-identity-client-id>"


Integrate Azure Key Vault for secrets via CSI driver

Never store secrets in OpenShift Secrets (base64 is not encryption). Use the Secrets Store CSI driver to mount Key Vault secrets directly into pods:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-secrets
  namespace: payments
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "<managed-identity-client-id>"
    keyvaultName: kv-prod
    tenantID: "<tenant-id>"
    objects: |
      array:
        - |
          objectName: db-connection-string
          objectType: secret
        - |
          objectName: api-key
          objectType: secret


Integrate with Azure Container Registry via private endpoint

# Create ACR with private endpoint — no public access
az acr create \
  --resource-group rg-aro \
  --name acrprodaro \
  --sku Premium \
  --public-network-enabled false

# Private endpoint in ARO spoke
az network private-endpoint create \
  --name pe-acr-prod \
  --resource-group rg-aro-network \
  --vnet-name aro-spoke-vnet \
  --subnet private-endpoint-subnet \
  --private-connection-resource-id $(az acr show --name acrprodaro --query id -o tsv) \
  --group-id registry \
  --connection-name pe-acr-conn

# Grant ARO pull access
az role assignment create \
  --assignee <aro-kubelet-identity> \
  --role AcrPull \
  --scope $(az acr show --name acrprodaro --query id -o tsv)


Apply OpenShift Security Context Constraints (SCC)

Never run pods as root. Use the restricted-v2 SCC (default in OpenShift 4.11+):

apiVersion: v1
kind: Pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        readOnlyRootFilesystem: true


Enable Microsoft Defender for Containers

az security pricing create \
  --name Containers \
  --tier Standard

Defender for Containers provides runtime threat detection, vulnerability scanning for images in ACR, and Kubernetes audit log analysis — all surfaced in Microsoft Defender for Cloud.


4. Observability Best Practices

Forward logs to Azure Monitor / Log Analytics

# Enable container insights on ARO
az aro update \
  --resource-group rg-aro \
  --name aro-prod \
  --enable-managed-identity

# Deploy the monitoring add-on via Helm
helm repo add microsoft https://microsoft.github.io/charts/repo
helm install azuremonitor-containers \
  microsoft/azuremonitor-containers \
  --set omsagent.secret.wsid=<workspace-id> \
  --set omsagent.secret.key=<workspace-key> \
  --namespace kube-system


Use Azure Monitor alerts for cluster health

AlertMetricThreshold
Node CPU pressurecpuUsageNanoCores> 85% for 5 min
Node memory pressurememoryWorkingSetBytes> 80% of capacity
Pod restart looprestartCount> 5 in 10 min
PVC near fullpvUsedBytes> 85% of capacity
Node not readynodeConditionNotReady > 2 min

5. Day-2 Operations Best Practices

Cluster upgrade strategy

ARO manages the control plane upgrade automatically — you control timing for worker nodes:

# Check available upgrade versions
az aro get-upgrade-versions \
  --resource-group rg-aro \
  --name aro-prod

# Schedule upgrade in maintenance window
az aro update \
  --resource-group rg-aro \
  --name aro-prod \
  --version 4.14.12

Use the EUS (Extended Update Support) channel for production clusters — it allows staying on a minor version for up to 18 months while still receiving security patches, avoiding the churn of mandatory minor version upgrades every 45 days.


Worker node upgrade — use surge capacity

# MachineConfigPool surge upgrade strategy
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker
spec:
  maxUnavailable: 1          # Upgrade one node at a time

Upgrade workers one node at a time to maintain application availability — pods are gracefully drained before each node reboots into the new RHCOS version.


Summary — ARO Best Practice Checklist

CategoryPractice
NetworkPrivate cluster — no public API or ingress
NetworkEgress via Azure Firewall with FQDN allow-list
NetworkDNS Private Resolver for private endpoint resolution
NetworkWorker subnet /22 or larger — never resize after
AvailabilityWorkers spread across AZs via 3 MachineSets
AvailabilityCluster autoscaler min 3 per zone
AvailabilityPremium_ZRS disks for stateful workloads
AvailabilityZone-redundant Azure Load Balancer
SecurityWorkload Identity — no credentials in pods
SecurityKey Vault + CSI driver — no base64 secrets
SecurityACR via private endpoint — no public pull
SecuritySCC restricted-v2 — no root containers
SecurityDefender for Containers enabled
ObservabilityContainer Insights → Log Analytics
ObservabilityAzure Monitor alerts on node and pod health
OperationsEUS channel for production stability
OperationsmaxUnavailable: 1 for worker upgrades

Optimize Azure Traffic Flow with UDR

Traffic Flow Through Azure Firewall via UDR

The UDR (User Defined Route) is the mechanism that forces all spoke traffic through Azure Firewall — overriding Azure’s default system routes which would otherwise send traffic directly between peered VNets, bypassing inspection entirely.


Why UDRs Are Necessary

Azure VNet peering by default creates direct routing between peered VNets — packets travel peer-to-peer without touching any intermediate device. This means without UDRs, a VM in Spoke 1 talking to a VM in Spoke 2 completely bypasses Azure Firewall:

Default behaviour (NO UDR):
  Spoke 1 VM (10.1.1.4) → Spoke 2 VM (10.2.1.4)
  System route: 10.2.0.0/16 → VNet peering (direct)
  Result: traffic bypasses firewall entirely 

With UDR applied to spoke subnet:
  Spoke 1 VM (10.1.1.4) → Spoke 2 VM (10.2.1.4)
  UDR overrides: 0.0.0.0/0 → 10.0.1.4 (Firewall private IP)
  Result: traffic hits firewall → inspected → forwarded 

The UDR wins because custom routes always override system routes — Azure’s route selection priority is custom UDR first, then BGP routes, then system routes.


Route Table Structure

A route table is an Azure resource associated with one or more subnets. Every subnet that needs inspection gets the same core UDR:

Route Table: rt-spoke1-subnets
Associated to: Spoke 1 subnet A, Spoke 1 subnet B

Routes:
  Name              Prefix          Next hop type        Next hop IP
  ─────────────────────────────────────────────────────────────────
  force-to-fw       0.0.0.0/0       Virtual Appliance    10.0.1.4

Deployed via ARM / Bicep:

resource routeTable 'Microsoft.Network/routeTables@2023-04-01' = {
  name: 'rt-spoke1-subnets'
  location: location
  properties: {
    disableBgpRoutePropagation: true   // ← critical — explained below
    routes: [
      {
        name: 'force-to-firewall'
        properties: {
          addressPrefix: '0.0.0.0/0'
          nextHopType: 'VirtualAppliance'
          nextHopIpAddress: '10.0.1.4'   // Azure Firewall private IP
        }
      }
    ]
  }
}

// Associate with spoke subnet
resource subnetAssociation 'Microsoft.Network/virtualNetworks/subnets@2023-04-01' = {
  name: 'spoke1/snet-app'
  properties: {
    addressPrefix: '10.1.1.0/24'
    routeTable: {
      id: routeTable.id
    }
  }
}


The Three Traffic Paths

Path 1 — North-South Outbound (spoke VM → internet)

Step 1: Spoke 1 VM (10.1.1.4) sends packet to 8.8.8.8
Step 2: Subnet route table consulted
UDR match: 0.0.0.0/0 → next hop 10.0.1.4 (Firewall)
Step 3: Packet arrives at Azure Firewall private IP
Step 4: Firewall evaluates application rules
Rule: allow src=10.1.0.0/16 dest=*.google.com proto=HTTPS → ALLOW
Step 5: Firewall SNATs packet
Source IP changed: 10.1.1.4 → Firewall public IP (20.x.x.x)
Step 6: Packet exits to internet from Firewall public IP
Return traffic arrives at Firewall public IP
Step 7: Firewall translates back → forwards to 10.1.1.4

SNAT is automatic for internet-bound traffic — the spoke VM’s private IP is never exposed to the internet. Azure Firewall’s public IP is the only address the internet sees.


Path 2 — North-South Inbound (internet → spoke VM)

Step 1: External client sends to Firewall public IP 20.x.x.x:443
Step 2: Firewall DNAT rule fires:
dest 20.x.x.x:443 → translated to 10.1.4.5:443 (spoke VM)
Step 3: Firewall forwards to 10.1.4.5 via VNet peering path
Step 4: Packet arrives at spoke VM — no public IP needed on VM
Step 5: VM responds to Firewall private IP (it sees FW as source)
UDR on VM subnet ensures return goes back through Firewall
Step 6: Firewall forwards return to external client

Path 3 — East-West (spoke 1 VM → spoke 2 VM)

This is the most important path for security — lateral movement between spokes must be inspected:

Step 1: Spoke 1 VM (10.1.1.4) sends packet to Spoke 2 VM (10.2.1.8)
Step 2: Spoke 1 subnet route table consulted
UDR: 0.0.0.0/0 → 10.0.1.4 (matches — more specific than system route)
Step 3: Packet arrives at Azure Firewall
Step 4: Firewall evaluates network rules
Rule: allow src=10.1.0.0/16 dest=10.2.1.8 port=443 → ALLOW
(or deny if no rule matches)
Step 5: Firewall forwards to 10.2.1.8 via peering to Spoke 2
Step 6: Spoke 2 subnet route table:
UDR: 0.0.0.0/0 → 10.0.1.4
Return traffic: 10.2.1.8 → 10.1.1.4
UDR forces return through Firewall too
Step 7: Firewall forwards return packet to Spoke 1 VM

Both directions of every connection traverse the Firewall — request and response. This is essential for stateful inspection — if only one direction went through the Firewall, the session state table would be incomplete.


disableBgpRoutePropagation — Why It Matters

Every route table has a disableBgpRoutePropagation flag. On spoke subnets this must be set to true:

disableBgpRoutePropagation: false (default)
→ VPN Gateway pushes on-premises routes into spoke effective routes
→ Spoke VM sends on-premises traffic directly to Gateway
→ Bypasses Firewall for on-premises bound traffic ❌
disableBgpRoutePropagation: true (required for spoke subnets)
→ VPN Gateway routes suppressed on spoke subnets
→ Only UDR routes active: 0.0.0.0/0 → Firewall
→ All traffic including on-premises bound goes through Firewall ✅

Forgetting this setting is one of the most common misconfiguration errors in hub and spoke deployments — on-premises traffic silently bypasses the Firewall even though the UDR looks correct.


UDR on GatewaySubnet — On-Premises to Spoke

To inspect traffic arriving from on-premises destined for spoke VNets, a UDR must also be applied to the GatewaySubnet:

Route Table: rt-gateway-subnet
Associated to: GatewaySubnet
Routes:
Name Prefix Next hop type Next hop IP
────────────────────────────────────────────────────────────────
to-spoke1 10.1.0.0/16 VirtualAppliance 10.0.1.4
to-spoke2 10.2.0.0/16 VirtualAppliance 10.0.1.4
to-spoke3 10.3.0.0/16 VirtualAppliance 10.0.1.4

Note this uses specific spoke prefixes rather than 0.0.0.0/0 — applying a default route to GatewaySubnet breaks the gateway’s ability to communicate with Azure control plane endpoints.


Effective Route Inspection

You can verify UDRs are working correctly by checking a VM’s effective routes in the Azure portal or CLI:

az network nic show-effective-route-table \
--resource-group rg-spoke1 \
--name vm-prod-01-nic \
--output table
Source State Address Prefix Next Hop Type Next Hop IP
──────── ─────── ──────────────── ────────────────── ──────────
Default Active 10.1.0.0/16 VnetLocal
Default Invalid 0.0.0.0/0 Internet ← overridden
User Active 0.0.0.0/0 VirtualAppliance 10.0.1.4 ✅
Default Active 10.0.0.0/16 VNetPeering
Default Active 10.2.0.0/16 VNetPeering

The default 0.0.0.0/0 → Internet system route shows as Invalid — it has been overridden by the custom UDR pointing to the Firewall. This confirms all traffic is force-tunnelled correctly.


Common Misconfiguration Pitfalls

Forgetting disableBgpRoutePropagation — gateway-learned routes override UDRs for on-premises prefixes, silently bypassing Firewall for hybrid traffic.

Missing return path UDR — if Spoke 2 subnet has no UDR, return traffic goes directly back to Spoke 1 via peering, creating an asymmetric routing loop that breaks TCP sessions.

Applying UDR to AzureBastionSubnet — Bastion requires direct internet connectivity for its management plane. A UDR with 0.0.0.0/0 → Firewall on AzureBastionSubnet breaks Bastion entirely. Bastion subnet must have no UDR or a very specific one that excludes Bastion management ranges.

Applying 0.0.0.0/0 UDR to GatewaySubnet — breaks gateway health probes and control plane communication. Always use specific spoke prefixes on GatewaySubnet, never a default route.

Firewall private IP not static — Azure Firewall’s private IP should be configured as static during deployment. If it changes, every UDR next-hop entry becomes invalid and traffic black-holes.


Key Takeaway

The UDR is a deceptively simple mechanism — a single route entry 0.0.0.0/0 → Virtual Appliance → 10.0.1.4 — that transforms Azure’s default direct peering behaviour into a fully inspected, security-enforced network. Applied correctly to every spoke subnet with disableBgpRoutePropagation enabled, it ensures no traffic — outbound internet, inbound DNAT, or lateral east-west — can bypass Azure Firewall, giving you complete visibility and control over your entire hub and spoke estate.

Simplify Your Azure Networking with Route Server

Azure Route Server

Azure Route Server is a fully managed service that acts as a BGP route reflector inside your hub VNet — it exchanges routes dynamically between your Network Virtual Appliances (NVAs) and Azure’s software-defined network, eliminating the need to manually maintain User Defined Routes every time your network topology changes.


The Problem It Solves

Without Route Server, every time an NVA learns a new route (a new branch office, a new subnet, a new peer) you had to manually update UDR tables on every spoke subnet:

Old approach — manual UDR maintenance:
NVA learns new branch: 192.168.50.0/24
Engineer must manually add UDR to:
- Spoke 1 subnet A route table
- Spoke 1 subnet B route table
- Spoke 2 subnet A route table
- Spoke 2 subnet B route table
- ... every subnet in every spoke
Miss one → black hole routing → outage
With Route Server:
NVA advertises 192.168.50.0/24 via BGP to Route Server
Route Server automatically programs the route
into effective routes of all peered spoke VNets
Done — zero manual intervention

How BGP Exchange Works — Step by Step

Step 1 — Route Server deploys into RouteServerSubnet

Route Server requires a dedicated subnet named exactly RouteServerSubnet with a minimum /27:

Hub VNet: 10.0.0.0/16
RouteServerSubnet: 10.0.6.0/27
→ Route Server Instance 0: 10.0.6.4
→ Route Server Instance 1: 10.0.6.5
→ Virtual IP (peering): 10.0.6.6

Route Server always deploys as two instances for high availability, both in the same subnet. Azure assigns them IPs automatically. Both instances must be peered with your NVA — you peer with each IP individually.

Route Server always uses ASN 65515 — this is fixed and cannot be changed.


Step 2 — NVA establishes eBGP sessions with both instances

Your NVA (Cisco CSR, Palo Alto VM-Series, Fortinet FortiGate, etc.) opens two BGP sessions — one to each Route Server instance. This is standard external BGP (eBGP) — the NVA and Route Server are in different ASNs:

NVA (ASN 65001) ←—eBGP—→ Route Server Instance 0 (ASN 65515, 10.0.6.4)
NVA (ASN 65001) ←—eBGP—→ Route Server Instance 1 (ASN 65515, 10.0.6.5)

Configuration on a Cisco CSR NVA:

router bgp 65001
bgp router-id 10.0.4.4
bgp log-neighbor-changes
! Peer with Route Server instance 0
neighbor 10.0.6.4 remote-as 65515
neighbor 10.0.6.4 ebgp-multihop 2
neighbor 10.0.6.4 soft-reconfiguration inbound
! Peer with Route Server instance 1
neighbor 10.0.6.5 remote-as 65515
neighbor 10.0.6.5 ebgp-multihop 2
neighbor 10.0.6.5 soft-reconfiguration inbound
! Advertise on-premises routes learned from VPN
network 192.168.0.0 mask 255.255.0.0
network 172.16.0.0 mask 255.255.0.0

The ebgp-multihop 2 is required because the NVA and Route Server are not directly connected at Layer 2 — they communicate over the VNet fabric.


Step 3 — NVA advertises routes to Route Server

The NVA tells Route Server about routes it knows — on-premises prefixes learned via VPN tunnels, SD-WAN routes, or any custom prefixes:

NVA → Route Server:
ADVERTISE 192.168.0.0/16 (on-premises HQ network)
ADVERTISE 172.16.0.0/12 (branch offices)
ADVERTISE 10.100.0.0/16 (SD-WAN overlay)

Route Server accepts these advertisements and stores them.


Step 4 — Route Server programs spoke VNet effective routes

Route Server takes the NVA-advertised routes and automatically injects them into the effective routes of every peered spoke VNet — no UDR required:

Spoke 1 VM effective routes (auto-programmed):
10.0.0.0/16 → VNet local
10.1.0.0/16 → VNet local
192.168.0.0/16 → 10.0.4.4 (NVA primary) ← from Route Server
172.16.0.0/12 → 10.0.4.4 (NVA primary) ← from Route Server
10.100.0.0/16 → 10.0.4.4 (NVA primary) ← from Route Server
0.0.0.0/0 → Internet

When a spoke VM sends traffic to 192.168.10.5 (an on-premises host), the effective route points it to the NVA, which forwards it through the appropriate VPN tunnel.


Step 5 — Route Server advertises Azure routes back to NVA

The exchange is bidirectional. Route Server tells the NVA about Azure VNet address spaces:

Route Server → NVA:
ADVERTISE 10.0.0.0/16 (hub VNet)
ADVERTISE 10.1.0.0/16 (spoke 1 VNet)
ADVERTISE 10.2.0.0/16 (spoke 2 VNet)
ADVERTISE 10.3.0.0/16 (spoke 3 VNet)

The NVA now knows all Azure prefixes and can route on-premises traffic destined for Azure correctly through its VPN tunnels — without anyone manually configuring static routes on the NVA.


Branch-to-Branch — The Key Feature

When branch-to-branch is enabled on Route Server, it becomes a route reflector between VPN Gateway and NVA, allowing on-premises sites to reach each other through Azure:

Branch A (192.168.1.0/24) ←—VPN—→ VPN Gateway
↕ BGP
Route Server
↕ BGP
Branch B (192.168.2.0/24) ←—VPN—→ NVA
With branch-to-branch ENABLED:
Route Server reflects Branch A routes → NVA → Branch B
Route Server reflects Branch B routes → VPN GW → Branch A
Result: Branch A can reach Branch B through Azure hub

This is how Azure Route Server enables transit routing — Azure becomes the backbone connecting your branch offices, without needing a separate SD-WAN overlay.

# Enable branch-to-branch
az network routeserver update \
--resource-group rg-hub-network \
--name hub-route-server \
--allow-b2b-traffic true

Active-Active NVA Pattern

Route Server is the enabler for active-active NVA deployments — both NVA instances advertise the same routes, and Route Server programs both next-hops into spoke effective routes using ECMP (Equal-Cost Multi-Path):

Both NVAs advertise: 192.168.0.0/16
Spoke VM effective routes:
192.168.0.0/16 → 10.0.4.4 (NVA primary) ← ECMP
192.168.0.0/16 → 10.0.4.5 (NVA secondary) ← ECMP
Traffic load-balanced across both NVAs
If one NVA fails → BGP session drops →
Route Server withdraws that next-hop →
All traffic shifts to remaining NVA automatically

This gives you sub-second failover without any manual intervention — the BGP hold-down timer (typically 90 seconds, tunable to as low as 1 second with BFD) triggers automatic route withdrawal.


RouteServerSubnet Requirements

PropertyRequirement
Subnet nameMust be exactly RouteServerSubnet
Minimum size/27 (32 addresses)
DedicatedNo other resources in this subnet
DelegationNone required
NSGNot supported on RouteServerSubnet
UDRNot supported on RouteServerSubnet

The restriction on NSGs and UDRs on the RouteServerSubnet is intentional — Azure manages all routing within this subnet internally, and applying UDRs would break the BGP sessions.


Route Server vs Manual UDRs — When to Use Each

ScenarioUse Route ServerUse Manual UDRs
NVA in hub for inspection
Dynamic on-premises routes via BGP
SD-WAN integration
Static force-tunnel to Azure Firewall
Simple hub with Azure Firewall only
Frequently changing branch topology
No NVA — just Azure native services

Route Server shines when you have a third-party NVA with dynamic routing requirements. If your hub uses only Azure Firewall (which does not speak BGP), stick with UDRs — Route Server adds no value without a BGP-capable NVA to peer with.


Key Limits

LimitValue
BGP peers (NVAs) per Route Server8
Routes advertised by each NVA1,000
Routes propagated to spoke VNets1,000 per VNet
Route Server ASN65515 (fixed)
NVA ASN restrictionsCannot use 65515, 65520, 12076
VNets the Route Server can peer withUnlimited (same region)

The 1,000 route limit per NVA is important for large enterprises with many branch offices — if you have more than 1,000 prefixes, use route summarisation on the NVA before advertising to Route Server.


Key Takeaway

Azure Route Server is the dynamic routing backbone of a hub and spoke network containing third-party NVAs. It replaces fragile, manually maintained UDR tables with automated BGP route exchange — the NVA advertises what it knows, Route Server programs every spoke automatically, and the whole network converges in seconds when topology changes. Combined with active-active NVAs and branch-to-branch enabled, it gives you a carrier-grade routing infrastructure entirely within Azure.

Understanding Azure Bastion: A Complete Guide

Azure Bastion

Azure Bastion is a fully managed PaaS service that provides secure RDP and SSH connectivity to your virtual machines directly through the Azure portal or native client — over TLS on port 443 — without exposing any public IP address on the VM itself.


Why Bastion Exists — The Problem It Solves

The traditional way to RDP or SSH into an Azure VM was to assign it a public IP and open port 3389 or 22 to the internet. This creates serious exposure:

Old approach:
VM has public IP 20.x.x.x → port 3389 open to internet
→ Constant brute-force attacks (thousands/day)
→ Any misconfigured NSG = immediate compromise
→ Public IP costs, management overhead
→ No session recording or audit trail
With Bastion:
VM has no public IP — completely unreachable from internet
→ Admin connects via browser to Bastion on port 443
→ Bastion proxies RDP/SSH over private VNet path
→ Session fully audited in Azure Monitor
→ No attack surface on the VM itself

How the Connection Works — Step by Step

1. Admin opens Azure portal or native client
Selects a VM → Connect → Bastion
2. Portal establishes WebSocket connection to Bastion
over HTTPS port 443 (same as normal web traffic)
→ passes through corporate firewalls without issue
3. Bastion authenticates the admin
via Azure AD / RBAC — checks if admin has
"Bastion User" role on the Bastion resource
4. Bastion opens RDP (:3389) or SSH (:22) connection
from its private NIC directly to the VM's private IP
entirely inside the VNet / peered spoke
5. RDP or SSH session streams back to browser
rendered as HTML5 — no RDP client needed
(Standard SKU supports native RDP/SSH client too)
6. Session ends → connection torn down
Audit log written to Azure Monitor / Log Analytics

The VM never sees a connection from the internet — it only sees a private IP connection from within the VNet.


AzureBastionSubnet Requirements

Bastion requires a dedicated subnet with a very specific name — Azure will refuse to deploy it anywhere else:

Subnet name: AzureBastionSubnet ← exact name required
Min size: /26 (64 addresses) ← Basic SKU minimum
Recommended: /24 or /25 ← room to scale instances
Location: Hub VNet only ← one Bastion serves all peered spokes

No other resources — VMs, firewalls, private endpoints — can share this subnet. The /26 minimum exists because Bastion deploys multiple managed instances internally and Azure reserves addresses for platform use.


Bastion SKUs

FeatureBasicStandardPremium
Browser-based RDP/SSH
Native client (RDP/SSH app)
Shareable links
IP-based connection (no Azure VM)
VNet peering support
Session scaling (instances)2 fixed2–50 scalable2–50 scalable
File transfer (upload/download)
Clipboard (copy/paste)✅ text✅ text + file✅ text + file
Session recording
Private-only Bastion (no public IP)
Kerberos authentication
Pricing~$140/month~$280/month~$470/month

Standard is the right choice for most enterprises — it adds native client support (so admins can use their local RDP or SSH client instead of the browser) and instance scaling for large teams.

Premium adds session recording — every RDP/SSH session is captured and stored in a storage account — critical for compliance environments (PCI-DSS, SOC 2) where you must prove what privileged users did during a session. It also supports fully private Bastion with no public IP at all, using a private endpoint instead.


NSG Rules Required

Two NSGs must be configured correctly — one on the AzureBastionSubnet and one on the target VM subnet:

NSG on AzureBastionSubnet

Inbound rules:

PrioritySourceDestPortProtocolActionPurpose
100InternetAny443TCPAllowAdmin HTTPS inbound
110GatewayManagerAny443TCPAllowAzure control plane
120AzureLoadBalancerAny443TCPAllowHealth probes

Outbound rules:

PrioritySourceDestPortProtocolActionPurpose
100AnyVirtualNetwork3389, 22TCPAllowRDP/SSH to VMs
110AnyAzureCloud443TCPAllowBastion diagnostics

NSG on target VM subnet

PrioritySourceDestPortProtocolActionPurpose
100VirtualNetworkAny3389TCPAllowRDP from Bastion
110VirtualNetworkAny22TCPAllowSSH from Bastion

The VM subnet must allow inbound from VirtualNetwork service tag — which includes all peered VNets including the hub where Bastion lives. Critically, there is no rule allowing port 3389 or 22 from Internet — the VM is completely shielded.


RBAC Roles for Bastion Access

Bastion access is controlled by Azure RBAC — not just network connectivity:

RoleWhat it allows
Reader on BastionSee the Bastion resource
Bastion ReaderConnect through Bastion (read-only session)
Virtual Machine Contributor on VMNeeded to initiate the connection
Bastion ContributorManage the Bastion resource

A typical setup grants an admin team the Virtual Machine Contributor role on specific spoke resource groups, plus Reader on the Bastion. They can then connect to VMs they have rights to — but cannot modify the Bastion configuration itself or connect to VMs outside their scope.


Bastion in Hub and Spoke — One Bastion for All Spokes

A single Bastion in the hub VNet reaches VMs in all peered spoke VNets — you do not need a Bastion in every spoke. This is a key cost and management advantage:

Hub VNet → AzureBastionSubnet → Azure Bastion
↓ peering allows Bastion to reach
Spoke 1 VMs (production)
Spoke 2 VMs (development)
Spoke 3 VMs (shared services)
Spoke 4 VMs (DMZ)

One Bastion instance, one Standard public IP, one monthly cost — covering your entire estate. The only requirement is that the hub-to-spoke peering has Allow gateway transit enabled on the hub side and Use remote gateways enabled on the spoke side (the same peering settings used for routing).


Key Takeaway

Azure Bastion eliminates the biggest attack surface in most Azure environments — publicly exposed management ports on VMs. By proxying all RDP and SSH through a hardened, Microsoft-managed PaaS service over HTTPS, it gives you secure, auditable VM access with no public IPs on your workloads, no VPN client requirement for browser sessions, and full RBAC control over who can connect to what.

Understanding Azure DDoS Protection Standard

Azure DDoS Protection Standard

Azure DDoS Protection Standard is a managed, always-on service that detects and mitigates volumetric, protocol, and application-layer DDoS attacks against your Azure public IP addresses — automatically, without any configuration changes during an attack.


The Three Attack Categories It Defends Against

Layer 3/4 — Volumetric attacks

These flood your network bandwidth with massive traffic volumes — UDP floods, ICMP floods, amplification attacks (DNS, NTP, memcached). Azure absorbs these at the network edge using its global 60+ Tbps scrubbing capacity, before the traffic ever reaches your VNet or gateway.

Layer 3/4 — Protocol attacks

These exhaust connection state tables on firewalls, load balancers, and gateways. SYN floods send millions of half-open TCP connections; Smurf attacks abuse ICMP broadcasts. DDoS Protection mitigates these by validating TCP handshakes and rate-limiting malformed packets at the edge.

Layer 7 — Resource layer attacks

HTTP floods, Slowloris, and application-specific attacks target your app’s compute rather than your bandwidth. DDoS Protection Standard alone does not fully mitigate Layer 7 attacks — these require Azure WAF on Application Gateway or Azure Front Door working alongside DDoS Protection. The two services are designed to be used together for full-stack protection.


How Adaptive Tuning Works

This is the core differentiator versus the free Basic tier. DDoS Protection Standard builds a per-public-IP traffic baseline using machine learning:

Normal Monday traffic profile for your VPN Gateway public IP:
- Avg 2,000 packets/sec
- Peak 8,000 packets/sec
- Protocol mix: 70% TCP, 20% UDP, 10% ICMP
- Geographic distribution: CA, US, EU
Attack detected when:
- Packets jump to 4,000,000/sec ← 500× normal
- 99% from single ASN in one region
- All UDP port 53 (DNS amplification)
Response: automatic mitigation within seconds
- Rate limit traffic matching attack signature
- Pass legitimate traffic through
- Alert via Azure Monitor

Baselines are built per public IP, per protocol, per port — so the service understands what normal looks like for your VPN Gateway vs your Application Gateway vs your load balancer, and tuning is automatic as your traffic patterns change.


DDoS Protection Tiers Compared

FeatureBasic (free)Network (Standard)IP Protection
Always-on monitoring
Automatic attack mitigation✅ basic✅ advanced✅ advanced
Adaptive ML tuning per IP
Attack analytics & metrics
Attack mitigation reports
Attack mitigation flow logs
Azure Monitor alerts
WAF policy integration
DDoS Rapid Response (Microsoft experts)
Cost protection (service credit)
ScopeAll Azure (shared)Per VNet (plan)Per public IP
PricingFree~$2,944/month + per IP~$199/IP/month

Network Protection (the classic “Standard” tier) is applied at the VNet level via a DDoS Protection Plan — every public IP in all linked VNets is automatically covered.

IP Protection is a newer, per-IP option introduced for smaller deployments where you only need to protect a handful of public IPs without paying for a full plan.


What a DDoS Protection Plan Covers

A single DDoS Protection Plan can be linked to multiple VNets across multiple subscriptions in the same tenant. This is the right model for hub and spoke — one plan at the hub subscription level covers everything:

DDoS Protection Plan (resource group: rg-network-hub)
↓ linked to
Hub VNet → VPN Gateway public IP protected
→ Azure Firewall public IP protected
→ Bastion public IP protected
Spoke 4 (DMZ) → App Gateway public IP protected
→ Load Balancer public IP protected

The first 100 public IPs are included in the plan price. Beyond 100, you pay per additional IP.


Monitoring and Alerting

During and after an attack, DDoS Protection surfaces detailed metrics in Azure Monitor:

MetricWhat it shows
Under DDoS attackBoolean — 1 if attack active on this IP
Inbound packets dropped DDoSPackets/sec being scrubbed
Inbound packets forwarded DDoSClean packets/sec passing through
Inbound bytes DDoSRaw attack volume in bytes/sec
Mitigation reasonSYN flood, UDP flood, etc.

Set an alert rule on Under DDoS attack = 1 to fire a notification to your security team or trigger a Logic App / n8n workflow the moment an attack begins.


When Should You Enable It?

Enable DDoS Protection Standard when any of these are true

Any public-facing production workload with real business impact if taken offline — an internet-facing Application Gateway, a VPN Gateway handling thousands of remote users, or a load balancer fronting a revenue-generating application — warrants the protection. The ~$3K/month cost is trivial compared to the revenue loss and incident response cost of a successful multi-hour DDoS attack.

You also need it when compliance frameworks require it. PCI-DSS, HIPAA, and ISO 27001 environments often require documented DDoS mitigation controls, and DDoS Protection Standard gives you the attack reports and flow logs needed to satisfy auditors.

The cost protection feature is a practical reason too — if an attack causes your Azure compute or bandwidth costs to spike (autoscaled VMs spun up to handle flood traffic, for example), Microsoft will credit those costs back when DDoS Protection was enabled.

You can skip it when

Dev/test environments, internal-only workloads with no public IPs, and resources entirely behind Azure Front Door or a third-party CDN that absorbs attacks upstream don’t need the plan — the Basic tier’s shared protection is sufficient, and the CDN/Front Door layer already absorbs volumetric attacks before they reach your origin.


In a Hub and Spoke Context

The recommended placement is one DDoS Protection Plan at the hub subscription, linked to the hub VNet and any spoke VNets that have public IPs (typically the DMZ spoke with App Gateway and WAF). Pair it with Azure Firewall for Layer 3/4 east-west filtering and Azure WAF on Application Gateway for Layer 7 protection, and you have defence-in-depth across all three attack categories.