Understanding ARO API Server Private Endpoint IPs

ARO API Server Private Endpoint IP

The ARO API server private endpoint IP is a private IP address automatically assigned by Azure from your master subnet’s address space when the cluster is created — it becomes the sole network entry point for all Kubernetes API traffic in a private cluster.


How the IP Is Assigned — the Full Mechanism

Azure subnet IP allocation order

Every Azure subnet reserves the first 5 IPs unconditionally:

10.1.0.0 — Network address (unusable)
10.1.0.1 — Default gateway
10.1.0.2 — Azure DNS
10.1.0.3 — Azure future use
10.1.0.4 — Broadcast address
─────────────────────────────────
10.1.0.5 → First assignable IP

After cluster provisioning, the master subnet fills up in this order:

10.1.0.5 → Master node 1 VM NIC (AZ1)
10.1.0.6 → Master node 2 VM NIC (AZ2)
10.1.0.7 → Master node 3 VM NIC (AZ3)
10.1.0.8 → API server private endpoint NIC ← auto-assigned
10.1.0.9 → Internal LB health probe IP
10.1.0.10+ → Future ARO platform components

The exact IP depends on provisioning order — Azure assigns the next available IP dynamically. You cannot pre-specify it, but once assigned it is static for the lifetime of the cluster.


The Private Endpoint NIC in Detail

The private endpoint is an Azure resource called a Private Endpoint — distinct from the VM NICs of the master nodes. You can inspect it:

# Find the ARO managed resource group (contains cluster infrastructure)
MANAGED_RG=$(az aro show \
--resource-group rg-aro \
--name aro-prod \
--query clusterProfile.resourceGroupId -o tsv)
# List private endpoints in the managed resource group
az network private-endpoint list \
--resource-group $MANAGED_RG \
--output table
# Output:
# Name ResourceGroup Location
# ───────────────────────────── ──────────────────── ─────────
# aro-prod-pe-apiserver aro-prod-cluster-rg eastus
# Get the private IP
az network private-endpoint show \
--resource-group $MANAGED_RG \
--name aro-prod-pe-apiserver \
--query 'customDnsConfigs[0].ipAddresses[0]' \
--output tsv
# Output: 10.1.0.8

What the Private Endpoint NIC Actually Is

The private endpoint is not a VM — it is a read-only synthetic NIC injected into your subnet by Azure’s network fabric. It has no OS, no compute, no management plane — it is purely a network construct:

Private Endpoint Resource
├── Name: aro-prod-pe-apiserver
├── Type: Microsoft.Network/privateEndpoints
├── NIC IP: 10.1.0.8 (from master subnet)
├── Target: ARO API server internal load balancer
│ (in Microsoft-managed ARO infrastructure)
├── Protocol: TCP
├── Port: 6443
├── Managed by: Microsoft / Red Hat (not customer)
└── Deletable: No — deleting breaks the cluster

Traffic arriving at 10.1.0.8:6443 is forwarded over Azure’s private backbone to the actual API server processes running on the master nodes — the customer never sees or touches the internal path.


How DNS Wires the IP to the FQDN

ARO automatically creates a Private DNS Zone and inserts an A record pointing the API server FQDN to the private endpoint IP:

# Find the private DNS zone
az network private-dns zone list \
--resource-group $MANAGED_RG \
--query "[].name" -o tsv
# Output:
# cluster.eastus.aroapp.io
# Inspect the A records
az network private-dns record-set a list \
--resource-group $MANAGED_RG \
--zone-name cluster.eastus.aroapp.io \
--output table
# Output:
# Name TTL Records
# ───── ──── ──────────
# api 300 10.1.0.8
# *.apps 300 10.1.1.100

The DNS zone is linked to the ARO spoke VNet automatically. You must manually link it to any other VNet (hub VNet, other spokes) that needs to resolve it:

# Link private DNS zone to hub VNet
az network private-dns link vnet create \
--resource-group $MANAGED_RG \
--zone-name cluster.eastus.aroapp.io \
--name link-hub-vnet \
--virtual-network /subscriptions/.../resourceGroups/rg-hub/providers/
Microsoft.Network/virtualNetworks/hub-vnet \
--registration-enabled false
# Verify resolution from jump host
nslookup api.cluster.eastus.aroapp.io 10.0.5.4
# Server: 10.0.5.4 (DNS Private Resolver)
# Address: 10.1.0.8 ← private endpoint IP returned ✅

Getting the API Server URL and IP Programmatically

# Get the full API server URL
API_URL=$(az aro show \
--resource-group rg-aro \
--name aro-prod \
--query apiserverProfile.url \
--output tsv)
echo $API_URL
# https://api.cluster.eastus.aroapp.io:6443
# Extract just the hostname
API_HOST=$(echo $API_URL | sed 's|https://||' | sed 's|:6443||')
echo $API_HOST
# api.cluster.eastus.aroapp.io
# Resolve to private IP (from inside VNet or connected network)
dig +short $API_HOST
# 10.1.0.8
# Verify TCP reachability on port 6443
nc -zv $API_HOST 6443
# Connection to api.cluster.eastus.aroapp.io 6443 port [tcp] succeeded!
# Login using oc
CREDS=$(az aro list-credentials \
--resource-group rg-aro \
--name aro-prod)
oc login $API_URL \
--username $(echo $CREDS | jq -r .kubeadminUsername) \
--password $(echo $CREDS | jq -r .kubeadminPassword) \
--insecure-skip-tls-verify=false

What Happens If You Try to Reach It From the Internet

The private endpoint IP (10.1.0.8) is a RFC 1918 private address — it is not routable on the public internet. From outside Azure, two things happen:

Scenario 1 — Public DNS lookup:
nslookup api.cluster.eastus.aroapp.io (from internet)
→ Returns NXDOMAIN or no answer
→ Public DNS has no record (zone is private only)
Scenario 2 — Direct TCP to port 6443:
curl https://api.cluster.eastus.aroapp.io:6443 (from internet)
→ DNS fails → connection never established
→ Even if DNS were somehow resolved, 10.1.0.8 is unreachable
from internet — packets dropped at Azure network edge

There is no public IP, no public DNS record, and no network path from the internet to the private endpoint — the attack surface is zero.


IP Stability — Does It Ever Change?

Once assigned at cluster creation time, the private endpoint IP is permanent for the cluster lifetime:

Cluster created: 10.1.0.8 assigned to API private endpoint
Cluster running: 10.1.0.8 (unchanged — days, months, years)
Master node reboot: 10.1.0.8 (private endpoint NIC is independent of master VMs)
ARO version upgrade: 10.1.0.8 (control plane upgrades don't change the PE IP)
Cluster deleted: 10.1.0.8 released back to subnet pool

This stability is intentional — UDRs, NSG rules, firewall rules, and DNS records all reference this IP. If it changed, every network policy referencing it would break. Azure guarantees it for the cluster lifetime without any reservation or static IP configuration needed on your part.


Key Takeaway

The ARO API server private endpoint IP is the next available IP after the master node NICs in your master subnet — automatically assigned by Azure during cluster provisioning, registered in a private DNS zone under aroapp.io, and permanently stable for the cluster lifetime. It is a synthetic NIC with no compute behind it — just a network fabric construct that forwards TCP port 6443 traffic over Azure’s private backbone to the actual API server processes on the master nodes. From the public internet it is completely invisible — no DNS record, no routable IP, no open port.

Leave a comment