Introduction
Azure Kubernetes Service (AKS) private clusters expose the Kubernetes API server only through a private endpoint, providing enhanced security by removing public access. However, this configuration requires proper private link setup, DNS resolution, and network connectivity for administrators and CI/CD systems to access the cluster. When connectivity fails, kubectl commands timeout, deployments cannot be applied, and cluster management becomes impossible. Understanding the private cluster architecture, DNS requirements, and network path is essential for troubleshooting and maintaining private AKS clusters.
Symptoms
When AKS private cluster connectivity fails, you will observe:
- kubectl commands hang or timeout with "connection refused"
- Unable to connect to API server from local machine or CI/CD
- "Unable to connect to the server: dial tcp: lookup" DNS errors
- Azure DevOps or GitHub Actions cannot deploy to cluster
- az aks get-credentials succeeds but subsequent kubectl commands fail
kubectl error messages:
``
Unable to connect to the server: dial tcp 10.0.0.4:443: i/o timeout
Or:
``
E1212 Unable to connect to the server: dial tcp: lookup myaks-abc123.hcp.eastus.azmk8s.io on 10.0.0.1:53: no such host
Azure portal shows: - AKS cluster is "Running" status - Private cluster enabled: Yes - API server endpoint shows private IP address
Common Causes
- 1.Missing private DNS zone - No private DNS zone for Azure privatelink region
- 2.DNS resolution failure - Cannot resolve private API server FQDN
- 3.Network not peered - Client VNet not peered to AKS VNet
- 4.Private endpoint not created - Private link endpoint missing
- 5.Firewall blocking access - NSG or Azure Firewall blocking port 443
- 6.VPN/ExpressRoute issues - On-premises connectivity not working
- 7.Wrong kubeconfig context - Using wrong cluster context
- 8.API server authorized IPs - IP restrictions blocking access
- 9.Jump host required - Trying to access from unsupported network
- 10.Azure Private Link service limits - Connection limits exceeded
Step-by-Step Fix
Step 1: Verify Cluster Configuration
Check the AKS private cluster configuration:
```bash # Get cluster details az aks show --resource-group myResourceGroup --name myAKSCluster --query "{apiServerAccessProfile: apiServerAccessProfile, networkProfile: networkProfile}"
# Check if private cluster az aks show -g myResourceGroup -n myAKSCluster --query "apiServerAccessProfile.enablePrivateCluster"
# Get API server endpoint az aks show -g myResourceGroup -n myAKSCluster --query "fqdn" -o tsv
# Check private link resources az network private-endpoint list --resource-group myResourceGroup -o table ```
Expected output for private cluster:
``json
{
"apiServerAccessProfile": {
"enablePrivateCluster": true,
"privateDnsZone": "system",
"privateLinkServiceResourceId": "/subscriptions/.../privateLinkServices/..."
},
"networkProfile": {
"networkPlugin": "azure",
"networkPolicy": null,
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16"
}
}
Step 2: Check DNS Resolution
Verify the API server FQDN resolves correctly:
```bash # Get cluster FQDN FQDN=$(az aks show -g myResourceGroup -n myAKSCluster --query "fqdn" -o tsv) echo $FQDN # Example: myaks-abc123.hcp.eastus.azmk8s.io
# Test DNS resolution from your network nslookup $FQDN
# Expected: Returns private IP (10.x.x.x) # Wrong: Returns public IP or NXDOMAIN
# Dig for more details dig $FQDN
# Check from a VM in the same VNet az vm run-command invoke -g myResourceGroup -n myVM --command-id RunShellScript --scripts "nslookup $FQDN" ```
If DNS returns public IP or fails: ```bash # Check if private DNS zone exists az network private-dns zone list -o table
# Look for the AKS private DNS zone az network private-dns zone list --query "[?contains(name, 'privatelink')].name" -o tsv
# Check zone for your region az network private-dns zone show --name privatelink.eastus.azmk8s.io --resource-group myResourceGroup ```
Step 3: Configure Private DNS Zone
Create and link the private DNS zone:
```bash # Create private DNS zone for AKS (if using custom DNS) az network private-dns zone create \ --resource-group myResourceGroup \ --name privatelink.eastus.azmk8s.io
# Get VNet ID VNET_ID=$(az network vnet show -g myResourceGroup -n myVNet --query "id" -o tsv)
# Link DNS zone to your VNet az network private-dns link vnet create \ --resource-group myResourceGroup \ --zone-name privatelink.eastus.azmk8s.io \ --name myAKSLink \ --virtual-network $VNET_ID \ --registration-enabled false
# Get the private endpoint IP PRIVATE_IP=$(az aks show -g myResourceGroup -n myAKSCluster --query "apiServerAccessProfile.privateApiServerEndpoint" -o tsv)
# Add DNS record az network private-dns record-set a add-record \ --resource-group myResourceGroup \ --zone-name privatelink.eastus.azmk8s.io \ --record-set-name myaks-abc123 \ --ipv4-address $PRIVATE_IP ```
For system-managed DNS (simpler approach): ```bash # When creating cluster, use system-managed private DNS az aks create \ --resource-group myResourceGroup \ --name myAKSCluster \ --load-balancer-sku standard \ --enable-private-cluster \ --private-dns-zone system \ --generate-ssh-keys
# For existing cluster, update DNS zone mode az aks update \ --resource-group myResourceGroup \ --name myAKSCluster \ --private-dns-zone system ```
Step 4: Verify Network Connectivity
Check network path to API server:
```bash # Get API server private IP API_IP=$(az aks show -g myResourceGroup -n myAKSCluster --query "privateFqdn" -o tsv) # Or get from DNS API_IP=$(nslookup $FQDN | grep Address | tail -1 | awk '{print $2}')
# Test connectivity from a VM in the same network az vm run-command invoke -g myResourceGroup -n myVM --command-id RunShellScript --scripts "nc -zv $API_IP 443"
# Check if you can reach the endpoint curl -k https://$FQDN/healthz # Expected: ok (or 401 Unauthorized is also fine, means server is reachable)
# Test from your local machine (if connected via VPN) Test-NetConnection -ComputerName $FQDN -Port 443 # PowerShell nc -zv $FQDN 443 # Linux/macOS ```
Step 5: Configure VNet Peering
If accessing from different VNet, configure peering:
```bash # Get AKS VNet ID AKS_VNET_ID=$(az aks show -g myResourceGroup -n myAKSCluster --query "nodeResourceGroup" -o tsv) AKS_VNET_ID=$(az network vnet list -g $AKS_VNET_ID --query "[0].id" -o tsv)
# Get your VNet ID MY_VNET_ID=$(az network vnet show -g myResourceGroup -n myVNet --query "id" -o tsv)
# Create peering (from your VNet to AKS VNet) az network vnet peering create \ --name myVNet-to-AKSVNet \ --resource-group myResourceGroup \ --vnet-name myVNet \ --remote-vnet $AKS_VNET_ID \ --allow-vnet-access
# Create reverse peering AKS_VNET_NAME=$(az network vnet list -g $(az aks show -g myResourceGroup -n myAKSCluster --query "nodeResourceGroup" -o tsv) --query "[0].name" -o tsv)
az network vnet peering create \ --name AKSVNet-to-myVNet \ --resource-group $(az aks show -g myResourceGroup -n myAKSCluster --query "nodeResourceGroup" -o tsv) \ --vnet-name $AKS_VNET_NAME \ --remote-vnet $MY_VNET_ID \ --allow-vnet-access ```
Step 6: Use Command Invoke (Quick Access)
For quick access without network configuration, use command invoke:
```bash # Run kubectl commands via Azure API az aks command invoke \ --resource-group myResourceGroup \ --name myAKSCluster \ --command "kubectl get nodes"
# Apply manifests az aks command invoke \ -g myResourceGroup \ -n myAKSCluster \ --command "kubectl apply -f deployment.yaml" \ --file deployment.yaml
# Run multiple commands az aks command invoke \ -g myResourceGroup \ -n myAKSCluster \ --command "kubectl get pods -n default && kubectl get services" ```
Step 7: Configure Authorized IP Ranges
If using authorized IP ranges, ensure your IP is included:
```bash # Check current authorized IP ranges az aks show -g myResourceGroup -n myAKSCluster --query "apiServerAccessProfile.authorizedIpRanges"
# Get your public IP curl -s ifconfig.me
# Add your IP to authorized ranges az aks update \ -g myResourceGroup \ -n myAKSCluster \ --api-server-authorized-ip-ranges 1.2.3.4/32,10.0.0.0/8
# Disable authorized IP ranges (for troubleshooting) az aks update \ -g myResourceGroup \ -n myAKSCluster \ --api-server-authorized-ip-ranges "" ```
Step 8: Set Up Jump Host Access
Create a jump host for accessing the private cluster:
```bash # Create jump host VM in the same VNet az vm create \ --resource-group myResourceGroup \ --name aksJumpHost \ --image UbuntuLTS \ --vnet-name myVNet \ --subnet default \ --generate-ssh-keys \ --public-ip-address aksJumpHost-pip
# SSH to jump host ssh azureuser@$(az vm show -g myResourceGroup -n aksJumpHost --query "publicIps" -o tsv)
# On jump host, install kubectl and az CLI curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash sudo az aks install-cli
# Get credentials az aks get-credentials -g myResourceGroup -n myAKSCluster
# Test connection kubectl get nodes ```
Verification
After configuration, verify connectivity:
```bash # Get fresh credentials az aks get-credentials -g myResourceGroup -n myAKSCluster --overwrite-existing
# Test kubectl kubectl get nodes
# Expected output: # NAME STATUS ROLES AGE VERSION # aks-nodepool1-12345678-vmss000000 Ready agent 1h v1.27.x # aks-nodepool1-12345678-vmss000001 Ready agent 1h v1.27.x
# Test other commands kubectl get pods -A kubectl cluster-info ```
Verify DNS resolution: ```bash # Should resolve to private IP nslookup $(az aks show -g myResourceGroup -n myAKSCluster --query "fqdn" -o tsv)
# Expected: 10.x.x.x (private IP range) ```
Prevention
To prevent connectivity issues:
- 1.Document network architecture:
- 2.
` - 3.- Client VNet: 10.0.0.0/16
- 4.- AKS VNet: 10.1.0.0/16
- 5.- Peering: Configured bidirectional
- 6.- Private DNS: privatelink.eastus.azmk8s.io
- 7.
` - 8.Use consistent DNS configuration:
- 9.```bash
- 10.# Always use system-managed DNS for simplicity
- 11.az aks create --private-dns-zone system
- 12.
` - 13.Monitor connectivity:
- 14.```bash
- 15.# Add to monitoring script
- 16.#!/bin/bash
- 17.FQDN=$(az aks show -g $RG -n $AKS --query "fqdn" -o tsv)
- 18.if ! nc -zv $FQDN 443 2>/dev/null; then
- 19.echo "ALERT: Cannot reach AKS API server"
- 20.fi
- 21.
` - 22.Use Azure Bastion for secure access:
- 23.```bash
- 24.az network bastion create \
- 25.--resource-group myResourceGroup \
- 26.--name myBastion \
- 27.--vnet-name myVNet
- 28.
` - 29.Configure CI/CD with proper network access:
- 30.```yaml
- 31.# Azure DevOps: Use self-hosted agent in peered VNet
- 32.pool:
- 33.name: 'PrivateAgentPool'
- 34.
` - 35.Keep kubeconfig updated:
- 36.```bash
- 37.# Refresh credentials regularly
- 38.az aks get-credentials -g $RG -n $AKS --overwrite-existing
- 39.
`
Related Articles
- [Technical troubleshooting: Fix Azure Aks Pod Crashloopbackoff Issue in Azure](azure-aks-pod-crashloopbackoff)
- [Technical troubleshooting: Fix Azure Api Management Policy Expression Runtime](azure-api-management-policy-expression-runtime-error)
- [Technical troubleshooting: Fix Azure App Configuration Feature Flag Not Refre](azure-app-configuration-feature-flag-not-refreshing)
- [Technical troubleshooting: Fix Azure App Service 503 Always On Disabled Issue](azure-app-service-503-always-on-disabled)
- [Technical troubleshooting: Fix Azure Application Gateway Err SSL Unrecognized](azure-application-gateway-err-ssl-unrecognized-name-alert)
<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "TechArticle", "headline": "Azure AKS Private Cluster Connectivity", "description": "Fix Azure AKS private cluster connectivity issues. Configure private endpoints, DNS zones, and network access for kubectl.", "url": "https://www.fixwikihub.com/fix-azure-aks-private-cluster-connectivity", "publisher": { "@type": "Organization", "name": "FixWikiHub", "url": "https://www.fixwikihub.com" }, "author": { "@type": "Person", "name": "FixWikiHub Editorial Team" }, "datePublished": "2025-12-11T07:28:26.055Z", "dateModified": "2025-12-11T07:28:26.055Z" } </script>