Azure private DNS is a great solution to simplify DNS resolution for cloud resources in Azure. However, chances are you have components in your infrastructure that do not natively integrate with Azure DNS zones. In this post, I will show you how you to enable your own DNS solution to resolve names from Azure private DNS zones with CoreDNS on Azure Kubernetes Service.
I will showcase how you can simplify the DNS resolution of Azure Private DNS zones from anywhere in your network using CoreDNS on a central, private AKS cluster. Adoption of Azure Kubernetes Services (AKS) has increased over the last few month and since AKS private clusters are globally available, the demand has grown even more. With new private clusters, new challenges are introduced as well - each private AKS cluster gets deployed with an private Azure DNS zone in which the Kubernetes Master API gets their DNS name in. This really opens the door to expand the capabilities of Azure Private DNS zones for other purposes as well.
DNS Resolution in Azure
In Azure, you can choose from three options for DNS resolution. Azure provided DNS, Azure Private DNS Zones and custom DNS. Each of this options has its benefits depending on the goal you want to achieve.
Azure provided DNS
The default when you deploy a new VNET. Resolution is done in the Azure backend. This is mostly not sufficient because it is not possible to alter DNS zone names nor can the lifecycle be managed.
Azure Private DNS Zones
Azure Private DNS zones can be used to resolve names to a specific domain in your VNET. It supports features like auto registrations for VMs and is a great way to create split-horizon resolution. However, for your VMs to be able to resolve the names of these zones, the zone must be linked to each VNET. This can conflict with auto registration in certain scenarios.
You can use your own DNS servers in Azure. This option offers more flexibility. For this, you need to configure the DNS server on each VNET.
When companies start using cloud services, it has been common practice to migrate virtual machines to Azure first and shift workloads to cloud native services afterwards. Therefore, some basic infrastructure gets set up in Azure as well - like domain controllers, DNS, VPN, networking and so forth. I want to take a look at the Domain Controllers and DNS servers and how they are being integrated into Azure. It is most likely, that a new AD site has been created and at least two domain controllers has been set up that also serve as DNS servers. Those DNS servers are configured on all VNETs to resolve names in the entire network.
The following image shows this a little better:
This is a reference architecture with a simplified view that does not include all components involved nor is it accurate in terms of how the components are connected - it is supposed to give an overview of the data flow.
When you deploy a private AKS cluster into this, a private DNS zone gets created as well and linked to the VNET the cluster got created in.
Architecture with AKS
At this point, the master API of the private AKS cluster akscluster-9123as9.15bf9f2f-f5df-492a-a83b-7e64b597e265.privatelink.westeurope.azmk8s.io can only be resolved (and therefore securely reached) from machines located in the VNET where the AKS got deployed in. Further along, you might want to deploy several AKS clusters or just private DNS zones by themselves and all of them need to be resolved from anywhere in your network.
Solution / Architecture
In general, to resolve addresses from Azure with your own DNS Server, the DNS request for the particular zone must be forwarded to the virtual IP 220.127.116.11. For private DNS zones, an additional zone link must be created:
Azure Private DNS Zone Link
The approach I’m proposing uses the DNS server solution that is already in place - in my case a Microsoft DNS server that also acts as an Domain Controller. This is very common but any regular DNS solution will work. The DNS server gets an conditional forwarder for the zone that points to CoreDNS running on AKS which will forward the requests to the virtual IP 18.104.22.168:
Custom DNS Forwarder Architecture
CoreDNS is a very lightweight, plugin based DNS solution that is perfect for this scenario. The AKS cluster will be placed in the shared network and all private DNS zones will be linked to the shared VNET. The Windows DNS server will be configured to forward the traffic based on requests of the zone privatelink.westeurope.azmk8s.io.
This architecture is based on the requirement to deploy several private AKS clusters. Each cluster comes with a private DNS zone in which the master API gets exposed in. In order to be more dynamic and to eliminate the task of creating or deleting a new conditional forwarder when a new cluster gets provisioned or decommissioned, one conditional forwarder gets created to a central AKS cluster that hosts CoreDNS. CoreDNS forwards all DNS requests to the virtual IP 22.214.171.124. When a new private AKS cluster is provisioned, the private DNS zone needs to be linked to the VNET of the shared AKS cluster and the resolution is established.
The following steps will describe the entire setup of a testing environment. However, in order to deploy this entirely on Azure, the on-premises part is excluded.
Resource Group / VNET
You might want to change some values so they fit into your environment. The entire code is written to be executed consecutively.
# add DNAT to Firewall to access DC and Term using RDP $dnatRuleDC = New-AzFirewallNatRule-Name"dc-rdp"-Protocol"TCP"-SourceAddress"*"-DestinationAddress$fwPip.IpAddress -DestinationPort"3389"-TranslatedAddress$dcNic.IpConfigurations.PrivateIpAddress -TranslatedPort"3389" $dnatRuleTerm = New-AzFirewallNatRule-Name"term-rdp"-Protocol"TCP"-SourceAddress"*"-DestinationAddress$fwPip.IpAddress -DestinationPort"3390"-TranslatedAddress$termNic.IpConfigurations.PrivateIpAddress -TranslatedPort"3389" $natRuleCollection = New-AzFirewallNatRuleCollection-Name"rdp-in"-Priority500-Rule$dnatRuleDC $natRuleCollection.AddRule($dnatRuleTerm) $azureFirewall.NatRuleCollections = $natRuleCollection Set-AzFirewall-AzureFirewall$azureFirewall
Configure Windows DNS Server
At this point, it is time to configure the Windows DNS Server.
install DNS role
install azure cli and kubectl (used later)
setup DNS Zone
# connect to the server mstsc /v $fwPip.IpAddress
I like to use scoop to install tools - you can also install the tools manually. Start a PowerShell as an Administrator and run the following commands:
To set up CoreDNS, we need to establish a connection from the DNS Server to the AKS cluster. Therefore, a temporary VNET link from the private DNS zone of the cluster to the IDM VNET needs to be created:
# restart aks vmss Get-AzVmss-ResourceGroupName"MC_$($rg)_sharedaks_$($location)" | Restart-AzVmss
# restart all vms Get-AzVM-ResourceGroupName$rg | Restart-AzVM
Now its possible to use any client in the network (that has the DNS server configured) to resolve the DNS name of the Azure Private Zone from the cluster.
Custom DNS Forwarder Architecture
# connect to the second server and test the DNS resolution there mstsc /v "$($fwPip.IpAddress):3390"
When a new private AKS cluster is provisioned, the private DNS zone just needs to be linked to the shared VNET and the DNS resolution is established.
If you want to use Azure Private DNS zones for other zones too, you need to create a new conditional forwarder on your DNS server for each zone that points to the CoreDNS load balancer.
When to use this?
Of course, just to forward some DNS requests, this is way too big in terms of administration, know-how and costs. However, if your organization uses container based applications, this solution is perfect for DevOps processes. With a central AKS cluster, you are also be able to consolidate several applications running on VMs into container - for instance Elastic, Splunk, HashiCorp Vault and much more.
Make it production ready
To use this in a production environment, some steps should be taken care of.
Based on how your network is set up, you might want to change the conditional forwarder to something that matches the layout. You don’t want your clients in America to resolve names in Europe 😉
AKS / CoreDNS
For production workloads, Microsoft recommends at least three node AKS cluster. The CoreDNS should get some affinity/antiaffinity rules to be distributed more evenly. If you use Helm and CI/CD, you might want to add the CoreDNS deployment into a Helm chart and deploy is with your pipeline of choice.
We are passionate about IT and we are passionate about sharing. The only possible way for us to express our needs is to collect, enrich and share our knowledge and our everyday experiences. This blog contains knowledge from the field and our goal is to provide helpful articles for everyone that comes across a similar problem or just wants to gain some practical knowledge about cloud technologies.
This blog series introduces a PowerShell module that automatically generated MarkDown documentation of your PowerShell Scripts and modules. It also gives an introduction into Abstract Syntax Trees (ASTs) in PowerShell.
With every recent Windows 10 update, and they happen a lot, Windows unfortunately also resets the power settings of the network adapters. Since I like to start both my PC and notebook from a remote location or from within the same network, I wrote a little PowerShell function to enable Wake-on-LAN (WoL) again.
The Azure Active Directory has for some time been offering the ability to assign licenses to users such as EMS, Office 365 (Exchange, SharePoint, etc.), but can also provide groups with licenses. As soon as a user is added to a group, if there are still enough licenses available, the user will receive the corresponding license assigned to the group. This works with synchronized groups from the local Active Directory as well as with Azure AD Security and dynamic groups.
If you are like me - at least in terms of lazyness - you automate the stuff that you face more than once. Recently, I came accross the reoccuring task of creating Azure DevOps projects with several teams over and over again.
This blog series explains what static site generators are, why we have chosen a static site generator for our blog, how static sites can be implemented using only Microsoft Azure technologies and when you should consider using them vs. a CMS like WordPress.
The SECURITY_DESCRIPTOR structure stores security related attributes of an object. It determines, who can access the object and which additional permissions are assigned. Because sometimes you are confronted with the “raw” NtSecurityDescriptor e.g. in Active Directory related scenarios, I tried give an overview about all parts of it.
During an Exchange online migration, some preparations must take plce in advance so users can be migrated easily to the cloud. A typical error in the mailbox migration process occurs because of the mail domain (property: smtp/proxyaddresses) with the message “Target mailbox doesn’t have an smtp proxy”.