In a container-based microservices approach to application development, application components must work together to process their tasks. Kubernetes provides various resources that enable this application communication. You can connect to and expose applications internally or externally.
To build highly available applications, you can load balance your applications. For security reasons, you may also need to restrict the flow of network traffic into or between pods and nodes. To allow access to your applications, or for application components to communicate with each other, Kubernetes provides an abstraction layer to virtual networking. Kubernetes nodes are connected to a virtual network, and can provide inbound and outbound connectivity for pods.
The kube-proxy component runs on each node to provide these network features. You can also distribute traffic using a load balancer. More complex routing of application traffic can also be achieved with Ingress Controllers.
Security and filtering of the network traffic for pods is possible with Kubernetes network policies. The Azure platform also helps to simplify virtual networking for AKS clusters. When you create a Kubernetes load balancer, the underlying Azure load balancer resource is created and configured.
As you open network ports to pods, the corresponding Azure network security group rules are configured. To simplify the network configuration for application workloads, Kubernetes uses Services to logically group a set of pods together and provide network connectivity.
The following Service types are available:. Good for internal-only applications that support other workloads within the cluster. NodePort - Creates a port mapping on the underlying node that allows the application to be accessed directly with the node IP address and port. LoadBalancer - Creates an Azure load balancer resource, configures an external IP address, and connects the requested pods to the load balancer backend pool.
To allow customers' traffic to reach the application, load balancing rules are created on the desired ports. For additional control and routing of the inbound traffic, you may instead use an Ingress controller. The IP address for load balancers and services can be dynamically assigned, or you can specify an existing static IP address to use.
Both internal and external static IP addresses can be assigned. Both internal and external load balancers can be created. Internal load balancers are only assigned a private IP address, so they can't be accessed from the Internet. The kubenet networking option is the default configuration for AKS cluster creation. With kubenetnodes get an IP address from the Azure virtual network subnet. Pods receive an IP address from a logically different address space to the Azure virtual network subnet of the nodes.
Network address translation NAT is then configured so that the pods can reach resources on the Azure virtual network. Nodes use the kubenet Kubernetes plugin. You can let the Azure platform create and configure the virtual networks for you, or choose to deploy your AKS cluster into an existing virtual network subnet.
This approach greatly reduces the number of IP addresses that you need to reserve in your network space for pods to use. For more information, see Configure kubenet networking for an AKS cluster.
These IP addresses must be unique across your network space, and must be planned in advance. Each node has a configuration parameter for the maximum number of pods that it supports.
Tutorial: Configure kubenet networking in Azure Kubernetes Service (AKS) using Ansible
The equivalent number of IP addresses per node are then reserved up front for that node. This approach requires more planning, as can otherwise lead to IP address exhaustion or the need to rebuild clusters in a larger subnet as your application demands grow.AKS reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure.
As a hosted Kubernetes service, Azure handles critical tasks like health monitoring and maintenance for you.
The Kubernetes masters are managed by Azure. You only manage and maintain the agent nodes. As a managed Kubernetes service, AKS is free - you pay only for the agent nodes within your clusters; not for the masters. This IP address range should be an address space that isn't used elsewhere in your network. The address range must be large enough to accommodate the number of nodes that you expect to scale up to. You can't change this address range once the cluster is deployed. As the cluster scales or upgrades, Azure continues to assign a pod IP address range to each new node.
If you modify it, use the single-line format - starting with "ssh-rsa" without the quotes. You can set these values to your service principal or load these values from environment variables:. When you create an AKS cluster, a network security group and route table are created. These resources are managed by AKS and updated when you create and expose services. Associate the network security group and route table with your virtual network subnet as follows.
You may also leave feedback directly on GitHub. Skip to main content. Exit focus mode. Is this page helpful? Yes No. Any additional feedback? Skip Submit. Send feedback about This product This page. This page. Submit feedback. There are no open issues. View on GitHub.By default, AKS clusters use kubenetand a virtual network and subnet are created for you.
With kubenetnodes get an IP address from a virtual network subnet. This approach reduces the number of IP addresses that you need to reserve in your network space for pods to use. These IP addresses must be unique across your network space, and must be planned in advance. Each node has a configuration parameter for the maximum number of pods that it supports. The equivalent number of IP addresses per node are then reserved up front for that node. This approach requires more planning, and often leads to IP address exhaustion or the need to rebuild clusters in a larger subnet as your application demands grow.
For more information on network options and considerations, see Network concepts for Kubernetes and AKS. Clusters configured with Azure CNI networking require additional planning. The size of your virtual network and its subnet must accommodate the number of pods you plan to run and the number of nodes for the cluster.
IP addresses for the pods and the cluster's nodes are assigned from the specified subnet within the virtual network. Each node is configured with a primary IP address.
By default, 30 additional IP addresses are pre-configured by Azure CNI that are assigned to pods scheduled on the node.
When you scale out your cluster, each node is similarly configured with IP addresses from the subnet. You can also view the maximum pods per node. The number of IP addresses required should include considerations for upgrade and scaling operations. If you set the IP address range to only support a fixed number of nodes, you cannot upgrade or scale your cluster. When you upgrade your AKS cluster, a new node is deployed into the cluster. Services and workloads begin to run on the new node, and an older node is removed from the cluster.
This rolling upgrade process requires a minimum of one additional block of IP addresses to be available. When you scale an AKS cluster, a new node is deployed into the cluster. Services and workloads begin to run on the new node. Your IP address range needs to take into considerations how you may want to scale up the number of nodes and pods your cluster can support.
One additional node for upgrade operations should also be included. If you expect your nodes to run the maximum number of pods, and regularly destroy and deploy pods, you should also factor in some additional IP addresses per node.
These additional IP addresses take into consideration it may take a few seconds for a service to be deleted and the IP address released for a new service to be deployed and acquire the address. The IP address plan for an AKS cluster consists of a virtual network, at least one subnet for nodes and pods, and a Kubernetes service address range.
The minimum number of IP addresses required is based on that value. If you calculate your minimum IP address requirements on a different maximum value, see how to configure the maximum number of pods per node to set this value when you deploy your cluster. Kubernetes service address range This range should not be used by any network element on or connected to this virtual network.
You can reuse this range across different AKS clusters. Don't use the first IP address in your address range, such as. The first address in your subnet range is used for the kubernetes. Docker bridge address The Docker bridge network address represents the default docker0 bridge network address present in all Docker installations.
While docker0 bridge is not used by AKS clusters or the pods themselves, you must set this address to continue to support scenarios such as docker build within the AKS cluster.
Default of Maximum pods per node The maximum number of pods per node in an AKS cluster is When you run modern, microservices-based applications in Kubernetes, you often want to control which components can communicate with each other.
How to: Azure Kubernetes Service + Custom VNET with Kubenet
The principle of least privilege should be applied to how traffic can flow between pods in an Azure Kubernetes Service AKS cluster. Let's say you likely want to block traffic directly to back-end applications. The Network Policy feature in Kubernetes lets you define rules for ingress and egress traffic between pods in a cluster. This article shows you how to install the network policy engine and create Kubernetes network policies to control the flow of traffic between pods in AKS.
Network policy should only be used for Linux-based nodes and pods in AKS. You need the Azure CLI version 2. If you used the network policy feature during preview, we recommend that you create a new cluster.
If you wish to continue using existing test clusters that used network policy during preview, upgrade your cluster to a new Kubernetes versions for the latest GA release and then deploy the following YAML manifest to fix the crashing metrics server and Kubernetes dashboard. This fix is only required for clusters that used the Calico network policy engine.
All pods in an AKS cluster can send and receive traffic without limitations, by default. To improve security, you can define rules that control the flow of traffic. Back-end applications are often only exposed to required front-end services, for example. Or, database components are only accessible to the application tiers that connect to them.
Network Policy is a Kubernetes specification that defines access policies for communication between Pods. Using Network Policies, you define an ordered set of rules to send and receive traffic and apply them to a collection of pods that match one or more label selectors.
These network policy rules are defined as YAML manifests. Network policies can be included as part of a wider manifest that also creates a deployment or service.
Azure provides two ways to implement network policy. You choose a network policy option when you create an AKS cluster. The policy option can't be changed after the cluster is created:. Both implementations use Linux IPTables to enforce the specified policies. Policies are translated into sets of allowed and disallowed IP pairs. These pairs are then programmed as IPTable filter rules. To see network policies in action, let's create and then expand on a policy that defines traffic flow:.
The network policy feature can only be enabled when the cluster is created. You can't enable network policy on an existing AKS cluster. For more detailed information on how to plan out the required subnet ranges, see configure advanced networking.
Note that instead of using a service principal, you can use a managed identity for permissions. For more information, see Use managed identities. It takes a few minutes to create the cluster. When the cluster is ready, configure kubectl to connect to your Kubernetes cluster by using the az aks get-credentials command. This command downloads credentials and configures the Kubernetes CLI to use them:.
Before you define rules to allow specific network traffic, first create a network policy to deny all traffic. This policy gives you a starting point to begin to whitelist only the desired traffic. You can also clearly see that traffic is dropped when the network policy is applied. For the sample application environment and traffic rules, let's first create a namespace called development to run the example pods:.
This back-end pod can be used to simulate a sample back-end web-based application. Create this pod in the development namespace, and open port 80 to serve web traffic.Update : This configuration is now officially documented. I thought I would share some of the insights I stumble upon.
For the latter, check our last article where we deploy advanced networking using ARM Template. As usual, the code used here is available in GitHub. To simplify the discussion, we assume we deploy services using internal load balancer. Basically, both pods and services get a private IP. Services also get a cluster-ip accessible only from within the cluster.
The first three parameters are related to the service principal we need to create. The fourth allows us to choose between Kubenet and Azure plugin. Once deployed we can connect to the cluster and deploy our service. This file deploys 3 pods in a deployment and a service to load balance them. Our service is the one named web-service. Its external IP belongs to the virtual network. The cluster-IP is something that can only be resolved within the cluster. The Kubenet plugin is related to basic networking.
This is used in the online documentation, for instance in the quickstart. They are therefore resolvable only from within the cluster. Here we are going to do something different than basic networking. Again, the first three parameters are related to the service principal we need to create. For that reason the Kubelet plugin consumes much less private IPs. We see the typical underlying resources of an AKS cluster. One we do not see in an advanced networking cluster is the Route Table. We should see there are three routes.
That configuration routes 3 cluster-IP ranges to the three primary IPs of the three nodes. This routing is necessary for requests to pods off node.By default, AKS clusters use kubenetand an Azure virtual network and subnet are created for you.
With kubenetnodes get an IP address from the Azure virtual network subnet. Pods receive an IP address from a logically different address space to the Azure virtual network subnet of the nodes.
Network address translation NAT is then configured so that the pods can reach resources on the Azure virtual network. This approach greatly reduces the number of IP addresses that you need to reserve in your network space for pods to use. These IP addresses must be unique across your network space, and must be planned in advance. Each node has a configuration parameter for the maximum number of pods that it supports.
The equivalent number of IP addresses per node are then reserved up front for that node. This approach requires more planning, and often leads to IP address exhaustion or the need to rebuild clusters in a larger subnet as your application demands grow.
This article shows you how to use kubenet networking to create and use a virtual network subnet for an AKS cluster.How to build and deploy a containerized app to Azure Kubernetes Service (AKS) - Azure Friday
For more information on network options and considerations, see Network concepts for Kubernetes and AKS. The use of kubenet as the network model is not available for Windows Server containers. You need the Azure CLI version 2. In many environments, you have defined virtual networks and subnets with allocated IP address ranges.
These virtual network resources are used to support multiple services and applications. With kubenetonly the nodes receive an IP address in the virtual network subnet. Pods can't communicate directly with each other.
Network concepts for applications in Azure Kubernetes Service (AKS)
You could also deploy pods behind a service that receives an assigned IP address and load balances traffic for the application. The following diagram shows how the AKS nodes receive an IP address in the virtual network subnet, but not the pods:. You can use Calico Network Policiesas they are supported with kubenet. Your clusters can be as large as the IP address range you specify. However, the IP address range must be planned in advance, and all of the IP addresses are consumed by the AKS nodes based on the maximum number of pods that they can support.
With Azure CNIa common issue is the assigned IP address range is too small to then add additional nodes when you scale or upgrade a cluster. The network team may also not be able to issue a large enough IP address range to support your expected application demands. As a compromise, you can create an AKS cluster that uses kubenet and connect to an existing virtual network subnet. This approach lets the nodes receive defined IP addresses, without the need to reserve a large number of IP addresses up front for all of the potential pods that could run in the cluster.
With kubenetyou can use a much smaller IP address range and be able to support large clusters and application demands. This cluster size would support up to 2, pods with a default maximum of pods per node. The maximum number of pods per node that you can configure with kubenet in AKS is These maximums don't take into account upgrade or scale operations.
In practice, you can't run the maximum number of nodes that the subnet IP address range supports.By default, AKS clusters have unrestricted outbound egress internet access.
This level of network access allows nodes and services you run to access external resources as needed. If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. Configure your preferred firewall and security rules to allow these required ports and addresses. This article details what network ports and fully qualified domain names FQDNs are required and optional if you restrict egress traffic in an AKS cluster.
This document covers only how to lock down the traffic leaving the AKS subnet. AKS has no ingress requirements. Blocking internal subnet traffic using network security groups NSGs and firewalls is not supported. To control and block the traffic within the cluster, use Network Policies.
You need the Azure CLI version 2. Run az --version to find the version. These actions could be to communicate with the API server, or to download and then install core Kubernetes cluster components and node security updates.
By default, egress outbound internet traffic is not restricted for nodes in an AKS cluster. The cluster may pull base system container images from external repositories. To increase the security of your AKS cluster, you may wish to restrict egress traffic. If you lock down the egress traffic in this manner, define specific ports and FQDNs to allow the AKS nodes to correctly communicate with required external services.
You can use Azure Firewall or a 3rd-party firewall appliance to secure your egress traffic and define these required ports and addresses.
AKS does not automatically create these rules for you. The following ports and addresses are for reference as you create the appropriate rules in your network firewall.
When you use Azure Firewall to restrict egress traffic and create a user-defined route UDR to force all egress traffic, make sure you create an appropriate DNAT rule in Firewall to correctly allow ingress traffic.
The issue occurs if the AKS subnet has a default route that goes to the firewall's private IP address, but you're using a public load balancer - ingress or Kubernetes service of type: LoadBalancer. In this case, the incoming load balancer traffic is received via its public IP address, but the return path goes through the firewall's private IP address.
Because the firewall is stateful, it drops the returning packet because the firewall isn't aware of an established session. Limiting egress traffic only works on new AKS clusters. For existing clusters, perform a cluster upgrade operation using the az aks upgrade command before you limit the egress traffic.
For existing clusters, perform a cluster upgrade operation using the az aks upgrade command to remove these rules. Please upgrade your existing firewall rules for the changes to take effect.
Some of the features below are in preview. The suggestions in this article are subject to change as the feature moves to public preview and future release stages. In this article, you learned what ports and addresses to allow if you restrict egress traffic for the cluster.
You can also define how the pods themselves can communicate and what restrictions they have within the cluster. For more information, see Secure traffic between pods using network policies in AKS.