Azure stack kubernetes cluster is not aks

If you’ve been test driving Azure Stack with a full stamp or just the ASDK, you may have decided to try out the Kubernetes Cluster template that is available in the marketplace syndication. This post is meant to walk through what the K8s template is, what it isn’t, and how it works.

The first thing to understand is that the Kubernetes Cluster template - herein KCT - is NOT the Azure Kubernetes Service (AKS). It is more akin to the Azure Container Service (ACS) that preceded the AKS. With ACS, Microsoft had developed a series of templates to roll out a container deployment using the orchestrator of your choosing. You could pick Mesosphere DC/OS, Docker Swarm, or Kubernetes. The template would help you roll out the deployment, but it wasn’t a managed service. It did not apply updates or automatically scale your cluster of nodes. That was up to you. Incidentally, ACS has been deprecated as an offering, replaced by AKS. Azure Kubernetes Service is a managed K8s cluster that allows you to easily scale the number of worker nodes and update the version of K8s in a managed fashion. The VMs running the master nodes in an AKS cluster are not even accessible to you.

Last year, Microsoft introduced a preview version of the Kubernetes Cluster on Azure Stack. It functions in a similar way as ACS, in that it deploys your K8s cluster through a set of ARM templates. In fact, as we work through the contents of the templates, we’ll see that it uses the same open-source acs-engine that the original Azure Container Service did. But it does not provide any kind of ongoing management of the cluster. It also does not have the same programmatic hooks as AKS. If you try to do a deployment from Azure DevOps using the AKS provider as a task, the deployment will fail with an error letting you know that the Microsoft.Containers resource provider was not available. That resource provider is what AKS uses, and it does not exist on Azure Stack. You can use a regular K8s deployment instead and provide the cluster dns name and kube config contents in the Pipeline.

That still leaves questions about how the KCT deploys the cluster and how you can manage it. How do you scale the workers? How do you update the version of K8s? How do you monitor the cluster health? Let’s take a look at the template deployment to get a sense of what is going on here.

The deployment is actually a nested deployment process. There are three deployments in total. The first two are executed directly, and the third uses an ARM template generated by the acs-engine running on a VM created for that purpose. The first template takes all of the parameters, creates some variables, and invokes a new deployment called the dvmdeployment. The dvmdeployment creates the following resources:

• Storage account
• Network security group allowing port 22 access
• Public IP address with a DNS label
• Virtual network with a single subnet
• Network interface with the virtual network, NSG, and PIP associated to it
• VM using the storage account and network interface already created
• Virtual machine extension using the Linux custom script extension

The important things here are the customData being passed to the VM and the Linux custom script being invoked. Let’s dissect those two items to see what’s going on with this dvm machine.

The customData is as follows

[base64(concat('#cloud-config\n\nwrite_files:\n- path: \"/opt/azure/containers/script.sh\"\n  permissions: \"0744\"\n  encoding: gzip\n  owner: \"root\"\n  content: !!binary | [binary blob]

I’ve omitted the binary blob, since it is super long and not helpful. Basically this is writing a file to /opt/azure/containers/script.sh. The custom script extension is running the following command:

[concat(variables('scriptParameters'), ' PUBLICIP_FQDN=', '\"', reference(resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName')),'2015-06-15').dnsSettings.fqdn,'\"',' /bin/bash /opt/azure/containers/script.sh >> /var/log/azure/acsengine-kubernetes-dvm.log 2>&1')]

Which is the JSON way of saying, run the script.sh bash script created in the customData field, pass it the PUBLICIPFQDN parameter and the variable _scriptParameters, and log output to acsengine-kubernetes-dvm.log. The variable scriptParameters has all of the deployment information that was part of the master template, including information like the subscription, tenant, service principal, and K8s cluster specifics. Which begs the question, what is in that script file? The script itself is 263 lines long, so I created a public gist. Feel free to peruse. I’ll hit the highlights.

First it copies the Azure Stack root certificate to /usr/local/share/ca-certificates/azsCertificate.crt. Then it installs pax, jq, and curl. Then it clones the azsmaster branch of the GitHub repo located here. Once it is done cloning, the script expands the archive examples/azurestack/acs-engine.tgz into a bin directory. Now it gets the json file necessary for deployment from the directory examples/azurestack/azurestack-kubernetes[version_number].json. Once it has that, it injects the storage account information into a temporary json file based on the primary json file. The temporary file is validated with a custom function and then copied back to the primary json file. Lastly, the credentials from the service account are added to the json file depending on whether the system is using ADFS or Azure AD for authentication. If everything is ready to go, the script finally runs the ./bin/acs-engine deploy command with all the necessary parameters passed in from the customScript invocation.

Got all that? The TL;DR is that we have created a Virtual machine to run an acs-engine process, and passed it all the settings from the original template filled out during deployment. Once that deployment is complete, this VM and all the accompanying resources could probably be deleted.

What is the acs-engine doing? Glad you asked. That is described is pretty good detail on the GitHub docs, so I won’t rehash it all. Suffice to say that it creates yet another ARM deployment, with its own associated template file. That final template does all the heavy lifting of creating the actual K8s cluster. The first 2098 lines of the template are simply the parameters and variable definitions. I’m not even going to post a gist of that. It’s a bit crazy pants. The only saving grace is that the template was programmatically generated. In terms of actual resource creation, the template creates the following:

• Route table for networking
• Network Security Group for ssh and kubectl
• Virtual network
• Public IP Address for master nodes
• Load balancer for master nodes
• Internal load balancer for master nodes API
• Network interfaces, one per each worker node
• Storage accounts, one per the variable linuxpool2StorageAccountsCount
• An availability set for the worker nodes
• A VM for each worker node
• An availability set for the master nodes
• Storage account for the master nodes
• Network interfaces, one for each master node
• A VM for each master node

And that is it. Each virtual machine has an associated customData and custom script extension command. The customData for the worker node is 8074 characters long. I’m not going to even try and parse all the stuff it is doing. Suffice to say that it is putting in place all the necessary components for a worker node in a K8s cluster. The custom script extension invokes the /opt/azure/containers/provision.sh script and logs to /var/log/azure/cluster-provision.log. The master node customData is 37,044 characters long. That is b/c of binary data being included for the kube-dns-deplyment.yaml file. Again, I am not going to try and parse through all of that. The master node custom script extension also invokes /opt/azure/containers/provision.sh script and logs to /var/log/azure/cluster-provision.log. If something goes wrong with your K8s cluster deployment, those are some of the log files you would want to investigate.

To get back to the original questions. How would you add another worker node to the cluster? Well, the short answer is that ;the acs-engine has a scale command that you could run from the dvmdeployment VM, in the same way that the deploy command was run. More information on the acs-engine scale command can be found in the docs. I haven’t tested it yet, but I plan to in the next few days. How would you update the version of Kubernetes? Acs-engine does not have an upgrade command. You might be able to use something like kubeadm, but I’m not sure. Since the cluster wasn’t built with kubeadm, I’m not sure if it can be used to do the upgrade. Again, more investigation is needed. Lastly, how do you monitor the cluster? That’s a bit more straight forward. Each VM has the Azure VM agent installed, and you can deploy the OMS agent on top of that. Then it’s a simple matter of getting the cluster plugged into Azure Monitor. Otherwise you can use the monitoring solution of your choice.

The offering is still evolving. And I expect that Microsoft will eventually replace this template with a more managed version. But until then, if you are trying to follow a hybrid application deployment model with Azure, Azure Stack, and Kubernetes then hopefully this helped you out.