Azure stack on azure part 1

With the introduction of Dv3 and Ev3 VMs in Microsoft Azure, it became possible to run nested virtualization on Azure. Since I’ve got Azure Stack on the brain these days, my immediate thought was, “I wonder if I can run Azure Stack on Azure?” (cue Inception music). Not only was the answer yes, but others had already started the process for me. Following in the footsteps of Daniel Neumann and Florent Appointaire, I was able to bet the process running. One of the engineers at Microsoft took some of that work, added their special sauce and rolled out a GitHub repo that helps you through the process. I have forked that repo, and started adding some automation myself.

In this series of posts I will show you how to deploy the Azure Stack Development Kit (ASDK) on Azure, perform the initial configuration tasks, and add additional services using resource providers. Let’s begin!

First of all, before we go any further, please be aware that this will cost money. The hardware requirements for the ASDK are 12 cores, 96BG RAM, and four data drives, each at a minimum of 140GB. Part of the reason to run the ASDK in Azure is that many IT Pros and Developers will not have a machine hanging around their home or work lab that meets these requirements. The VM size I will be using is an E16S v3. It will be using premium storage with 4x 256GB data disks and a 256GB OS disk. I will also be using a Azure Hybrid Use license. Using the Azure cost calculator, I can see that running in the East US region is going to cost $778.85 per month for the VM and another $190.06 per month for the storage, bringing it to a grand total of $968.91 per month to run the whole thing. Of course, you can shut the VM down when you’re not working on it, bringing the price down to $1.06 per hour for the VM. The premium storage, however, is charged while the disk exists. If you plan to follow along and deploy this yourself, it is going to cost you money in Azure.

Second disclaimer - wow now I feel like a lawyer - this scenario is supported for the ASDK, but it is not meant for any type of production workload. That goes for the ASDK in general, but especially running the ASDK on Azure. The nested virtualization can get a little buggy sometimes, and the process has been known to BSOD during the network configuration.

OK, with all the hand waving out of the way, here’s the process. First deploy the template from my GitHub repo. There’s a nice shiny button, or you could use PowerShell. As much as I love PowerShell, I cannot resist a nice shiny button. Clicking said button takes you to the portal where you can fill out the required parameters. Auto Shutdown is enabled by default - trying to save you money - but I usually turn it off. Once everything looks good, accept the terms and conditions and then deploy the template. That is going to take a while, so let’s look at what it’s actually doing. To the JSON!

"resources": [
    {
        "type": "extensions",
        "name": "CustomScriptExtension",
        "apiVersion": "2016-04-30-preview",
        "location": "[resourceGroup().location]",
        "dependsOn": [
            "[parameters('virtualMachineName')]"
        ],
        "properties": {
            "publisher": "Microsoft.Compute",
            "type": "CustomScriptExtension",
            "typeHandlerVersion": "1.8",
            "autoUpgradeMinorVersion": true,
            "settings": {
                "fileUris": [
                    "[variables('fileUri')]"
                ],
                "commandToExecute": "[concat('powershell.exe -ExecutionPolicy Unrestricted -File ', variables('scriptFileName'), ' ', variables('adminUsername'))]"
            }
        }
    }
]

In the resources section there is the customScriptExtension, which executes a PowerShell script defined in the variables.

"scriptFileName": "post-config.ps1",
"scriptPath": "https://raw.githubusercontent.com/ned1313/AzureStack-VM-PoC/master/scripts/",
"fileUri": "[concat(variables('scriptPath'), variables('scriptFileName'))]",

The script in this case points to the GitHub repository, in the scripts folder, to a script called post-config.ps1. That scripts does some helpful things.

1. Create a local directory (AzureStackOnAzureVM) to place files in
2. Disable IE’s Enhanced Security Configuration so that you can actually download files and visit websites
3. Make a bunch of registry key changes to enable Credential Delegation and allow the ASDK VMs to interact with the host VM as needed
4. Add some of the Windows features needed to function as the ASDK host
5. Rename the local Administrator to Administrator
6. Set the PowerShell execution policy to Unrestricted
7. Download a PowerShell script for the ASDK installation process
8. Download the ASDK downloader executable 9. Add the rest of the Windows features that require a restart, and then restart the VM

Once all those steps are complete, we will need to log into the VM to continue the process. Now remember that the local admin account has been renamed to Administrator, so you will use .\Administrator as the username to login. On the desktop you should see the next step of the process, a shortcut to the Install-ASDK.ps1 file that was downloaded in during the post-config.ps1 script. Rather than run it directly, let’s take a look at the contents. Open the file in an Administrative PowerShell ISE window.

You can run the Install-ASDK.ps1 script now, and it will prompt you for all the necessary values. This script checks to see what versions of the ASDK are available, starting with version 1711. Then it will prompt you for the local Administrator password and verify it. Then it will prompt for a Tenant Admin to use with the ASDK. I created a Tenant Admin user for this purpose called AzsAdmin1@azslab.us. Now the scripts asks which version of the ASDK to install, if you’re not sure just pick the most recent. If everything looks good, press any key and the process will begin.

The script will now download the appropriate ASDK files and merge them into the CloudBuilder.vhdx. Then it will mount the vhdx file and extract the necessary components for the host VM to become the ASDK host. In a typical deployment of the ASDK, you would set the vhdx as a boot option and reboot directly into it. Since that is not an option when running in Azure, instead we pull out the deployment files from the vhdx. Then the script makes a few setting tweaks to allow the ASDK to run in a nested virtualization mode, rather than on a physical server. With that work out of the way, the script then downloads some additional scrips from the GitHub repo and schedules a job to run at next restart, invoking the ASDKCompanionService.ps1 script. This script is what configures the networking component for the BGPNAT VM, and it will kick off after the host VM is restarted. Finally, the script kicks off the ASDK installer PowerShell script, passing it a parameter block for configuration.

This process will take a while, like a really long time. My first run took about 3 hours. Now would be a good time to get a cup of coffee, run a half marathon, and write your mother back since she hasn’t heard from you in a while and you know how she worries.

During the network configuration, you will likely be disconnected from the RDP session. That is normal, and it should come back after about one minute. Later you will be logged out of the system, this is because the host VM will be joined to the local ASDK Active Directory. After the reboot, you can log back in as AzureStack\AzureStackAdmin - using the same password as the local Administrator - to continue watching the process. It is at this point that the ASDKCompanionService starts checking to see if the networking configuration of the ASDK has created the internal switch and the BGPNAT VM. It will create an additional internal switch, change the BGPNAT VM adapter and fix the IP addressing. This is where the process typically throws a BSOD if it is going to. If you get past this point, the deployment will likely complete successfully.

Now we’ve got ourselves a deployment of Azure Stack running in Azure. In the next post I will through the process of registering the Azure Stack with Azure for marketplace syndication, and creating some offers and plans for Azure Stack users.