Using TOKEN substitution in Azure DevOps

There are many different approaches to configuring and deploy .NET web applications for multiple environments using Azure DevOps. And some of the built-in tasks in Azure DevOps work just fine. But let’s say we have the following constraints:

Constraints

• No secrets in source code.
  We don’t want to put them in XML transformation files within the solution.
• No code rewrites.
  While we can configure NLOG via code, we want a general solution that will work for any type of config.
• No local changes for debugging.
  Ensure that the local developer experience continues to work and that no extra configuration or steps needed to run locally, just hit F5 and go.
• No extra builds in DevOps.
  Build the application once in DevOps. We don’t want to have a separate build for each environment. The build pipeline will run the build the app, run the code tests and produce a deployment artifact (Zip file)
• General solution
  Have a solution that will work for secrets that are stored in all types of XML files and not just config files with appSettings  or connectionStrings sections.

XML Transformation

Without this last requirement, the solution would be quite simple. Use web.config transformations with XDT (XML Document Transform) and XML variable substitution in the Azure App Service deploy task.

With XML transformations, we can create a transformation file for each environment and the deploy task will transform the config file with the environment specific settings. The problem is that the secrets are stored the transformation files – which are in the source code. To avoid that, Microsoft have the XML variable substitution step which will replace the matching variables from the DevOps pipeline. The drawback is that this only works for appSettings  and connectionStrings sections.

What if we want to change a connection string stored in another XML configuration file that does not use these sections?

For instance, let’s say we have a simple NLOG.config file that writes data to Azure Table Storage when in production. We want to inject this value from Azure KeyVault during deployment to production.

While there are many different ways to develop a solution, here we outline a solution that makes use of XML transformations and token replacement in Azure DevOps.

Overview of solution

We want to take the zip file that the build pipeline produced and during the release pipeline we want to replace the variables in the configuration files that are scoped to the current environment.

To ensure that the local development experience is not altered, we will leave the original config file alone and create a transformation file that will hold the tokens. We will modify the zip file, replace the tokens with the actual values in the transform file and then let the Deploy task do its thing.

The steps to implement a solution are:

1. Create an XML transformation configuration with tokens
2. Create tasks in DevOps to extract the files, replace the files and re-archive them
3. Configure DevOps deploy task

XML Transform and Tokens

Let’s start with the NLOG.config file. We want to change the connection string from “UseDevelopmentStorage=true” to the actual production connection string.

This is a simplified version of our config file:

First, we create an NLOG.release.config file, where “release” is the environment that we are targeting. The file is similar to our original XML file, but has reference to the XDT schema and Transform and Locator attributes to tell the transformation engine what to match.

The important thing to note here is that we add the values as tokens prefixed by #{ and suffixed by }#. The token terminators are customisable in the Replace Tokens task if you want to use something else.

The reason we are adding the tokens in the transform file is that we want to keep the original config file intact for local development. Next, check this in and let Azure DevOps create a build artifact.

Modify the Release Pipeline

If we enable XML transformations in the Azure App Service Deploy task, we will just get tokens in our config file. And the XML variable substitution will not work on our config file. So to get around this we need to replace the tokens before the deploy step. We can use a Replace Tokens task from the marketplace to help with this. This task does not work with zip files and so we also need to add tasks to open the file and close it when we’re done.

The sequence of steps is:

• Extract the build artifact to a working folder
• Replace the tokens in the transform config files
• Zip up the file again and overwrite the original
• Deploy the archive with XML transformation enabled.

In other words, we are using the Replace Tokens task to switch our tokens in the transform file with values from DevOps and then when we put the zip file back in place, we let Azure deploy the archive as normal.

Extract Files

In this task, specify a temp working folder to unzip the build artifact and keep the defaults for the other values.

Replace Tokens

Set the Root directory to the same location that the files were extracted to. And configure the target files. In our case, we are only targeting the NLOG.release.config file.

If you want to use different token prefixes and suffixes, you can edit them in the Advanced section of the task.

Archive Files

Next, overwrite the original zip file with the Archive files task. Set the root folder to the working folder path specified in previous tasks. Ensure you uncheck the “Prepend root folder name to archive paths” setting to prevent your app from being buried in a bunch of nested folders.

Deploy Task

The last step is to deploy the archive. Specify the zip file and ensure you check XML transformation. The Replace Tokens task has already done the variable substitution so we don’t need this set.

Configure Variables

For everything to work, the tokens that were specified in the XML transformation config file must be present in the Variables section in Azure DevOps. Configure the values to your actual values and scope. To get the values from Azure KeyVault into DevOps variables, well, that’s another story!

Summary

To keep secrets out of source code, have a clean development build experience, and have non-standard XML configurations applied during deployment with Azure DevOps, we used XML Document Transform files to hold the tokens to be substituted. In the release pipeline, we unzipped the archive into a working folder, ran the Replace Tokens task on the transform files and then zipped up the archive again with the untokenized transform files and let the Deploy task transform the config files during deployment.

Greenfinch Technology is a software development company based in Dublin, Ireland. Our focus is on using cutting-edge technologies to build world-class mobile and web applications for the BES (Business Enterprise Sector).