Introduction

The aim of this article is to provide an overview of how to build an effective CI/CD strategy for Azure Datafactory (ADF) that will enable configuration changes to be promoted through multiple environments from development environment through to production using Azure DevOps (ADO) Release tasks. For this walkthrough we are going to assume just 3 ADF deployment environments – build, test and production.

Assumptions

• An empty repository in ADO with a single branch called – build
• Previous experience creating/using Release tasks in ADO
• Previous experience creating ADF instances within the Azure Portal, script or IaC
• Previous experience creating resources inside of ADF instances e.g. pipelines

flowchart BT
    B["Test"] -- deploy --> C["Production"] 
    A["Build"] -- create release/deploy --> B
    G["*adf-build-instance*"] -- publish changes --> A

The order we will be deploying/promoting updates to our ADF instances

Create your ADF Instances

In the Azure Portal create your ADF instances. I will refer to them as adf-build, adf-test and adf-production.

In this demo adf-build will be the development environment instance where all infrastructure changes will be configured and tested before being promoted to the ADF test and production instances (adf-test followed by adf-production) using ADO Release tasks.

To generate the ARM templates required to create the Release tasks in ADO we need to make at least one configuration change in the adf-build instance. A quick way to do this is to create a new pipeline, adding an activity and immediately deactivating said activity it will be enough to trigger the create on the ARMTemplateForFactory.json and ARMTemplateParametersForFactory.json files.

Connect the ADF build instance to your Azure devops Repo build branch

Now we need to create a relationship between the infrastructure in you adf-build instance in ADF and the build branch in ADO

• In the Azure portal, open your build ADF instance (adf-build) and click the Launch studio button on the overview page.
• Click Manage > Git Configuration > and click the Configure button
  • Repository Type: Azure Devops Git
  • Select the Cloud Radio button
  • Microsoft Entra ID: Select your Microsoft Entra Tenant then click Continue
  • Select the Select Repository Radio button
  • Azure Devops organization name: Select your Azure Devops Organisation
  • Project name: Select your Project
  • Repository Name: Select your Repository
  • Collaboration branch: Select the build branch (or whatever you chose to name it)
  • Publish branch: Select the build branch (or whatever you chose to name it)
  • Import resource into this branch: Select the build branch (or whatever you chose to name it)
  • Leave the other defaults and click Apply
  • Click Publish All to push our build ADF instance configuration i.e the infrastructure ARM templates to the build branch in ADO.

Configure the Release tasks in ADO for each environment

Now we need to configure the Release tasks so we can promote our changes from the build branch through to the ADF test and production instances. We will begin by creating the test release task followed by production release task

• in ADO open your Azure Datafactory repository
• Under Pipelines select Releases
• Click New+ > New release pipeline or if its your first one click New pipeline
• Select Template: Empty Job
• If the Stage window opens close it – we will come back to it.
• Now we need to add the Artifacts i.e the ARM templates we will be deploying to each environment
  • Click Add an Artifact
  • Source type: Azure Repository
  • Project: Select your Azure Datafactory Project
  • Source (repository): Select the Azure Datafactory repository
  • Default branch: build
  • Leave all the other defaults and click Apply
• Now we need to add the test release task
  • Click the Stage 1 box and in the pop up context window change the Stage name to test and close the context window
  • Click the Lightening bolt to update the Pre-deployment conditions. Change to Manual Only
  • In the newly named test stage click the 1 job, 0 task link
  • In the Agent Job click the + sign to add a task agent
  • In Add Tasks search for ARM template deployment and click Add
    • Display Name: Give the task a suitable name
    • Deployment Scope: Resource Group
    • Azure Resource Manager Connection: Select a preexisting connection or create a new one
    • Subscription: Select your subscription
    • Action: Create or Update Resource Group
    • Resource Group: Select the Resource group that contains your test ADF instance (adf-test or whatever name you chose for your test adf instance)
    • Location: Select the location your resource group is in
    • Template location: Linked Artifact
    • Template: Select the ARMTemplateForFactory.json file
    • Template Parameters: Select the ARMTemplateParametersForFactory.json file
    • Override Template parameters: update the factoryName to adf-test (or whatever name you chose for your test adf instance)
    • Leave all other defaults and click Save
    • On the Pipeline tab, in the Stages box click Add > + New stage and repeat the steps for the production ADF instance. Be sure to update the resource group i different and override the factoryName parameter with adf-production (or whatever name you chose for your production adf instance)

Test the deployment/promotion process end to end

Now we can test the deployment and promotion process end to end

• In the Azure portal make a change to the ADF adf-build instance e.g. add a new pipeline or change the existing pipeline
• Publish your changes to the build branch in ADO
• In ADO open the Azure Datafactory Release pipeline
• Click Create new Release and click Create
• Select the test release stage task and deploy
• Once deployed, confirm you are happy with the changes in the ADF test instance – if not you can rollback to a known working release until you resolve the issue
• Once you are happy with your deployment to the ADF test instance repeat the process of deploying to the ADF production instance.

Summary

We have been through the process of describing how to build a CI/CD pipeline enabling promotion from dev through to production environments.

In ADO we created a repository with a single branch called build

In the Azure portal we created 3 ADF instances for build (develop), test and production

In the Azure portal in ADF Studio, after generating the ARM templates in the ADF build instance we proceeded to link the ADF build instance it to the build branch in ADO

In ADO we create a Release Pipeline linked it to the build branch artifacts and then created a test release stage task and a production release stage task – making sure to override any parameters

We then proceeded to test the process end to end by making changes to the ADF build instance in the Azure portal, pushing the changes to the build branch in ADO, creating a release and then promoting the changes to the ADF test instance and then the ADF production instance

Thanks for reading and happy building!