Azure Resource Manager, or ARM, “allows you to provision your applications using a declarative template.” So says the Azure Quickstart Templates page. ARM templates are JSON and allow administrators to import and export Azure resources using varying management patterns. I really like ARM templates for implementing infrastructure as code in Azure. In this post I show a very simple example of how to use ARM templates to export and then import a basic ADF (Azure Data Factory) pipeline.
Export an ADF Pipeline
To demonstrate, I created the Simplest ADF Pipeline Ever containing a single Wait activity configured to wait for 10 seconds. I named the pipeline “Wait Pipeline” and the Wait activity, “Wait 10 seconds.”
When the pipeline is open in the ADFv2 editor, click the dropdown labeled, “ARM Template” and then click “Export ARM Template”:
A zip file is downloaded. Decompress the file:
Delete the Pipeline
After exporting the ARM template, I deleted the original pipeline:
Import the Template
To import the template, click the ARM Template dropdown and click “Import ARM Template”:
On the Custom Deployment page, click the “Build your own template in the editor” link to open the template editor:
When the template editor opens, click the “Load file” button to load a template file:
Select an ARM template file to upload:
Note: The file – even the number of files – will change as the resource increases in complexity.
Save the template:
Configure the “Custom deployment” blade by selecting your Subscription, Resource group, and checking the
“I agree to the terms and conditions stated above” checkbox. Don’t let the “Purchase” button throw you, click Purchase to deploy the ARM template:
Once the ARM template is deployed, the resource(s) described therein – a very simple Azure Data Factory pipeline, in this case – is deployed and available:
Summer o’ ADF 2019 Edition
this post is part of a series titled the Summer o’ ADF, 2019 Edition! Click that link to see more posts and learn more about Azure Data Factory.
Need Help Getting Started or Updating SSIS or Azure Data Factory in Your Enterprise?
Enterprise Data & Analytics specializes in training and helping enterprises modernize their data engineering by lifting and shifting SSIS from on-premises to the Azure-SSIS integration runtime in Azure Data Factory. Our experienced engineers grok enterprises of all sizes. We’ve done the hard work for large and challenging data engineering enterprises. We’ve earned our blood-, sweat-, and tear-stained t-shirts. Reach out. We can help.
Hi,
Is it possible to import just one pipeline using ARM templates into an existing ADF that contains multiple pipelines?
Thanks in advance.
Hey Matthew just adding something more to your question, as of now i dont think you can deploy individual resources through default arm template.
I have simillar issues as you mentioned :
Consider we have dev test and prod environment of ADF v2. The dev environment is git integrated. The developers have debuged their changes and merged with collaboration branch after pull request. The changes are published and deployed to test environment first. Here many features are getting tested but few are ready for prod and few are not, how do we move the ones which are ready since the arm template takes the entire factory?
How do we hande this type of deployment scenario in Microsoft recommended CICD model of git/vsts integrated adf v2 through arm template
One thing can be that the debug is itself your test environment for developers, however since we cant apply trigger testing in debug mode hence we do need a test environment. Also we can test in dev enviroment data factory mode but then we should stop continuos build and release trigger as we need to first test and then deploy manually.
Hi
I have 5 pipelines in my azure data factory which are live for now.
I had exported ARM template, so now I am importing that ARM_Template, is there need to delete existing pipelines or should I change the name of that pipelines in exported json file to check file is importing in correct way or not?
Why not go a step further and use powershell to import the template using:
Connect-AzureRmAccount
Select-AzureRmSubscription -SubscriptionId “here is the subscription identifier”
Set-AzureRmDataFactoryV2Pipeline -ResourceGroupName “your resource group” -DataFactoryName “NameOfYourDataFactory” -Name “PipelineName” -File “C:\GitRoot\PipelineName.json”
Can I import multiple pipelines within a ADF resource in a single go? I have 100 ADF pipelines which I want to import and export.
Hello Arabinda,
Did you get any workaround for this
Susant
Hi Everyone,
Microsoft recently announced – and have since updated – an open source project named Bicep. Bicep is designed to help manage ARM templates (get it? ARM, Bicep?). I blogged about it here: https://andyleonard.blog/2020/09/microsoft-azure-announces-project-bicep-an-arm-dsl/
Hope this helps,
Andy
I followed the steps to import a customized arm template but I don’t see the purchase options , there is a review and create option I click on that but I don’t see the pipeline getting created in my adf could you please help with this? also I have add both the parameters and template json file in custom setting options