One Way to Incrementally Develop ADF Pipelines

A friend pinged me recently to ask about rolling back Azure Data Factory (ADF) pipeline versions. My response was a question: Are you using source control with ADF? That did not help the current situation.

I thought of the way I often build ADF pipelines and shared my methodology, which is relatively simple (it has to be simple for me to understand it!):

Building Incrementally

I start by building a pipeline and getting something – anything – to work. Next, I click the ellipsis beside my pipeline in the Factory Resources blade and then click “Clone”:

I add some more functionality and get it working, and then I rinse and repeat – cloning this new version of the pipeline.

If I hit a snag, I decide whether to clone the current (snag-y) pipeline and beat work on it some more from where I’m stuck, or simply back up a step and clone a fresh copy of the pipeline when it last worked.

When I’m done, I delete pipelines I do not need.


Need Help Getting Started with SSIS and ADF?

Enterprise Data & Analytics specializes in helping enterprises modernize their data engineering by lifting and shifting SSIS from on-premises to the cloud. Our experienced engineers grok enterprises of all sizes. We’ve done the hard work for large and challenging data engineering enterprises. We’ve earned our blood-, sweat-, and tear-stained t-shirts. Reach out. We can help.

Enterprise Data & Analytics

Andy Leonard

Christian, husband, dad, grandpa, Data Philosopher, Data Engineer, Azure Data Factory, SSIS, and Biml guy. I was cloud before cloud was cool. :{>

2 thoughts on “One Way to Incrementally Develop ADF Pipelines

  1. Cool Andy. I didn’t know about the clone functionality but then again I have just started with ADF. This is a great tip.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.