One Way to Incrementally Develop ADF Pipelines

A friend pinged me recently to ask about rolling back Azure Data Factory (ADF) pipeline versions. My response was a question: Are you using source control with ADF? That did not help the current situation.

I thought of the way I often build ADF pipelines and shared my methodology, which is relatively simple (it has to be simple for me to understand it!):

Building Incrementally

I start by building a pipeline and getting something – anything – to work. Next, I click the ellipsis beside my pipeline in the Factory Resources blade and then click “Clone”:

I add some more functionality and get it working, and then I rinse and repeat – cloning this new version of the pipeline.

If I hit a snag, I decide whether to clone the current (snag-y) pipeline and beat work on it some more from where I’m stuck, or simply back up a step and clone a fresh copy of the pipeline when it last worked.

When I’m done, I delete pipelines I do not need.

:{>

Need Help Getting Started with SSIS and ADF?

Enterprise Data & Analytics specializes in helping enterprises modernize their data engineering by lifting and shifting SSIS from on-premises to the cloud. Our experienced engineers grok enterprises of all sizes. We’ve done the hard work for large and challenging data engineering enterprises. We’ve earned our blood-, sweat-, and tear-stained t-shirts. Reach out. We can help.