A friend pinged me recently to ask about rolling back Azure Data Factory (ADF) pipeline versions. My response was a question: Are you using source control with ADF? That did not help the current situation.
I thought of the way I often build ADF pipelines and shared my methodology, which is relatively simple (it has to be simple for me to understand it!):
Building Incrementally
I start by building a pipeline and getting something – anything – to work. Next, I click the ellipsis beside my pipeline in the Factory Resources blade and then click “Clone”:
I add some more functionality and get it working, and then I rinse and repeat – cloning this new version of the pipeline.
If I hit a snag, I decide whether to clone the current (snag-y) pipeline and beat work on it some more from where I’m stuck, or simply back up a step and clone a fresh copy of the pipeline when it last worked.
When I’m done, I delete pipelines I do not need.
:{>
Need Help Getting Started with SSIS and ADF?
Enterprise Data & Analytics specializes in helping enterprises modernize their data engineering by lifting and shifting SSIS from on-premises to the cloud. Our experienced engineers grok enterprises of all sizes. We’ve done the hard work for large and challenging data engineering enterprises. We’ve earned our blood-, sweat-, and tear-stained t-shirts. Reach out. We can help.
Cool Andy. I didn’t know about the clone functionality but then again I have just started with ADF. This is a great tip.