One Complex Example of ADF Pipeline Return Value

Posted on Updated on by Andy LeonardCategories:ADF, ADF Execute Pipeline Activity, ADF Pipeline Return Value, REST API

This post is part of a short series on Azure Data Factory pipeline return value functionality, which is in preview at the time of this writing.

The Problem I am Trying to Solve

I want to develop an Azure Data Factory (ADF) design pattern for calling focused, unit-of-work, function-y ADF pipelines that perform focused tasks. Some of these “worker pipelines” will need to return values to the calling pipeline.

Credit to Mr. Paul Andrew for sharing how to obtain a data factory’s SubscriptionId in his post titled Get Data Factory to Check Itself for a Running Pipeline Via the Azure Management API. Like the characters in the Pirates of the Caribbean movies, I’ve been conditioned to reply, “Mr. Paul Andrew” whenever I hear someone say, “Paul Andrew.”  If you read this blog and do not read Mr. Paul Andrew’s blog, you should.

In This Post…

In this post, I demonstrate one way to create a child pipeline that returns the SubscriptionId for a data factory. I then call the child pipeline from a parent package.

To build this demonstration, please follow the instructions that follow.

Create a New Branch

As in the previous post titled One Simple Example of ADF Pipeline Return Value, I recommend we create a new branch. Click the name of the current branch (my current branch is the branch I used in the previous post. It is named “PipelineReturn”) and then click “+ New branch”:

When the “Create a New Branch” window displays, click on the “Base on” dropdown and select “PipelineReturn branch”:

  1. Set the “Branch name” property to “GetSubscription”
  2. Click the “Create” button:

Create a New Folder

ADF folders help isolate work and simplify searching for related ADF pipelines. Creating a new folder is a good idea, in my humble opinion.

  1. Click the ellipsis beside the Pipelines category
  2. Click “New folder”:

I named my new folder “Get Subscription”.

  1. Type the name of your folder into the “Create a folder” textbox
  2. Click the “Create” button to create the folder:

Create a Child (or Worker) Pipeline

Once the folder is created:

  1. Click on the ellipsis beside the folder name
  2. Click “New pipeline”:

When the new pipeline displays, rename the new pipeline “PL_Util_GetSubscriptionId”:

Configure the PL_Util_GetSubscriptionId Pipeline

Add a Web activity to call the ADF REST API:

  1. Expand the “General” category on the Activities blade
  2. Drag a “Web” activity from the Activities blade, “General” category, to the pipeline canvas
  3. Select the “Web” activity
  4. On the “Web” activity’s General tab, change the “Name” property to “Get SubscriptionId”:

  1. Click the “Web” activity’s Settings tab
  2. Set the “URL” property to “https://management.azure.com/subscriptions?api-version=2020-01-01”
  3. Set the “Method” property to “GET”
  4. Set the “Authentication” property to “System Assigned Managed Identity” (also known as “SAMI”)
  5. Set the “Resource” property to “https://management.azure.com”:

Let’s Test the Configuration

Click the Debug button on the upper pipeline menu:

Optimally, you want to see a successful execution:

To view the results, click the “Get SubscriptionId” Output:

You can view the value returned from the call to the ADF REST API in the output JSON:

 

Yay, success.

The most common error I encounter when making calls to the ADF REST API is a permissions error that I blogged about in a post titled An Introduction to Azure Data Factory REST API.

Continue PL_Util_GetSubscriptionId Pipeline Configuration

Add a pipeline variable to the “PL_Util_GetSubscriptionId” pipeline.

  1. Click in the white space of the “PL_Util_GetSubscriptionId” pipeline
  2. Click the Variables tab:

  1. Click the “New” button
  2. Supply a “Name” for the pipeline variable (I use “returnSubscriptionId”):

Add a “Set variable” activity:

  1. Expand the “General” category on the Activities blade
  2. Drag a “Set variable” activity from the Activities blade, “General” category, to the pipeline surface
  3. Connect a Success output from the “Get SubscriptionId” Web activity to the new “Set variable” activity
  4. Select the “Set variable” activity
  5. On the “Set variable” activity’s General tab, change the “Name” property to “Set returnSubscriptionId”:

Configure the “Set returnSubscriptionId” “Set variable” activity’s Settings.

  1. Click the “Set returnSubscriptionId” “Set variable” activity’s Settings tab
  2. Make sure the “Variable type” property is set to “Pipeline variable”
  3. Select the “returnSubscriptionId” pipeline variable from the “Name” property dropdown
  4. Click inside the “Value” property textbox to display the “Value” property “Add dynamic content” link
  5. Click the “Value” property “Add dynamic content” link:

Configure the dynamic content expression for the “Set returnSubscriptionId” “Set variable” activity’s “Value” property.”

  1. On the “Activity outputs” tab, click the tile labeled:
    “Get SubscriptionId
    Get SubscriptionId activity output”

Note the dynamic content textbox reflects your selection by displaying “@activity(‘Get SubscriptionId’).output.

Edit the expression text.

  1. Edit the expression in the dynamic content textbox to “@replace(activity(‘Get SubscriptionId’).output.value[0].id,’/subscriptions/’,”)”
  2. Click the OK button:

Your “Set returnSubscriptionId” “Set variable” activity’s Setting tab should now appear similar to what is shown here:

Let’s Test Again

We’ve reached another good spot for a test. Back in the day, we called these “unit tests.”

Click the Debug button to start a test execution:

If all goes as planned, the test execution succeeds. Click the “Output” for the “Set SubscriptionId” “Set variable” activity:

The Output displays the SubscriptionId value for the data factory, which is now stored in the pipeline variable named returnSubscriptionId:

The blocked out part contains the SubscriptionId for my Azure subscription.

Continue PL_Util_GetSubscriptionId Pipeline Configuration Again

Add a new “Set variable” activity named “Set SubscriptionIdOutput”:

Begin configuring the “Set SubscriptionIdOutput” “Set variable” activity’s Settings tab.

  1. Click the “Set SubscriptionIdOutput” “Set variable” activity’s Settings tab
  2. Set the “Set SubscriptionIdOutput” “Set variable” activity’s “Variable type” property to “Pipeline return value (preview)”
  3. Click the “+ New” button:

When the “+ New” button is clicked, pipeline return value configuration properties display.

  1. Configure the “Name” property for the pipeline return value (I entered “SubscriptionId”) – remember this value
  2. Select a return value type from the “Type” property dropdown (I selected “Expression”)
  3. Click inside the “Value” property textbox to display the “Add dynamic content” link
  4. Click the “Add dynamic content” link

When the “Pipeline expression builder” blade displays:

  1. Click the Variables tab
  2. Click the “returnSubscriptionId” variable to set the dynamic content to “@variables(‘returnSubscriptionId’)”
  3. Click the OK button:

Let’s Test This Pipeline Again

Click the pipeline Debug button to begin another test execution:

If all goes as planned, this latest test execution should succeed.

Click the output for the execution of the “Set SubscriptionIdOutput” “Set variable” activity:

As configured, the “Set SubscriptionIdOutput” “Set variable” activity output sends a pipeline return value key / value named “SubscriptionId” (key) and the value set to the SubscriptionId (value) for the data factory.

Next, let’s build a parent pipeline to call this child pipeline.

Create a Parent Pipeline

In the Factory Resources blade, click the ellipsis beside the “Get Subscription” folder (1), and then click “New folder” (2):

Name the new pipeline “PL_Parent_GetSubscriptionId”:

Configure the PL_Parent_GetSubscriptionId Pipeline

  1. Expand the “General” category in Activities
  2. Drag an “Execute Pipeline” activity onto the ADF canvas
  3. Click the “Execute Pipeline” activity
  4. On the “Execute Pipeline” activity’s General tab, change the “Name” property to “Execute PL_Util_GetSubscriptionId”:

Continue configuring the Execute pipeline activity:

  1. Click the Settings tab
  2. Set the “Invoked pipeline” property to “PL_Util_GetSubscriptionId”
  3. Make sure the “Wait on completion” property checkbox is checked:

Add a variable to the parent pipeline:

  1. Click the white space on the pipeline canvas
  2. Click the Variables tab
  3. Click the “+ New” button
  4. Configure the new variable:
    1. Name: SubscriptionId
    2. Type: String
    3. Default value: <blank>

Add and begin configuring a “Set variable” activity:

  1. Expand the General; category on the Activities blade
  2. Drag a “Set variable” activity onto the ADF canvas
  3. Select the “Set variable” activity
  4. Set the Name property of the “Set variable” activity to “Set SubscriptionId”:

  1. Click the Settings tab of the “Set SubscriptionId” “Set variable” activity
  2. Make sure the “Variable type” property is set to “Pipeline variable”
  3. Select the “SubscriptionId” variable from the “Name” property dropdown
  4. Click inside the “Value” property textbox to surface the “Add dynamic content” link
  5. Click the “Add dynamic content” link:

When the “Pipeline expression builder” blade displays, click
“Execute PL_Util_GetSubscriptionId
Execute PL_Util_GetSubscriptionId pipeline return value (preview)”
on the “Activity outputs” tab to add “@activity(‘Execute PL_Util_GetSubscriptionId’).output.pipelineReturnValue” to the dynamic content textbox:

  1. Edit the dynamic content text, appending “.SubscriptionId” to the existing expression so that the expression now reads “@activity(‘Execute PL_Util_GetSubscriptionId’).output.pipelineReturnValue.SubscriptionId”
  2. Click the OK button:

Let’s Test PL_Parent_GetSubscriptionId

If all goes as planned, the test execution of PL_Parent_GetSubscriptionId should succeed.

Click the output of the “Set SubscriptionId” “Set variable” activity:

If the functionality works as designed, the Output should return the SubscriptionId value from PL_Util_GetSubscriptionId to PL_Parent_GetSubscriptionId, and the Output of the “Set SubscriptionId” “Set variable” activity should confirm:

Key Points / Gotchas

Here are some important steps to remember when developing this complex ADF solution that makes use of pipeline return value(s):

  • In the child pipeline:
    • I used two “Set variable” activities because – at the time of this writing – I could not get a single “Set variable” activity to read the output of the Web activity and set that value as the pipeline return value. Maybe the Microsoft Azure Data Factory Team will make that functionality available once pipeline return values goes GA (Generally Available – not preview). This two-step approach will likely continue to work, though.
    • If you want to return the results of an expression, set the Return Value Type property to “Expression”
  • In the parent pipeline:
    • The parent pipeline must access the pipeline return value using the Name of the value returned- in this case, SubscriptionId – from the child pipeline.
    • I recommend the value returned from the child pipeline be stored in a parent pipeline variable.

Conclusion

Azure changes daily. As time passes between today (24 Jun 2023) and when you read this post, I’m pretty sure some of the screenshots will be outdated. I pray the concepts remain similar (or similar-enough) for you to grasp the concepts herein and solve The Problem You Are Trying To Solve®.

As always, I welcome your thoughts, feedback, comments, and suggestions.

Andy Leonard

andyleonard.blog

Christian, husband, dad, grandpa, Data Philosopher, Data Engineer, Azure Data Factory, SSIS guy, and farmer. I was cloud before cloud was cool. :{>

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.