This post is part of a short series on Azure Data Factory pipeline return value functionality, which is in preview at the time of this writing.

The Problem I am Trying to Solve

I want to develop an Azure Data Factory (ADF) design pattern for calling focused, unit-of-work, function-y ADF pipelines that perform focused tasks. Some of these “worker pipelines” will need to return values to the calling pipeline.

In this example, I started by reading Mark Kromer‘s (excellent) article titled You can now customize the return value from your pipeline! I then crafted the simple example shown in this post to make sure I understood the principles involved before using pipeline return value (preview) functionality in more robust ADF patterns.

Follow the steps I outline below to build a simple example for an ADF pipeline that returns a value!

Create a New Branch

There are two types of developers:

1. Those who use source control
2. Those who will use source control

It’s a best practice to always start a new branch when making changes to code (and ADF is code).

Click the name of the collaboration branch (mine is named the default, main) and then click “+ New branch”:

ADF integrates well with git. “Managing Code in ADF” covers this topic. It is a course I offer as part of my Azure QuickStart and Premium Level offerings.

When “Create a new branch” displays, enter the name of the new branch and click the “Create” button:

 

Create a New Folder

Creating a new folder is more of an Andy practice than a best practice. It helps me keep organized, and Lord knows I need all the help I can get keeping organized.

To begin, click the ellipsis beside the Pipelines category, and then click the “New folder” option:

Enter a name for the new folder and then click the “Create” button:

I named my new folder “Pipeline Return”.

Create a New Pipeline

Click the ellipsis beside your new folder and then click “New pipeline”:

When the new pipeline displays (it still has that new pipeline smell!), rename the new pipeline “PL_Child”:

Configure the PL_Child Pipeline

Add a “Set variable” activity:

1. Expand the “General” category on the Activities blade
2. Drag a “Set variable” activity from the Activities blade, “General” category, to the pipeline surface
3. Select the “Set variable” activity
4. On the “Set variable” activity’s General tab, change the “Name” property to “Set ReturnValue”:

1. Click the “Set variable” activity’s Settings tab
2. Set the “Variable type” property to “Pipeline return value (preview)”
3. Click the “+ New” button:

When you set the “Variable type” property to “Pipeline return value”, you must define the return value by setting the Name, Type, and Value properties. This can become complex. I use the following settings in this simple example:

• Name: “ReturnValue”
• Type: “String”
• Value: “This is a Returned Value.”

We will use the name of the Pipeline return value – “ReturnValue” – when configuring the parent pipeline, which is the next step.

Create a New Pipeline

Click the ellipsis beside your new folder and then click “New pipeline”:

When the new pipeline displays, rename the new pipeline “PL_Parent”:

Configure the PL_Parent Pipeline

1. Click the Variable tab for the PL_Parent pipeline
2. Click the “+ New” button
3. Configure the Variable Name property: “ReturnedValue”
4. Set the Variable Type property to “String”
5. Configure the Variable “Default value” property: “Default Value”

Add an “Execute Pipeline” activity:

1. Expand the “General” category on the Activities blade and drag an “Execute Pipeline” activity from the Activities blade, “General” category, to the pipeline surface
2. Select the “Execute Pipeline” activity
3. On the “Execute Pipeline” activity’s General tab, change the “Name” property to “Execute PL_Child”:

1. Click the “Execute Pipeline” activity’s Settings tab
2. Select “PL_Child” to set the “Invoked pipeline” property
3. Make sure the “Wait on completion” checkbox is checked:

1. Expand the “General” category on the Activities blade and drag a “Set variable” activity from the Activities blade, “General” category, to the pipeline surface
2. On the “Set variable” activity’s General tab, change the “Name” property to “Set ReturnedValue”:

1. Click the Settings tab
2. Set the “Variable type” property to “Pipeline variable” (default)
3. Select the ReturnedValue variable for the “Name” property
4. Click inside the “Value” property textbox to display the “Add dynamic content” link, and then click the “Add dynamic content” link:

When you click the “Add dynamic content” link, the “Pipeline expression builder” blade displays.

On the “Activity outputs” tab, click the

“Execute PL_Child
Execute PL_Child pipeline return value (preview)”

option:

When you click the “Execute PL_Child, Execute PL_Child pipeline return value (preview)” option, the dynamic content textbox will display “@activity(‘Execute PL_Child’).output.pipelineReturnValue”:

Edit the value in the dynamic content textbox, which reads “@activity(‘Execute PL_Child’).output.pipelineReturnValue”, appending the current expression – with the name of the value returned from the PL_Child pipeline (ReturnValue) – to read “@activity(‘Execute PL_Child’).output.pipelineReturnValue.ReturnValue”:

Click the OK button to update the “Set ReturnedValue” Set variable activity’s Value property.

The “Set ReturnedValue” Set variable activity is now configured to store the value of the child pipeline (PL_Child) returned variable – named “ReturnValue” – in the parent pipeline’s variable named “ReturnedValue”.

Before proceeding, let’s review:

1. Recall the PL_Parent pipeline’s variable named “ReturnedValue” has its “Default value” property set to “Default Value”.
2. Click the “Debug” button to test-execute the PL_Parent pipeline:

If all goes as expected, the test execution succeeds:

1. On the PL_Parent pipeline’s Output tab, hover over the “Set ReturnedValue” Set variable output execution details:

Click the “Set ReturnedValue” Set variable output execution details to display the JSON Output:

Note the value of the “ReturnedValue” variable in the PL_Parent pipeline is not longer the default value (“Default Value”), but instead is the value returned from PL_Child pipeline, which is “This is a Returned Value.”

Key Points / Gotchas

Here are some important steps to remember when developing a simple ADF solution that makes use of pipeline return value(s):

1. The child pipeline must contain a “Set variable” activity configured to use “Pipeline return value (preview)” as the Variable type, and the pipeline return value configuration must configure Name, Type, and Value properties
2. The parent pipeline must access the pipeline return value using the Name of the value returned from the child pipeline.
3. I recommend the value returned from the child pipeline be stored in a parent pipeline variable.

Conclusion

Believe it or not, this a simple example of how to return a value from a child pipeline. In an upcoming post, I demonstrate one use case for pipeline return values. You will see more complexity in that demonstration.

Stay tuned.