On last week’s exciting and thrilling episode of Data Engineering Fridays, I was stymied for a bit. You can watch me struggle with my code here – the struggle lasted for a while. I eventually figured it out. As an aside: I absolutely love streaming live-coding! Believe it or not, failing is one of my favorite parts. I can hear some of you thinking, …

“Why do you like to fail – live, online, in front of God and everybody – Andy?”

That’s a fantastic question. I’m so glad you asked! I never want to convey to developers – or those who aspire to become developers – the (absurdly false) impression that software development consists of writing code that Just Works over and over again, as you skip from mountain top to mountain top singing show tunes.

I’d much rather share reality with you and the reality is: Most of the time, you’ll write code, test-execute it, and it will fail.

The Problem I Am Trying to Solve

I’m building an Azure Data Factory execution orchestration framework. In an ADF parent pipeline, I want to select a case from a Switch activity based on an execution result value returned from a child pipeline execution:

The problem? No matter the value returned from the Execute Pipeline activity, the Switch activity executed the Default case. My test child package failed, and as I troubleshot it (I refuse to type the word “troubleshooted” ever again in my career. I got in some target practice where I shot one of my pistols the day after I troubleshot this pipeline), I realized this was the case for all child pipeline executions in my framework.

In the video, you can hear me asking myself questions. I compared this way of “thinking out loud” to one of my favorite scenes in the movie, “The Hunt for Red October.” “Why is the default case executed?” “Why is the default case executed when the execution result is ‘Failed?'” “Why is the default case executed regardless of the execution result?” “How do you get a crew to want to get off a nuclear sub…?”

More Questions

When “Failed” was returned from the child pipeline named “executor,” the string “Failed” was sent to a variable named “executor_status_return.” The Switch activity selected the appropriate case based on the value of executor_status_return variable.

“Why isn’t the Switch activity selecting the ‘Failed’ case?”
Because the executor_status_return variable value did not equal the value “Failed.”
The executor_status_return variable value equaled something else.

Data Types, Again

The executor pipeline returns an Expression:

The expression is configured to hold the output.status property of the “Check pipeline run status” activity. When the @activity(‘Check pipeline run status’).output.status expression evaluates to “Failed,” the object contains a string; the string “Failed.”

An expression is an object, not a string.
An object can be a string, but that doesn’t make the data type of the object a string.

To read the string value of the object in such a way that the string value may be evaluated, say… by a Switch activity, for example, the object can be cast or converted to a string. That’s what I did using the string conversion function in ADF expressions:

That worked.

But…

The more I thought about it, the more I wanted to Just Read the pipeline-returned value without having to mess with converting it. I tested applying to conversion to the expression in the child pipeline:

That also worked.

To me, this design feels a little cleaner. I plan to start Friday’s Data Engineering Fridays show with this change. Also, the show this Friday will be restreamed to TikTok. I’m just trying to keep up with the Fabric Fridays folks!