Data Engineering Execution Orchestration Frameworks in Fabric Data Factory Series Landing Page

The Data Engineering Execution Orchestration Frameworks in Fabric Data Factory series includes the following posts: Tinkering with Fabric Data Factory Framework Functionality – 13 Feb 2024 One Way to Invoke Fabric Data Factory Pipelines Using the Fabric REST API – 30 May 2024 Use Azure Data Factory in Fabric Data Factory – 2 Oct 2024 …
Continue reading Data Engineering Execution Orchestration Frameworks in Fabric Data Factory Series Landing Page

Fabric Data Factory Design Pattern – Dynamically Start a Collection of Child Pipelines

In the first post in this series – a post titled Fabric Data Factory Design Pattern – Basic Parent-Child – I demonstrated one way to build a basic parent-child design pattern in Fabric Data Factory by calling one pipeline (child) from another pipeline (parent). In the second post, titled Fabric Data Factory Design Pattern – …
Continue reading Fabric Data Factory Design Pattern – Dynamically Start a Collection of Child Pipelines

Fabric Data Factory Design Pattern – Dynamically Start a Child Pipeline

In an earlier post titled Fabric Data Factory Design Pattern – Basic Parent-Child, I demonstrated one way to build a basic parent-child design pattern in Fabric Data Factory by calling one pipeline (child) from another pipeline (parent). In a later earlier post titled Fabric Data Factory Design Pattern – Parent-Child with Parameters, I modified the parent …
Continue reading Fabric Data Factory Design Pattern – Dynamically Start a Child Pipeline

Fabric Data Factory Design Pattern – Parent-Child with Parameters

In an earlier post, I demonstrated one way to build a basic parent-child design pattern in Fabric Data Factory by calling one pipeline (child) from another (parent). In this post, I modify the parent and child pipelines to demonstrate calling a child pipeline that contains a parameter. In this post, we will: Clone and edit …
Continue reading Fabric Data Factory Design Pattern – Parent-Child with Parameters

Fabric Data Factory Design Pattern – Basic Parent-Child

The Microsoft Fabric Team released a new Invoke Pipeline Activity in September 2024. This post describes one way to implement a parent-child design pattern using the new activity. In this post, we will: Create a new workspace Build a child pipeline Build a parent pipeline Test Create a New Workspace Connect to Fabric Data Factory. …
Continue reading Fabric Data Factory Design Pattern – Basic Parent-Child

One Way to Invoke Fabric Data Factory Pipelines Using the Fabric REST API

I’m an automation weenie, that’s a fact. It’s why I’m drawn to frameworks – especially data engineering execution orchestration frameworks. Speaking of frameworks, if you haven’t checked out Martin Schoombee’s series titled Building a Framework for Orchestration in Azure Data Factory: A Series, you are missing out. First, Martin is scary smart. Second, he shares practical …
Continue reading One Way to Invoke Fabric Data Factory Pipelines Using the Fabric REST API

Scaffold Design Pattern in Fabric Data Factory Pipeline

When assigned a project, it’s tempting – and dangerous – to Just Start Coding. If you suffer from the urge to develop first and design later, you are not alone (there’s at least one other developer like you and he’s typing this post). Do yourself a favor and… Design First As the definition of “design” …
Continue reading Scaffold Design Pattern in Fabric Data Factory Pipeline

Scheduling and Monitoring a Fabric Data Factory Pipeline Execution

The more I work with Microsoft Fabric Data Factory, the more I like it. In this post, I share one way to schedule a Microsoft Fabric Data Factory pipeline, a different way to test execute a pipeline, and then one way to monitor the execution. Schedule a Pipeline Connect to Microsoft Fabric Data Factory and …
Continue reading Scheduling and Monitoring a Fabric Data Factory Pipeline Execution