Your pipeline has tests. Not as many as you’d like. But they exist, they run in CI, and they pass. Last Thursday the pipeline execution instance reported green at 3:00 AM and red by 9:00 AM. Nobody had touched the code. I can hear some of you thinking, “If the output changed, Andy, then something …
Continue reading We Borrowed Testing, Too
Tag:Data Engineering
The Data Engineering Patterns Series
I recently wrote a series of articles about Data Engineering Patterns. The four articles focus on the differences between software engineering and data engineering and the need to apply software engineering principles to data engineering. But I also cover why that’s not as simple as it sounds. the series, in order, is as follows: We Borrowed …
Continue reading The Data Engineering Patterns Series
The Words for the Work
What a month of naming posts was really all about [originally posted at Engineer of Data Substack] I have spent a chunk of a career in the part of data that has no name. Not the dashboards. Not the models everyone lines up to demo. The layer underneath – the part that moves the data, …
Continue reading The Words for the Work
You Can’t Govern What You Can’t Name
Governance is downstream of a name [originally posted at Engineer of Data Substack] Last week, I wrote: Observable Run – every execution leaves a durable, queryable record of what ran, with what, and how it ended. I called it a pattern. It is also a door. Behind it is the question every data team eventually gets …
Continue reading You Can’t Govern What You Can’t Name
A Pattern Language for Orchestration
Crafting the beginning of a vocabulary [originally posted at Engineer of Data Substack] Last week, I wrote: Stateful problems need their own names. We never wrote them. You’ve written idempotent loads, restart logic, dependency chains, promotion configs. By hand. Correctly enough to ship. You already build every one of these. You’re not missing the knowledge. You’re …
Continue reading A Pattern Language for Orchestration
We Borrowed the Tools, Not the Patterns
Data engineering in a software engineering world [originally posted at Engineer of Data Substack] Your orchestration runs. It Just Works every night. You built something real and that is not nothing. The ache isn’t that it’s broken – it’s that it lives in three people’s heads and a folder of conventions nobody wrote down. You …
Continue reading We Borrowed the Tools, Not the Patterns
Separating Concerns in SSIS
I recently read a social media post from a data engineer that I found both encouraging and heartbreaking. This data engineer shared that he used an AI engine to successfully convert an SSIS package, roughly 1MB in size, into a different data engineering platform. Why encouraging? First, it is genuinely encouraging that AI engines are …
Continue reading Separating Concerns in SSIS
My Name is Andy, and I Vibe Code
Originally published at Engineer Of Data as a newsletter. Subscribe to my newsletter here. 50 Years Last month, I Celebrated 50 Years in Software. That link will take you to a recording of a show in which my youngest daughter, Emma, interviewed me to discuss this milestone. I think Emma is an excellent interviewer, but I biased. Embracing …
Continue reading My Name is Andy, and I Vibe Code
Presenting Data Engineering Fundamentals with Fabric Data Factory at PASS Data Community Summit 2025
Data Engineering Fundamentals with Fabric Data Factory – a Full-Day Pre-Conference Training Session 17 Nov 2025 I am honored to present Data Engineering Fundamentals with Fabric Data Factory at the PASS Data Community Summit 2025 17 Nov 2025 in Seattle! Abstract This one-day training introduces data engineering concepts for enterprise data warehousing using Microsoft Fabric …
Continue reading Presenting Data Engineering Fundamentals with Fabric Data Factory at PASS Data Community Summit 2025
The John Barker Mentoring Project
John Barker retired from the US Air Force in or around 1975. He returned home to rural Blackstone Virginia, in part to help care for his aging parents. He had experience in electronics as a technician and built a Southwest Technical Products M6800 computer kit. Although I’m not sure this is an accurate image of …
Continue reading The John Barker Mentoring Project