I recently wrote a series of articles about Data Engineering Patterns. The four articles focus on the differences between software engineering and data engineering and the need to apply software engineering principles to data engineering. But I also cover why that’s not as simple as it sounds. the series, in order, is as follows: We Borrowed …
Continue reading The Data Engineering Patterns Series
Category:Patterns Series
The Words for the Work
What a month of naming posts was really all about [originally posted at Engineer of Data Substack] I have spent a chunk of a career in the part of data that has no name. Not the dashboards. Not the models everyone lines up to demo. The layer underneath – the part that moves the data, …
Continue reading The Words for the Work
You Can’t Govern What You Can’t Name
Governance is downstream of a name [originally posted at Engineer of Data Substack] Last week, I wrote: Observable Run – every execution leaves a durable, queryable record of what ran, with what, and how it ended. I called it a pattern. It is also a door. Behind it is the question every data team eventually gets …
Continue reading You Can’t Govern What You Can’t Name
A Pattern Language for Orchestration
Crafting the beginning of a vocabulary [originally posted at Engineer of Data Substack] Last week, I wrote: Stateful problems need their own names. We never wrote them. You’ve written idempotent loads, restart logic, dependency chains, promotion configs. By hand. Correctly enough to ship. You already build every one of these. You’re not missing the knowledge. You’re …
Continue reading A Pattern Language for Orchestration
We Borrowed the Tools, Not the Patterns
Data engineering in a software engineering world [originally posted at Engineer of Data Substack] Your orchestration runs. It Just Works every night. You built something real and that is not nothing. The ache isn’t that it’s broken – it’s that it lives in three people’s heads and a folder of conventions nobody wrote down. You …
Continue reading We Borrowed the Tools, Not the Patterns