Data engineering in a software engineering world
[originally posted at Engineer of Data Substack]
Your orchestration runs. It Just Works every night. You built something real and that is not nothing. The ache isn’t that it’s broken – it’s that it lives in three people’s heads and a folder of conventions nobody wrote down.
- You wrote the retry logic from scratch in three different projects, a little different each time.
- A new hire can’t safely touch the orchestration for weeks, because none of it has names they’d find in a book.
- Two pipelines solve the same sequencing problem two incompatible ways, and both are “correct.”
- “How do you handle restartability?” gets answered with a story instead of a term.
There are few advantages to living through 51 years of software development. One advantage? I’ve seen the long game. Software engineering stood in this exact same spot once. Slinging code felt so good (it still feels good). After trial, after error, it still makes me giggly inside to see the code not only succeed, but do what I wanted. We, Programmers, (re-) solved the same structural problems until a small group named the recurring solutions.
Christopher Alexander’s pattern language first.
The Gang of Four.
It wasn’t merely a technical breakthrough. It was shared vocabulary. Naming converted reinvention into reuse. Design patterns made the work reviewable, testable, teachable, comparable.
Data engineering adopted software’s tooling: source control, workflow management, packaging. We bolted on CI/CD. We never adopted the vocabulary.
We borrowed the tools, not the patterns.
The gaps show up where concerns converge:
- retries
- idempotency
- restartability
- sequencing
- code promotion
- observability
Software addressed these concerns in a software-centric way.
That way had a trick: push the state to the edges, keep the moving parts stateless.
A stateless component is trivial to retry. Nothing to corrupt, nothing to resume.
Data can’t take that deal. The state is the whole point.
So when we borrowed their tools – the excellent tools of software engineering – we inherited an impedance mismatch that dogged our efforts in a manner difficult to quantify because it was barely visible. Data engineering looked and felt like software development because data engineering *is* software development.
Statelessness was never on the table for data engineers.
Not a chasm we failed to cross; a door that was never open.
Which means borrowing their patterns was never enough.
Stateful problems need their own names. We never wrote them.
Data forces us to deal with heterogeneous sources, schema drift, late-arriving data, fragile environments – with a toolset that never got its vocabulary layer.
Software engineering got thirty years and a library of canonical books.
Data integration got tools and a shrug.
Software engineering solved the naming problem years ago, and data engineering mostly didn’t.
That is the gap.
Not you. Not your team.
Your orchestration isn’t unmaintainable because your team fell short.
It’s unmaintainable because the work was never given a name.
Patterns are how a craft stops starting over.
If you’ve rebuilt the same retry logic for the third time, you already understood this.
You just never had the word for it.
You’re not behind.
The field is.
Comments