What a month of naming posts was really all about

[originally posted at Engineer of Data Substack]

I have spent a chunk of a career in the part of data that has no name.

Not the dashboards. Not the models everyone lines up to demo. The layer underneath – the part that moves the data, in the right order, at the right time, and puts it back together when something tears at 2:00 AM.

The plumbing.
The glue.
The work that is invisible when it goes right and catastrophic when it goes wrong – and either way, nobody has a word for it.

I have built that layer for over twenty years.

I have written the same restart logic more times than I can count, a little different each time, on teams that never knew they were re-solving a problem the team down the hall had already solved. I have stood in rooms and explained how our orchestration worked and watched four smart people nod at four different pictures.

I spent this June trying to give that work its words.

It came out in three pieces, and I want to tell you what they were really about, because I am not sure I knew when I started. In “We borrowed the tools, not the patterns,” I wrote about how we took software engineering’s tooling and skipped its vocabulary. In “A pattern language for orchestration,” I named some of the patterns we had been building blind. In “You can’t govern what you can’t name”, the names turned out to be the thing governance had been waiting on the whole time.

Three weeks.
One argument.
I couldn’t see the shape of it until it was done.

Here is the shape

They were not three topics. They were three rungs:

1. Vocabulary;
2. Then governance; and
3. Then trust

Each one stands on the one beneath it. You cannot govern what you cannot name, so governance stands on vocabulary. The business does not extend trust to what it cannot govern, so trust stands on governance. Pull out the bottom rung and the whole ladder is just pipelines that run – until the day someone asks you to prove it.

This is the distance I have helped teams cross for twenty years.

The distance between a script or package or pipeline that runs and an estate you can govern. It turns out that distance has a length, and the length is exactly one vocabulary.

I watched software engineering cross it a long time ago. Not because I, or it, was smarter. Because at some point it stopped treating its recurring work as a thousand private solutions and started treating it as a discipline – a discipline of the named, the structured, the shareable, the teachable, the answerable.

Data orchestration never quite had that.

We have tools.
We have heroics.
We have a great deal of undocumented competence walking around in three people’s heads.

What we never quite had is the discipline.

I think it is time we named it

So name it.

Not with a new word. This whole series has been one lesson: the work was there all along, and only the names were missing. The discipline is no different. We do not need to coin it. We need to claim it.

Data engineering.
Used the way software engineering uses its name – not as a job title, but as a discipline.

Named, structured, governable, answerable. The practice of treating the movement and assembly of data as a craft that can account for itself, instead of a thousand private solutions held together by competence and memory.

Not better SSIS.
Not better Databricks or Data Factory.
Not a tidy habit.
It is what software engineering became when it grew up, finally reaching the field that moves the data.

That is the name. It has been on my own masthead for years – Engineer of Data – and I am only now making it mean what it should.

I spent a month giving the work its words; words it took me a career to find.

The building starts where the naming ends.