Crafting the beginning of a vocabulary
[originally posted at Engineer of Data Substack]
Last week, I wrote:
Stateful problems need their own names. We never wrote them.
You’ve written idempotent loads, restart logic, dependency chains, promotion configs.
By hand.
Correctly enough to ship.
You already build every one of these.
You’re not missing the knowledge. You’re missing the words. Missing words have a cost:
- You explained your restart approach in a standup and watched three people nod at three different mental models.
- You reviewed a pipeline, knew something was missing, and had no word for what.
- You onboarded someone by walking them through “how we do it here,” because none of it was in a book they could read.
- You solved the same problem on two teams and called it two different things, so nobody noticed it was the same problem.
The Gang of Four applied Christopher Alexander’s definition of “patterns” to define software design patterns. I led a team of authors to write two editions of SSIS Design Patterns and, in so doing, applied the form to data engineering while leaving out Alexander’s metaphysics.
A pattern is not a blueprint.
Alexander’s whole claim was that a pattern lets you solve the same problem a thousand times and never once the same way. So the implementation was never part of it.
Naming the pattern is the work.
Hence, the words.
A starting vocabulary
Idempotent Write – a step may run more than once (retry, restart, backfill); the result must match running it once.
Nameless cost: every team reinvents “safe to re-run” and gets it subtly wrong.
Checkpointed Restart – a long run fails partway; you resume from a known position, not from zero and not by hand.
Nameless cost: restart means rerun-everything or surgery.
Declared Dependency – order stated as structure, not implied by schedule time or folder layout or memory.
Nameless cost: “it works because A usually finishes before B.”
Promotable Configuration – the same artifact moves dev to test to prod unchanged, because environment lives outside the logic.
Nameless cost: “we edit the package when we deploy.”
Parameterized Execution – one artifact serves many inputs instead of forty near-identical copies.
Nameless cost: copy-paste sprawl nobody dares refactor.
Observable Run – every execution leaves a durable, queryable record of what ran, with what, and how it ended.
Nameless cost: every post-mortem is archaeology.
This is not a list.
This is merely the beginning of a vocabulary, not a closed set.
Languages grow.
And like words in any language, the patterns listed presuppose each other. Checkpointed Restart presupposes Idempotent Write. Both presuppose Observable Run – you can’t resume or trust what you can’t see. Declared Dependency presupposes Observable Run – you sequence on what actually happened.
I left Alexander’s metaphysics out of the SSIS Design Patterns books. I’m not sure I would again. He didn’t name patterns for reuse. He named them to make places more alive, easier to live in. A pattern language for orchestration isn’t only about tidiness. It’s about making the work humane to inhabit, for the next engineer and for you at 2:00 AM.
A pattern you can’t name is a solution you have to rediscover. Name it once, and the rediscovery stops.
Software engineering didn’t pull ahead because it got smarter.
It pulled ahead because it got a vocabulary.
Your move.
Comments