We Borrowed Testing, Too

Your pipeline has tests.
Not as many as you’d like. But they exist, they run in CI, and they pass.

Last Thursday the pipeline execution instance reported green at 3:00 AM and red by 9:00 AM. Nobody had touched the code.

I can hear some of you thinking, “If the output changed, Andy, then something changed.”
Well, you’re not wrong.

Maybe the source data changed. Maybe a reference value changed. Maybe an upstream team quietly changed the meaning of a column. This is where data engineering testing gets tricksy: The code can remain unchanged while the thing being tested moves underneath it.

…and it’s not necessarily a gap in your testing discipline. It may be a sign that the vocabulary does not quite fit the work.

What software engineering named

Software engineering named:

• Unit
• Integration
• Regression
• Fixture
• Mock
• Stub
• Coverage
• Red, green, refactor.

Every one of those words is a boundary. Each word tells you what’s inside and what’s outside the test, what you control and what you are pretending to control. Software engineering spent years refining testing, and the arguments were productive because of a shared vocabulary.
Everyone knew what the other person meant.

Data engineering borrowed the runners:

• pytest
• assertion libraries
• CI gates
• green checkmarks

All of it good, most of it free, none of it wrong to take. The tools (mostly) transferred. The vocabulary did not transfer as cleanly.

I have been thinking about this for a while.

In 2007, I co-authored Professional Software Testing with Visual Studio 2005 Team System and wrote the chapter titled “Testing the Database.” Since I hold data integration / engineering is software development, I maintain this applies when testing data engineering.

Almost twenty years later, I have learned that the principles sometimes transfer more cleanly than the vocabulary.

Where the words break

Regression – in software engineering, regression testing often asks whether a change to the code, configuration, or dependencies changed existing behavior. In data engineering, the output can move even when the pipeline code does not. The input changed. A reference value changed. An upstream system changed. Same symptom, but a different root cause. In my experience, changes in input, reference data, and upstream systems account for a surprising amount of what actually goes wrong.

Flaky – in software, a flaky test is a broken test. Quarantine it, fix it, or delete it. In data engineering, a test that passed Tuesday and failed Wednesday may be the most honest thing in the building. The test did not change. The world did. Calling that test flaky blames the witness.

Coverage – software engineering can measure line coverage, branch coverage, function coverage, and more. None of those measures is perfect, but at least the terms describe something countable. What does coverage mean for a pipeline? Rows? Deviations from previous data profiling results? The branches of a business domain nobody ever fully enumerated? Because data coverage is rarely defined, it is rarely measured. We fall back to reporting coverage of the code, even when the greater risk lives in the data.

The gate – a software test often acts as a gate during a build or deployment. A data test may need to act as a gate every night, forever, against inputs owned by people who do not report to you. That changes the conversation. A failed build can prevent a release. A failed data test at 3:00 AM may prevent the CFO from receiving a report at 9:00 AM. Someone has to decide whether to stop the load, continue with a warning, use yesterday’s data, or wake up the owner of the source system. That is more than a test result. It is an operational decision. We often file this under operations or observability. That can leave ownership unclear. Sometimes it ends up belonging to the wrong person. Worse, sometimes, it belongs to no one.

You already know this part

• You ran it in dev and eyeballed the row count.
• You compared it to last month. Last month was also unverified.
• The check that matters is a WHERE clause someone added in 2019. With no comment. Still load-bearing.
• The pipeline execution displays green on the Enterprise ETL dashboard and you still query the table before you tell anyone it finished.

That last one is the tell.
Green is not trust.
If it were, you would not look.

Words for what you are already doing

One way to improve this conversation is to name the checks data engineers are already performing:

1. Contract Assertion – what must be true of the input before the run is allowed to proceed. Shape, nullability, domain, arrival. This is a test of the world rather than of your code. Software tests can sometimes mock the world away. Data engineering cannot because the world is the subject.
2. Invariant Check – what must be true of every run regardless of what arrived. Totals tie. Keys do not duplicate. No orphan references. You do not need to know today’s input to write one.
3. Reference Run – frozen input, known output. The closest thing to determinism genuinely available to you, and the one place software’s testing vocabulary transfers cleanly. It is a smaller place than anyone wants it to be.
4. Reconciliation – the output agrees with an independent computation of the same fact. This is not a test of your code. It is a test of your claim.
5. Drift Detection – the input moved and the code did not. This is often what data engineers mean by “regression”: The code remained still while the world moved.

These are not necessarily new terms, and I do not pretend they are the only terms. They are one way to distinguish five different questions that are too often compressed into the word “test.”

Name the check in the pipeline. Name it in the alert. Name it in the runbook.

At 3:00 AM, “Contract Assertion failed: CustomerStatus domain changed” is more useful than “Data quality test failed.”

Future you will thank you.

The verdict

Software engineering has spent decades developing language for asking whether a change broke something. Data engineering has borrowed much of that language, but we have not always named the surprises that begin outside our code. A passing software test tells us something about a particular version of the code under particular conditions.

A passing test is never the whole truth.
In data engineering, the timestamp is part of the result.