I recently read a social media post from a data engineer that I found both encouraging and heartbreaking.

This data engineer shared that he used an AI engine to successfully convert an SSIS package, roughly 1MB in size, into a different data engineering platform.

Why encouraging?

First, it is genuinely encouraging that AI engines are now capable of converting a data engineering artifact of that size into another platform. In my opinion, that is a real milestone and one worth celebrating.

Why heartbreaking?

I describe large SSIS packages as monoliths. In my SSIS training, I call monoliths an anti-pattern. In fact, one of the slides is titled “Monoliths Must Die!”

The slide includes a screenshot from one of the most memorable Star Trek: The Original Series episodes (The City on the Edge of Forever). In that episode, Kirk meets Edith Keeler, an activist whose well-intentioned actions in the past accidentally alter Earth’s history enough to erase the future timeline in which the Enterprise exists. The episode echoes themes explored in The Man in the High Castle by Philip K. Dick. Both works required real courage to publish so close to the end of World War II. But I digress…

Monoliths must die because maintaining them is usually a miserable exercise. Troubleshooting a failed monolithic SSIS package execution often means stepping through a long chain of tasks that are functioning correctly. That work adds no diagnostic value and consumes an enormous amount of time and focus. In my opinion, it is a colossal waste of data engineering attention.

At this point, I can hear many of you asking:

“Is there an alternative, Andy?”

That’s an excellent question. I’m so glad you asked!

The remedy for the monolith anti-pattern is a well-established SDLC principle called Separation of Concerns.

In practical terms, separation of concerns means writing modular code. Think about a C# or JavaScript application. A monolith in those ecosystems would look like a single main() method containing 14,000 lines of code, possibly wrapped inside one endless if-else statement. (I wish I were exaggerating.)

Would such code compile? Often, yes. Would it run? Also yes. But should it be written that way? Absolutely not.

Separation of concerns argues for small, single-purpose units of code. Each unit does one thing and does it well. This matters because small, focused code is easier to test, easier to reason about, and far easier to troubleshoot. Applications built from collections of single-purpose components avoid many problems. Even if they solved no other issues, that benefit alone justifies abandoning monoliths.

Downsides?

Are there downsides to separation of concerns?
Sometimes, yes.

Returning to SSIS, how do you apply separation of concerns in practice? Breaking monolithic packages into smaller packages is an excellent start. But doing so introduces a new challenge. How do you execute tens, dozens, or even hundreds of SSIS packages in a controlled and reliable way?

There is a solution, and this solution itself (like many engineering solutions) poses additional challenges, but this post is long enough.

I will have more to say about one effective approach to orchestrating large numbers of SSIS packages in early 2026. If you want to follow along, the best way to keep up with my thinking is to subscribe to my newsletter.