An Example of Data Integration Lifecycle Management with SSIS, Part 0

What is Data Integration Lifecycle Management (DILM)?

Here’s one way to think about DILM:

Data Integration Lifecycle Management (DILM) is applying software Application Lifecycle Management (ALM) best practices to Data Integration development and operations (DevOps), version control, release management, and configuration.

I can hear you thinking, “But Andy, why would we apply software best practices to a data integration platform like SQL Server Integration Services (SSIS)?” I’m glad you asked. The answer is: “Because SSIS development is software development.”

SQL Server Integration Services suffers from having the name of a popular relational database engine – SQL Server – baked into its name. Don’t let that throw you, SSIS development is software development. Yes, SSIS packages are tightly-coupled to data sources and destinations, and to the data the packages move.

I Object!

I hear from many people who wear several hats in small shops. Many of them object to my thoughts about DILM for SSIS. If you are one of those people who wear many hats and object to my thinking about DILM, I want to say two things to you:

1. You ROCK! Wearing many hats is hard. I’ve been there, done that, and have the blood-, sweat-, and tear-stained t-shirt.
2. Do whatever works for you.
3. BONUS 3rd thing: You may find some benefit from DILM practices if you give them a shot.

Who Needs to Practice DILM?

It depends on  the problem(s) you are trying to solve. Here are a few questions to help you determine your need for implementing DILM:

• Have you ever lost code?
• Has a server ever crashed and part of the solution involved re-developing code?
• Have you ever received a phone call from work while on vacation?

If you answered, “Yes,” to any of these questions, implementing some flavor of DILM may help.

This Series

I don’t have all the answers, so I cannot possibly provide all the answers to you. What I can do, though, is share some things I’ve learned implementing SSIS solutions for the past decade. I’ve led a team of 40 ETL developers building multiple enterprise-class projects simultaneously. I’ve parachuted into enterprises on fire as a lone wolf consultant and helped douse the flames. I’ve joined teams and formed teams to solve enterprise data integration problems. This breadth of experience has taught me priorities that are different from the priorities of some of my compatriot SSIS professionals.

You may read some of my thoughts and think, “That’s overkill.” To which I will respond, “Yeamaybe.” I understand. Really I do. As I wrote to the small-shop-people, do what works for you.

I want to tell the story of a data integration project, to follow it through its lifecycle as it starts, matures, and grows. Although the project is interesting, we will focus on lifecycle. I hope everyone finds some value in this series for that is my goal.

:{>