Maybe you noticed this and wondered, “Is Andy bailing on SSIS?” The answer to that question is:
Goodness no! If you only knew what I have on the drawing board and in the pipeline, you’d understand me abandoning SSIS is nothing to be concerned about. (I don’t blame you for not knowing – I’ve not shared more about these endeavors… yet!)
I’ve repeatedly shared that I enjoy learning. I’ve advised readers of this blog and those who follow me on social media to continually educate themselves. I’ve equally warned folks: If you don’t like lifelong learning, technology is not for you and you should go into another field.
I like learning!
I’ve invested some time and money in more formal training via Microsoft Virtual Academy and edX (although you can take all the courses I’ve taken for free… you have to pay only if you want a certificate).
Learning a New Language
Frank and I chat often. In a recent chat I shared my excitement at discovering patterns and frameworks in Apache Storm and Spark – frameworks and patterns that are remarkably similar to frameworks and patterns I’ve developed for SSIS. Why was I excited? It’s validating to me when smart people – people smart enough to build open source technology platforms like Spark and Storm – include the same functionality I’ve built for SSIS in their platforms and solutions. I’ve debated smart SSIS developers on occasion who believe frameworks in general, or specific frameworks, are not necessary. I’m not good at these debates because I can rarely remember the reason why something is a good or bad idea; I only remember that the idea is good or bad. I think the reason my brain works this way is because I’m an engineer and more interested in solving problems for customers than winning an argument… but I digress.
Frank shared an anecdote from his days of learning Deutsche. Paraphrasing:
I had a hard time remembering when to use ‘who’ and when to use ‘whom’ in English grammar until I learned German. Now I get it.
I have developed patterns and utilities (such as DILM Suite) to address architectural concepts I identified as unclear or less-clear or even missing from SSIS. While it’s validating (to me) to find this same functionality built into open source platforms, the biggest thrill remains learning new stuff. As Frank and I discuss in an upcoming Data Driven show, I bring a bunch of context when learning about open source data integration solutions – and Frank brings a bunch of context when learning about the software architecture that underlies Data Science.
Why is context so important? Context is what we mean most of the time when we use the word “experience.”
Like Frank’s German lessons helped him with English grammar, my open source data integration training is helping me articulate arguments for frameworks and patterns – in SSIS, even. I’ve gained additional depth in my career as a result of cross-training in other data integration platforms. I’m better at SSIS because I’ve learned Spark and Storm. Bonus: I’m getting lots of experience in the Azure Data Engineering platforms.
But – please trust me – I’m nowhere near leaving SSIS! In fact, I have a more lucid understanding of where SSIS excels when compared to other data integration platforms.
You’re still stuck with me writing about SSIS. I hope that’s ok.
I’m learning more about Apache Storm which is used for stream processing for near-real-time analytics in Azure HDInsight. I grabbed the image above from an AWS presentation on slideshare (link).
I got all giggly inside as I learned about Storm architecture. Why? Because I built this much of this same functionality as part the DILM Suite and we implement many of these same SSIS Design Patterns (SSIS Framework Editions, Controllers, Application Restartability, etc.) when delivering data integration awesomeness to our Enterprise Data & Analytics consulting customers.
It’s kinda cool to see similar architecture, patterns, and functionality in a completely separate toolset and environment. I find it… validating.
It just goes to show you, architecture is architecture, regardless of the platform or tools.
I’m slowly working my way through the Data Science Certification offered by Microsoft Virtual Academy along with edX. This course is not part of the official curriculum, but I’ve been wanting to learn more about Spark so I took it.
Description What is Intelligent Data Integration? SSIS packages developed using tried and true design patterns, built to participate in a DevOps enterprise practicing DILM, produced using Biml and executed using an SSIS Framework.
Attend a day of training focused on intelligent data integration delivered by an experienced SSIS consultant who has also led an enterprise team of several ETL developers during multiple projects that spanned 2.5 years. And delivered. Attendees will learn:
– a holistic approach to data integration design. – a methodology for enterprise data integration that spans development through operational support. – how automation changes everything. Including data integration with SSIS.
Topics include: 1. SSIS Design Patterns 2. Executing SSIS in the Enterprise 3. Custom SSIS Execution Frameworks 4. DevOps and SSIS 5. Biml, Biml Frameworks, and Tools
If you are learning Azure Command Line Interface (CLI) 2.0 – especially if you’re learning it by reading articles or watching videos based on v1.0 – you may see references to the “azure” command. But if you type “azure” in a command prompt, you may receive this error:
‘azure’ is not recognized as an internal or external command, operable program or batch file.
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.