A storm is coming…

I’ve been spending some time learning other data integration and analytics open-source-y stuff. Perhaps you’ve noticed recent blog posts like:

• Apache Storm Stream Processing in Azure HDInsight
• Achievement Unlocked: Certified in Implementing Predictive Analytics with Spark in Azure HDInsight
• Achievement Unlocked: Completed Processing Big Data in Hadoop with Azure HDInsight

If you read MSDN Magazine, you may have noticed a familiar name at the end of Frank La Vigne’s (blog | @Tableteer) new column, Artificially Intelligent – especially the October, November, December (2017), and January 2018 articles – I’ve been helping Frank with some technical editing for his column.

Maybe you noticed this and wondered, “Is Andy bailing on SSIS?” The answer to that question is:

No.

Goodness no! If you only knew what I have on the drawing board and in the pipeline, you’d understand me abandoning SSIS is nothing to be concerned about. (I don’t blame you for not knowing – I’ve not shared more about these endeavors… yet!)

I’ve repeatedly shared that I enjoy learning. I’ve advised readers of this blog and those who follow me on social media to continually educate themselves. I’ve equally warned folks: If you don’t like lifelong learning, technology is not for you and you should go into another field.

I like learning!

I’ve invested some time and money in more formal training via Microsoft Virtual Academy and edX (although you can take all the courses I’ve taken for free… you have to pay only if you want a certificate).

Learning a New Language

Frank and I chat often. In a recent chat I shared my excitement at discovering patterns and frameworks in Apache Storm and Spark – frameworks and patterns that are remarkably similar to frameworks and patterns I’ve developed for SSIS. Why was I excited? It’s validating to me when smart people – people smart enough to build open source technology platforms like Spark and Storm – include the same functionality I’ve built for SSIS in their platforms and solutions. I’ve debated smart SSIS developers on occasion who believe frameworks in general, or specific frameworks, are not necessary. I’m not good at these debates because I can rarely remember the reason why something is a good or bad idea; I only remember that the idea is good or bad. I think the reason my brain works this way is because I’m an engineer and more interested in solving problems for customers than winning an argument… but I digress.

Frank shared an anecdote from his days of learning Deutsche. Paraphrasing:

I had a hard time remembering when to use ‘who’ and when to use ‘whom’ in English grammar until I learned German. Now I get it.

I have developed patterns and utilities (such as DILM Suite) to address architectural concepts I identified as unclear or less-clear or even missing from SSIS. While it’s validating (to me) to find this same functionality built into open source platforms, the biggest thrill remains learning new stuff. As Frank and I discuss in an upcoming Data Driven show, I bring a bunch of context when learning about open source data integration solutions – and Frank brings a bunch of context when learning about the software architecture that underlies Data Science.

Context Matters

Why is context so important? Context is what we mean most of the time when we use the word “experience.”

Like Frank’s German lessons helped him with English grammar, my open source data integration training is helping me articulate arguments for frameworks and patterns – in SSIS, even. I’ve gained additional depth in my career as a result of cross-training in other data integration platforms. I’m better at SSIS because I’ve learned Spark and Storm. Bonus: I’m getting lots of experience in the Azure Data Engineering platforms.

But – please trust me – I’m nowhere near leaving SSIS! In fact, I have a more lucid understanding of where SSIS excels when compared to other data integration platforms.

You’re still stuck with me writing about SSIS. I hope that’s ok.

;{>