Shannon Lowder (@shannonlowder | shannonlowder.com) did a fantastic job presenting the true story of a data science project. He presented the problem he was trying to solve, the places he looked for data, issues and obstacles he encountered, and solutions he developed. Enjoy! Links Storm Events files NOAA Hail Data files Microsoft Azure Machine Learning: …
Continue reading The Recording for “A Data Science Story” by Shannon Lowder is Available
Category:Data Engineering
Free Webinar: Troubleshooting SSIS – 13 Mar at 12:00 PM ET
Join Microsoft Data Platform MVP, author, blogger, and Enterprise Data & Analytics Chief Data Engineer Andy Leonard for a free webinar titled Troubleshooting SSIS 13 Mar 2019 at 12:00 PM ET. Register today! Here’s a free preview!
Mean Time to Identify Failure
While managing a team of 40 ETL developers, I wanted to track lots of metrics. Some of the things I wanted to track were technical, like SSIS package execution times. Some metrics were people-centric. Andy’s First Rule of Statistics states: You can use statistics for anything about people, except people. Andy – circa 2005 It was …
Continue reading Mean Time to Identify Failure
Biml, Book Reviews, and Metadata-Driven Frameworks
I occasionally (rarely) read reviews at Amazon of books I’ve written. If I learn of a complaint regarding the book I often try to help. If a reader experiences difficulty with demos, I often offer to help, and sometimes I meet with readers to work through some difficulty related to the book (as I offered …
Continue reading Biml, Book Reviews, and Metadata-Driven Frameworks
Presenting Moving Data with Azure Data Factory at SQL Saturday Charlotte!
I am honored to present Moving Data with Azure Data Factory at SQL Saturday 806 in Charlotte, NC 20 Oct 2018. This is the first time I am delivering this session. It still has that new presentation smell! Abstract Azure Data Factory – ADF – is a cloud data engineering solution. ADF version 2 sports …
Continue reading Presenting Moving Data with Azure Data Factory at SQL Saturday Charlotte!
On Data Engineering in 2018
A few years ago I had a conversation with Scott Currie, the CEO of Varigence and inventor of Biml. Scott is one of the smartest people I know and I know a lot of very smart people. Whenever I have the opportunity to communicate with the very smart people I know, I often ask them – …
Continue reading On Data Engineering in 2018
Announcing the Fundamentals of Azure Data Factory Course!
I am excited to announce a brand new course (it still has that new course smell) from Brent Ozar Unlimited and honored to deliver it! This one-day, live, online course is titled Fundamentals of Azure Data Factory and it’s designed to introduce you to Azure Data Factory (ADF). There will be demos. Live demos. Lots …
Continue reading Announcing the Fundamentals of Azure Data Factory Course!
Anatomy of a Catalog Reference Script in SSIS Catalog Compare
A Catalog Reference script is automatically generated from several methods in SSIS Catalog Compare’s Catalog object named CatalogBase. Header Documentation The script begins with a header documentation similar to that shown here: When executed, the Project Reference script header documentation portion displays in the Messages output window of SQL Server management Studio (SSMS): The statements …
Continue reading Anatomy of a Catalog Reference Script in SSIS Catalog Compare
Honored to Present Faster SSIS at SQL Saturday Boston 22 Sep 2018!
I am honored to present Faster SSIS at SQL Saturday Boston 22 Sep 2018! Check out this schedule – there are a bunch of smart people presenting – plus me! Abstract Ever wonder why SSIS runs so slow? Watch SSIS author Andy Leonard as he runs test loads using sample and real-world data and shows …
Continue reading Honored to Present Faster SSIS at SQL Saturday Boston 22 Sep 2018!
Lift and Shift an SSIS Catalog…
… to a different instance on-premises. … to Azure Data Factory SSIS Integration Runtime. This screenshot is from the SSIS Catalog Compare documentation. You can script an entire SSIS Catalog using SSIS Catalog Compare! This awesome functionality can save you hours (days, weeks, even!) when starting a new project or updating an existing project. This …
Continue reading Lift and Shift an SSIS Catalog…