I got a great question from a student in the December delivery of Expert SSIS. A student asked, “Why wouldn’t you use Environments in Visual Studio (Dev, Test, and Prod), and deploy accordingly the mapped project parameters and package parameters?” I’ve looked into using SQL Server Data Tools (SSDT, or Visual Studio) configurations in the …
Continue reading SSIS and Visual Studio Configurations
Category:SSIS
On Data Frameworks…
You may not realize this, but Apache Spark is a framework. Spark is cluster-computing engine that manages parallel executions extremely well. Spark enables other technologies including Java, Scala, Python, R, and graph processing. Spark stitches together previously-disparate functionality into a cohesive, syntactically-similar set of commands. Spark’s architecture is library-driven and includes the following libraries: Spark SQL …
Continue reading On Data Frameworks…
I Love Delivering Training
In December 2017, I delivered Expert SSIS training in cooperation with Brent Ozar Unlimited. I’m a Star Trek fan and I include a couple references to the series in the course. One reference is to the episode titled The City on the Edge of Forever. In my humble opinion, the plot for this episode is better …
Continue reading I Love Delivering Training
Presenting A Day of Intelligent Data Integration in NYC 18 May!
I’m honored to present A Day of Intelligent Data Integration – a SQL Saturday NYC precon – 18 May! Description What is Intelligent Data Integration? SSIS packages developed using tried and true design patterns, built to participate in a DevOps enterprise practicing DILM, produced using Biml and executed using an SSIS Framework. Attend a day …
Continue reading Presenting A Day of Intelligent Data Integration in NYC 18 May!
SSIS Design Patterns: Sniffer (File Watcher)
Nutmeg, the Wonder Dog, Sniffing Have you seen the SSIS File Watcher demos where a developer puts an SSIS package in an infinite loop while it waits for a file to show up in a directory? It’s a popular pattern, especially when files are FTP’d (or SFTP’d) to enterprises for processing. I’ve seen way too …
Continue reading SSIS Design Patterns: Sniffer (File Watcher)
Data Integration Life Cycle Management with SSIS is Available!
My latest book – Data Integration Life Cycle Management with SSIS – is now available! In this focused (185-page) book, I walk through ways to manage SSIS in the enterprise using what ships in the box. I point out what works well and what’s missing. I share workarounds for the stuff that’s missing, and then I …
Continue reading Data Integration Life Cycle Management with SSIS is Available!
OLE DB is *NOT* Deprecated
I cannot count how many times I’ve been asked about the deprecation of Microsoft OLE DB for SQL Server. My best guess? Hundreds of times over the years. My reply to questions has been: “I do not believe Microsoft will deprecate the OLE DB Driver for SQL Server.” In data integration circles, the announcement in …
Continue reading OLE DB is *NOT* Deprecated
Presenting A Day of Intelligent Data Integration at SQL Saturday Richmond!
I am honored to present my day-long precon titled SSIS Design Patterns and Biml: A Day of Intelligent Data Integration at SQL Saturday 705 – Richmond 23 Mar 2018! Description What is Intelligent Data Integration? SSIS packages developed using tried and true design patterns, built to participate in a DevOps enterprise practicing DILM, produced using Biml and executed using …
Continue reading Presenting A Day of Intelligent Data Integration at SQL Saturday Richmond!
SSIS Performance Pattern – Loading a Bajillion Records
Do you need to use SSIS to load some data from a large source? This post covers some thoughts on a couple/three patterns you may apply to help. I can hear you thinking, “Which pattern should I use, Andy?” That’s an excellent question! The answer is, “It depends.” You should test each pattern and see …
Continue reading SSIS Performance Pattern – Loading a Bajillion Records
Parsing SSIS Catalog Messages for Lookup Performance, v2
A couple years ago I wrote Administering SSIS: Parsing SSIS Catalog Messages for Lookups. I’ve updated the script to return all Lookup Transformation performance messages captured by the SSIS Catalog with Basic logging enabled (the SSIS Catalog’s default Logging Level). Some smart data integration people recommend setting the SSIS Catalog Logging Level to Verbose and …
Continue reading Parsing SSIS Catalog Messages for Lookup Performance, v2