Biml, Book Reviews, and Metadata-Driven Frameworks

I occasionally (rarely) read reviews at Amazon of books I’ve written. If I learn of a complaint regarding the book I often try to help. If a reader experiences difficulty with demos, I often offer to help, and sometimes I meet with readers to work through some difficulty related to the book (as I offered here). About half the time, there’s a problem with the way the book explains an example or the sample code; the other half the time the reader does not understand what is written.

I own both cases. As a writer it’s my job to provide good examples that are as easy to understand as possible. Sometimes I fail.

In very rare instances, I feel the review misrepresents the contents of a book –  enough to justify clarification. This review afforded one such opportunity. None of what I share below is new to regular readers of my blog. I chose to respond in a direct manner because I know and respect the author of the review, and believe he will receive my feedback in the manner in which it was intended – a manner (helpful feedback) very similar to the manner in which I believe his review was intended (also helpful feedback).

Enjoy.

My Reply to This Review of The Biml Book:

Thank you for sharing your thoughts in this review.

I maintain successful technology solutions are a combination of three factors:

  1. The problem we are trying to solve;
  2. The technology used to solve the problem; and
  3. The developer attempting to use a technology to solve the problem.

I believe any technologist, given enough time and what my mother refers to as “gumption” (a bias for action combined with a will – and the stubbornness, er… tenacity – to succeed), can solve any problem with any technology.

I find your statement, “It doesn’t really teach BIML but rather teaches a specific optional framework for those who want to do everything with BIML rather than increase productivity for repetitive patterns” inaccurate. Did you read the entire book? I ask that question because I most often encounter readers who state the precise opposite of the opinion expressed in this review (some of your fellow reviewers express the opposite opinion here, even). I disagree with your statement, but I understand your opinion in light of other opinions expressed to me by readers during my past decade+ of writing several books. I share one example in this blog post. The short version: we often get more – or different – things from reading than we realize.

There is a section in The Biml Book – made up of 3 of the 20 chapters and appendices – that proposes a basic metadata-driven Biml framework which is the basis of a metadata-driven Biml framework in production in several large enterprises. I wrote those 3 chapters and the basic metadata-driven Biml framework in question. Varigence has also produced a metadata-driven Biml framework called BimlFlex – based upon similar principles – which has been deployed at myriad enterprises. The enterprise architects at these organizations have been able to improve code quality and increase productivity, all while decreasing the amount of time required to bring data-related solutions to market. They do not share your opinion, although they share at least some of the problems (you mentioned ETL) you are trying to solve.

Am I saying Biml – or the framework about which I wrote or even BimlFlex – is the end-all-be-all for every SSIS development effort? Goodness no! In fact, you will find disclaimers included in my writings on Biml and SSIS frameworks. I know because I wrote the disclaimers.

Misalignment on any of the three factors for successful technology solutions – the problem, the technology, and/or the developer – can lead to impedance mismatches in application and implementation. For example, Biml is not a good solution for some classes of ETL or data integration (points 1 and 2). And learning Biml takes about 40 hours of tenacious commitment (which goes to point 3). This is all coupled with a simple fact: data integration is hard. SSIS does a good job as a generic, provider-driven solution – but most find SSIS challenging to learn and non-intuitive (I did when I first started using it). Does that mean SSIS is not worth the effort to learn? Goodness, no! It does mean complaints about simplifying the learning process are to be expected and somewhat rhetorical.

Architects disagree. We have varying amounts of experience. We have different kinds of experience. Some architects have worked only as lone-wolf consultants or as members of single-person teams. Others have worked only as leaders of small teams of ETL developers. Rarer still are enterprise architects who are cross-disciplined and experienced as both lone-wolf consultants and managers of large teams in independent consulting firms and large enterprises. Less-experienced architects sometimes believe they have solved “all the things” when they have merely solved “one of the things.” Does this make them bad people? Goodness, no. It makes them less-experienced, though, and helps identify them as such. Did the “one thing” need solving? Goodness, yes. But enterprise architecture is part science and part art. Understanding the value of a solution one does not prefer – or the value of a solution that does not apply to the problem one is trying to solve – lands squarely in the art portion of the gig.

Regarding the science portion of the gig: engineers are qualified to qualitatively pronounce any solution is “over-engineered.” Non-engineers are not qualified to make any such determination, in my opinion.

By way of example: I recently returned from the PASS Summit. Although I did not attend the session, I know an SSIS architect delivered a session in which he made dismissing statements regarding any and all metadata-driven frameworks related to ETL with SSIS. If I didn’t attend his session, how do I know about the content of the presentation? A number of people in attendance approached me after the session to share their opinion that the architect, though he shared some useful information, does not appreciate the problems faced by most enterprise architects – especially enterprise architects who lead teams of ETL developers.

My advice to all enterprise architects and would-be enterprise architects is: Don’t be that guy.

Instead, be the enterprise architect who continues to examine a solution until underlying constraints and drivers and reasons the solution was designed the way it was designed are understood. Realize and recognize the difference between “That’s wrong,” “I disagree,” and “I prefer a different solution.” One size does not fit all.

Finally, some of the demos and Biml platforms were not working at the time you wrote this review. Many have been addressed since that time. In the future, updates to SSIS, Biml, and ETL best practices will invalidate what this team of authors wrote during the first half of 2017. As such, the statements, “And much of the online information is outdated and no longer works. It’s a shame someone hasn’t found a way to simplify the learning process.” is a tautology that defines progress. Learning is part of the job of a technologist. I encourage us all – myself most of all – to continue learning.

Peace.

Stairway to Biml 9 is Live!

I am pleased to share Stairway to Biml 9 – Refactoring a Simple Custom Biml Framework is now live at SQL Server Central!

Typing that, I realize I neglected to publicize the fact that two other Stairway to Biml articles were recently published at SQL Server Central:

These three articles are taken from two chapters in The Biml Book:

  • Chapter 6: A Custom Biml Framework
  • Chapter 7: Using Biml as an SSIS Design Patterns Engine

Level 7 is focused on collecting and storing database metadata that will be used by Biml to automatically generate SSIS packages for a data integration solution.

Level 8 centers on one method for using the collected metadata to build a simple Biml-driven framework that automatically generates the SSIS packages for such a solution.

Level 9 describes and demonstrates one way to refactor this simple Biml framework, decoupling design patterns that were hard-coded in previous examples.

Enjoy!

:{>

 

“Do I Need a Biml Framework, Andy?”

No. No, you do not need a Biml Framework.

In the same way that I do not need a vehicle powered by an internal combustion engine, you do not need a Biml framework.

You see, I live about 5 miles from Farmville Virginia. I could walk to town each time my family needs groceries. I could pack as many groceries as I can carry into a backpack and walk the 5 miles back home. There are several benefits to this, not the least of which is the exercise that I would get walking to the store and then walking back – carrying a load of groceries, even!

“Is walking to the grocery store efficient, Andy?” Well, if you’re going get all persnickety about it… I suppose not. But think of all the money I would save on vehicle maintenance and fuel – plus the benefits of exercise!

Does this reasoning sound silly to you? It does to me (and I wrote it). Let’s add a dose of reality, shall we?

The Art of Data Integration Architecture, Part 1

Like many sciences, data integration is part art and part science. The art part is just good judgment. My lovely bride, Christy, has an awesome saying about good judgment: “Good judgment comes from experience, and experience comes from bad judgment.” With that in mind, I share the following in the hope that it saves you some… experience:

Don’t Use Biml…

If you are building a data integration solution that loads data from one source to one destination, do not use a Biml Framework. Heck, don’t even use Biml for this. Build this solution manually in SSIS or T-SQL or some combination thereof. It’s quicker, easier, and less-frustrating to construct a loader manually.

Use Biml…

If you are building a data integration solution that loads data from a couple dozen sources to a collection of destination tables, use Biml but do not use a Biml Framework. It takes about 40 hours to become proficient in Biml. To learn Biml (or anything, really), you first need a real-world problem to solve. Next, you need the gumption to try and solve that problem using a tool you have not yet used to solve a real-world problem.

Why wait until you need to build a couple dozen SSIS packages? It takes me about 2 hours to build and test an SSIS package that incrementally loads data. The math works:

24 x 2 = 48

If I take the time to learn Biml, which will take about 40 hours, I can complete two dozen loaders in less time than it would have taken me to build those SSIS packages manually. Plus I’ve learned something. More on this in a bit…

Use a Biml Framework…

After you’ve invested the initial 40 hours in learning Biml… If you are building a data integration solution that loads data from one or more sources to a collection of destination tables, use a Biml Framework. Once your framework is built, you can decide when it makes sense to use your automation and when it makes sense to build loaders manually.

I wrote about a Biml Framework here.
You can download an early example of a Biml Framework here.

Future-Proofing

Perhaps you’re looking at all the innovation and automation happening around you and thinking, “This is just some passing fad. The djin will be put back in the bottle soon and we’ll return to the days of carving our own integrated circuit chips out of wood.” Ok, maybe you’re not thinking precisely that. But I hope and pray you get my drift.

If the announcements of the latest Microsoft conference have taught you anything, they should have taught you that automation is following the arrow of time. Automation is on the increase in both volume and complexity. One benefit of having lived over half a century (and doing technology for most of that time) is perspective:

I promise you, the djin no longer fits in the bottle.

If I’m scaring you, good.

I’m not arguing the merits of automation.
I am arguing the facts of automation.

I am not attempting to assess the potential good – or potential harm – of automation. I am stating that it’s here to stay – and grow.

If you want to live and work in a field where you can master a craft and simply do the same thing until you retire, technology is not the field for you. Technology is the field for you if you enjoy learning and growing.

“How much time should I spend learning new stuff, Andy?” Experience informs me the answer is “about 25% of your time should be spent learning and growing.”

“Nummers”

My brother and friend Kevin Hazzard (DevJourney | LinkedIn | (awesome and fascinating) DataDriven interview) refers to “numbers” as “nummers.” There are a nummer of reasons I like this mispronunciation… perhaps I will explain another time. I bring up nummers here because – as my friend and brother Aaron Lowe used to tell me (before he passed away earlier in 2018 – I miss him…):

Math is hard.

I once delivered a solution that contained several hundred SSIS packages using Biml. The unit-of-measure for my estimate, based on the nummers I shared earlier (2 hours per package based on an incremental load SSIS design pattern) was “months.”

I delivered the solution in days using Biml.

It’s possible your employer isn’t good at math.
It’s possible your employer is more interested in providing you a job than in making a profit by remaining competitive (I love it when politicians speak of providing jobs… don’t get me started…).

The nummers argue that one day things may will change for your employer. If When that day arrives, things may will also change for you.

(Did I mention that I hope I am scaring you?)

So… Learn!

At the time of this writing, it is not possible for others to take your knowledge away from you. I pray that day never comes, but more than half-century of existence has trained me to not by surprised by anything.

So… learn!

Learn as much as you can. Consider knowledge a hedge against future changes. Add knowledge of the new and shiny to your experience. I don’t care how old or how young you are, start today.

I share this in the same spirit of you wanting me to get more exercise by walking to town to buy groceries, except that I am right about this and you were… less efficient. 😉

Start at SQL Server Central. It’s free.

Well, you have to give them your email address.

I understand that bothers some folks, but I encourage you to carefully weigh the benefits of sharing your email address with an entity that provides this much free education against falling behind in technology trends, application, and education. It’s totally your call, but you may find yourself in possession of the cleanest Inbox Zero of anyone who is uninformed and increasingly less-employable…

The Stairways are awesome.

Head over to edX. Many courses there are free. If you want, you can pay for certificates that you can post on LinkedIn. But – please – go get the knowledge.

 

If you want to dive deeper, sign up for Biml Academy or SSIS Academy or Fundamentals of Azure Data Factory! I’m still building the material so the costs for subscribing to the academies are low.

Peace.

It’s Biml 2018 Release Day!

Scott Currie and the team at Varigence announce the release of BimlFlex 2018, BimlStudio 2018, and BimlExpress 2018!

This much-anticipated release includes awesome features for experienced Biml professionals as well as for data engineers new to automating processes!

BimlFlex 2018

BimlFlex is a mature and complete data engineering automation framework. Out of the box, BimlFlex automates several data models, including:

  • Staging Database
  • Persisted Staging Area (ODS)
  • Raw Data Valut
  • Business Data Vault
  • Dimensional Data Warehouse
  • Data Marts
  • Cubes
  • Tabular

The benefits of BimlFlex:

  1. Flexible Data Framework
  2. Robust Metadata Mapping Tool
  3. Avoid Data Debt
  4. Shorter Delivery Times
  5. Simplified Maintenance
  6. Upfront Pricing

 

I absolutely love the BimlFlex Savings Calculator near the bottom of the page (click to enlarge)!

BimlStudio 2018

BimlStudio 2018 includes support for all Azure Data Factory (ADF) version 2 items (as of 06/20/2018). Which ADFv2 items does BimlStudio 2018 support?

  • Linked Services
  • DataSets, Sources, and Sinks
  • Pipelines and Activities (including control flow)
  • Triggers

Also included is support for BimlScript PreCompiled Assembly Package (BSPCAP) files, which include the preprocessed binary assets for all of the BimlScript files in your project.

BSPCAP promises faster processing – especially with large codebases – for builds in the interactive designer and for command line builds. I’m looking forward to learning more about this feature!

There are too many new features to list here – go to the BimlStudio 2018 page and check them all out!

BimlExpress 2018

Varigence keeps giving away cool stuff! Nowhere is Varigence’s commitment to community more evident than in the feature list for BimlExpress 2018. The previous version – BimlExpress 2017 – included the Preview Pane. BimlExpress 2018 includes the ability to Convert SSIS Packages to Biml:

How cool is that? And it’s in the free (FREE!) version!

As with BimlFlex and BimlStudio, there are too many cool features to list here. Head over to the BimlExpress 2018 feature page to learn more.

And Happy Biml’ing!

:{>