Biml, Book Reviews, and Metadata-Driven Frameworks

I occasionally (rarely) read reviews at Amazon of books I’ve written. If I learn of a complaint regarding the book I often try to help. If a reader experiences difficulty with demos, I often offer to help, and sometimes I meet with readers to work through some difficulty related to the book (as I offered here). About half the time, there’s a problem with the way the book explains an example or the sample code; the other half the time the reader does not understand what is written.

I own both cases. As a writer it’s my job to provide good examples that are as easy to understand as possible. Sometimes I fail.

In very rare instances, I feel the review misrepresents the contents of a book –  enough to justify clarification. This review afforded one such opportunity. None of what I share below is new to regular readers of my blog. I chose to respond in a direct manner because I know and respect the author of the review, and believe he will receive my feedback in the manner in which it was intended – a manner (helpful feedback) very similar to the manner in which I believe his review was intended (also helpful feedback).

Enjoy.

My Reply to This Review of The Biml Book:

Thank you for sharing your thoughts in this review.

I maintain successful technology solutions are a combination of three factors:

  1. The problem we are trying to solve;
  2. The technology used to solve the problem; and
  3. The developer attempting to use a technology to solve the problem.

I believe any technologist, given enough time and what my mother refers to as “gumption” (a bias for action combined with a will – and the stubbornness, er… tenacity – to succeed), can solve any problem with any technology.

I find your statement, “It doesn’t really teach BIML but rather teaches a specific optional framework for those who want to do everything with BIML rather than increase productivity for repetitive patterns” inaccurate. Did you read the entire book? I ask that question because I most often encounter readers who state the precise opposite of the opinion expressed in this review (some of your fellow reviewers express the opposite opinion here, even). I disagree with your statement, but I understand your opinion in light of other opinions expressed to me by readers during my past decade+ of writing several books. I share one example in this blog post. The short version: we often get more – or different – things from reading than we realize.

There is a section in The Biml Book – made up of 3 of the 20 chapters and appendices – that proposes a basic metadata-driven Biml framework which is the basis of a metadata-driven Biml framework in production in several large enterprises. I wrote those 3 chapters and the basic metadata-driven Biml framework in question. Varigence has also produced a metadata-driven Biml framework called BimlFlex – based upon similar principles – which has been deployed at myriad enterprises. The enterprise architects at these organizations have been able to improve code quality and increase productivity, all while decreasing the amount of time required to bring data-related solutions to market. They do not share your opinion, although they share at least some of the problems (you mentioned ETL) you are trying to solve.

Am I saying Biml – or the framework about which I wrote or even BimlFlex – is the end-all-be-all for every SSIS development effort? Goodness no! In fact, you will find disclaimers included in my writings on Biml and SSIS frameworks. I know because I wrote the disclaimers.

Misalignment on any of the three factors for successful technology solutions – the problem, the technology, and/or the developer – can lead to impedance mismatches in application and implementation. For example, Biml is not a good solution for some classes of ETL or data integration (points 1 and 2). And learning Biml takes about 40 hours of tenacious commitment (which goes to point 3). This is all coupled with a simple fact: data integration is hard. SSIS does a good job as a generic, provider-driven solution – but most find SSIS challenging to learn and non-intuitive (I did when I first started using it). Does that mean SSIS is not worth the effort to learn? Goodness, no! It does mean complaints about simplifying the learning process are to be expected and somewhat rhetorical.

Architects disagree. We have varying amounts of experience. We have different kinds of experience. Some architects have worked only as lone-wolf consultants or as members of single-person teams. Others have worked only as leaders of small teams of ETL developers. Rarer still are enterprise architects who are cross-disciplined and experienced as both lone-wolf consultants and managers of large teams in independent consulting firms and large enterprises. Less-experienced architects sometimes believe they have solved “all the things” when they have merely solved “one of the things.” Does this make them bad people? Goodness, no. It makes them less-experienced, though, and helps identify them as such. Did the “one thing” need solving? Goodness, yes. But enterprise architecture is part science and part art. Understanding the value of a solution one does not prefer – or the value of a solution that does not apply to the problem one is trying to solve – lands squarely in the art portion of the gig.

Regarding the science portion of the gig: engineers are qualified to qualitatively pronounce any solution is “over-engineered.” Non-engineers are not qualified to make any such determination, in my opinion.

By way of example: I recently returned from the PASS Summit. Although I did not attend the session, I know an SSIS architect delivered a session in which he made dismissing statements regarding any and all metadata-driven frameworks related to ETL with SSIS. If I didn’t attend his session, how do I know about the content of the presentation? A number of people in attendance approached me after the session to share their opinion that the architect, though he shared some useful information, does not appreciate the problems faced by most enterprise architects – especially enterprise architects who lead teams of ETL developers.

My advice to all enterprise architects and would-be enterprise architects is: Don’t be that guy.

Instead, be the enterprise architect who continues to examine a solution until underlying constraints and drivers and reasons the solution was designed the way it was designed are understood. Realize and recognize the difference between “That’s wrong,” “I disagree,” and “I prefer a different solution.” One size does not fit all.

Finally, some of the demos and Biml platforms were not working at the time you wrote this review. Many have been addressed since that time. In the future, updates to SSIS, Biml, and ETL best practices will invalidate what this team of authors wrote during the first half of 2017. As such, the statements, “And much of the online information is outdated and no longer works. It’s a shame someone hasn’t found a way to simplify the learning process.” is a tautology that defines progress. Learning is part of the job of a technologist. I encourage us all – myself most of all – to continue learning.

Peace.

Two (or More) Kinds of Developers

I’ve made statements about “two kinds of developers” for years. These statements are false inasmuch as all generalizations are false. The statements are not designed to be truisms. They are designed to make people think.

Last week – while presenting a full-day pre-conference session at the PASS Summit 2018 and again when delivering a session about Faster SSIS – I repeated the sentiment shown at the top of this post:

There are two kinds of developers:
1) Those who use source control; and
2) Those who will.

I follow up with: “Because if you do not use source control, you will lose code one day and it will break your heart.” Audience members laugh and the point is made.

More Than Two

There are myriad types of developers. And that’s a good thing. Why? Because there are myriad problems developers face in the wild and those problems need to be solved.

There are no one-size-fits-all solutions. If you attended the PASS Summit last week you likely saw some really cool demos. I know I did. And you may have thought – or even been told – that this, this right here is the answer for which you’ve searched your entire career.

There’s a possibility that the people selling you on whatever-this-is are absolutely correct.
There’s a greater possibility that they are less than absolutely correct.

I write this not to disparage anyone’s solution (or my own solutions). Promise.
I write this to dissuade the disparaging of anyone else’s solution (because that also happens).

The Bottom Line

Goldilocks. The bottom line is we all want the Goldilocks solution. We want the just-right solution that maximizes efficiency and minimizes complexity.

That’s just hard.

I can hear you thinking, “Why is maximizing efficiency and minimizing complexity hard, Andy?” I’m glad you asked. Solutions are a moving target, part art and part science, and the only way to learn where and when to draw the art-science line is experience.

A Moving Target

Maximizing efficiency and minimizing complexity is hard because it’s not at all the bottom line; it’s a line in the middle – a balancing of mutually-exclusive demands on your time, expertise, and energy.

Plus, it shifts.

Everything scales. Things scale either up and / or out or they scale down and / or in. In general (generality warning!), down and in is “bad” and up and out is “good.”

Experience Matters

Experienced architects understand subtle nuances; the art part of the art / science of enterprise software. When you speak with an experienced architect, you may hear her say, “It depends,” often. Good architects will finish the sentence and share at least some of the things upon which “it depends.” Less-experienced architects will present naked scalars and polarized advice.

Naked Scalars

Naked scalars are numeric values in a vacuum. They are unsupported because most are unsupportable. In other words, they are lies. Now, “lies” is a pretty harsh word. I prefer an engineering definition for the word “truth” that sounds an awful lot like the oath witnesses are asked to swear in US courts:

“Do you promise to tell the truth, the whole truth, and nothing but the truth, so help you God?”

This oath covers falsehoods that are shared, yes; but it also covers omissions of the truth.

Examples of naked scalars:

  • “97% of engineers believe ____.”
  • “10% of the people I know have practiced ____ successfully.”

Polarized Advice

Polarized advice can be a special case of naked scalars, advice focused on 0% and 100%. Polarized advice may or may not include scalars (naked or otherwise).

Examples of polarized advice:

  • “I’ve never seen a good use case for ____.”
  • “You should always ____.”

Are naked scalars and polarized advice always bad and wrong? Nope. That would be a generality (and we covered generalities already, did we not?).

Managing the Risk of Inexperience

What exactly is a consultant communicating when they engage naked scalars or polarized advice?
They are signalling a lack of experience.
They are, in effect, stating, “I do not have experience with ____.”

How do you manage the risk of inexperience?
You hire people – architects, especially – who understand there are good reasons systems are designed as they are. They will say things like, “I’m not sure why this was designed this way,” and mean it. It’s not a criticism; it’s an admission of curiosity. Trust me on this: You want curious consultants. They are more likely to identify a solution that solves the problem you are trying to solve in a way that doesn’t create new problems. Returning to the good reasons systems are designed as they are…

  1. Are (or were) the good reasons, well, good? Sometimes.
  2. Do the good reasons scale? Sometimes.
  3. Do the good reasons stand the test of time? Sometimes.

Good architects discern the baby from the bath water. Their experience separates good architects from the crowd. Not-as-good architects are less flexible, less willing to learn, and loathe to admit mistakes.

Let’s face facts, though: All architects and developers know what they know and don’t know what they don’t know. Better architects recognize these uncomfortable truths and mitigate them.

One way to mitigate inexperience – the best way, in my opinion, is to work with others.

The Story Of Us

At Enterprise Data & Analytics, our consultants and architects work together as a team. Our diverse group is a strength, bringing perspective to bear on the problems you are trying to solve. Our experience levels vary, the software and tools with which we work vary, and our demographics vary. As owner, I am honored to lead a team from diverse cultural – as well as diverse technical – backgrounds.

I didn’t set out to build a cultural- / age- / gender-diverse team. I set out to find the best people – to do what Jim Collins describes as “getting the right people on the bus.”

I found, though, that focusing on getting the right people on the bus had the side-effect of building a cultural- / age- / gender-diverse team.

As an added bonus, people of different genders approach problem-solving differently. People of different ethnicity pick up on stuff – especially cultural stuff, including enterprise culture – that people of other cultures miss.

EDNA‘s diversity is a strength that emerged unintentionally, but emerged nonetheless. As Chief Data Engineer, it’s very cool to watch our less-experienced consultants growing into more-experienced consultants and architects, while at the same time watching our people interact and perform as a team – each member catching stuff and contributing ideas because of their unique perspectives.

Cost Value

I can hear some of you thinking, “We’re on a budget here. Don’t good architects cost more than less-than-good architects, Andy?” I feel you. The answer is, “No. Good architects cost less than less-than-good architects.”

I can prove it. Because math. (Read that post for more information…)

It’s often accurate that good architects cost more per hour than less-than-good architects. Do you know why good architects charge more per hour?

Because they are worth more per hour.

(Generality!)

But consider this: “Time will tell” is a tried and true statement. Like good wine, the likelihood a generality is accurate improves with age. If enterprises continue to hire an organization – like Enterprise Data & Analytics or any other firm – to help them solve the problems they are trying to solve, then the folks shouting them down may be doing so in an effort to compete. Competition is fine, but I never hire anyone who talks bad about other clients or the competition. Why? They’ve demonstrated the capacity to talk bad about me at some later date.

Conclusion

I love our team!
I love our expertise!
I love our diversity!
I love that we always deliver value!

Contact me to learn more.

:{>

Free Stuff for People Who Give Back: Announcing 2019 Scholarships

I haven’t advertised this in the past and… I’m not sure why: I donate licenses for SSIS Catalog Compare and (non-free) SSIS Framework Editions – and subscriptions to Biml Academy and SSIS Academy – to individuals who work for charities and non-profit organizations. I am honored to announce our 2019 Scholarships.

I was inspired to make this public after reading this post over at Brent Ozar Unlimited.

Free Stuff for Charities and Non-Profit Organizations

  

Do you work for a charity or non-profit organization? Submit your application today.

Discounted Consulting Services

In addition to donating free licenses to our software and online training sites, Enterprise Data & Analytics offers a discounted rate to charities and non-profits for consulting services.

We are here to help.™ How may we serve you? Contact us today and let us know!

It Costs Precisely $0.00USD to be Nice

Be nice. We all have bad days. I know I do. It seems like I get interrupted about 1,000 times more when I’m busy than when I’m not busy. Why is that? Is it some vast universal conspiracy to rob me of productive work? Is it all in my head?

I am not sure.

One thing I am sure of, though, is that it ultimately comes down to me. I read this years ago and it stuck:

If the Comic Sans font bothers you, please reread the message.

I Am Here to Help™

I get a bunch of messages from recruiters. Many of them via LinkedIn. Most of the communication follows a similar pattern:

  1. I receive a Connection Request from someone with “recruiter” or “personnel” or “people” in their title.
  2. I accept. Why? I like people. I’m just that kind of guy.
  3. I get a message – usually within 24 hours – that reads in part something like the following: “I came across your profile searching for someone to fill a position for a ______. I think you may be a good fit for the position. If you are not interested, please share with any qualified individual in your network.” This is sometimes followed by a promise of referral recompense, though I’ve never – not once, to date – ever been compensated for recommending someone.

Recruiting

Back in my Linchpin People days,  I actually did a little recruiting. I’m not entirely sure, but I think I made more money per hour doing recruiting than I have ever made – period. I mean, I didn’t do it for the money (and I don’t recommend people for referral compensation, I was just pointing that out…). I was actually trying to help a friend, or customer, or both out of a jam.

In short, IT recruiting pays well. So I understand why recruiters behave they way they sometimes do.

There was a time when I would have done almost anything to have recruiters reaching out to me almost daily. It wasn’t that long ago, actually – just a couple decades. I remember begging recruiters to just give me a chance! I knew I could do the work. I could learn anything and I have a strong work ethic. I knew I would succeed. But…

I Lacked Experience

‘nough said.

Lately, Though…

Most recruiters who contact me these days do so via LinkedIn following the communications pattern I shared earlier.

I don’t mind. I used to mind, but I no longer do.

“Why Don’t You Mind, Andy?”

I’m glad you asked! That’s an excellent question.

There are actually several reasons. Please allow me to share one of them. Last year Enterprise Data & Analytics was hired to help a team deliver medical-related data integration with SQL Server Integration Services (SSIS). The customer succeeded with our help and all was well with the world. But there was this one twist that bears mentioning:

We were hired by a company for which I worked years ago.
And my boss used to report to me.

“It’s a good thing you weren’t a jerk to that person, huh, Andy?”

Almost. That sentence has an extra five words in it. There at the end.

Don’t get me wrong. It is good I wasn’t a jerk to that person. It’s better that I wasn’t a jerk to anyone – well, almost anyone – at that place of business. Why?

Because you never know.

Back to the Recruiters…

So… here we are communicating with people whose specialty and experience currently lies with hiring IT professionals… and you decided it’s a good idea to “teach them a thing or two?”

Really? One question: Have you thought that one through?

Are you never going to need to change jobs again? Ever?
Are you going to leave the IT field when you do?
If so, is it possible – even remotely – that you might encounter even one of the recruiters with whom you interact on LinkedIn?

One Way to Respond

How do I respond? I have this note saved as a text file. It takes about 15 seconds to find, open, copy, paste, and edit:

Be Nice

Peace.

My Day – Saturday, 21 Jul 2018 – #SQLCareer

5:30-9:30 AM

Awaken after hitting snooze thrice.

Weigh in, pray, read the Bible, check the burn barrel.
This morning I weigh 201 pounds.
There’s a bunch to pray about. I start with praise followed by gratitude, thanking God for another day.
This morning I read Psalm 150 again (now reading Psalms from last to first), the second half of Romans 14, and Colossians 1:9-14.
Coffee.

I help Christy and Emma get the van loaded for the Farmer’s Market today. They sell all sorts of goodies.

The fire in the burn barrel is out and a few coals are smoldering. I decide to let is cool down and empty the ashes, then start over.

Breakfast.

9:30-11:00 AM

I get the boys up – those boys can sleep. Stevie Ray needs to finish up some laundry and homeschool testing and Riley has some cleaning to do. I clean up the kitchen some and then head to the office.

Slides for the Data Platform Summit 2018 in Bangalore, India are due tomorrow. I’ve delivered each presentation before. But this is the first delivery of the Intelligent Data Integration precon with Azure Data Factory material. Exciting!

11:00 AM-2:30 PM

I head outside for some “pine therapy.” It’s Saturday and I want to get the trunk of this large-ish tree on the ground. That means either finishing or all but finishing the limbing of the tree. To turn it into exercise (well, more exercise) I decide to use the ax instead of the chainsaw. I accomplish my goal. Four large piles of pine brush lie drying in the Farmville sun.

2:30 PM

I call it for the day. I may have overdone it a little. Exercise is a good idea and getting outside for a stretch was awesome. Perhaps the ax wasn’t such a good idea… My hands and elbows are sore.

I do something I rarely do – plop down on the couch and watch some television.

:{>

My Day – Friday, 20 Jul 2018 – #SQLCareer

5:06 AM

– awaken after hitting the snooze on my phone alarm once.

5:06 – 9:30 AM

– Weigh in, pray, read the Bible, coffee, check the burn barrel.
This morning I weigh 201 pounds.
There’s a bunch to pray about. I start with praise followed by gratitude, thanking God for another day.
This morning I read Psalm 150 and the first half of Romans 14, which later prompts a couple social media posts about how I should stop despising and judging.
Coffee: ’nuff said.
The fire in the burn barrel did not do well last night. The bigger pieces I left to smolder overnight are still there. It’ll take a couple tries to get it burning this morning, I imagine. I start Try #1 with some dead twigs.
I comment on Brent’s post about #SQLCareer and fill up the Buffer buffer for the day.
I open Outlook and see an email from Kent Bradshaw.
I meet with Kent via SkypeForBusinessLync to discuss the goings-on at Enterprise Data & Analytics. Consulting is slow now, which is normal for this time of year. In about the month we’ll start getting pings for work. That’s fine, we both have other things going on (vacations, a trip to the Data Platform Summit 2018 in Bangalore, India, etc.).
Christy and I have breakfast (keto-friendly). She started baking yesterday for the Farmer’s Market tomorrow. After breakfast, I resist the small tray of brownies on the counter… for now. I had a couple yesterday and they are, as ever, awesome.
Stevie Ray finishes camp today and he needs to be picked up at 5:30. It’s about an hour away so I will likely knock off mid-afternoon today.
Riley is finishing up his homeschool testing this weekend. Since he’s doing math I’m on for questions. This will likely consume much of the morning.
Write. I add to my Notepad document to record my day for the next few days.
Startup vDemo – my demonstration virtual machine – in preparation for recording the SSIS Academy Data Flows 2 lab (video only, no audio). Let’s see if we can capture the video before I need to proctor Riley’s exam…

9:30 AM – 12:00 PM

Capture the video for the SSIS Academy Data Flows 2 lab which validates the lab document. Save the lab doc as PDF.
Proctor Riley’s exam.
Christy is cooking like a mad-woman for the Farmer’s Market and doesn’t want to stop to eat lunch (she somehow managed to make supper in the crockpot already!), so I grab some leftovers and eat in the office.

12:00 – 3:30 PM

Worked on SSIS Academy Data Flows 2 course. Added some files to the Data Flows 1 collateral.
I did about an hour’s worth of “Pine Therapy” – keeping the burn barrel going and enjoying some time outside in this lower humidity limbing up a pine I felled a couple days back and piling brush.
I decided to call it quits a little early as we prepare to pick up Stevie Ray from camp. It’s about an hour’s drive into the sticks. You have to put “the sticks” in context and realize that I’m writing that from five miles outside of Farmville Virginia…

My Day – Thursday, 19 Jul 2018 – #SQLCareer

#SQLCareer post for Thursday, 19 Jul 2018

4:26 – 5:30 AM

– Wake up (beating the alarm), weigh in, pray, read the Bible, coffee, check the burn barrel.
Christy and I have been on a keto diet since 5 Mar. I started out at 238 pounds. This morning I weigh 199 pounds. I’ve been bouncing in the 196-203 range since early May. Part of the reason for the plateau is cheating on the diet (diet fatigue). Part of the reason is I’ve probably lost what I can with diet alone and need to add more exercise. My exercise lately has been work around the farm.
There’s a bunch to pray about. I start with praise followed by gratitude, thanking God for another day.
I’ve been reading through the Book of Psalms recently and this morning I read 149 (of 150). I’m also studying the books of Romans and Colossians.
Coffee: ‘Nuff said.
I’m slowly burning limbs and brush from areas I’m clearing around the farm. There are hot coals from the three larger pieces I put in the barrel last night. I add three more larger pieces.

5:30 – 6:00 AM

– Write. I create a Notepad document to record my day for the next few days. After reading Brent Ozar’s first entry last night – in which he reminded me of Steve Jones’ SQLCareer post earlier this month – I decided to share a few days.
Startup vDemo – my demonstration virtual machine – in preparation for today’s Summer O’ ADF webinar titled Use SSIS Catalog Compare to Lift and Shift SSIS to ADF.
While vDemo is starting, I log into Azure Data Factory and start my Integration Runtime. The webinar is over 6 hours from now but I don’t want to wait until the last minute – it takes 20-30 minutes for ADFIR to start. I note the free Azure account I set up for the Summer O’ ADF has $181 left and is probably down to about 10 days remaining in the free month with $200 of credit. I note the accounting doesn’t (yet) reflect the 5 hours or so I had Integration Runtime running yesterday. My Azure spend has latency and I am attempting to monitor the latency. At present I see resource usage reflected in billing 36-60 hours after the resource usage.
I fire up vSQL2014 to begin testing SSIS Catalog Compare v3.2.1.1 (Preview). I’ve been working on this latest release of Catalog Compare for a year. Or more. Reasons for the delay, in order:
0. I bit off a lot. I made major changes to the graphical (treeview) representation of the SSIS Catalog. If you’ve been playing along at home, you can see these changes in the free DILM Suite utility, Catalog Browser. Catalog Browser and Catalog Compare share a lot of code.
1. Enterprise Data & Analytics landed two large data engineering projects at about the same time one year ago. One client doubled their request at the beginning of 2018 which resulted in doubling our people. Both projects are done but the both clients indicate they will have more work for us in the future. I still deliver consulting work – designing and building SSIS and database solutions – while managing the company. Nick Harris manages the business development cycle; a skill at which he is most gifted and for which I have neither proclivity or the inclination to proclive (…to quote the sage wisdom of Tom Smykowski [Office Space], “Engineers are not good at dealing with customers.”). Consulting and management took time away from software development.
2. I am a C# n00b. A year ago I was working my way up to becoming a C# n00b. (This maybe should have been first…)
While walking through the script for today’s webinar, I found a bug in the scripting of SSIS parameter literals. Booleans are… difficult. The string value of a Boolean is “true” or “false.” The corresponding T-SQL values are 1 or 0. Since SSIS Catalog parameter values are stored as sql_variants, getting the value right is vital. True/false not only doesn’t work, it causes the T-SQL script to fail. Hence, SSIS Catalog Compare v3.2.1.0 (Preview) was updated yesterday to SSIS Catalog Compare v3.2.1.1 (Preview). I’ll put it through some paces this morning by running through the webinar script again. If all goes well, I will drop it into the Dropbox folder and let the Early Adopters know there’s a new version available.
Software development is hard. Product development is harder. Product management is harder still. I am getting there…

6:00 – 6:30 AM

– Check the Buffer(.com) buffer. Load it up with posts for today from andyleonard.blog and a few other sites.
Check the results of yesterday’s newsletter campaign.
Look over stats from the business websites:
Enterprise Data & Analytics
DILM Suite
Biml Academy – more sales this month compared to last month, most likely due to new content.
SSIS Academy – launched this week and has decent traffic for a newly-launched site.
Poke around on social media for a few minutes. I start with LinkedIn and use it more than Twitter and Facebook these days. I really like the Daily Rundown notification and read at least the headlines each day. The article that catches my attention today is about drug companies retracting price hikes or announcing they will not increase drug prices in response to a tweet by the president. I’m drawn to this on a couple levels:
1. Many of Enterprise Data & Analytics’ customers are medical-related.
2. I am a Type-2 diabetic and one of my medications is expensive (~$500/month). (Related: the keto diet I mentioned earlier has helped. I now take 1/3rd of the medications I used to take and my blood sugar is lower.)
I also read about Amazon’s Prime Day and how it improved revenue for other online retailers, specifically Target.

6:30 – 7:45 AM

– I get a low battery warning on the laptop. I don’t always bring the laptop downstairs after work but I did yesterday hoping I’d get a chance to look at the SSIS Catalog Compare bug I’d identified earlier in the day. I did have time and I fixed it (I think) – more testing today… I shut down the laptop for now.
I make a keto-friendly breakfast for Christy and me and fry up some bacon for Emma and Riley (Stevie Ray is at camp this week). I put Christy’s omelette (spinach and cheddar) in the preheated-and-then-shut-off oven along with the bacon for the kids. I have an omelette and some fried bologna. The keto diet is weird if you’re used to non-fat diets. Fat is part of the keto diet, as are complex carbs. And it’s time for my second liter of coffee.
I catch up on some reading while eating. I am currently reading The Holiness of God by R. C. Sproul during mornings and The Culture Series by Iain M. Banks before sleeping. Reading is important and I enjoy both reading and writing.
I check the fire a couple times to make sure the wood I added is catching up. It’s not, so I add some more, smaller pieces this time. The wood I put in before sunrise was damp with the light dew. Sitting on the coals has dried it out and the twigs catch in a few minutes.
I load and start the dishwasher. I forgot about the clothes in the washer last night – it was a busy and late night for me with praise band rehearsal – so I re-wash that load.
I empty the trash and make a run to the landfill, checking the fire on the way out and when I return. It’s burning nicely.

8:00-8:20 AM

– I get to the office and settled a little before 8:00 AM. The laptop battery is happy (well, happier, as it charges).
I contemplate live-blogging the day and decide I will not be able to provide many updates, but it’s worth starting, and so I do. After some proofreading, it’s ready to publish.

Update 1: 8:20-8:45 AM

– I proofread (again!) this post and correct typos and misspellings and verb tenses and… you get the picture. I promise I proofread it in Preview. But for some reason, it’s not the same as reading it after it’s posted.

I pop open Outlook and check my email. Nick has closed another gig for what we call “SSIS Review.” I need to add a page to the Enterprise Data & Analytics website about this service. We’ve been doing a lot of this lately. We typically set the contract at not-to-exceed 40 hours (minimum) and only invoice for hours as they are consumed (unless the customer requests otherwise… some customers have budgets that they must consume within a fiscal year. If the customer requests it, we invoice up front.).

I also see a Customer Status Update from Penny. Penny has been working with Enterprise Data & Analytics for a couple years now. She’s my daughter, Mom to three of my favorite five grandchildren, and quickly becoming a senior SSIS developer. I’m proud of her.

Time to hit Update and post about this post on social media.

More later…

Update 2: 8:45-9:45 AM

I restarted vDemo and starting walking through the script for today’s webinar.

I put the wash in the dryer and started a new load. I propped open the dishwasher so the load that just finished will dry faster.

I checked the fire again and, durnit! Still not going like I want. I added some more small stuff and let it set for a while. It’s started burning well as you can see here.

I rescued a small bird from a couple of our cats that must’ve just left the nest.

I posted a LinkedIn video about the upcoming webinar:

Update 3: 9:45-10:45 AM

Webinar script rehearsal is complete and was successful.

Update 4: 10:45-11:45 AM

Took a break away from the laptop. Why? I like to do something else before presenting – something… different. Plus it’s a gorgeous day in Farmville Virginia with surprisingly low humidity for this time of year. Emma and I took advantage of the remaining coolness of the morning and piled some brush near the burn barrel.

It’s now time to start the webinar.

Update 5: 11:45 AM-1:15 PM

The webinar is done and, by all indications, was a success. It’s lunch time!

Update 6: 1:15-2:30 PM

Lunch was keto-friendly, of course. A salad and a meat-stuffed squash.

Christy and I dropped off her van across town. It needs new tires. The shop tells us they can get it done today which is great because we have a couple other errands to run this afternoon – we’re meeting friends from church and I am getting a haircut (or several hairs cut…).

Update 7: 2:30-4:30 PM

I posted The Recording for Use SSIS Catalog Compare to Lift and Shift SSIS to ADF is Available.

I got a haircut.

I restocked the burn barrel.

I edited the PowerPoint and lab document for the upcoming SSIS Academy Data Flows 2 course. Online courses take probably more time to build than you think – especially if you do it right. I spent a lot of time documenting the Data Flow 1 course. I’m spending just as much time going through the documentation for the Data Flow 2 course, editing and clarifying. Why? I don’t want to have to try to remember (or figure out) the mechanics of publishing a course each time. I want to spend my brain cycles on developing awesome courses!

At this point, I’ve been at it for over 12 hours. Do you know what entrepreneurs call a 12-hour shift? A half day.

Update 8: 4:30-6:00 PM

I updated pages at Biml Academy, SSIS Academy, and Enterprise Data & Analytics.

The shop finished with Christy’s van so I took her across town to pay for the work and pick it up.

I folded the last of the clothes.

I captured the video and began editing the audio script for SSIS Academy Data Flow 2 lecture. I hope to record the lecture audio tomorrow but it may have to wait until Saturday – tomorrow we pick Stevie Ray up from camp.

It’s time to take another look at the burn barrel, grab supper with the sexiest redhead on the planet, watch some Food Network, read some Culture Series, and then get a good night’s rest.

I am thankful. Today was a pretty good day.

:{>

The Cloud Costs Money

It’s true: the cloud costs money.

Yesterday, when I logged into my Azure Portal to set up an Azure Data Factory Integration Runtime for my presentation, the balance for my Azure free subscription (which allows me $100USD per month) was sitting around $82USD. This morning, after leaving a demo instance of ADFIR running overnight, my balance has dropped:

I absolutely love the cloud! This is why I’ve been delivering webinars about the cloud and blogging and presenting about Azure Data Factory.

I would just like to clarify a few things:

The Cloud Is Not Free

This much is true. And while the cloud can save enterprises – especially smaller enterprises – lots of cash on “burst-y” workloads via scale-on-demand – the cloud can actually cost more money when unneeded services are accidentally left running.

The Cloud Will Not Fix Bad Code

In a previous consulting life, a customer contacted us and asked for an evaluation of their architecture. They were attempting to scale the business and encountering… obstacles. A team looked over their software and database designs and recommended a rewrite of their custom code. We supplied a proposal (including an expensive estimate) to deliver the re-architecture, redesign, and rewriting of their code.

They felt the price was too high.

We understood, so we countered with a proposal to coach their team to deliver the same result. This would cost less, the outcome would be the same, and their team would grok the solution (because their team would build the solution). The catch? It would take longer.

They felt this solution would take too long.

We understood, so we offered to help in any way they would allow. The company sought help from other firms. A competing consulting firm offered to migrate their code and databases to a cloud provider. The company liked this solution and embarked on a migration.

The migration succeeded.
It took longer than expected.
The company hemorrhaged cash and clientele throughout the process, and is now a fraction of its previous size in people, clients, cash flow, and profit.

The lesson? Migrating to the cloud wasn’t a bad move for them.
Ignoring the advice of experienced architects was a bad move for them.

Did this client need to move to the cloud? Perhaps. More than anything, this client needed to pay off a tremendous amount of technical debt. By the time we were called, they were well into technical inflation and slipping into a technical depression.

Yes, the cost for a fix was high.
No, the cost for a fix wasn’t as high a price as the costs they have paid for not fixing their code. And…

… the cloud did not fix their bad code.

Heartbreaking

This was heartbreaking to witness. We could have helped them. Why were we unable to help them? We told them the truth; our competition told them what they wanted to hear.

As Brent Ozar pointed out in his post titled What It’s Like to Work for a Bad Contracting Company:

See, bad contracting companies have a huge sales force that goes out and asks customers, ‘Hey, what problems are you having? Sure, we can solve those. Just sign here.’ They make impossible promises about their – well, YOUR – capabilities and timelines.

Brent is right. This is why I ask, “Are you getting the best technical help available? or are you being sold technology by the best sales people?” There is sometimes a difference between consultants who are good at sales (or who hire good sales people) and consultants who are good at technology.

I’m going to take that one step further:

The very best technologist – the one most able to solve your problem for the lowest cost – may be someone who personally irritates you and who charges an hourly rate higher – perhaps multiples higher – than the competition.

Allow me to be (more) blunt:

  1. Did the company in the (true) account above believe they were going to save money?Yes, they did.
  2. Did the company in the (true) account above save money?No, they did not.

In my post titled A True Story of Value vs. Hourly Rate I share another true story of how our awesome, experienced team at Enterprise Data & Analytics was able to deliver value to a client. Even though we charged more per hour than the competition we delivered more value. How do we know?

The client did the math.

I’m writing this post to beg you to also do the math.
I’m writing this post to ask you to buy consulting services (if you need them) and not just be sold consulting services (whether you need them or not).
I’m writing this post to let you know that the cloud is awesome. But the cloud is not free.

“How Much Does Azure Cost?”

I hear this question a lot and it is difficult to answer. You can certainly browse over to the Azure Pricing Page to learn more. The difficulty is estimating how many __U’s you are going to use (see what I did there?).

Brent asked a really good question on Twitter and got this awesome answer:

The website is azureinstances.info and it. is. awesome.

How to Fix Being Charged for an Unused ADF Integration Runtime?

Stop ADF IR:

Delete unused cloud resources:

Peace.

Learn More:
Free Webinar – The Azure Data Factory Controller Design Pattern – 28 Jun at noon EDT

Honored to Present an SSIS and ADF Precon at Data Platform Summit 2018!

Expert SSIS – Live, Online, 2.5 Days – 10-12 Sep 2018 – delivered in cooperation with Brent Ozar Unlimited

Honored to Deliver Intelligent Data Integration – a PASS Summit 2018 Precon

“You Do Not Know What You Are Doing”

Peeves make lousy pets.

Knowing this doesn’t help; I still keep a few pet peeves. One of my pet peeves is this statement, “You don’t know what you are doing.” Why is this a pet peeve? It denies the obvious fact that everyone one of us, everywhere, is still learning.

“My Name is Andy and I Own and Operate a Consulting Company.”

“But Andy, you don’t know how to own or operate a consulting company.” That may or may not be a true statement. What is a truer statement? I may not know everything there is to know about owning and operating a consulting company, but I can learn.

“My Name is Andy and I Built a Software Product.”

“But Andy, you don’t know how to build a software product.” That may or may not be a true statement. What is a truer statement? I may not know everything there is to know about building a software product, but I can learn.

Interesting sidebar: SSIS Catalog Compare is not only the first product I’ve ever written, it’s the first complete application I’ve written in C#.

“My Name is Andy and I Co-Host a Successful Podcast”

“But Andy, you don’t know how to co-host a successful podcast.” That may or may not be a true statement. What is a truer statement? I may not know everything there is to know about co-hosting a successful podcast, but I can learn.

I Can Learn

I know I can learn because I have demonstrated this fact many times over. I proved it last month (at the time of this writing – April 2018 thereafter) when I completed the Microsoft Professional Program for Big Data. I proved it by learning enough C# to write Catalog Compare, Catalog Browser, and Framework Browser.

I promise I am learning more every day about owning and operating Enterprise Data & Analytics and building and managing the software solutions and products that make up the DILM Suite – including  products like SSIS Catalog Compare and the SSIS Framework – and co-hosting Data Driven, with Frank La Vigne (@Tableteer).

“I couldn’t so you shouldn’t.”

What I Know

What is someone truly saying – what do they truly mean – when they say or write someone doesn’t know what they’re doing?

They’re making this statement about themselves: “I couldn’t so you shouldn’t.”

No one brings this point home better than Grant Cardone in his book (get the audio book – you are welcome), Be Obsessed or Be Average, or #BOBA. The followup to his (awesome) book, The 10X Rule, Be Obsessed or Be Average complements and completes Cardone’s thoughts on the hard work and time required to achieve success.

“What is the Point, Andy?”

When people make statements like “You don’t know what you are doing,” they are saying, “I gave up so you should give up, too,” or, “I didn’t get what I wanted so you don’t deserve what you want, either.”

This is very fair thinking.

When I write the word “fair” I shudder at what “fair” has come to mean and how it’s been used to justify junk and the crap it’s been used to rationalize.

Conclusion

I am not going to quit learning.
I will continue to try to make old things work better.
I will continue to try new things.
I will fail more often than I succeed (this is how I learn).
I will not stop until I go home.

My advice, encouragement, exhortation:

  • Don’t quit.
  • Make the problems give up before you do.
  • Listen to people who have succeeded (or are succeeding).
  • Do not listen to people who have given up.

I have more to learn and I know that.

Peace,
Andy