Two (or More) Kinds of Developers

I’ve made statements about “two kinds of developers” for years. These statements are false inasmuch as all generalizations are false. The statements are not designed to be truisms. They are designed to make people think.

Last week – while presenting a full-day pre-conference session at the PASS Summit 2018 and again when delivering a session about Faster SSIS – I repeated the sentiment shown at the top of this post:

There are two kinds of developers:
1) Those who use source control; and
2) Those who will.

I follow up with: “Because if you do not use source control, you will lose code one day and it will break your heart.” Audience members laugh and the point is made.

More Than Two

There are myriad types of developers. And that’s a good thing. Why? Because there are myriad problems developers face in the wild and those problems need to be solved.

There are no one-size-fits-all solutions. If you attended the PASS Summit last week you likely saw some really cool demos. I know I did. And you may have thought – or even been told – that this, this right here is the answer for which you’ve searched your entire career.

There’s a possibility that the people selling you on whatever-this-is are absolutely correct.
There’s a greater possibility that they are less than absolutely correct.

I write this not to disparage anyone’s solution (or my own solutions). Promise.
I write this to dissuade the disparaging of anyone else’s solution (because that also happens).

The Bottom Line

Goldilocks. The bottom line is we all want the Goldilocks solution. We want the just-right solution that maximizes efficiency and minimizes complexity.

That’s just hard.

I can hear you thinking, “Why is maximizing efficiency and minimizing complexity hard, Andy?” I’m glad you asked. Solutions are a moving target, part art and part science, and the only way to learn where and when to draw the art-science line is experience.

A Moving Target

Maximizing efficiency and minimizing complexity is hard because it’s not at all the bottom line; it’s a line in the middle – a balancing of mutually-exclusive demands on your time, expertise, and energy.

Plus, it shifts.

Everything scales. Things scale either up and / or out or they scale down and / or in. In general (generality warning!), down and in is “bad” and up and out is “good.”

Experience Matters

Experienced architects understand subtle nuances; the art part of the art / science of enterprise software. When you speak with an experienced architect, you may hear her say, “It depends,” often. Good architects will finish the sentence and share at least some of the things upon which “it depends.” Less-experienced architects will present naked scalars and polarized advice.

Naked Scalars

Naked scalars are numeric values in a vacuum. They are unsupported because most are unsupportable. In other words, they are lies. Now, “lies” is a pretty harsh word. I prefer an engineering definition for the word “truth” that sounds an awful lot like the oath witnesses are asked to swear in US courts:

“Do you promise to tell the truth, the whole truth, and nothing but the truth, so help you God?”

This oath covers falsehoods that are shared, yes; but it also covers omissions of the truth.

Examples of naked scalars:

  • “97% of engineers believe ____.”
  • “10% of the people I know have practiced ____ successfully.”

Polarized Advice

Polarized advice can be a special case of naked scalars, advice focused on 0% and 100%. Polarized advice may or may not include scalars (naked or otherwise).

Examples of polarized advice:

  • “I’ve never seen a good use case for ____.”
  • “You should always ____.”

Are naked scalars and polarized advice always bad and wrong? Nope. That would be a generality (and we covered generalities already, did we not?).

Managing the Risk of Inexperience

What exactly is a consultant communicating when they engage naked scalars or polarized advice?
They are signalling a lack of experience.
They are, in effect, stating, “I do not have experience with ____.”

How do you manage the risk of inexperience?
You hire people – architects, especially – who understand there are good reasons systems are designed as they are. They will say things like, “I’m not sure why this was designed this way,” and mean it. It’s not a criticism; it’s an admission of curiosity. Trust me on this: You want curious consultants. They are more likely to identify a solution that solves the problem you are trying to solve in a way that doesn’t create new problems. Returning to the good reasons systems are designed as they are…

  1. Are (or were) the good reasons, well, good? Sometimes.
  2. Do the good reasons scale? Sometimes.
  3. Do the good reasons stand the test of time? Sometimes.

Good architects discern the baby from the bath water. Their experience separates good architects from the crowd. Not-as-good architects are less flexible, less willing to learn, and loathe to admit mistakes.

Let’s face facts, though: All architects and developers know what they know and don’t know what they don’t know. Better architects recognize these uncomfortable truths and mitigate them.

One way to mitigate inexperience – the best way, in my opinion, is to work with others.

The Story Of Us

At Enterprise Data & Analytics, our consultants and architects work together as a team. Our diverse group is a strength, bringing perspective to bear on the problems you are trying to solve. Our experience levels vary, the software and tools with which we work vary, and our demographics vary. As owner, I am honored to lead a team from diverse cultural – as well as diverse technical – backgrounds.

I didn’t set out to build a cultural- / age- / gender-diverse team. I set out to find the best people – to do what Jim Collins describes as “getting the right people on the bus.”

I found, though, that focusing on getting the right people on the bus had the side-effect of building a cultural- / age- / gender-diverse team.

As an added bonus, people of different genders approach problem-solving differently. People of different ethnicity pick up on stuff – especially cultural stuff, including enterprise culture – that people of other cultures miss.

EDNA‘s diversity is a strength that emerged unintentionally, but emerged nonetheless. As Chief Data Engineer, it’s very cool to watch our less-experienced consultants growing into more-experienced consultants and architects, while at the same time watching our people interact and perform as a team – each member catching stuff and contributing ideas because of their unique perspectives.

Cost Value

I can hear some of you thinking, “We’re on a budget here. Don’t good architects cost more than less-than-good architects, Andy?” I feel you. The answer is, “No. Good architects cost less than less-than-good architects.”

I can prove it. Because math. (Read that post for more information…)

It’s often accurate that good architects cost more per hour than less-than-good architects. Do you know why good architects charge more per hour?

Because they are worth more per hour.

(Generality!)

But consider this: “Time will tell” is a tried and true statement. Like good wine, the likelihood a generality is accurate improves with age. If enterprises continue to hire an organization – like Enterprise Data & Analytics or any other firm – to help them solve the problems they are trying to solve, then the folks shouting them down may be doing so in an effort to compete. Competition is fine, but I never hire anyone who talks bad about other clients or the competition. Why? They’ve demonstrated the capacity to talk bad about me at some later date.

Conclusion

I love our team!
I love our expertise!
I love our diversity!
I love that we always deliver value!

Contact me to learn more.

:{>

AndyWeather Internet of Things (IoT)

AndyWeather.com is a site I’ve maintained for about 10 years now. I use the site and related hardware, software, and services to test concepts and perform experiments.

I then apply my experience in delivering Internet of Things (IoT) solutions for Enterprise Data & Analytics customers and for SSIS and Biml training, such as my upcoming course titled Fundamentals of Azure Data Factory delivered in cooperation with Brent Ozar Unlimited.

It all started when GoDaddy created a DMZ for SQL Server databases. I found this functionality in 2008 and asked myself, “Self, how might we use this?”

Since That Time…

There have been two major iterations of AndyWeather. I use weather data collected during the first iteration for training purposes at SSIS Academy and when delivering training to Enterprise Data & Analytics customers.

AndyWeather v2

The setup of the second iteration is fairly straightforward:

  1. The Acurite Weather Station consists of an instrument pack plus a base station. The instruments collect weather measurements and transmit them to the base station.
  2. The base station is connected to an older e-Machine running Windows 7 Ultimate (32-bit) on 2GB RAM.
  3. An Acurite application interfaces with the base station and the application stores data locally in a single CSV file.
  4. I wrote a very simple C# console application named “abt” (an acronym for “Azure Blob Transfer”) to transfer the CSV file to Azure Blog Storage.
  5. An Azure Data Factory pipeline that loads an Azure SQL DB staging table.
  6. The AndyWeather website which reads the latest weather data from the Azure SQL DB staging table.
  7. I wrote another very simple C# application named “awt” (an acronym for “AndyWeather Tweets”) that tweets updates to the @AndyWeather twitter account.

Acurite Weather Station

The latest iteration began in early 2018 when I purchased an updated package of instruments and a new base station made by Acurite. So far, I like this station a lot. It was less expensive than the previous station and appears more rugged (again, so far – time will tell).

I recently relocated the weather station to improve connectivity between the instruments and the base station. I recorded a Data Driven *DataPoint* about it:

(Pay no attention to the exploding pecans in the background…)

The e-Machine

I intentionally use an under-powered PC for the server. Why? I want to learn how the base station – and then everything downstream of the base station – responds to busy server conditions. This is Engineering 101 stuff and I’ve learned a lot:

I love this old machine!

Acurite Application

The Acurite people maintain an application for communicating with base stations:

(click to enlarge)

The PC Connect application allows me to configure how and when weather data is collected from the base station – which collects measurements from the instruments. The application lets me configure the units-of-measure and file location – and I can even share my weather data with Weather Underground. How cool is that?

The Azure Blob Transfer Console Application

The Azure Blob Transfer (abt) application is a very simple console application written in C#. It picks up the CSV file containing weather data stored by the Acurite PC Connect application and writes the file to an Azure Blob Storage container:

(click to enlarge)

The CSV file in Azure Blob Storage is overwritten each time abt successfully executes. You can download a copy of the abt solution here.

Azure Data Factory Pipeline

An Azure Data Factory (ADF) pipeline calls a stored procedure that first truncates a staging table in a Azure SQL DB using a Stored Procedure activity, followed by a Copy Data activity that copies the weather data from the CSV file in Azure Blob Storage to an Azure SQL DB staging table:

At the time of this writing, ADF version 2 is current.

You can download the ARM template for the pipeline here.

The AndyWeather Website

The AndyWeather website has been around since the days of the first iteration of AndyWeather – the one that stored data in a SQL Server instance hosted at GoDaddy’s DMZ. It’s fairly straightforward code, which helps it perform fairly on desktops and mobile devices:

The biggest performance hit comes from executing the stored procedure against an Azure SQL DB, which can sometimes take 5-10 seconds to complete.

The AndyWeather Tweets Console Application

I snagged some C# code and a TwitterAPI class from a project named called TweetSharp to help build the awt console application:

You can download a copy of the awt solution here.

The @AndyWeather Twitter Account

It makes me happy every time I see a tweet from @AndyWeather:

I tell people, “It’s just a dumb little app,” but I really had fun building it. I learned a bunch, too!

Conclusion

The AndyWeather IoT solution uses hybrid technology – on-premises instruments and servers, combined with cloud services – to deliver weather data to a website and Twitter account. It’s accessible from social media and the web from desktops and mobile devices.

Just so you know, this isn’t everything I’ve built using the AndyWeather instruments. There’s a bunch more – some of which is still in the experimental phase. I’ll share more as time permits. But I want you all to know, I consider Azure a great big cyber-playground!

:{>

“You Do Not Know What You Are Doing”

Peeves make lousy pets.

Knowing this doesn’t help; I still keep a few pet peeves. One of my pet peeves is this statement, “You don’t know what you are doing.” Why is this a pet peeve? It denies the obvious fact that everyone one of us, everywhere, is still learning.

“My Name is Andy and I Own and Operate a Consulting Company.”

“But Andy, you don’t know how to own or operate a consulting company.” That may or may not be a true statement. What is a truer statement? I may not know everything there is to know about owning and operating a consulting company, but I can learn.

“My Name is Andy and I Built a Software Product.”

“But Andy, you don’t know how to build a software product.” That may or may not be a true statement. What is a truer statement? I may not know everything there is to know about building a software product, but I can learn.

Interesting sidebar: SSIS Catalog Compare is not only the first product I’ve ever written, it’s the first complete application I’ve written in C#.

“My Name is Andy and I Co-Host a Successful Podcast”

“But Andy, you don’t know how to co-host a successful podcast.” That may or may not be a true statement. What is a truer statement? I may not know everything there is to know about co-hosting a successful podcast, but I can learn.

I Can Learn

I know I can learn because I have demonstrated this fact many times over. I proved it last month (at the time of this writing – April 2018 thereafter) when I completed the Microsoft Professional Program for Big Data. I proved it by learning enough C# to write Catalog Compare, Catalog Browser, and Framework Browser.

I promise I am learning more every day about owning and operating Enterprise Data & Analytics and building and managing the software solutions and products that make up the DILM Suite – including  products like SSIS Catalog Compare and the SSIS Framework – and co-hosting Data Driven, with Frank La Vigne (@Tableteer).

“I couldn’t so you shouldn’t.”

What I Know

What is someone truly saying – what do they truly mean – when they say or write someone doesn’t know what they’re doing?

They’re making this statement about themselves: “I couldn’t so you shouldn’t.”

No one brings this point home better than Grant Cardone in his book (get the audio book – you are welcome), Be Obsessed or Be Average, or #BOBA. The followup to his (awesome) book, The 10X Rule, Be Obsessed or Be Average complements and completes Cardone’s thoughts on the hard work and time required to achieve success.

“What is the Point, Andy?”

When people make statements like “You don’t know what you are doing,” they are saying, “I gave up so you should give up, too,” or, “I didn’t get what I wanted so you don’t deserve what you want, either.”

This is very fair thinking.

When I write the word “fair” I shudder at what “fair” has come to mean and how it’s been used to justify junk and the crap it’s been used to rationalize.

Conclusion

I am not going to quit learning.
I will continue to try to make old things work better.
I will continue to try new things.
I will fail more often than I succeed (this is how I learn).
I will not stop until I go home.

My advice, encouragement, exhortation:

  • Don’t quit.
  • Make the problems give up before you do.
  • Listen to people who have succeeded (or are succeeding).
  • Do not listen to people who have given up.

I have more to learn and I know that.

Peace,
Andy

Who is Exhibiting at the PASS Summit 2018? Enterprise Data & Analytics, That’s Who!

I am honored and excited to announce that Enterprise Data & Analytics will be an exhibitor at the PASS Summit 2018!

If you browse on over to the PASS Summit Sponsors page and scroll to the Exhibitors section, you’ll find us listed:

Honored and excited – that’s me!

I see – and have lived – this virtuous cycle in the SQL Server and PASS communities:

  • A person discovers the Community and is overwhelmed at our openness and genuine willingness to help others. They realize they are not alone.
  • They learn more and become better at their jobs which, in turn, positively impacts their quality of life.
  • Some desire to give back to the community, so they develop a presentation and submit it to a User Group or SQL Saturday.
  • Some are selected to deliver their presentation.
  • Some presentations are well-received and increase the visibility of the presenter in the community.
  • As presentations are honed over time, some are used as a springboard to develop and deliver other presentations, further increasing the visibility of the presenter.
  • Some presenters achieve enough visibility to become a brand.
  • Some presenters are selected to present at larger events, like the PASS Summit.
  • Some presenters use their newfound greater visibility and brand awareness to join a consultancy practice or to become independent consultants.
  • The continued care and feeding of the brand of some consultants drives business growth.
  • The businesses of some consultants grows to the point where they can become sponsors and exhibitors at events such as User Groups, SQL Saturdays, and – eventually – the PASS Summit.

This cycle can be broken (or quashed) at any point by any number of actions, inactions, missteps, mistakes, and/or competitive overreach. In fact, I promise you will make mistakes and take missteps along the way (ask me how I know), but those mistakes and failures can tear you down or build you into more than you were – and the outcome is 100% your choice.

I advocate for the next generation of presenters. I want to see you engage, learn, share, grow, build their brands, and give back – just like I did.

Go get ’em!

:{>

PS – Need some help with your data? Contact us! We are here to help and by hiring Enterprise Data & Analytics you support some great communities!

Experience Matters

This is a picture of Kent Bradshaw, Microsoft Certified Data Scientist. Kent and I work together at Enterprise Data & Analytics delivering data engineering, business intelligence, and data-based solutions that help businesses turn their data into actionable information. We do not do this alone, we are surrounded by an excellent team that holds diverse skills, perspectives, and work and life experience.

In this post I want to focus on experience because I believe experience is the most valuable component a consultant brings to a customer.

When Experience Hurts

Experience can hurt if we allow it. We can become locked into that’s-the-way-it’s-done syndrome. There are new frameworks and methodologies introduced every day (literally). It’s too much for any one person to keep up with the latest and greatest and most shiny new stuff.

This is why a team delivers value to customers.

A diverse team – like the team at Enterprise Data & Analytics – is incredibly valuable. We’re not just a bunch of old guys, we are also Millennials. Our younger team members are constantly learning about new technologies and methodologies and bringing them the attention of our older team members. The energy our younger team members bring to our projects is inspiring and amazing, and…

One of the most rewarding parts of my job as Chief Data Engineer is watching our younger folks thrive, grow, and succeed.

I love how our team works together! I love it when we all pile on WebEx to solve a problem for a customer or design part of a data warehouse solution. It’s thrilling! I’m proud beyond words of our team.

Although we haven’t advertised it much, the team at Enterprise Data & Analytics has grown – nearly doubling in 2018 (so far).

Diversity in backgrounds, experiences, and levels of experience helps us overcome the downsides of experience.

When Experience Helps

Kent and I (and others on the Enterprise Data & Analytics team) have delivered many data-related solutions. I like to joke and say, “when we started working with data we had to carve our own chips out of wood!” That’s not entirely accurate, but I learned Motorola 6800 machine code in 1975 (I was 11) and Kent started when punch cards were the norm.

Having lots of experience gives Kent and me perspective. We understand the maturity cycles of technological, corporate-cultural, and even projects. Why? We’ve experienced a number of said cycles. When the inevitable “bumps in the road” occur, we don’t panic or lose perspective. We’ve likely experienced something very similar in the past, made mistakes attempting to recover, and learned the best way forward from those mistakes.

Do we still make mistakes? Yep. But fewer, and we have a deeper, richer well from which to draw experience-based, time-tested solutions. Perhaps most important to our customers, we have the confidence that only comes from recovering from a mistake (or several).

Plus, our more experienced team members have the privilege and honor of sharing our experience with our younger team members. Together, we learn from each other. I love that!

Our more experienced team members bring leadership and management experience to bear on each and every Enterprise Data & Analytics project. My experience leading a team of 40 ETL developers at Unisys gives me a deeper appreciation for the total cost of ownership of any solution (which can dwarf the cost of development – especially short-sighted development).

Snapping It Together

With our diverse backgrounds, experiences, and levels of experience, Enterprise Data & Analytics is able to deliver the best solutions, period.

How may our team serve you? We offer:

  • Data Warehouse Rescue – is your data warehouse project stalled? We can help.
  • SQL Server and MySQL database performance tuning, development, and support – both on-premises and in the cloud (AWS and Azure).
  • SSIS (SQL Server Integration Services) coaching, development, performance tuning, and training.
  • Biml (Business Intelligence Markup Language) coaching, development, and training.
    • Coaching is my favorite! We navigate and you fly.
  • Data science to help you visualize your data, and data engineering to prepare your data for data science and business intelligence.
  • Automation solutions including SSIS Catalog Compare, Catalog Browser, SSIS Framework Community Edition, the BimlExpress Metadata Framework, and more!
  • Our team writes the books others read to learn SSIS.

Contact us today for more information.

:{>

The Recording for Introduction to Loading the Cloud with SSIS is Now Available

Enterprise Data & Analytics

The recording for the free webinar titled Introduction to Loading the Cloud with SSIS is now available (registration required).

Abstract

The cloud is coming, ready of not. In this session, presenter and author Andy Leonard discusses and demonstrates some patterns for loading cloud destinations using SSIS.

For more information on data integration in the cloud, SSIS and Biml consulting and training, and “lift and shift” data operations, please contact us! We are here to help.

:{>

Free Webinar – Introduction to Change Detection with SSIS 15 Mar 2018

Join me 15 Mar 2018 at noon ET for Introduction to Change Detection with SSIS, another free webinar sponsored by Enterprise Data & Analytics.

Change detection is the central component for Incremental Load data integration patterns. In this presentation I discuss and demonstrate three change detection patterns:
– No Change Detection
– Brute-Force Change Detection
– Hashed Change Detection

I hope to see you there!

Register today.

:{>

Website Ch-ch-changes…

Inspired by Frank La Vigne’s (blog | @Tableteer) snappy update to the Data Driven home page, I decided to spruce up a couple / three of my websites including this one, andyleonard.blog.

I added the sidebar shown on the right of the site. It contains a more-readily-available search box and links to other websites I maintain – or co-maintain, as is the case with Data Driven , co-maintained with Frank. At Data Driven, Frank and I share interviews with people doing machine learning, artificial intelligence, data science – and working in related fields. At the time of this writing, we’ve been live for just over six months and have almost 100 shows posted. We’re honored to have over 30k downloads!

Biml Academy is an online training site that contains free and paid Biml training, like the Basic Biml Training course. Read this post in which I explain why Basic Biml Training is a good deal if you purchase it now instead of waiting (spoiler: as new content is added the price will increase but you will get access to the new content as it becomes available at no additional charge if you purchase now).

DILM Suite hosts Data Integration Lifecycle Management (DILM) tools and utilities. Most of the tools and utilities are free and many are open source. We charge for a few, like SSIS Catalog Compare and SSIS Framework Commercial and Enterprise Editions. But there’s the free and open source SSIS Framework Community Edition, and the free Catalog Browser, and a free Basic SSIS Catalog Dashboard in Power BI, and more! I updated the logo (seen here) and made a few cosmetic changes.

Enterprise Data & Analytics, to which we lovingly refer to as “EDNA” (not the character in The Incredibles… but no capes!) is a consulting firm specializing in data integration, data science, training, and independent verification and validation. We deliver data. I spiffed up the content at the EDNA site, adding verbiage, more images, and cleaning up a few visualizations.

Believe it or not, that’s not all. I have a couple other business endeavors that I enjoy. As a serial entrepreneur, I’m always thinking about the next product or service – and sometimes – the next website…

A True Story of Value vs. Hourly Rate

A little over a year ago I wrote a post titled Value. If you have a few minutes Value provides good background for this post. Unlike most of my posts about business and client interactions (I have a standing rule to not blog about stuff until at least one year has passed), the stuff I share in this post just happened.

Enterprise Data & Analytics was hired to deliver data integration with SSIS. Our point of contact is an enterprise data integration architect for a large corporation. She’s been doing this kind of work for decades and has experience with other data integration platforms.

She knows her stuff.

We are some number of months into the gig at the time of this writing. During a status call earlier today she shared the following information:

“We estimated over 1,400 hours to complete the work we sent your team at Enterprise Data & Analytics. Based on what you’ve delivered plus the latest status of work-in-progress, your team is delivering the work in 43% of the estimated time. I’ve shared this with management.”

This enterprise considered hiring offshore teams to deliver the same work. The offshore companies promised SSIS developers in the $80-$90/hour range. Our customer has some experience with offshore SSIS development and… some of that experience has been positive. Many companies are very good at selling themselves as being able to deliver based on the number of developers in their employ, and that alone.

Software development – especially complex software development – requires more than lots of warm bodies.

The data integration architect shared her team “had to spend lots more time answering questions” when working with offshore teams, due to relatively inexperienced developers. That adds to the $80-$90/hour. She shared these teams deliver in the range of 120%-200% compared to estimates.

I know $80-$90/hour sounds like a bargain but let’s do some math:

Best case: $80/hour * 120% = $96/hour (plus team support)
Worst case: $90/hour * 200% = $180/hour (plus team support)

Inexpensive developers are not as inexpensive as they appear.

At the time of this writing, our team delivers data integration with SSIS for $250/hour. If the contract is for more than 200 hours (like this one) and if we are able to negotiate favorable terms, we reduce that rate to $185/hour.

I understand $185/hour seems high but let’s do some more math:

$185/hour * 43% = $79.55/hour

We cost less! That was my point in the Value post. Here’s evidence.

And customer team support for our team? All but non-existent. We are experienced. We know how to manage our time and the time of the team working with us. We read and understand the documentation because we’ve been doing this for decades. After reading the documentation we set up an initial call with the analyst supporting our project, walk through the details, and ask 90% of our questions during that 30- to 60-minute meeting.

Expensive developers are not as expensive as they appear.

I’m glad the data integration architect did the math. I’m even happier she shared the results with us. I am happy to share this with you as you consider hiring consultants to help your team with your next data integration project.

Contact us today!