Timo Elliot [Blog | @timoelliott] wrote yesterday that BI is Dead and Julie Koesmarno [Blog | @mssqlgirl] was kind enough to tweet about it. The compelling point in Mr. Elliott’s post is:
“The most charitable view is that Gartner feels it has to exaggerate the demise of BI in order to get customers to pay attention to the changes before it’s too late.”
Mr. Elliott goes on to point out Gartner is redefining the term “BI” to apply only to what most BI folks now refer to as “self-service BI.” But Gartner also admits:
“’IT-modeled data structures… promote governance and reusability across the organization’ and ‘In many organizations, extending IT-modeled structures in an agile manner and combining them with any of the sources listed above is a core requirement’”
If you’re not confused you’re reading it wrong. I can hear you asking, “Andy, what do you think?” I’m glad you asked!
What I Think
I won’t pretend to understand the subtleties of Gartner’s logic at the time of this writing. I promise to dig more and try to get my head around what they’re saying and writing. I doubt, however, that Gartner is being wishy-washy. I think they’re trying to tell us something about a trend in Business Analytics.
What’s the Trend?
<trend>Self-service BI is gaining traction.</trend>
I’ve been in the Business Intelligence field since before I knew what it was called. When I did learn what it was called, we called it Decision Support Systems (DSS) and (in manufacturing) Manufacturing Execution Systems (MES). Later, I learned that the part of I really enjoyed was called Data Acquisition, a vital part of Supervisory Control and Data Acquisition (SCADA). Which led me to a career in Data Integration.
I’ve heard variations on this theme of “data integration is dead” for two decades. It’s not true. I believe it may one day become true, but today’s not the day.
“Why Isn’t the Report of BI’s Demise Accurate, Andy?”
Consider Grant Fritchey’s [Blog | @grfritchey] recent foray into R. Grant is, in my humble opinion, one of the smartest and most capable data minds on the planet. He’s an engineer who gets the value of analysis. Don’t take my word for it, ask anyone who’s read his books or heard Grant present. I promise I am not picking on Grant, I am using his post to make a point. And that point is this:
“Data analytics is often hard.” – Andy, circa 2016
Self-service BI is in better shape today than at any time in the past. The plethora of available tools are awesome and empower users like never before.
But…
Raw data is rarely in a format that lends itself to consumption by self-service BI tools.
I believe this is the lesson of Grant’s post.
It’s Not Just Big Data, Either
There’s a lot of buzz surrounding Big Data, and the buzz is right and proper most of the time. But most of my customers do not have data volumes that reach the threshold of Big Data. As I wrote a few years back, Little Data Remains Important and most of my customers have Complex (or Difficult, or Hard) Data – not Big Data.
Just Add Garbage…
Data Quality is important because garbage-in = garbage-out. But what you may not realize is data collected into any store for analysis is going to be aggregated. Small variations in data quality, when summed or multiplied, bend predictive analytics lines in less-than-accurate directions. So much so that – sometimes – the ratio of signal-to-noise (good quality data to bad quality data) can be north of 99% when analytics solutions become so inaccurate that they’re useless for predictive analytics. Note: this isn’t the DQ threshold for all solutions but it is for some. All data-based solutions become useless once data quality dips below some threshold, and I’d hazard a guestimate that most solutions cross that line with a good:bad ratio in the 90-something percentile.
Combined, complexity and data quality limit the effectiveness of self-service BI. It’s not the fault of the individual or the individual self-service BI tool. It’s the data. As I stated, data analytics is often hard.
Conclusion
In my opinion, Gartner is right to raise the flag about the shift towards self-service BI because this shift will impact the workloads for many IT shops. For now, there’s still plenty of data collection, master data management, integration, and cleansing work to be had; much of it in support of self-service BI tools.
:{>
Learn more:
Related Training:
IESSIS1: Immersion Event on Learning SQL Server Integration Services
IESSIS2: Immersion Event on Advanced SQL Server Integration Services
Related Articles:
Stairway to Integration Services
Stairway to Biml
Related Books 
Help!
Enterprise Data & Analytics – We are here to help.
You know I have little confidence in Gartner. I have seen them completely blow it too many times. I’ve been on calls with Gartner analysts I suspected of doing Internet searches because what they were proposing happened to match exactly the return order of the Internet searches I was doing checking their "advice."
However, I think you are right about self-service. I think it’s becoming increasingly viable. However, there is still a lot of plumbing to make self-service work. And it’s not new. If it was, people wouldn’t have been proclaiming dashboards, etc., in SharePoint for years now.
Who cares what Gartner calls it. BI is not going anywhere until such time as we bomb our civilization back to the stone age.
🙂
For my understanding, Self-service BI mainly indicates self-service reporting. ETL is a important part of BI and it is not easy for self-service. ETL is also important for big data analysis. At least ETL is not dead.
I generally rely on IT to provide me credentials for source systems and database provisioning. I then rely on the business owners to better understand the context of the source data. From there, I prefer to perform the data analytics (BI) myself. I use SQL Server or SAS for the heavy lifting and then Power Query and Power Pivot for analytical data modeling. I’ve been doing this for over 20 years and the only constant has been Excel
While all signs point to self-service BI as being white hot right now, I think we all could still benefit by recognizing its challenges as well. Most notably being concerns around trustworthyness, which may be why decision support needs can never be fully satisfied through "unofficial" self-service BI solutions alone, but rather work better when supplanted with traditional/robust BI reporting solutions. While self-service may bring speed and empowerment to business users, the analysis done using these solutions are "unofficial" in nature. Instead, self-service reporting serves best as a starting point to guide direction for analysis. Furthermore, any comprehensive self-service solution would also be likely to heavily depend on back-end data models that are developed and optimized by…you guessed it…a conventional and technical BI/DW developer.