And up through the ground came a bubblin' crude

Anyone not convinced that “doing BI” is both very hard and very expensive need only consult this link.  Yes it’s from 2006, but have things really changed since then? Probably the opposite.  In his postings, SSAS guru Dave Rodabaugh describes the hiring process for BI/DW architects in a really compelling five-part story.  He starts off by stating “I admit to being a real hammer when conducting an interview.” – Quite honestly, I don’t blame him given these folks are charging anywhere north of $250/hr these days.  At that price, you _better_ damn well know your stuff!

Here are a couple other posts on the trials and tribulations of hiring good DBAs: http://tinyurl.com/ct75os and http://tinyurl.com/dksywb

And finally, here’s what I thought was an interesting set of SQL Server interview questions (understanding them alone is challenging).

What’s the point of all this?  DW/BI is indeed rocket science.  Its High Priests are DBA and DW Architects. And with time, they become irreplaceable in the enterprise.  If a programmer, a software architect, or even an enterprise architect makes a mistake, you can usually catch it in time and if not, there is usually an opportunity to fix the problem downstream – it’s like losing one of several engines in flight.  It’s not fun, it can get bumpy, but chances are no one is going to die.  On the other hand, a database BI/DW management mistake can be fatal because data can get whacked, irreversibly.   Or it can get stolen, or corrupted for example, with dramatic legal repercussions for the business.  This is like an explosion in flight.  It’s not something you can fix in time, and survival is not likely.  That’s why competent DBA and DW architects rule the BI/DW world.  If you want to see what these guys really do for a living and why they commend the big bucks, check out the following for starters:

http://dbscience.blogspot.com/

http://prodlife.wordpress.com/

http://www.fulltablescan.com/

http://jesseorosz.spaces.live.com/

http://optimaldba.blogspot.com/

http://www.petefinnigan.com/weblog/entries/

http://www.sqlservercentral.com/

So I have two questions about the existing state of BI.  First, why did it get so complicated and second, is it good or bad for business? 

I think the first question is simple enough.  Things are like this because the darn products are simply too hard to use.  You shouldn’t need a PhD to properly setup, configure, deploy, tune and question a database engine.  Forgive me for liking simplicity, but last I checked, operations manuals for behemoths like DB2, Oracle and SQL Server were in the thousands of pages combined.  To me, when a set of product manuals have more pages than the Federal Budget, that’s cause for concern.  I’m an avid follower of the KISS principle.

Recently I purchased this book which is essentially a Microsoft SQL Server Analysis “bible” covering every part of that ecosystem including SSAS, SSIS, SSRS and Excel.  It runs 624 pages.  And quite honestly, unless you have significant mastery of each subject matter, you’re not going to be very effective doing BI at the enterprise level (at least on the Microsoft stack, but the others are typically hairier anyway, let’s face it).  In my opinion, this book represents the _minimum_ one needs to know to be effective in this business.  And believe me that’s not for the casual weekend DBA.

The second question is a bit more subtle.  How do you define “bad for business”?  For the past forty years, business leaders (CIOs) and bean counters (CFOs) have computed the total costs of owning and exploiting these databases (salaries, hardware, software, licenses, power, time wasted, failure rates etc.) and determined that the rewards justified the costs.  But do they really?  In light of the well-documented failure rates among business intelligence projects in the past years, can they possibly be right?  It isn’t too challenging to pull up a myriad of articles such as these decrying and documenting the dismal success rates of BI project all over the planet: http://tinyurl.com/cr8lt8 or http://tinyurl.com/cnc36h

And in some cases, you also need a PhD to even figure out _how_ to calculate ROI on a BI endeavor as this 2002 44-slide PowerPoint presentation from Jonathan Wu will attest to.

So from the looks of it, countless millions of dollars and hours of intellectual effort have been spent in the past decades achieving mostly failure.  The same argument could be made (and has often times) about software development in general, but it looks like BI is faring even worse!  When’s the last time you heard about a successful enterprise BI project that wasn’t over budget, over schedule or over-complicated?  We know what happens when complexity is used to hide questionable assumptions, processes and results.  Ask anyone at AIG. If it’s too complex, too obfuscated, and too inaccessible, then chances are it’s bad business. 

And so it is in light of this that I have recently started to think differently about this “cloud” offering in the BI world sometimes referred to as BaaS, BIaaS or even DaaS.   Having been in software technology for twenty years, I’ve seen my share “new improved”, world-changing, and “revolutionary” concepts come and go.   But people are missing the point about this one as it applies to BI.  

The BI “cloud” isn’t about technology, delivery, security or governance hurdles per say. It’s about abstraction.  And abstraction is the antidote to complexity. This is why I feel many players in today’s BI world are threatened by this concept, and why there’s so much gold in them thar hills when applied to enterprise business intelligence in judicious ways.