Baking MapReduce into Database Engines - Worth the Reduction Sauce?

MapReduce (implementations include Hadoop and CloudDB) has gained popularity in the industry. It also serves as marketing fodder for several new-breed ADBMS vendors who now claim to support it in various forms. So what is really behind this magic pixel dust, what problems does it solve, and how relevant is it to someone deciding on a new (or additional) ADBMS platform these days?

First, let’s point out MapReduce is not a technology but an algorithm. Wikipedia defines an algorithm as “an effective method for solving a problem using a finite sequence of instructions.” In MapReduce’s case, the problem being solved is the processing and analysis of very large data sets. The solution is a parallelized divide-and-conquer approach and works like this. First, you split up the “problem” into small manageable chunks. Second, you fan out each chunk in parallel to individual “work units” (maps). Third, you take individual results from each unit and recombine them into your final result (reducers). In SQL parlance, conceptually, it’s like doing a select aggregate with a group by.

There are file based and database-centric applications of MapReduce in existence. Of course, the presumption is that your “problem space” can be split up in distinct pieces and recombined without information loss. And not all problems are either large enough or mathematically suited to this approach. But luckily data management is, by definition, perfectly well suited because, as a mathematician once told me “every data management task can be broken down into two and only two activities: partitioning and equivalence”.

Some folks think MapReduce is a modern breakthrough concept, but they’re wrong. The application of this algorithm to the management of large data is nothing new, as pointed out by Dr. Stonebraker in a 2008 posting. What’s “new” about MapReduce is that Google has popularized it. And the thought is, if Google can process and analyze the entire world via MapReduce, then clearly MapReduce must be the Holy Grail of monster data management. But Google has unique challenges (gargantuan data volumes), and some very impressive (and plentiful) gray matter at its disposal.

Because the interesting thing about this “divide and conquer” approach is that, although fairly easy to conceptualize, it’s incredibly hard to implement properly. The human brain is really not “wired” to think in parallel. Research has shown that the top brains can at best juggle seven different objects simultaneously (and for short time periods). To understand the intellectual challenges at play here, I strongly recommend watching this Google video series.

As I understand it, implementing MapReduce correctly and efficiently is probably as hard as conquering multi-threaded programming. And in twenty years, I have met three people who really understood multi-threading correctly and two of them were Russian PhDs. I've had battle-tested architects tell me they would rather shave with broken glass than tackle the risk and difficulty of multi-threading (luckily, they weren’t designing operating or flight-control systems!). My point is, it takes some pretty special skill and talent to do it right. Nothing inherently wrong with that, but it’s neither quick nor cheap.

So why then would database vendors race to support MapReduce? After all, dealing with and managing relational systems is complicated enough as is. But at least people have been trained in the art for decades, and SQL is lingua franca. So the pitfalls and solutions are well established. Additionally, Codd’s premise guaranteed abstraction by separating the logical layer (SQL and a normalized schema) from the physical one (hardware and storage). But MR is heavy with non-standard cross-layer implementation details by necessity. Clearly a step backward from the KISS principle (even a “major step backwards” if you buy into Dr. Stonebraker’s argument that MapReduce is offensive).

Regardless, three well-known new-breeders, namely Aster Data, Vertica and Greenplum jumped on the bandwagon early on and announced “MapReduce implementations” for their product. I wondered what compelled them to invest time and resources into something that didn’t seem essential (or cheap) to the market at large. Are users really clamoring for MapReduce support in their warehouse engines?

To learn more, I went to YouTube and checked out Aster’s video “In Database MapReduce Applications”. In it, I learned that graph theory problems (think: travelling salesman) were well suited to MapReduce but not SQL. Examples included social networking (LinkedIn), Government (intelligence), Telecom (routing statistic), and retail (CRM, affinities), and finance (risk, fraud). Pretty much anything that can be modeled using interconnected nodes. But a connection from a node to another is really a “relation”, and so clearly well suited to a “relational engine”. So I might have missed something.

I also learned that existing applications typically extracted data from the database, performed some analytic work on it, and then pushed the data back into the store. In other words, they couldn’t perform processing inside the database. I found that generalization hard to swallow but reminiscent of numerous past battles on whether “business logic” belongs in the application or database layer.

Aster’s implementation of MapReduce is “deep inside” their engine, from what I understand. One example I could find was yet another YouTube video called “In-Database MapReduce Example: Sessionize”. In it, Shawn Kung shows a MapReduce function being used inside a SQL statement to “sessionize” user IDs in a clickstream context. Aster also provides very basic how-to’s on their website and blog. Clearly Aster is targeting this new MapReduce capability at the DBA side of their users, and it looks a lot like leveraging UDFs to me. Aster’s conclusion: “we need to think beyond conventional databases.” I’m all for that!

Next, I wanted to learn about Vertica’s implementation. Especially since Vertica’s own Dr. Stonebraker had initially nailed MapReduce pretty hard as mentioned above. But Vertica’s new position seems to be that MapReduce is a-ok after all, provided it remains external and doesn't pollute the purity of the relational engine. I couldn't find much on YouTube or their website save for a press release dated 8/4/09 stating “With version 3.5, Vertica also introduces native support for MapReduce via connectivity to the standard Hadoop framework”. It seems the “scoop” on the Vertica/MapReduce wedding is best described in their corporate blog. Basically Vertica is OK with "integrating" or connecting to but not "ingesting" MapReduce (via Hadoop) if I understand clearly.

I was also able to glean some tidbits from Omer Trajman on Twitter. Namely that Vertica supports Hadoop “adapters” which allow you to read and write into the database (which is basically the press release). I wish I had more in-depth information about Vertica’s MR functionality but even a basic search for the term on their overly busy website yields zero information and, unless I missed it, I couldn’t find any relevant webcasts either.

Greenplum were, if I am not mistaken, first to support MapReduce. Greenplum has the best MR resource online if you ask me. It’s clear, detailed and full of insight. Greenplum has a merged/cooperative DBA/programmer approach in their offering. Programmers can write maps and reducers in their language of choice, leveraging DBA generated data sets as (and if) needed, and DBAs can use MR functions along with SQL without (presumably) getting their hands dirty. There isn’t much to add to this excellent resource so I won’t.

So having mapped out all these facts, what can we reduce from it (I’m so funny) and more importantly, should any of this stuff matter to prospects when evaluating ADBMS vendors? IMHO, you might benefit from a MR-enabled ADBMS if:

(1) You have petabytes (or more) of data, an MPP architecture, and a search, scientific research, or mining problem a high-performance SQL engine cannot handle.

(2) You don’t have heavy legacy systems. Integrating (or migrating) existing business and relational code with a new-breed MR-enabled engine can’t be fun, quick or cheap. You might be one of the lucky few with pet projects on the table.

(3) You’re in Academia and have access to numerous cheap and competent programming resources, lots of metal, plenty of time, and limited pressure to succeed.

(4) Your organization has a track record of successful projects dependant on symbiotic working relationships between your DBAs and your programmers. In my experience, DBAs and programmers don’t work well together. They have different goals and approaches. And it seems intellectual and political integration of both resources would be a sine qua non condition to success with an MR database product.

Short of that, I can’t imagine too many people lining up at the MR-ADBMS vendors’ doors simply based on their MapReduce capabilities. And I don’t think vendors make that case either. In my opinion, supporting MR in the product simply says “Hey, look at me, I’m at the forefront of technology. See how smart I am.” But as a buyer, I’d be a little concerned about overreach.

In fact, I wonder how these vendors spread resources efficiently (and economically!) between database engine building, cloud provisioning (which Aster and Vertica now pitch), and MapReduce integration. I suppose marketing requires less focus than engineering as a discipline but still, that’s a lot on one’s plate.

SELECT SUM(blessings) FROM working_in_bi

If you’re working in the business intelligence industry, you should really count your blessings.  Mind you, you can do that if you’re employed anywhere these days.  But there’s something special about what I call the “BI Family”.  I’m not referring to a specific segment of BI, and I am not focusing on any particular job function. I’m talking about working in the BI industry as a whole.

I feel somewhat qualified to comment on this because, in the past twenty years, I’ve worked in numerous industries.  To name a few: life sciences, research, semi-conductor, telecom, payroll, accounting, systems integration, consulting, audio-visual, online media, financials, accounting, and insurance.  So I have a lot of background to compare from.  And believe me, there are worse adoption outcomes than membership in the “BI Family”.  Here’s my subjective top ten list of why working in this industry is really cool (no specific order).

#1. Brains

People are really smart. I’m not suggesting other industries spawn dummies, but the proportion of high IQs in the BI world always amazes me.  It’s often humbling and always stimulating.

#2. Strength

As we all know, the BI market is not only huge, but getting larger and growing yearly to the tune of 8-10% a pop. Other industries are not so healthy to say the least.   Simply put, BI is clearly not a fad, and no one questions its future.  It’s one of the fuels of our free-market system by protecting businesses and providing them with competitive tools.

 #3. Meaning

Guy Kawasaki says: “make meaning”.  He’s right.  Life’s too short to be in a meaningless industry.  And if BI isn’t about making meaning then I don’t know what is.  The whole purpose of the industry is to make meaning and support critical decision making.  This industry yields real-life significant solutions to crucial sectors like health, research, medicine, and defense (to name a very few).  

#4. Quality of life

In light of ominous HR predictions, news of recurring layoffs, and current employment trends in the IT industry, the BI sector has been relatively spared. Clearly I haven’t done any formal polling but I get the “vibe” that people are generally pretty happy to be in this game.  And why not. Compensation is generally good.  And location-wise, BI companies are clustered around national funding clusters namely Northern California and the New York/Boston areas.  These comprise some of the most magnificent (and most expensive, granted) landscapes and vibrant urban areas in the country.  Other industries have centers in, shall we say, less compelling geographic areas.

#5. Funding

I lamented the VC situation in my previous post, and clearly this doesn’t apply to all segments of BI, but if you have any sort of compelling BI proposition with the word “cloud” in your business plan, trust me you will get a VC’s attention.  Maybe not an official invite to pitch in person, but most likely a phone call.  In other industries, that opportunity is long gone.

#6. Gratification

BI projects used to take many months (sometimes years) to implement (when they even got completed).  But nowadays the industry is in “agile” mode.  And those who don’t embrace that won’t likely be in this business much longer.  This means you get to build solutions and see results quickly.  That’s gratifying. 

#7. Globalism

BI is world-wide.  True, so are most other industries, but from my experience, there is less of an “us versus them” attitude.  It has a fraternal feel to it.  Hands and minds seamlessly reach across continents in ways I have not experienced elsewhere. (I’ll go hug a tree now).

#8. Bozos

Most people are genuinely nice and unassuming.  I know this sounds naïve at best but it’s true.  I have not seen the level of ego, axe grinding, or personal animosity frequent in other industries.  Every contact I’ve initiated from top analysts to CEOs in this industry has been followed up promptly with courteous, genuine and insightful discussion. I’ve found most people to be more generous with their time and advice than in many other industries. Maybe they fear less for their jobs or fancy titles. In either case, the BI industry is fairly low on the bozo scale.

#9. Passion

People in BI are passionate about their field.  I’m not saying they get out of bed every morning to go save the world (onward BI soldiers), but overall they value their work and their contribution.  Most people I’ve met in this business are workaholics.  They know their stuff inside-out and boy do they love to talk about it.  Passion signals a great, vibrant industry.  Additionally (and this key), there seems to be better customer advocacy in this industry than others.  It’s not perfect, but vendors often do listen and react accordingly.

#10. Innovation

I hate to reveal this well-kept secret (don’t tell anyone!), but there isn’t a lot of desire to innovate in the financial, accounting or insurance fields, for example.  I’d be preaching to the choir by pointing out the myriad of new-breed ADBMS players out there, but also the multitude of new OLAP, data mining and analysis products, approaches and new (non-relational) ways of looking at data, cloud BI, EC2, etc.  We’ve seen orders of magnitude of both hardware and software innovation in the BI world.  It is a rich intellectual field teaming with innovation levels typical of a “new frontier” because it is.

So what’s the point of this apologist diatribe?  Just to remind people in this field to count their blessings. There are many worse places and industries to be in.  And it’s easy to take things for granted in the heat and excitement of daily business life.     

In the past eighteen months I’ve met many challenges in this business.  From coding to QA, to technical writing, from sales engineering to evangelism, from product management to market analysis. You name it.  So I’ve seen a lot of the facets in a very intense, very short amount of time.  

And I’ve also been lucky to interact with numerous players in the industry, many of which have generously spent time and resources supporting my self-education efforts with their insight, connections, and advice.  You guys (and gals) know who you are and I thank you for the help. There are many mensches in this business.

For the first time in my life, I think I can say I’ve found a home here in the BI industry.  I’ve never felt this way in the past twenty years, and I’m not exactly sure how to explain it, but like the old pair of shoes my wife keeps insisting I ditch, it just feels right and I’d like to keep it that way.

Please Stop Making More ADBMS Sausage

If you’re thinking about building a new startup in the high-performance analytical database (ADBMS) market, hat’s off to you: kudos and respect my brother. I’ve been in the kitchen, and I’ve seen the sausage being made. But let me tell you something: you might be a day late and a dollar short to the party.

In the past years, I’ve often pondered where the new-breed high-performance analytical database industry was headed. Will the existing players manage to survive? And is there a chance in hell for new ones to succeed in this market? If you had asked me to peek into my newly minted BI crystal ball in early 2009, I would have said “no way”. Why? Because at the time I was predicting the demise of a majority of the twelve or so players in this space based on the observation that the field was too crowded and too expensive. I figured, in this cut-throat competitive space, and with tough economic times ahead, we’d be lucky to see two, maybe three survivors come 2010.

Since then, not only have several other significant players, technologies and business models popped up (for example, Groovy, XtremeData, VectorWise, Hadoop/MR and the OSS guys), but we have clearly not seen the level of attrition I was anticipating. Nobody has (officially) gone out of business save Dataupia as best I can tell, and Datallegro got a check from Steve Ballmer. Sure, some folks are experiencing tougher times than others (I dare say several are hanging by a thread) but overall, resilience has been the name of the game. So what gives?

From my point of view, there are two types of new-breeders really: those living in “comfortably numb” mode (quietly outliving peers may not be a bad strategy these days), and those kicking it into high-gear with a vengeance. In the latter category I can’t help but think of Vertica, Netezza, Aster Data and ParAccel. It takes a lot of “cojones“, cash and luck to build a new ADBMS company. But even blessed with all these, and given the proper planetary alignments, I would advise anyone considering a start from scratch nowadays to ponder the following points.

First, it is insanely expensive and complex to develop systems software to produce a complete analytical database engine (and I mean “complete” in a holistic Product Management sense). Tony Bain highlights some of the database startups challenges in his excellent series starting here (things are not significantly different between OLTP and OLAP in this respect). But systems software is a different animal than your run-of-the-mill corporate enterprise application. On average you’re looking at 200,000 man-hours and anywhere from $60M to $100M to fund such a venture to completion. This is just to get rolling. Additionally, you cannot just put out a database product and call it a day.

Maintaining (enhancing) a behemoth of Oracle or SQL Server stature runs hundreds of millions of dollars every single year. Everything from equipment to talent costs more when developing database software. Believe you me, the folks who will work on your SQL optimizer, inter-fabric communications, parallel or compression schemes better not be affordable newbies. Your development platforms won’t likely be the average oh-hum laptop attached to cheap storage. An efficient QA or Performance Group will cost a small fortune in payroll and redundant equipment. And seasoned performance architects don’t run the streets. You cannot assemble a database engine product by cobbling together open source bits and distributed talent like a new Web 2.0 RIA venture. You can’t take a Kia (even a dozen of them) to the Indy 500.

Second, I think it’s no longer possible to find sufficient levels of Venture Capital funding for such endeavors. My feelings on this issue are re-enforced when I read articles like this, or this. I think the writing was on the wall for several years now. Reports of the VC’s demise are greatly exaggerated, but the funds, the endurance, and the risk acceptance levels are gone. Small bets on small returns are in. Large bets on IPO-driven returns are out (for now). Even if you manage to score a major industry name like Mike Stonebraker on your Board, I think VCs in this space (those that are left and not now engaged in M&A) will say “talk to the hand”. Investors in numerous existing new-breeders are biting their nails to the bone (or looking for ways out). So to me, the train has left the station. And unless you can pony up your own seed money, trying to fund such a project via institutional money is currently, in my opinion, an exercise in futility.

Third, the field is already too crowded and spread out very thin. For a great overview of the major players out there, don’t miss Bloor Research’s competitive analysis paper. A lot of existing players do not have sufficient “boots on the ground” to make headway against larger ones, much less established powerhouses. Heck, even going against Vertica’s deep-pocketed marketing is no piece of cake. Worse yet, in this business, success is not guaranteed by technical superiority. I know it sounds heretic saying this about an industry dominated by performance claim testosterone, but it’s true.

Besides technical prowess, you need to get the word out louder than everyone else. Unfortunately, everyone has the same “word”. In a crowded space this means you have to yell “Fire!” pretty darn loud and relentlessly to get noticed. I think a lot of “database deals” are sealed on the golf course, more so than POC or bake-offs. Mind you, this is probably the case with most enterprise software. But to get your foot in the door, you need BOD and Primadona action. BOD are well-connected heavy hitters on your board. Primadonas are the star sales guys (or gals) currently working for your competition. Those you’ll have to poach with sweetheart deals to come work for you, a totally unproven new-breeder with a year of runway to go.

Fourth, pricing pressure in this business is relentlessly choking. This is a consequence of my previous point. A little over a year ago, word on the street was $100K/TB retail ($50K/TB street price) but now we’re seeing $20K/TB retail (TwinFin land), which probably means you can do $10K/TB on the street. Aster Data is pitching an appliance for $50K (1TB, includes Dell hardware), and Oracle’s new improved Exadata V2 (SATA storage) even touts $5,700/TB so I mean, at these margins, you’re basically talking about giving stuff away, and in a lot of cases, I suspect that’s what’s going on. So unless you’ve been around the block a bit and have some ammo in the bank, I don’t know how a newcomer can sustain this type of pricing “carpet bombing”. As if that weren’t enough, you have OSS and cloud players breathing down your neck. Customers expect more for less and perception of BI as a “commodity” is growing. In this pricing environment, survivability for a newbie is improbable at best.

Fifth, several windows of “technology opportunity” for ADBMS are closing. For example, if your great idea for a new ADBMS company involves a columnar approach, you might be too late to the party. If your “innovation” hinges on massive parallelism, compression, in-memory caching/cubing schemes, super-fast intra-nodal fabrics, hybrid MPP/SMP, hybrid row-column storage (PAX-like), or yet another SQL chip accelerator or super-duper FPGA, you might have missed the boat (on the other hand, if you figured out how to do analytics on compressed encrypted data, then you might be on to something).

I believe the top new-breeders did all the technical legwork in the past 4-5 years. It took Mike Stonebraker long enough, but Vertica pretty much put columnar on the map. And most of that engineering is now mature enough (and well proven) to warrant acquisition interest from the big boys. Initially, the big guys took a “wait and see” attitude (remember, OLTP butters their bread anyway, not analytical OLAP) but now, having seen results and traction on others’ dime, I think they’re ready to pony up some cash (classic buy vs. build decision) to absorb the bits and pieces suiting their marketing strategies. By doing so, the good ole boys re-invent themselves and say “hey look, we have columnar technology as well now!” (How Sybase didn’t corner this market with IQ is beyond me, especially having read Seth Grime’s excellent paper about it).

Better yet, by integrating new technologies into existing code bases, the big dogs can say “hey, we have the best of both worlds for OLTP and OLAP” (Oracle’s latest Exadata comes to mind). And perhaps “look, we have SMP on the processing side and MPP in storage layer”, or vice-versa, thereby returning to the old “one-size-fits-all” GP-RDBMS paradigm so criticized by Stonebraker (but so convenient for the corporate user).

And clearly, given the growing popularity of “operational analytics”, an OLTP+OLAP offering is compelling. So I think the “proof-of-concept” window for many new-breed technologies, specifically MPP columnar (but others as well, for instance, acceleration hardware, where Ingres is picking up VectorWise, or MPP where Microsoft snapped up Datallegro), has closed. The winners (and their results) are in and acquirers will likely make their move in 2010. This dovetails nicely with a recent TDWI survey claiming half the respondents plan to replace their DW platform between 2010 and 2012 (apparently this is presented here on October 7^th).

All this being said, is it possible that a brand new software endeavor currently in stealth-mode development in Nepal might suddenly dominate the analytical database scene within months? How about a revolutionary FPGA/SQL Chip/Flash/Optical hardware contraption that could blow the hinges off industry standards and benchmarks? Sure why not. Real innovation is usually unexpected, and often unintended. But I don’t see it being driven by the classic “VC funds startup makes big database scores big IPO” model much longer.

When I look at things like Hadoop and current developments in the OSS space for VLDB analytics, I still have trouble grasping the business model, but I clearly see “life force” innovation at work here. A year ago I would never have expected a place like Visa to stray from the “Big Threes” but nowadays these guys are messing with Hadoop! People are also doing amazing things with MapReduce implementations and BigTable KV types of massive data storage systems.

How does open source fare against the five points mentioned above? Pretty darn well if you ask me. Costs are significantly lower, venture capital is not needed or minimal, engineering is crowd-sourced, there’s more breathing room, market entry is viral and massive, distribution and testing self-fueled, and pricing (or lack thereof) better controlled.

Additionally, open source seems shielded from the “Borg Effect”. I don’t see how massive proprietary shops like Oracle, IBM or Microsoft can successfully “absorb” these entities. I don’t think Larry has a clue what to do with MySQL. He can’t really unload it, but he can’t really integrate it either. Darn Trojan horse! In 2008, Infobright went Open Source and raised $10M in the process. Looking at the results, I think these guys were smart!

So if you’re thinking about building yet another high-performance analytical database engine the classical way (and not going OSS), my advice to you is simple: unless you have $60M in the bank and technology significant (and new) enough to impress people like Daniel Abadi (good luck on that one), you might be climbing up a greased pole. I'm not saying it's impossible mind you, but there’s been a lot of cooks making the same sausage over the last five to six years to last us a while. Maybe it's time to look at the next curve.

Vaya Con Dios - The Day I left XSPRADA

Effective today, I will no longer be working at XSPRADA. There, I said it. Catharsis, take me away! It was definitely a tough decision but this is par for the course in the startup world, and God knows I’ve been there done that, but in this case, my close long-time personal relationship with the founders and unwavering worship of the technology for the past ten years make this particularly bittersweet.

I was very lucky to work with some of the best, most resilient people this industry has to offer and I don’t claim this lightly. As you know, I am parsimonious with compliments. But "they" say that tough times never last, only tough people do. This is why I am still convinced that XSPRADA technology will someday take its proper place in the world come what may.

A failed endeavor is only one during which you have learned nothing. And in this case, the XSPRADA opportunity has enriched me in personal and professional ways beyond my wildest expectations. So I depart a richer, more experienced man for it. In an industry too often clouded by “smoke and mirrors”, I am always reminded of the following advice I received a while back:

1. Don't profess about things you are not sufficiently familiar with.
2. Don't assume something isn't true that might be, or is that isn't
3. Listen at least twice as long as you talk.
4. Ask twice as many questions as you answer and LISTEN to the answers.
5. When asked about something and you don't know, say "I don't know."
6. If there is even the slightest possibility that you could be wrong, acknowledge it....
7. and don't forget to thank people for their time and advice.

You combine that advice with the one found here and believe you me you’ve got yourself one kick-ass sales engineer there. They don’t run the streets. I’d say these points could (or should) constitute the Seven Commandments of the Sales Engineer (and probably any other profession, except perhaps politician). They were given to me when I started by Chris Piedmonte, founder of XSPRADA, and a guy whose courage, integrity and technical brilliance are, in my book, without equal.

Be that as it may, I must soldier on. Naturally, I will not be able to discuss topics pertaining to XSPRADA technology as an insider from now on, but Lord knows there is sufficient ADBMS/BI material out there that’s interesting enough to cover and discuss on a regular basis. Namely, the new “PAX Analytica” movement as originally brought up by Curt Monash and professionally laid-out by Daniel Abadi as usual in his excellent post.

Also, the old one-size fits all (OLAP+OLTP+whatever) versus dedicated engines (columnar OLAP) has been revived with the recent ORCL announcement touting the new improved Exadata V2 (exit HP, enter Sun). This deserves addressing in more detail. It leads to serious implications for current and potential customers.

Relevant as well is a discussion about the future. Will the “big boys” end up swallowing “new-breed” technology and integrating it (what I call the Borg effect, as discussed on Daniel Lemire's excellent post). Or will they become obsolete allowing the new-breeders to survive long-term as independent replacement entities?

And then there’s a recent thread about analytical speed which is very relevant at this point in time I believe. Finally, a little bird told me there’s about to be some really interesting rumble (again) pertaining to the infamous (or not, depending on which side you’re on) TPC organization. Indeed, there’s no lack of interesting topics out there!

Additionally, I think there’s a compelling story behind employment search and provisioning in the BI industry, so I’ll be penning some thoughts about that as I go along. In my experience, you can infer a lot about an industry’s state by checking its recruiting culture and pulse. Joy Chen claims in a recent blog posting that 54% of workers plan to resign after the recession. If this prediction is correct, the impact on our industry is sure to be felt and that, IMHO, is worth discussing.

So what’s really happening behind the employment scene in BI? I’ll be sharing some thoughts about that as I embark on the new path the BI Gods have charted for me. So thanks for sticking around, and as they say where I come from (well, ok maybe a little further South) Vaya con Dios!

Mole Whackers Need not Apply

Two completely different events caught my attention lately. One of them is a post by Curt Monash called Bottleneck Whack-A-Mole, and the other is the much-publicized alliance for “BI in the Cloud” comprising RightScale, Talend, Jaspersoft and Vertica.

In the post, Curt describes software development (or developing a good software product) as “a process of incremental improvement”. Fair enough. The analogy he draws is between constantly fixing and improving performance bottlenecks and the annoying (if entertaining) arcade game of Whack-A-Mole where you have to be fast enough to clobber enough of the critters as they randomly pop up from below. He then makes the point that “Improving performance in, for example, a database management system has a lot in common with Whack-A-Mole.” Having spent most of my life designing, developing, improving and testing commercial and enterprise software applications, I have to say I don’t totally agree with his analogy for several reasons.

First, call me old-fashioned, but I’m an ardent believer in the fact that software building is deterministic. Whack-A-Mole engineering is not. The age-old controversy about software being more of an art than a science may never be resolved, but at the end of the day, I feel software is (should be) a scientific, engineering-driven, deterministic endeavor like any other engineering discipline. With Whack-A-Mole engineering, buildings and airplanes fall to the ground. That’s not good. In my experience, those who seek to “romanticize” software engineering are typically adverse to proper planning, design and testing as being too “dry” or unworthy an endeavor. That’s nonsense.

Second, there is a distinct difference in the way you develop “regular” software from “system software” and I’ve learned this from sitting in the front row the past several years at XSPRADA watching database software being built from the ground up. It’s a little bit like the difference between building a tree house and a major commercial skyscraper. And I believe that playing Whack-A-Mole games while trying to bring up a building is a scary proposition at best (especially for future tenants). And yet, the example Curt provides involves Oracle’s Exadata, of all products! He states: “When I spoke to Oracle’s development managers last fall, they didn’t really know how many development iterations would be needed to get the product truly unclogged” – This statement is mind-boggling to me.

Because for one thing, it suggests that Exadata is “clogged” (ouch) but worse, that their engineering people have no clue as to how they might eventually (if ever) snake the blockages out of it! So, basically it’s a trial and error approach to building a database. Notwithstanding their “professed optimism” that it wouldn’t take “many iterations at all” to finally figure things out, it certainly doesn’t give me (or any reasonable person) a warm feeling about a multi-million dollar product claiming to be the world's ultimate analytical machine.

I think there’s a lot to be said for sound engineering practices, proper planning and testing, setting expectations and deterministic engineering management practices in the world of system software. That Oracle (or Netezza for that matter, also referenced in the post) might just be going along whacking moles instead is a scary proposition indeed. Even if this little game is limited to “performance engineering” as Curt suggest (as if there was a more important endeavor in an ADBMS), that’s a serious allegation in my book. I say leave the arcade games to the kids, and let the real engineers design and implement database and system software please. There’s no room for amateurs in this game.

On to my next point of interest: the new Gang of Four in the Cloud (with apologies to design pattern aficionados) comprising RightScale, Talend, Vertica and Jaspersoft have recently promoted and demonstrated a “bundled” on-demand package for the cloud. I attended their webcast yesterday and was impressed, but with reservations.

Each of these vendors is impressive on its own, no doubt about it. But it seems to me the bundled proposition might be confusing at best to the unwary customer. This new offering is billed by the marketing folks as “Instant BI, just add water” which drives me nuts. Look, it might be simple in theory, and it might take a few minutes to setup the stack on your own (as Yves de Montcheuil from Talend claims) but it’s still a long way to actually accomplishing anything serious in a few simple clicks. Sorry, not going to happen anytime soon.

You still have to work your way through provisioning and instance management (RightScale), data integration and loading (Talend), feeding and configuring the database (Vertica), and setting up the reports/analytics you might need (Jaspersoft). All of which can be accomplished just as easily (or not) internally by the way. It’s true you’d still have to purchase or license Vertica internally, which may or may not match the SaaS pricing I’m not sure (and either way, Vertica has a SaaS offering as well) but the other components are open source so, I’m not sure I see the big advantage there.

An interesting thing I noticed as well is that some people didn’t seem to understand what RightScale’s role was in the whole offering. This tells me they don’t really grasp the intricacies of “the cloud” – because instance and infrastructure management for enterprise in the cloud is not trivial and you do need something like RightScale to grease the wheels (it’s an abstraction layer really), but I think many people assume moving to the cloud is “magic” and makes all these issues disappear. If that were the case, you wouldn’t need RightScale in the mix. Beware undermanaging expectations I'd say.

Additionally, the pricing model (which is supposed to be so much simpler in the cloud) is confusing at best as each vendor has its own menu. The best answer to that I can remember was “starting at $1,700 per month” – I’m not sure what to make of that. So I think from an engineering/technical standpoint, this endeavor is noble, but from a “let’s make things simpler and transparent for the user” perspective, there’s still a lot of work to be done. In other words, it's a nice play for the vendors holding hands, but I'm not sure how beneficial it might be to the average enterprise user.

As usual, caveat emptor – Beware promises of a holy grail in BI as there is no such thing. It’s all about work. Hard, detailed and careful work with proper planning and budgeting. In that respect, setting up successful BI solutions is a lot like running and implementing software projects. There are no shortcuts, and it’s not a job for mole whackers.

Oh yeah? Well my database is SMALLER than your database!

Contrary to popular edict, smaller is not always better, unless of course you’re talking about analytical database engines. In that respect, it’s hard to find an ADBMS that can fit on hard media like a CD or a USB stick. For example, I don’t think SQL Server, Oracle, DB2, Greenplum, Aster, ParAccel, or the myriad of other ADBMS vendors can fit all their bits in a tight spot. Even in the open source realm, I doubt you can wedge InfoBright (MySQL) or IceBreaker (Ingres) onto a stick, much less shlep their bits around as an email attachment.

One exception to this is the V-stick from Vertica. When I first read about this, I initially thought it was a hoax but apparently not. It’s pretty cool too because it includes the O/S, web server, GUI and the engine all together on a 16GB thumb drive. How an engine like Vertica, designed around distributed MPP, can possibly operate representatively (using terabyte-size data) on a thumb drive is beyond me, and I’ve never heard of anyone actually using this gizmo but I’d sure love to get my hands on one and review it if it’s still available.

The other exception of course is RDM/x, the XSPRADA database engine. The reason is simple: its total deployment footprint is around 10MB. That includes the 32/64 ODBC drivers and a couple DLLs. The engine itself is currently around 6MB. Last I looked the installer clocked in at 16,760KB. This means you can actually deploy RDM/x onto a memory stick if you want to. I tried it, it works. It’s pretty cool. But after a while I wondered, why would anyone care about this?

The reason is two-fold. First, it’s really easy to try out software that is small and self-contained without expanding large amounts of time and resources. Yes, you can download RDM/x from our website but in many cases (like secured firewalled enterprises), that’s not an option.

Second, it means we’re a good candidate for embedded applications. Because if I can fit my database engine on a stick (or in an email), I can probably embed it in instruments and devices as well either as raw C++ code or libraries.

But for quick POCs, size and simplicity really does matter. Say you’re suddenly tasked with evaluating solutions to deploy a BI solution inside your company. Suppose you’re a Microsoft shop. Suppose additional capex is not an option, and suppose further you have a week to show results (namely a set of nicely formatted reports, pivot tables or dashboards). Now what? If you have significant in-house experience with SQL Server and associated SSAS, SSIS, SSRS, and Excel (and assuming you have a clear and deep understanding of the business scope and goals to begin with) you’re probably going to:

(1) Figure out where your source data is coming from (connection strategies)

(2) Model your DW (figure out grain on facts, dimensions etc, need to figure out BIDS and SSAS)

(3) Establish some preliminary ETL process (including incremental loads, need to figure out SSIS)

(4) Load your warehouse (if you screw it up, then need to drop and do it over)

(5) Setup an SSAS cube structure (figure out SSAS via SSMS or BIDS then publish the thing)

(6) Figure out what queries to generate (talk to DW DBA or learn MDX)

(7) Figure out what BI tool to use (Excel or browser, depends on policies and audience)

(8) Generate the reports (canned or ad-hoc)/dashboards/pivot tables for the POC

Now, if you have no prior experience with the Microsoft BI toolset, and you can whip this little project up in a week, guess what, you need to quit your job and start a consulting company because clearly, as a NYC recruiter once told me “you’re so money”. But if you’re a normal person with little prior BI experience (and the terms ROLAP, MOLAP, SCD and MDX don’t ring a bell), you’re in a bind.

So another thing you can do is download a tiny analytical database (say, the XSPRADA RDM/x engine, for example) and throw, say, 100GB of data at it (this is just a small POC remember?), then plop Excel on top of it and generate some really cool reports or pivot tables to show the boss (in under a week) it can be done. How hard is that to do? This hard:

Figure out where your source data is coming from.

Yup, that one is pretty universal in the BI world. Difference here is all your data sources will export as CSV to feed the XSPRADA engine. So at least that’s consistent across all sources (be they structured, semi-structured or not). CSV is data format lingua-franca so your connection "strategy" is this: get everything out as CSV. Plain and simple.

Model your data warehouse.

That’s always a smart thing to do for obvious reasons although the XSPRADA engine is schema-agnostic and you can feed it normalized or star/snowflake models at will. The secret phrase is: “we don’t care”! So for a quick POC, if you find yourself "forced" to feed RDM/x a 3NF model, no worries.

Establish some preliminary ETL process.

RDM/x runs against initial CSV data islands directly off disk. Point to the CSV files using the XSPRADA SQL extensions for DDL and you’re done. You’ll likely be doing this via script or code (C++, Java or .NET to the ODBC driver directly or via a JDBC-ODBC bridge). For incremental loads, just plop the new CSV files on disk and point RDM/x to them using the INSERT INTO…FROM extension. This process can be done in real time without disruption while other queries are running. No hassle there.

Load your warehouse.

That’s executing a single line of SQL DDL code such as

CREATE TABLE ….FROM “c:\file1.csv;c:\file2.csv…c:\file32.csv”; or INSERT INTO…FROM “c:\file1.csv;c:\file2.csv…c:\file32.csv”;

Made a mistake of want to modify the schema and “reload” real quick? Not a problem. Simply re-issue the same DDL command and the table/schema is instantly updated. From a trial and error perspective (which, in a POC situation, is fairly typical), that’s a high-five.

Setup an SSAS cube structure (figure out SSAS via SSMS or BIDS)

There is no concept of cubes inside the XSPRADA engine. RDM/x automatically slices and dices based on incoming queries in real time. So if you want to “cube” just feed the engine slicing OLAP queries. RDM/x automatically restructures and aggregates in real time. No need to pre-define or pre-load cubes, deal with hierarchies or materialized views. I blogged about this earlier. RDM/x is a lot like Luke 11:9 – Ask and you shall receive.

Figure out what queries to generate (talk to DW DBA)

That’s where an external tool using MDX (along with an MDX expert!) can come in handy (most people don’t roll their own SQL for OLAP, although it can certainly be done in POC mode). One cool thing about RDM/x is its ability to “withstand” poorly-formulated SQL because the queries are optimized against the internal mathematical model. RDM/x is typically more “SQL-forgiving” than most other engines. And a poorly formulated query is likely transformed internally to still yield optimal performance. So even if you’re no SQL guru, the RDM/x engine is still on your side.

Figure out what BI tool to use (Excel, no brainer)

Connect Excel to the XSPRADA engine directly via ODBC or connect Mondrian to RDM/x (via bridge) then connect Excel to Mondrian via the SimbaO2X ODBO/XMLA connector. Alternatively, make the argument that using OSS like Pentaho or Jaspersoft against RDM/x directly is more flexible and accessible (not to mention cheaper!) than messing with Excel. Depending on your user base and corporate standards, that argument may or may not hold water.

Generate the reports/dashboards/KPI/Pivot Table/ad-hoc queries required by management.

Exactly the same way you would using any other tool and/or SQL.

At the end of the day (or in our case, the week), it’s all about “time to results” and “pain to results”. In those types of situations, smaller and simpler clearly has a significant advantage over the rest. And speaking of smaller, I have run over my allocated space for this posting :)

Bits & Pieces Summer Posting

I thought I would do a “freeform post” today to celebrate the lazy 2009 Summer and the fact that most of the planet (but not the BI world for some reason) seems to be on vacation at the moment.

As you know I’ve been following ParAccel with interest for a short while now wondering how they would deploy those $22M of Sales & Marketing greenbacks they just scored. I read this interesting article about them lately and it looks like “customer acquisition” might be part of their strategy. Good move. Unfortunately, I couldn’t determine who the “other database products” or the other “columnar-MPP database” vendor might refer to. I can only surmise it might be Vertica. If anyone knows who else OfficeMax looked at, please share the wealth.

For all my whining about missing TDWI in my own backyard (San Diego) lately, it seems I didn’t miss much after all according to Merv Adrian who posted about the conference shortly thereafter. From the looks of it, the highlight might have been a sunset ride on the Lyzasoft yacht in the San Diego bay. Talk about good PR!

Andy Heyler wrote a good piece about the demise of Dataupia called No Data Utopia. In it he refers to the “awkwardly named” Dataupia. Yes Dataupia is a weird name. But so are several others such as Kickfire, Tokutek, or Calpont, for example. And although XSPRADA is admittedly rather funky, at least it’s an acronym (Extended Set Processing for Rapid Algebraic Data Access). To me the money quote in there is “you need to have a clearly differentiated position in such a crowded market”. It’s pretty much what I’ve been saying for a while (and common sense if you ask me). At the moment, I don’t see any of the players in this market besides XSPRADA with a “clearly differentiated position” on anything. At the end of the day, it’s still all about the prisons that are columns and rows.

Netezza announced the long-awaited (not) TwinFin product line prompting a flurry of nasty posts from competitors like Kognitio, and an interesting Monash post about data warehouse pricing. Much like traditional software license pricing, ADBMS prices seem to be reaching for the bottom (stay tuned for $19.95 per terabyte while supplies last!). And as someone recently said, the bottom is open source. Should be interesting to see what happens. Personally, I think a lot of this stuff is going to become commoditized. The play looks a lot like printers or razors, where the actual hardware is sold dirt cheap, but the paper or blades cost a fortune to replenish. Caveat emptor.

A relatively new ADBMS vendor called XtremeData has emerged. It looks like they’re based in the US (Schaumburg, IL to be precise) but the actual brains of the operation are in India somewhere. They’ve certainly been vocal on several BI blogs (namely, DBMS2). To me the funniest thing is their “ChalkTalks with Faisal” screencasts. Faisal is apparently their India-based CTO. The entire presentation is like a Netmeeting whiteboard session where Faisal keeps talking while drawing stuff on a whiteboard in a Flintstonish manner. All that’s missing are the stick figures. It wouldn’t be so bad if they realized the audio is horrible, due to the incessant noise from the marker writing on the board. It sounds exactly like squealing puppies in the background. It’s totally distracting albeit very amusing.

SQL Server 2008 R2 is out but without the Gemini (or Madison née Datallegro) pieces I guess. That’s Office 2010, parts of which are going into the cloud if I understand correctly. This whole Madison/Gemini “revolution” in BI is starting to get a little, shall we say, boring for lack of materialization. Not sure what’s going on with Microsoft lately but I’m getting more and more concerned. Even .NET seems to be taking a backseat to Java/J2EE. It wasn’t like that a year ago. I am sensing a scary downward spiral. One thing’s for sure, I have yet to see anything remotely connected to .NET or C# in the BI programming world, save for those .NET C# extensions Aster provided in their engine recently for doing MapReduce (and of course the PushBI initiative but even there…). To me it seems the entire ADBMS/BI code stack is Java on Linux (SuSE, Red Hat and CetnOS). I’m talking about the supporting/ecosystem tools of course, not the underlying engines (those are mostly C/C++ I believe).

I could be wrong but, it’s not looking good for Microsoft. My prediction: this behemoth will eventually split off into a myriad of smaller entities, some of which will survive, some of which won’t. The sum of the parts may be worth more than the whole.

Ingres has actually managed to generate some buzz in the US lately by announcing it is teaming up with a company called VectorWise (well, it’s a research outfit actually) to develop a “project”. No customers yet but a lot of very fancy PhD types in Amsterdam (they did MoneyDB/X100) and a first option to buy VectorWise is rumored should the venture be successful. Time will tell. It always amazes me why Ingres isn’t more of a household name in the US. In Europe, they’re very popular (at least that’s what my French Ingres experts tell me ).

Finally, a recent article is claiming that BI is used by only 8% of employees in the enterprise and that’s, of course, only counting the shops that have implemented it to begin with. Actually I find that number high and would have guessed more like 5%. This is not surprising in light of the recent economic woes and scandals we’ve witnessed recently. In most of these cases, it wasn’t a lack of technology or resources at play but rather a conscious choice to ignore reality. The tools are there. The desire is not.

In my opinion, this disconnect is very apparent in numerous retail outfits. Places like Home Depot, Whole Foods, AT&T or Circuit City (RIP) for example, who clearly have resources and tools to perform and exploit top-notch business intelligence but still manage to provide mediocre service or product at best consistently.

At Home Depot, they can’t (or won't) keep track of inventory correctly. I was once told it was because there was too much theft (both internal and external) to update the databases frequently enough! Consequently, they can’t tell you if they have some items in the store or not. Not all items, just some of them. And when they can, they’re unable to locate them physically inside the store (as in what aisle and section). That’s shocking to me given what we constantly hear about RFID and data warehousing investments at these large box places. But Lowes does a much better job of this so clearly, it’s not a technology issue.

At Whole Foods (at least the one in Irvine where I live) the quality and quantity of their product is inconsistent at best. Some days the self-serve fish is fresh, and sometimes it is not (and has been sitting out for too long looking like a nice fat food poisoning lawsuit waiting to happen). Like the proverbial “box of chocolates”, you never know what you’re going to get. Similarly, their checkouts are never balanced. You’ll see huge lines at several of them while employees sit idle at empty others. Every time I go there some happy-go-lucky line manager with a bright idea of the week makes it harder for me to shop and enjoy it there which is why I never set foot in the place any more (instead, I go to an even more expensive supermarket). Apparently no one is keeping track of this. Least of all the department managers who seem to frequently act on impulse in trial-and-error fashion. Yet surely Whole Foods has an uber-BI stack running somewhere in Austin giving them a “big picture” on a store by store basis right? You’d think someone would be paying attention (or maybe even using it)? But clearly they are not, or someone would be fixing these problems (or at least genuinely addressing customer complaints, which they won’t because it's "inconvenient" for them, as they like to put it).

I don’t think it’s so much about the difficulty of implementing and leveraging BI as the article suggests. I think it’s about genuine laziness on the part of upper management. Because, at the end of the day, if you’re the CEO of a place like this, you need to get your highly compensated butt out on the floor to truly see, taste and smell what’s going on. You need to talk to your employees, your customers, and get in their shoes (incognito if possible). I’m always shocked to hear C-level people lamenting the fact that their CRM system isn’t giving them enough visibility into their customers. Or better yet, they need to “understand the customer” better. Who’s kidding who? Fact is they simply don’t give a hoot most of the time. And if they’re too lazy or too self-important to do that, they’re not likely to pay much attention to BI tools and warehouses either no matter how fancy or ubiquitous the software might be. That's really just the nature of what "service" has become in the US lately. In order to improve BI usage, we will have to improve the quality of Management first and put real folks back in charge.

How I Learned to Love Mondrian (confessions of a WISA guy)

I’ve been playing with Pentaho’s Mondrian for almost a year now on and off. I have to say, everytime I mess with that stack I am more and more impressed by its richness and capabilities. And twelve months ago, when I started learning it, I was what you could call “severely LAMP-challenged”. I’ve sure made a lot of progress since then and wanted to talk about this as I figured it might help other Microsofties out there needing (or wanting) to put a toe in these mysterious LAMP/OSS waters.

The first thing I ever did with Mondrian was figure out how to install it on a Windows platform. The reason I did was twofold. First, we didn’t have appropriate Linux hardware/software in house at the time, and second, I have way more experience on Windows so it’s a lot easier for me, and third, I wanted to do it locally and avoid dealing with cross-platform bridging at the moment (our ODBC drivers are Windows only as well). Path of least resistance is an engineering mantra in my book.

Lucky for me I had worked with Java and Apache Tomcat in the dot-com days so I had no trouble pulling and installing the JRE/JDK and the web server itself (which comes as a Windows service). Next, I deployed the Mondrian WAR file into the Tomcat webapps folder which caused it to be automatically “deployed” as a web application. Way easier than deploying ASP.NET applications (but you didn’t hear this from me ).

Then, I fired up the Mondrian landing page, clicked on the Jpivot link and, of course, kaboom. Yes, without a JDBC driver, Mondrian is not a happy camper. It took me a little longer to figure out the Sun JDBC-ODBC bridge and how to plug corresponding connection string it into numerous Mondrian files to replace the default connections there (which are all for MySQL if I recall).

The Mondrian documentation isn’t great but if you Google long enough you can usually find some other poor slob with a similar problem and, with luck, published solutions online. [Side note: I once had the nerve to email Julian Hyde, their Chief Architect, about some technical question. He abruptly suggested I don’t bother him and use the “community” forums instead]. Unfortunately those forums are often useless for non-enterprise (read: non-paying) users. There are flurries of unanswered questions and problems up there. This is a generic OSS problem I suppose. You get what you pay for J

So finally I had my bridge setup, along with a DSN called MondrianFoodMart (the default) pointing at the default Access database (distributed with Mondrian). And now I was able to fire up Mondrian on top of the database and do a couple drills, run a couple MDX queries. Bliss.

Next, I took that Access database and exported it to CSV format with corresponding DDL. This is the way to feed the XSPRADA engine. Fired up our RDM/x server and ran the DDL scripts. Then re-defined the MondrianFoodMart DSN to point to us via our ODBC driver (32-bit only, 64-bit won't fly with the bridge - painful lessons learned...). Reloaded the Mondrian page, and voila! Mondrian was now talking to RDM/x and displaying the Sales cube.

One point of the exercise was being able to show Mondrian OLAP on top of our database. Another was being able to show our database’s behavior in time as more and more queries come in (hint: it gets faster). Now, with Mondrian, this is a little tricky because the platform is heavily cache-based. Mondrian shoots initial queries at a relational system and proceeds to cache heavily as it aggregates results. So the more you use it, the more it caches. Obviously the reason for this is that Mondrian is designed to run on top of relational databases, and not “OLAP-intelligent” engines such as ours. It has to translate the MDX into straight SQL queries every time, as it fills its caches initially. Nevertheless it does re-hit the database as needed when you start slicing and dicing on new dimensions or facts, as one would expect. So you can actually see our engine’s “dinosaur tail” behavior as I once described in a previous post.

Now, the MondrianFoodMart database is fine for setting up the stack, but it’s not particularly interesting in so far as data volume goes if you’re in the VLDB space. More recently, I attempted to setup the TPC-H/SSB sample data under Mondrian, meaning I tried to manually create the XML defining some SSB cube. There is a whole fairly complex XML language Mondrian uses to define and connect fact tables (measures) with associated dimensions. They have a UX driven tool called Workbench but I could never get it to work on my system (and didn’t have enough time to keep messing with it). With online help and using the FoodMart.xml sample file, I was able to get a basic cube up in about a day. Nothing fancy, but now I can OLAP into arbitrarily large data sets and that’s a good thing.

As cool as it is seeing our stuff run under Mondrian, I always dreamed of doing the same thing under Excel (as in 75% market share, yeah I want to support that please). Until recently, I thought this would not be possible until we implemented MDX in the engine but then I saw the light. It is called the SimbaO2X connector and it rocks!

[Start Commercial] Did I mention how much I love this company Simba? They pretty much wrote the book on data connectivity. Within a day they had me a 30-day trial version of their O2X offering, no questions asked. And follow-up to boot. Their stuff works, and they know how to take care of people. What a concept! [End Commercial]

This SimbaO2X puppy lets ODBO clients (say like Excel) talk to XML/A OLAP servers (say like Mondrian). Note, there is a similar offering from Pentaho called Pentaho Spreadsheet Services. It carries a small yearly license fee from what I understand. Supposedly you can email Pentaho sales for additional information and a local contact. I’m still waiting for their reply. Hey it’s OSS…Did I mention you get what you pay for?

Either way, the relevant fact is that the SimbaO2X connector works without a hitch. I am finally able to create and manage pivot tables from Excel, talking to Mondrian (via XML/A), talking to RDM/x (via ODBC)! This is the bomb! I need to really get a deeper understanding of MDX capabilities now. But the more I learn about it the more impressed I get, and the better demos I can do.

The Canary in the Gold Mine?

I’ve been claiming for a while that data mining and predictive analytics (PA) were the new hills to conquer in BI and this morning the news came out that IBM had plopped down big money for SPSS. IBM is also investing R&D dollars in ways to manipulate data directly while encrypted and/or compressed. This particular research fascinates me because I believe it will be key to SaaS acceptance, where security is still a significant push-back for obvious reasons. This means analytics might actually have a future on the cloud. And this is important IMHO because this allows for significant progress in the UX systems required to use (drive) mining engines efficiently. The kind of improvements that cannot be generated and deployed quickly enough with fat client implementations. I’m thinking of really interesting things like www.spezify.com for example.

Another interesting trend is pushing analytical capabilities deep into the database engine either via stored procedures or user-defined functions in one or more programming languages (much like .NET inside SQL Server, for example). All this leads me to believe that insightful BI players have been turning their guns on solving the next big pain point of BI which is, IMHO, data mining and predictive analytics. This embedded capability relates to the deep kind of analytics I once blogged about in the context of Greenplum’s MAD paper.

So does this mean we’re all done with OLAP? Not likely, but I think a certain peak has been reached where OLAP has become “bearable”. I don’t really have a 3-5 year “future outlook” on OLAP at this point. Is it still hard to cube and do MDX? Yes. Is it still a pain in the behind to setup large SSAS analytics? You bet. Is setting up a production version of Pentaho’s Mondrian ROLAP for the faint of heart? Not exactly. But there are now multiple alternatives out there in both hardware (faster COTS components, FPGAs, GPUs, MPP) and software (columnar, ALGEBRAIX) realms.

Our own ADBMS at XSPRADA is designed and tuned specifically for OLAP workloads in its present form. Product such as ours have helped “commoditize” OLAP work by shifting design and pre-structuring efforts (cubing, slicing and dicing) from the user (DBA) to the software itself. This is done automatically and based on queries coming in. There is no need to configure cubes, mixed workloads are supported, and all the user really has to do is ask questions. It’s that simple really. Let the software worry about the darn cubes!

So I guess my point is, if there are people still struggling (read: losing time and money) with OLAP in the enterprise, I have to say it’s because they’re either poorly advised or simply not opening their eyes to new tools and techniques currently available. At this point OLAP pain is no longer a necessity. It’s an uneducated choice. From a technical standpoint, it has been addressed. Let’s move on to the next problem please. This is why I think the industry is poised to tackle another challenge now, namely data mining and predictive analytics. Even Curt Monash in a recent blog about the SPSS acquisition writes:

“So far business intelligence/predictive analytics integration has been pretty minor, because nobody’s figured out how to do it right, but some day that will change. Hmm — I feel another “Future of … ” post coming on”.

Sorry Curt, I beat you to it J

Mining is a totally different segment of the business intelligence endeavor. When you do OLAP, you’re asking “tell me what happened and why”. When you do mining, you have no clue what happened and much less why. In mining you’re asking “tell me what I should be looking at” or “tell me what’s interesting in this data?” And predictively, you’re asking “tell me what’s likely to happen” – as in, show me the crystal ball. Mining is not a pre-structured, pre-indexed kind of “cubing” world. It’s an ad-hoc discovery process. It’s iterative. Much like the way a human brain functions when discovering information, and trying to make sense of it. This “human-like” behavior is actually one of QlikView’s usability pitches. In mining, the relational model is a hindrance, not an asset, because relationships are not necessarily canned or static. Predictive analytics are more of an art than a science as well. These concepts don’t fit nicely in pre-structured, tabulated formats.

Additionally, mining and PA are creative endeavors (whereas OLAP is not). This is why it’s important to let users define their own “stuff” so they can trial-and-error through the problem. Conventional database engines don’t support this type of workload elegantly. It’s simply not “structured” nicely like OLTP or OLAP. You can’t easily (or cost-effectively) try, erase and re-start with conventional engines. They're not forgiving.

So what’s needed are systems that can first intelligently process data upstream in ELT mode because acquiring statistic on incoming data (at varying rates) is an important step for analytics. XSPRADA’s engine starts analyzing data statistically upon initial presentation. More importantly, it keeps doing so automatically in real time, and continuously via comprehensive optimization. This is a unique feature that causes the system to continuously re-evaluate system resources against queries and data to seek out additional or more effective optimizations.

Next, you need systems that can tell you where NOT to look. Because in this type of work, pertinent data is often clustered in very specific areas (as in 5% of 100TB perhaps). And user questions tend to hit within small percentages of those clusters. Yes there are always exceptions, but generally-speaking, that’s what happens. So what you DON’T want are systems that spend a lot of time scanning boatloads of data (needle in the haystack). What you need is intelligent software that can quickly eliminate vast areas of informational “no-man’s land” based on incoming queries. In such a problem space, throwing additional monies at ever more powerful metal is a self-defeating approach. It’s the software stupid! J

As it turns out, XSPRADA’s ALGEBRAIX technology is very good at eliminating "useless" (read: at a given time) data spaces. Not only that, but it also shines at inferring subtle relationships between different entities. The kind of relationships a human wouldn’t even think of asking on her own. It’s also very good at recognizing patterns (both in queries and targeted result sets).

In a way, you would expect that a system built on pure mathematical foundation would be particularly well suited to data mining workloads. And it sure is. This is the beauty of having a “wide” and rich enough technology that is as easily and readily applicable to a multitude of different BI problems. It means you don’t need to re-invent the wheel or re-architect your system every time a new problem space opens up. And that, in the business intelligence technology world is a rare find indeed.