Big Honking Databases

Insert Cricket Sound...

2010-01-28T20:46:00.000-08:00

I've been so jammed up lately I never got a chance to update this blog. As you might have heard (if you were kind enough to check back here for new content) I have left the IT/database industry for the horology one, have started a new job about a month ago, and am in the process of moving to Switzerland to pursue a new position and embark on a new career. I won't go into the details here and will just point you to my new blog for all the juicy news.

If you've been following my posts on here for the past year or so, I thank you for your readership and, most of all, your precious participation. I've learned a boatload of valuable lessons working on this blog and interacting with you since the early days.

I won't be discussing software and BI issues on here any longer obviously, but if you also happen to be interested in social media, marketing, brand management, community building (oh yeah, and watches!) then as they say back in NYC where I grew up - Meetcha 'round the corner! :)

Peace.
Jerome.

Social Media Marketing 101 - The New Fountain of Youth

2009-12-21T09:36:00.000-08:00

I was recently asked to opine on what makes a social media manager and his (or her) strategy successful in today's market. Having dabbled in the field for a little while now (perhaps what can be considered a long time in this emerging medium) it forced me to stop and reflect on my own trials and tribulations in the realm of social media marketing (SMM). The first thing I pondered is how to even define "success" in social media and community management. It's not something that can be pinned down easily like an engineering project where criteria are clearly defined. Namely, does the product work as expected, is budget under control, and are the customers happy? Managing a brand or product via social media channels is more akin to wine making or political campaigning. It's something that takes considerable time, involves a myriad of tools and tactics, and, quite honestly, more than a little bit of luck. And it also takes a special kind of personality.

So first off, what is "social media"? It is a set of web-based communication channels connecting a brand with existing and potential customers. These typically include a website, Twitter, video channels (YouTube or Vimeo, for example), several social networking sites like MySpace, Facebook, or LinkedIn, maybe some IM services, throw in a couple photo sites like Flikr, and perhaps some opt-in email or SMS push channels. Social media marketing (SMM) and social media optimization (SMO, a subset), is the art of juggling all these channels simultaneously to build brand recognition and increase customer acquisition. The reality is in fact a bit more complicated, but those are the basics.

Note that I called this SMM endeavor an "art" - and purposely so - because to the best of my knowledge, there are no clear well-defined best-practice recipes for success in this business and most companies tend to fly this bird by the seat of their pants. Some better than others. Yes, there are numerous self-proclaimed social media experts and consultants out there. People like Brian Solis, for example, who clearly know what they're doing and contribute substantially to educating the community about what works and what doesn't but in reality, no one really knows what works for sure. There are no magic formulas or guarantees. Just lists of obvious pitfalls to avoid and generally-accepted best practices, but that's about it. So here are my distilled two cents on SMM strategy based on personal and vicarious experiences of the past couple of years.

You know your SMO strategy is starting to pay off when:

1. Your web analytics indicate growing popularity and visit stickiness.

2. You see increased public participation in all your social media outlets.

3. You start getting unsollicited interest from the press and industry influencers.

4. You find it easier to establish relationships and partnerships with distribution channels.

5. You start getting attention from competitors.

6. Customers start evangelizing your product to their peers.

7. Your brand awareness starts spreading outside your targeted segments.

8. You start seeing measurable and sustained growth in customer acquisition.

9. You start seeing repeat orders from customers.

10. You pick up positive comments about the experience of doing business with you.

What does it take to become successful at this game? A couple of tips here:

1. Do not profess expertise on topics you know little about. Eventually, it will show.

2. Always remain honest. Never lie. Your most important asset is credibility. You can fix almost any mistake except credibility damage.

3. Be genuine. Maintaining a special "online" personality is not genuine.

4. Be obsessed about customers. Walk in their shoes. They're always right.

5. Do not let fear of mistakes (or failure) paralyze you. It's OK to make mistakes. What's important is quick admission and correction.

6. Be either loved or hated. Middle ground is not compelling.

7. It's not a job, it's a way of life and a mission. Understand the expectations. It's a marriage, not a date.

8. At the end of the day, it's all about the "story". Understanding show business is crucial.

9. If you suck at building relationships in real life, you will suck at it online. The inverse is not necessarily true.

10. Question everything, keep an open mind, but be wary of "experts" (the field is too new).

I've had my share of SMM boo-boos in the past years; online successes and failures. Stuff I thought would never work (that did) and things I knew for sure would take off (but never did). There's nothing like this new media to bang some realism back into one's ego and topple pre-conceived notions. That's why I love it. The beauty of this new "frontier" is its never-ending capacity to change, evolve and teach. Try it. It'll keep you young :)

The Wailing Wall of Open Source BI

2009-12-07T12:34:00.000-08:00

Henry David Thoreau once wrote: "The mass of men lead lives of quiet desperation". Much the same can be said of the multitude of users struggling with open source reporting and analysis tools like Mondrian or Jaspersoft. The difference, of course, if that those folks happen to be pretty vocal. And nowhere more so than on those vendors' own "support" forums.

I double-quote the term purposely. Because what you witness on these forums falls far (very far) short of what I consider to be minimally acceptable customer support levels. Now, it's a fact that many people find solace in these communities - after all, given the massive amounts of questions posed there, some are bound to get answered quickly and (hopefully) correctly. And it's a fact that many people achieve success (or some level of it) with open source BI solutions. At what cost, we can only surmise, but clearly, resilient, persistent and courageous people are getting some work done on these platforms.

But more often than not, the levels of post abandonment (ignored questions) and the arrogant responses are high enough to be shocking. I am stunned at the number of times where posting users are treated like it's their fault. The implication is that they are stupid or negligent. As a matter of fact, if you spend the time analyzing the language semantics, what you see are fearful, timid, often desperate users mustering the courage to post questions in the hope that somehow, someday, they will be answered by the Grand Wizards of [fill in the OSS BI vendor name]. This is not unlike the Wailing Wall in Jerusalem, where masses of people stick written prayers between the stones.

I find that these forum "masters" are too often condescending and consequently, they generate a "master-pupil" environment I find somewhat repulsive. It reminds me a lot of the old school systems in Europe, where Professors would stand on an elevated stage above the class to seal their social superiority. There's nothing wrong with respect but in my opinion, it has to be earned. When you berate, ignore or insult those who seek to learn from you, you are far from deserving respect, Grasshopper.

I do realize that these vendors provide some level of "enterprise" support for those wishing to license the software but given what you see in the public forums, it's hard to imagine it can be any better on the paid side because, quite honestly, Open Source companies simply don't "get" service. They are too often blinded by the magnificence of their code or product and forget that without users and healthy bi-directional communities, there wouln't be a company, Open Source or not.

My experience navigating through several Open Source forums was similar. At the time I figured, it's just me. I'm an impatient SOB and it's my problem. But when you start analyzing these forums (and I spent hours doing so), you quickly realize the disease is wide-spread. All of a sudden, it's not just you anymore - almost everyone has the darn infection! I often wonder why people put up with this nonsense. Deep inside, I know the answer of course: because they don't have a choice. Either they can't find an easier product to use, or they are compelled to use it for organizational reasons (as in, my boss told me to check this out <sigh>). I do believe the advent of SaaS BI analytical tools spell the end of an era for the multitudes who must suffer through Open Source to get even the simplest of reports and dashboard out in less than three months!

But don't take my word for it. After all, I work on the SaaS side of BI and am likely biased. Even worse, I actually blogged a while ago about the merits of using Mondrian for OLAP. Indeed, after many months of putzing with it, I was able to accomplish something useful but I have twenty years of experience in IT. Read that again: two decades of messing around with difficult stuff and figuring out how to make it work. So sure, for someone like me (and if forced into it) you bet open source can work. But what about the poor guys who don't have that kind of experience? What about the people tasked with just getting reports or analytics done? I feel bad for those folks who invest time and considerable effort in open source solutions only to be left at the altar of success. And if you don't believe me when I say they're out there en masse, then feast your eyes on the following selected quotes and links from Pentaho and Jaspersoft OSS BI forums (they are reproduced verbatim, grammar, spelling and emotional content intact).

"It seems I first have to dive in pentaho source code to figure how the printing function in pentaho is working"

"I am new user to Pentaho. I downloaded mondrian-3.1.2.13008. Now please advise me the steps I need to follow to configure modrian in my system."

"Although I'm no expert, I think Spreadsheet services has been discontinued, well I've never seen a Pentaho Employee mention otherwise"

"I've been trying to use IIF all morning but I'm getting nowhere."

"I have two questions, which I barely dare to ask, but I am sitting here and can't get over my problem:"

"I tryed all combinations. The only combination that works is fact table,dimension tables and aggregates tables in the same Mysql schema, and a datasource that point to that schema."

"Hi everybody. Though I've run the demo applications and read almost every thread about this, still I can't get an exact understanding of the relation existing between Pentaho Server and Mondrian."

"I think that the preconfigured examples of JSPs, catalogs etc. are confusing at best - bad practice at worst."

"I am fairly new to Mondrian, and I am stuck at ths point"

"This might be very trivial for some of you but I'm really having a hard time figuring this out. So any help is greatly appreciated. Thanks so much."

"I've goggle this one down to my knuckles. I suspect my chopsticks are tangled on the Mondrian/Pentaho side? I would appreciate any thoughts the community might have."

"Nobody can help? If so, does anyone know where else I can go to find the answer ... I've pretty much exhausted Google's list of MDX links, and am getting a little desperate now. Don't want to tell my client "it can't be done" without good cause..."

"Desperate for assistance. Last big issue to overcome and deliver my pentaho bi solution."

"Need some Help again.. I told my Boss that I will get the prototype ready by tommorow. Any help would be appreciated"

"I'm quite desperate.... ... please... help...."

"Helloo...???? Anyone there?? My question may be silly, still any of your ideas would be much helpfull..Am really desperate to find the logic.. Am not good in java and am in verge of my project and solving this issue resolves half of my task...Please respond..!!!"

"We are pretty desperate by now. We tried to manually re-insert the UNC server part into the file name, but this is getting dirtier and dirtier as we build new jobs."

"Sorry bothering you again, but I'm in a desperate situationn. We are trying to run the new Pentaho GA, but we have a lot of problems when we try to run this with Oracle."

"I'm really fed up now trying to make charts working over couple of days. I have created a database connection and given the field names correctly. Can someone please help me? I'm really desperate to get this working.."

"I know this has been logged by both myself and others before, however I am desperate for an answer here."

"I checkd out all the sample dashboards of cdf.. but am clueless to create a dashboard of my own.. is there a dashboard builder for cdf??"

"I dont consider myself a fool when it comes to working with technology, and I have found myself frustrated and confused beyond belief at what should be a seemingly simple task..."

"And...regarding the timestamp....how in the heck to I tell that to not show up if I have no idea where it's coming from? Sorry if I seem frustrated, but I am."

"I've just recently begun working with Pentaho, so forgive me if this is a known issue or feature...I did a search on these forums and couldn't find an answer...a frustrated contractor in DC."

"I got frustrated that i cannot get a decent quality, well formatted pdf output."

"However, I do have to say I really had to grind through the first 10 days or so in order to get this thing down to a point of not hitting hurdles every step of the way. In fact, I've got a colleague who is really pretty frustrated and trying to jump ship"

"A few weeks ago I was very frustrated that I was completely lost withing thePentaho suite. I cannot afford training (I'm broke, plus I live too far away from the traing venues) so my only means at this stage is all of the above"

"I’m not trying to start a ruckus or anything I just wanted to state when I am coming from and what I would like to see from Pentaho. Judging from the many unanswered post on this subject I think there are many more like me out there."

"Sorry about the begginer question. But i'm getting frustrated"

"I am getting increasingly more frustrated it is quite hard to get started. The documentation seems only to touch the very basics. "

"I am a bit frustrated right now as I haven't been able to solve any of my problems."

"...first, *I* don't think i'm an idiot. what frustrated me was the lack of TASK-ORIENTED information i found. i had a fairly simple, fairly small TASK i needed to GET DONE... i dont care about kettle or apis or what great problems Kettle could solve... I had a TASK and an IRATE BOSS."

"I'm pretty frustrated at this point: I mean seriously how much more simple could a transformation be!?!?!?!?!?!"

"If I am posting about that, it is because I ve been frustrated much!"

"I was very excited to find Pentaho and eagerly wanted to create my first Dashboard. Now I’m just completely frustrated. And as I search these posts, I see thousands of people are looking for the same basic answer with little to no response from Pentaho: How do you create/deploy a non-sample dashboard; i.e. how do you actually use this thing?"

"Maybe a non-IT guy doesn't want to deal with SQL queries at all...

Now let's look at a couple of "answers" to many of these posts (unfortunately, questions outnumber answers significantly):

"You use the search button. Version 2/3/3.5 brings in lots of changes to the way Pentaho does things, please mention which version of the BI server you are running.
Join the Unofficial Pentaho IRC channel on freenode. Server: chat.freenode.net Channel: ##pentaho - Please try and make an effort and search the wiki and forums before posting!"

"If all that is true then it is clearly a bug in mondrian and you should report it in jira."
Here, the moderator seems to question whatever the premise of the question was (as if people bothered posting lies...)

"The procedure is exactly as I have told you. You must have done something wrong somewhere. Just check through carefully;"
Typical "you must have screwed up something" response. How encouraging and a little condescending if you ask me.

"If not, please post the error. We're not clairvoyant."

"There might be a simpler way... search the doc."

In this post, a user actually bothers pasting a novel sized exception asking for help. Specifically, he asks: "Any help or even pointers to documentation more than Spring/Acegi provides would be greatly appreciated".

What does the Jasper guy reply with?

"JAAS authentication can certainly be done. Have a look at the Acegi documentation...The forum is also a great resource. - Is he kidding? Did he even read the post?

Here's another classic: a user is complaining about the advanced search in the Jasper forums. The moderator's suggestion is terse: "it is what it is". Then he suggests using Google! Are these people serious?

How about this comment: "I am desperate to use JasperReports , however I am having hard time understainding it". Or this one: "Still can't compile - getting desparate...I am getting nowhere trying to get my reports to compile."

What can we conclude from this mass of desperation, frustration and confusion? Why would these vendor create and maintain such an environment to begin with? Because quite honestly, if I were looking for a BI platform for a critical project or client, and came upon these forums in the process, I'd run so fast the other way it's not even funny.

As I mentioned above, if you have some serious software development and IT background, the luxury of time, (the right hardware and software) and a propensity for hacking complex nebulous systems, or the courage to read a 600 page book on working with Pentaho, for example, these open source solutions might just do the trick. If your organization or client is insisting on going the Open Source route, and you have no say in the tooling selection, then clearly you have to deal with the pain, hope for the best, and pray for a check (or a job) at the end of the ordeal.

But if you're just a normal analyst or consultant tasked with a BI implementation on a tight deadline with an impatient boss breathing down your neck, there is an alternative. And if you're just a guy (or gal) who needs to get the job done now (as in, this week) without the drama and headaches of open source theatrics, you'll likely be looking at a SaaS BI solution. Now, SaaS BI may not turn out to be your bag, but you owe it to yourself to at least give it a whirl and here's why (putting my pitching hat on):

It takes very little time, effort and money to try it out. In many cases, it's free. There's nothing to install or configure. It takes hours or days, not weeks or months, to build and showcase prototypes. Most of the time (as in 80%) the functionality is sufficient to do the job. And features/functionality increases dramatically with each release. Oh, and there's updated documentation. But here's the kicker: you'll actually get just-in-time support from people who actually care about your success. As a matter of fact, they're invested in it. They will welcome your questions and use them to improve the offering. They won't ignore you and they won't treat you like an inferior species.

I know this is sounding a lot like "Miracle on 34th Street". Must be the season. Truth be told, SaaS BI is not for everyone, and your mileage may vary. But it's worth an honest shot. Because the alternative is an endless Wailing Wall of pain, misery and isolation that no one in BI should have to put up with anymore.

Of CRM Golden Calves and Ten Commandments of SaaS Landlordship.

2009-11-29T17:40:00.000-08:00

I have a pet peeve concerning two things I’ve been meaning to write about lately, namely CRM and multi-tenancy. I recently read an excellent article referencing both concepts and thought it might be a good stepping stone to a quick blog post.

In his article Don’t Get Conned – The many disguises worn by software-as-a-service, Matt Wallach discusses CRM software and the concept of SaaS multi-tenancy. Matt’s business is life sciences, and he discusses the state of CRM software for his industry. His argument is along the lines of “be wary of on-premise wolves in SaaS sheep clothing”. The wolves, in this case, are on-premise CRM vendors who simply throw servers over a wall and call it a SaaS day. The sheep (aka the good guys) are those genuine SaaS vendors riding a multi-tenant architecture. He then proceeds to define what multi-tenancy means using the well-known “neighborhood versus apartment” analogy. Sounds innocuous enough (and the piece is well-written) but here’s the problem I have.

I am going to get a lot of heat for this statement, but truth be told, CRM is a hoax. There is no such thing as CRM. It’s all smoke and mirrors, and you cannot nurture customer relationships (or any other form of relationship) using a bunch of bits. It’s a cop-out. I know of CRM market segments and sizes. I am well aware of CRM dissemination and popularity in corporate America. I don’t question its omnipresence in virtually all businesses in one form or another. My claim, however, is that it does not and cannot work as billed. In numerous cases, it is simply a waste of “feel-good” money. Here’s why.

The companies and industries making the most use of CRM packages are those with the worst customer service. This in itself should be eye-opening. Airlines are perfect examples. Can anyone think of an industry with worse customer service? I can’t. Yet they spend gazillions on CRM and boast about it frequently. How about retail? When’s the last time these guys used CRM to do anything besides manage their “rewards programs” (who are they rewarding please tell me?) or push advertizing gimmicks? How about Telecoms? How many people are happy with their cell carriers in the US? When was the last time your interaction with a carrier’s customer service department was either useful, efficient, or pleasant? How about the automobile industry or even car rental outfits? When was the last time you were wowed by one of those big corporate CRM consumers?

Now, I’m not saying all companies who use CRM manage customers poorly. Some do get lucky. And others use it as a tool, not a means to an end, which is perfectly fine. What I am pointing out is that customer service quality seems inversely proportional to the resources spent on CRM at the corporate level. And that’s ironic. The problem is in the C-suite. These folks see CRM as a panacea but inherently, deep down, these people do not have customer service “genes”. They have Board of Directors genes, profit margin genes, or MBA ones, but they don’t know the first thing about worshipping customers – It’s a form of autism on their part.

Because in my experience, customer service is not something you can learn on the fly (although I suspect it is taught in business schools). And it’s not something you can acquire via software. Companies, like people, are either born with it or not. It’s a nature not a nurture trait. And no amount of software or resources will change the behavior of a company whose culture isn’t obsessively centered on the customer. It sounds obvious, and everyone talks about it, but most of the time, it’s nothing more than lip service. It’s a sham I tell you.

Luckily, the solution is simple: throw the software out the window. Really, honestly, you don’t need it. You can’t even prove you need it anyway. Most of the time, you don’t know how to interpret or filter the results. And worse yet, you have too much information. You get overloaded with it in a drunken stupor and it ends up clouding your judgment. And shortly thereafter, you throw common sense out the window. Malcom Gladwell makes excellent anecdotal arguments against information overload in Blink, one of his classics. It’s a fascinating read (specifically I refer to section 4 of chapter 4). CRM and software overload have the same effect on corporate management as those described by Gladwell affecting ER doctors’ decision making. Too much information leads to disastrous results.

I’m not suggesting the demise of customer or performance management software as a whole obviously. Keep the databases, the warehouses and the BI in-house (because analysis and insight are still obviously crucial) but take your big honking CRM software and pull the plug. You will likely learn more (and spend a whole lot less) by talking to your five top customers. If you want to stay in control and get 360 customer views, sign up for something like TeamSupport – it’s all you need. Most importantly, do something truly revolutionary: get out of the building and walk a mile in your customers’ shoes.

Order your own product, fly your own airline (in coach), rent your own cars, dial your customer service numbers repeatedly – from a roaming zone, for example – visit your retail stores (incognito) and talk to your customers. Order your own food, visit your own bathrooms, wear your own clothes. Engage customers one on one, on the streets, in the stores, at their home, face to face, and listen like you mean it. Listen like your life (and job) depended on it – because it does. You cannot manage customer relationships from a spreadsheet or a SQL query any more than you can steer a marriage successfully via the web (yes I know, there’s probably an iPhone app for that). It’s that simple. Your fancy CRM system is the blue pill. Take the red one!

So now, having sufficiently ticked off everyone in the CRM industry, let me tackle the SaaS “badge of honor” also known as multi-tenancy. Here’s a reality check: no one really knows what multi-tenancy means. How do I know this? Because if you ask three people what it means you will get five different answers. And when you search for it online, you get multiple descriptions, some more convincing than others. So unless I missed something, there is no official definition, only qualified opinions.

To me, the gist of it involves partitioning software application space very quickly and cheaply to accommodate exponential mass market adoption cycles (there’s a mouthful). Exactly how you achieve this, from a technical perspective, is open to interpretation. Some people partition inside the database and share schemas (like Salesforce.com, where every tenant shares database tables), others partition on the application layer where processes may serve multiple requests, some consider virtualization to be a form of multi-tenancy, and yet others implement a hybrid approach.

At the end of the day, multi-tenancy is a design methodology. Multi-tenancy (however you implement it) allows faithful cloud landlords to follow basic rules. If Moses had come down from “The Cloud” instead of Mount Sinai, he might have brought these back instead:

(1) You shall provision tenants quickly without affecting other parts of the system.
(2) You shall be able to provide deterministic system metrics on a 24/7 basis.
(3) You shall let non-technical staff (or software) handle provisioning.
(4) You shall monitor your platform’s health day and night.
(5) You shall implement and frequently simulate doomsday scenarios.
(6) You shall make it as easy to provision one tenant as it is for one hundred.
(7) You shall segregate and protect thy tenant’s data at all cost.
(8) You shall upgrade transparently and automatically for all tenants.
(9) You shall allow tenants some degree of customization but not more.
(10) You shall have a simple, clear and consistent pricing model.

In my opinion, what goes on behind the scenes to support this is irrelevant because it’s irrelevant to the user. The user just wants to be “on-boarded” and provisioned pronto and safely without hassle. If you happen to be able to pull this off technically and cost-effectively, then you’re multi-tenant! If not, you have some pain coming your way.

Whether or not you can support thousands of tenants per instance (however you define “instance”) is, for all intents and purposes, really your problem, not the user’s. And I have yet to meet a prospect who’s first and foremost concern involves “multi-tenancy”.

This might suggest we cease using the term as a marketing asset because I don’t think it resonates clearly. In my experience, the only things that matter to a cloud user are simplicity, service levels, trust, and cost. So let’s give users exactly what matters to them, explain it simply, and keep it real.

Yours in BI.

BRBR45DDTK3B

S&M: Where the SaaS rubber doesn’t meet the BI road.

2009-11-27T09:31:00.000-08:00

I was lucky enough to be at Dreamforce 2009 last week and wanted to pen down a few thoughts while the event is still fresh in my mind. I don’t think there was any earth-shattering news there, and I got the feeling (both onsite and online) that a lot of people didn’t really grasp the value of Benioff’s announcement (or strategy) about “socializing” the platform with Chatter. I, for one, certainly couldn’t make sense of Colin Powell’s presence at one of the keynotes (not sure what he can possibly offer the world of SaaS but maybe I missed something). But overall it was an enlightening conference and here are some of my impressions (and they pertain mostly to the SaaS BI realm).

First, the sheer number of bodies at the event was impressive. I understand 18,000 people took part and that is quite a large crowd given how undersold (to put it mildly) other conferences have been this year. Businesses have been reluctant to invest in conferences in 2009 as evidenced by abysmal attendance numbers and the rising popularity of “ virtual conferencing”. So if one conference was preferred over all others, it must have been Dreamforce 2009, because it seemed like everybody and his mother sent people there.

Second, I was impressed by the level of “education” the typical attendee exhibited. I didn’t really see or hear people asking basic “big picture” questions. Rather, the inquiries were very focused, deep, and to the point, revealing mature customers (buyers) who had done some serious homework. Actually, most of these folks have had meaningful experience in the cloud (some good, some bad) and knew how to hit the right vendor pressure points. From my standpoint, it is always vastly better (and more enriching) to deal with well educated buyers in a no-nonsense approach. This is exactly the user profile I experienced at Dreamforce.com manning the GoodData booth.

Third, and I realize this is subjective, but to be honest, there are a lot of small clueless companies out there having nothing to do with cloud per say who clutter these shows for the publicity of slapping “cloud” onto their marketing literature. I’m not going to name names, but let’s just say when you sell gardening shoes, mailboxes, or kitchen countertops, I’m not sure you should be spending marketing dollars on Dreamforce.

Fourth, I didn’t pick up any “religious” fervor at the show from either buyer or vendor sides. I assumed this was going to be a major rah-rah for everything cloud (with Open Source type of fervor) but I found the discussions to be much more measured and rational with most people objectively comparing both approaches (when applicable) with few pre-conceived notions. I believe this is a sign of industry maturation as people are getting better at separating the wheat from the chaff. I feel for the most part that SaaS limitations are well understood by most (not all) people and expectations are becoming more reasonable for the most part.

Fifth, I believe that SaaS beachheads have been claimed. This is particularly true in the BI space. This has a lot to do with perception obviously but in my opinion, the winners and losers have already been tagged. Most companies (if not all) are fairly new in the cloud space yet already, they have public reputations as in “these guys aren’t serious” or “these folks are the ones you want to talk to”. Obviously first-to-market matters a great deal in any industry and cloud is no different. Except with SaaS of course, you can be first to market with a virtual product (vaporware, in the on-premise world), and it takes longer for people to read the fine print but, eventually, they do. When people have an interest in a particular SaaS domain, they go directly to the “top dogs” without stopping anywhere else. I believe there is plenty of space for new companies in the cloud but for those having established an early lead (perceived or not, and with compelling technology), the future seems bright.

Sixth, everybody in the BI cloud space faces the exact same problems. And no one has clear answers so this is still a very “trial and error” process. The difference is between vendors who admit this, and those who don’t (mostly to themselves). This is a sweeping statement but overwhelmingly, when you talk to other vendors, the same themes come out time and again. Namely, how to scale fast enough (or onboard efficiently with minimal customer “touch”), and how to control sales and marketing costs which are turning out to be higher than anticipated. Although adoption is growing, in my opinion, the technical hurdles are not remotely as high as the business ones.

Most of the BI cloud vendors have managed to get by on minimal engineering costs. Offshore labor is cheap enough that you can afford competent engineering teams in India, Central Europe or China (to name a few) for literally dollars a day and minimal liability. Some of these vendors are making ends meet with two-man engineering teams! And only two I know of have engineering teams exceeding ten people. So clearly, the money pit is elsewhere.

And elsewhere is S&M (no, not the fun kind) namely Sales and Marketing. The original proposition for cloud was that the “service”, unlike traditional enterprise software, was going to kind of sell itself. S&M budgets were going to be minimal. No more travelling field sales force, expensive face-to-face customer visits, pre or post-sales engineers. It was all going to be “automatic” and on the web. But my limited experience contradicts this.

Because now, adoption and competition are growing. For example nowadays in SaaS BI, you have dozens of vendors. You skim enough to get to the “serious” ones (see #5 above) and now you’re left with maybe four or five guys. Next year, there will likely be ten serious contenders. The more competition you have, the higher sales cycles and costs get. Next thing you know, you’re back to boots on the ground and to a more traditional enterprise software sales models. This is the danger facing many SaaS players these days. CEOs and investors are edgy about this emerging trend. It breaks the anticipated mold.

The other problem is what I call customer “touch-too-much”. In a SaaS model, efficient on-boarding is crucial. This is not only about flipping the proverbial switch – because properly-engineered multi-tenant systems achieve this quite well – but more about the time it takes to get a user’s business problem solved. Namely, the costly interaction spent on a given customer to handle specific needs and the amount of customization needed to achieve satisfaction (and final sign-off on the purchase order). POCs, sales cycles and marketing costs are growing.

This is a huge problem in the BI space because the requirements phase can be long and even there agility is not necessarily a Holy Grail. The basic problem is simple: it is very difficult to remove the “human factor” when implementing BI. Cookie cutter never satisfies a particular business problem entirely. The money’s in the customization and the subject matter expertise. You must solve difficult business problems, not engineering ones. The same challenges apply to software engineering, and history has seen a flurry of “blissful automation” endeavors fail in that space (remember 4GL?). At the end of the day, you can’t remove what’s between the chair and the keyboard, and you can’t efficiently and consistently automate it in software – whether in the cloud or not.

Now, this is not so bad for a company like Salesforce.com because they’re a platform play. So by definition, they provide efficient functionality (large offering surface), but it’s all fairly mediocre and “cookie cutter” – Users are free to (try and) customize their modules as they see fit. In the analytics space, for example, Salesforce reporting and analysis is shallow. Consequently, users looking for customer data insight and trending statistics (say for pipeline analysis) will look for integrated solutions fitting their specific business needs.

But for the after-market players who “plug” into something like Salesforce, it’s a major hurdle. Unless they can very quickly and cheaply customize their offerings for a myriad of different business cases, and move through POCs quickly, the SaaS hosting and costing model won’t do them much good. In my opinion, most existing BI SaaS vendors are currently struggling with this conundrum. Lucidera seems to have as well. Its demise was a shot across the bow. The first SaaS BI player to move past this problem wins the game and keeps the investors happy.

I have a hard time thinking of a SaaS BI vendor currently striking a reasonable balance between zero S&M and massive S&M. On one end of the scale, I see folks adamantly opposed to spending a dime on marketing and expecting serendipitous results. On the other end, I see vendors placing heavy bets on misguided or highly-targeted verticals. I see both strategies as precarious and hope for more level-headedness in the coming months.

No matter which way this goes, we're in for a wild ride. No prisoners will be taken. It's a great time to be in this business.

Yours in BI.

Of Birthrights and DNA in the BI Cloud World

2009-10-27T12:04:00.000-07:00

I have been so busy getting ramped up on GoodData that I haven’t had a minute to post here in over a week. On Friday, I am headed out to our R&D Center in Prague, Czech Republic - looks horrible doesn't it? :) - for a week to do a brain-meld with our engineering team and deep-dive the GoodData architecture. I’ve learned a lot about it in the past two weeks but clearly, there’s nothing like sitting down with the hard-core developers and discovering how the sausage has really been made for the past two years.

One of the unique things about this platform is the fact it was designed for the cloud from the get-go. “Born and bred in the cloud” as they say. And that was several years ago, before the cloud “explosion”, when the idea of running a full BI stack in the cloud was just a fantasy to most people. They said it couldn’t be done (matter of fact, some folks still claim that, but reality proved them wrong in 2009) but Roman Stanek and his guys did it and I think the results speak for themselves (here's one of many examples).

I think the “born and bred in the cloud” stamp is important because there is so much hype and hyperbole about all things “cloud” these days. Especially in the BI space. A lot of times when you hear or read things about clouds, it tends to feel like this (with apologies to my Thailand readers who obviously understand that script).

To separate the wheat from the chaff, you really have to ask the right questions. Namely, is your architecture really cloud-based or did you simply throw bits over the wall onto a bunch of hosted boxes (or VMs)? Do you understand the meaning and implications of real multi-tenancy? Are you simply shifting technology because it’s hip (marketing-wise), or do you truly possess cloud DNA? And on the non-technical side, do the economics make sense?

In the mid-1990s, ASPs (application service providers) were all the rage. Everyone wanted to “go ASP”. I worked for some of them. But the “software” crawled, got in the users’ way, was hell to maintain and deploy, and the costing model didn’t work. Many (most) went belly-up. But hey, at the time, it sounded good and the VCs wrote checks. But already back then, painful lessons were learned. Namely that it’s virtually impossible to change horses in the middle of a race.

The inherent value of “born and bred in the cloud” platforms is not simply technical to me. Clearly the engineering is intricate and “cool”, but what’s really at stake is what Roman calls “price-based costing”. (Speaking of pricing and clouds, check out Roman’s latest blog post, it’s à propos!). This is a Peter Drucker concept but it applies really nicely to cloud-based economics.

Price-based costing is defined as “choosing a desired sales price and costing out production to meet that sales price with a desired profit margin”. The idea is to first determine what you want to charge for a product or service, and then work the development economics backwards to achieve that price point. When you’re starting from scratch with a cloud-based technology, you can pull that off, because parameters, cost and elasticity are linearly deterministic. The up-front costs are fairly low and the flexibility is fantastic. Additionally, the tools are cheap, commonly available and simple.

Consider that the GoodData backend is almost all coded in Perl on Linux with a REST interface! (I’m not suggesting Perl or Linux are “simplistic” but I worked the MSFT development stack all my life, and I can see orders of magnitude in complexity reduction here).

So now, if you have legacy bits and a “ground-based” engineering platform (and people), you’re stuck on a price structure in this bearish economy, and you can’t just flip to a cloud model as a panacea, no matter what your Marketing people tell you. Whether you sell proprietary or open source software, you’re in the same predicament. Sure, you can provide less functionality or cripple your offering (or worse yet, try to make do with less people), but just throwing it inside a “cloud” won’t do it economically, and the outcome will likely be painful. The DNA just isn’t there.

So when some BI vendors simply slap some code (open source or not) onto AMIs (say like Mondrian for example) and call it a BI on-demand offering with a ROLAP engine, it makes me wonder if they “get” the concept at all. Mind you I ponder the same thing about big league BI tooling players who simply re-package their (oh so heavy) bits into virtual boxes and call it a cloud day as well.

With a genuine “born and bred” cloud model, you have fine grained leeway in adjusting the pricing dial. Guess what, you can even turn it down all the way to zero and still survive for a long time at given data volumes. At GoodData, for example, you can get 10MB accounts for free. Not a huge amount, certainly, yet sufficient to “play with”. But no matter how many people sign up for that, GoodData’s cloud platform can handle it both volume wise and cost wise at ridiculous scale and at the flip of a switch. Scalability in a “born and bred” cloud platform is very (did I say very?) cheap. And pricing can be finely tuned by throttling the technology, not the other way around. And this isn’t about taking a loss for marketing purposes. GoodData, like any other sound business (except OSS maybe), isn’t about giving stuff away for free, believe you me.

So I guess my point would be: caveat emptor. The next time you’re considering replacing that big honking on-premise BI tool with the vendor’s “new improved” cloud-based on-demand version, check under the hood and make sure that “born and bred” DNA is there (hint: not likely). And the next time you’re considering a “native” SaaS cloud vendor for your BI analytics, ask the right questions about what makes their backend tick. You might be in for an interesting smoke and mirrors show.

On Bringing the Good News While Clearing Out Fungus.

2009-10-19T10:46:00.000-07:00

I wanted to spend a little time sharing some good news with you. I am going to be Technical Evangelist for a relatively new company called GoodData. As evangelism means "bringing the Good News" to the world, this is truly a match made in Heaven for yours truly, as I will now be bringing the GoodData news to the world.

Many of you will have heard about GoodData before of course. Most people in our industry know it was founded by serial entrepreneur Roman Stanek of NetBeans and Systinet fame. Sun acquired NetBeans circa 1999 for $10M and HP grabbed Systinet in 2006 for a mere bag of shells ($100M of them actually).

GoodData is backed by heavy-hitting investors including Andreessen Horowitz (Marc was also an initial angel investor) but also O'Reilly AlphaTech Ventures and General Catalyst. It's always nice when you can point to Marc Andreessen pitching your company on CNNMoney. GoodData also cleared another funding round last week as anyone following our industry will have heard. This kind of news does not go unnoticed these days.

So what is GoodData about? GoodData is an on-demand business intelligence platform for collaborative analytics (phew!). This means the software runs in the cloud (Amazon's EC2 to be precise), so clearly the model benefits from all the usual technical, deployment, and financial benefits SaaS brings to the table. But more importantly, and what really sold me on the concept, is the following equation (and those who follow me know my affinity for all things KISS):

gd = BI - BS

In English, this says GoodData is business intelligence without all the "bullshiitake" (term borrowed from my hero Guy Kawasaki). So let's talk a little bit about that breed of mushroom.

When you need to implement a BI project but depend on IT to setup infrastructure and purchase software before you can even get started, that's bullshiitake.
When you need a PhD or significant training to start using a complicated overkill BI tool, that's bullshiitake.
When it takes you weeks or even months to analyze time-critical data for your company and show results to an impatient boss, that's bullshiitake.
When you can't collaborate with your peers and customers (internal or external) or integrate with outside apps while doing agile BI, that's bullshiitake.

GoodData represents a new way of thinking and acting about BI by removing these "mushrooms". In that sense, GoodData is really re-inventing how BI consumers and producers interact, behave, grow, and produce results together. And that, to me, goes way beyond a pure technology play. It is a “next curve” move and an industry-shifting vision I wanted to help forge.

So in the coming weeks and months, I am going to be talking about the technical and social aspects of this industry-shifting platform. You'll be able to reach me at my new work email and I will be twitting on the @gooddata stream as well.

My pitch to BI professionals will be simple. If you seek independence, control of your environment, quick and deep results, and the flexibility to enhance your company’s bottom line, then you need to try out GoodData. We represent a fundamental new way of thinking about BI. My role will be to show you how and why.

BI in a New York Minute is Born

2009-10-11T15:48:00.000-07:00

In my last post, I mentioned I was going to do a "trial & error" run on a recent concept of providing weekly BI tidbits (news items) for people on the go. Having had some time to reflect on this endeavor a little further over the weekend, and after consulting with some key people, I've decided to try another tack. Fact is, I prefer keeping my main blog for "deeper" less frequent posts.

So instead, I've decided to create another way to dessiminate this information. Additionally, I'm giving up on the idea of posting items on a weekly basis. Why cover weekly periods when in fact, BI news is occurring pretty much 24/7 in real time nowadays including weekends.

To deliver better, I have created a new blog called "BI News in a New York Minute" where I simply post tidbits in real time. When I get them, you get them. People can subscribe to this stream via the blog (obviously) or by pulling feeds from http://biminute.blogspot.com/atom.xml. Additionally, I've linked BI News in a New York Minute to Twitter so updates are also broadcast there when available (almost, but not annoyingly, in real time). So if you aren't following me in twitland yet, point here and click "follow" :)

This is just one modality of a more general media concept I am working on for the Business Intelligence industry. In the meantime, I am hopeful (and grateful if) you will give me feedback and suggestions!

Yours in BI.
J.

Your Weekly BI News in a “New York Minute”

2009-10-09T16:13:00.000-07:00

I’m a glutton for BI information to the point of addiction. Aday rarely goes by without my “scanning” three hundred or so feeds, blogs, twits,websites and other news and information sources pertaining to businessintelligence. I did the same thing whenI was on the software development side, mind you, but using significantly less onlinesources and way more books. I find BI to be so dynamic an industry that by thetime most books are published, the information is already obsolete. And that’s just on the technical side. On the business front, things change andevolve even faster.

So I started looking around for an aggregated “digest” ofthe day’s salient BI news points. I was looking for something that spanned themultitude of industry areas in BI from engineering and product news (platformto front-end), to business development, partnerships, personnel (who quit,joined or got fired), distribution (SaaS and cloud issues), venture funding, importantconferences, and so on. In a nutshell,all the areas that affect our industry one way or another. To my surprise, I couldn’tfind anything readily available. Somethingsomeone could take on a train or pull down to a mobile device during a dailycommute to get a quick “scoop” on the week’s happenings in BI.

Then I figured, why not try my hand at compiling such a list ona weekly basis. I know this is fairlysubjective, but while scanning my own sources, I always “pull over” severalitems for more in-depth examination. I do this very quickly (I’m a very fastreader) and often “bucketize” these links using Delicious.com for furtherexamination and/or analysis. Clearlypeople have different areas of interest but I figured this pruned list couldbenefit some folks with no more than a “New York Minute” to spare.

I’m not sure how best to format and present this yet. I didn’twant to start off with some sort of mailing list offering because, quitehonestly, I’ve never stuck with a mailing list subscription more than a coupleof weeks myself. So for now I’m going totry and post this on my blog on a weekly basis every Friday. If I detect minimal interest, then I’ll try anotherapproach. I'll also include more items in future weeks. This is more of a "trial and error" endeavor at this point. Meanwhile, and without furtherado (or order), here’s my BI Digest for the week of October 5^th,2009 "in a New York Minute".

Discrete SaaS player 1010Data scores a big contract in the retails space by signing up Dollar General.
MicroStrategy is far from Open Source but giving away software for free nevertheless now. Trojan horse or good value?
Awesome webinar on what was learned in the past 20 years in BI compliments of Claudia Imhoff and WhereScape
"But can it core a apple"? On integrating OLTP and OLAP in the same database engine. One of the most commented (and educational, for me) Curt Monash posts I’ve seen.
First of a two-part in-depth series on MapReduce vs. relational databases.
You might not be paying your DBA enough money to put up with this (alternatively, you might be using the wrong DBMS platform ).
BI and DW “implementors“ take heed. You may need more duct tape after all (this doesn’t just apply to the software development community, believe me).
For a deep (and I mean deep) dive into what customer analytics in retail are all about.
A really novel (read: smart) way to engage your market without the BS, hassle and expenses of conventional physical conferences as described by Merv Adrian who tried it.
Dashboards in Excel 2010 (yawn). Sorry it’s been hard to get excited about MSFT BI in, oh say about 18 months now.
Can (or should) InfoBright actually kick MonetDB’s rear-end? The numbers are (almost) in.
In-memory databases and the Kognitio men who love them.
Well, looks like nobody is using EC2 after all (NOT!)
Larry invites Marc to Oracle Open World next week. See rumors fly...
At least one person is excited about Gemini (but it’s a monkey so..).
If you live and breathe “fabric” and grok fiber over Ethernet, this one’s for you.

And that's the way BI is. Goodnight, and enjoy your weekend!

Baking MapReduce into Database Engines - Worth the Reduction Sauce?

2009-09-30T10:43:00.000-07:00

MapReduce (implementations include Hadoop and CloudDB) has gained popularity in the industry. It also serves as marketing fodder for several new-breed ADBMS vendors who now claim to support it in various forms. So what is really behind this magic pixel dust, what problems does it solve, and how relevant is it to someone deciding on a new (or additional) ADBMS platform these days?

First, let’s point out MapReduce is not a technology but an algorithm. Wikipedia defines an algorithm as “an effective method for solving a problem using a finite sequence of instructions.” In MapReduce’s case, the problem being solved is the processing and analysis of very large data sets. The solution is a parallelized divide-and-conquer approach and works like this. First, you split up the “problem” into small manageable chunks. Second, you fan out each chunk in parallel to individual “work units” (maps). Third, you take individual results from each unit and recombine them into your final result (reducers). In SQL parlance, conceptually, it’s like doing a select aggregate with a group by.

There are file based and database-centric applications of MapReduce in existence. Of course, the presumption is that your “problem space” can be split up in distinct pieces and recombined without information loss. And not all problems are either large enough or mathematically suited to this approach. But luckily data management is, by definition, perfectly well suited because, as a mathematician once told me “every data management task can be broken down into two and only two activities: partitioning and equivalence”.

Some folks think MapReduce is a modern breakthrough concept, but they’re wrong. The application of this algorithm to the management of large data is nothing new, as pointed out by Dr. Stonebraker in a 2008 posting. What’s “new” about MapReduce is that Google has popularized it. And the thought is, if Google can process and analyze the entire world via MapReduce, then clearly MapReduce must be the Holy Grail of monster data management. But Google has unique challenges (gargantuan data volumes), and some very impressive (and plentiful) gray matter at its disposal.

Because the interesting thing about this “divide and conquer” approach is that, although fairly easy to conceptualize, it’s incredibly hard to implement properly. The human brain is really not “wired” to think in parallel. Research has shown that the top brains can at best juggle seven different objects simultaneously (and for short time periods). To understand the intellectual challenges at play here, I strongly recommend watching this Google video series.

As I understand it, implementing MapReduce correctly and efficiently is probably as hard as conquering multi-threaded programming. And in twenty years, I have met three people who really understood multi-threading correctly and two of them were Russian PhDs. I've had battle-tested architects tell me they would rather shave with broken glass than tackle the risk and difficulty of multi-threading (luckily, they weren’t designing operating or flight-control systems!). My point is, it takes some pretty special skill and talent to do it right. Nothing inherently wrong with that, but it’s neither quick nor cheap.

So why then would database vendors race to support MapReduce? After all, dealing with and managing relational systems is complicated enough as is. But at least people have been trained in the art for decades, and SQL is lingua franca. So the pitfalls and solutions are well established. Additionally, Codd’s premise guaranteed abstraction by separating the logical layer (SQL and a normalized schema) from the physical one (hardware and storage). But MR is heavy with non-standard cross-layer implementation details by necessity. Clearly a step backward from the KISS principle (even a “major step backwards” if you buy into Dr. Stonebraker’s argument that MapReduce is offensive).

Regardless, three well-known new-breeders, namely Aster Data, Vertica and Greenplum jumped on the bandwagon early on and announced “MapReduce implementations” for their product. I wondered what compelled them to invest time and resources into something that didn’t seem essential (or cheap) to the market at large. Are users really clamoring for MapReduce support in their warehouse engines?

To learn more, I went to YouTube and checked out Aster’s video “In Database MapReduce Applications”. In it, I learned that graph theory problems (think: travelling salesman) were well suited to MapReduce but not SQL. Examples included social networking (LinkedIn), Government (intelligence), Telecom (routing statistic), and retail (CRM, affinities), and finance (risk, fraud). Pretty much anything that can be modeled using interconnected nodes. But a connection from a node to another is really a “relation”, and so clearly well suited to a “relational engine”. So I might have missed something.

I also learned that existing applications typically extracted data from the database, performed some analytic work on it, and then pushed the data back into the store. In other words, they couldn’t perform processing inside the database. I found that generalization hard to swallow but reminiscent of numerous past battles on whether “business logic” belongs in the application or database layer.

Aster’s implementation of MapReduce is “deep inside” their engine, from what I understand. One example I could find was yet another YouTube video called “In-Database MapReduce Example: Sessionize”. In it, Shawn Kung shows a MapReduce function being used inside a SQL statement to “sessionize” user IDs in a clickstream context. Aster also provides very basic how-to’s on their website and blog. Clearly Aster is targeting this new MapReduce capability at the DBA side of their users, and it looks a lot like leveraging UDFs to me. Aster’s conclusion: “we need to think beyond conventional databases.” I’m all for that!

Next, I wanted to learn about Vertica’s implementation. Especially since Vertica’s own Dr. Stonebraker had initially nailed MapReduce pretty hard as mentioned above. But Vertica’s new position seems to be that MapReduce is a-ok after all, provided it remains external and doesn't pollute the purity of the relational engine. I couldn't find much on YouTube or their website save for a press release dated 8/4/09 stating “With version 3.5, Vertica also introduces native support for MapReduce via connectivity to the standard Hadoop framework”. It seems the “scoop” on the Vertica/MapReduce wedding is best described in their corporate blog. Basically Vertica is OK with "integrating" or connecting to but not "ingesting" MapReduce (via Hadoop) if I understand clearly.

I was also able to glean some tidbits from Omer Trajman on Twitter. Namely that Vertica supports Hadoop “adapters” which allow you to read and write into the database (which is basically the press release). I wish I had more in-depth information about Vertica’s MR functionality but even a basic search for the term on their overly busy website yields zero information and, unless I missed it, I couldn’t find any relevant webcasts either.

Greenplum were, if I am not mistaken, first to support MapReduce. Greenplum has the best MR resource online if you ask me. It’s clear, detailed and full of insight. Greenplum has a merged/cooperative DBA/programmer approach in their offering. Programmers can write maps and reducers in their language of choice, leveraging DBA generated data sets as (and if) needed, and DBAs can use MR functions along with SQL without (presumably) getting their hands dirty. There isn’t much to add to this excellent resource so I won’t.

So having mapped out all these facts, what can we reduce from it (I’m so funny) and more importantly, should any of this stuff matter to prospects when evaluating ADBMS vendors? IMHO, you might benefit from a MR-enabled ADBMS if:

(1) You have petabytes (or more) of data, an MPP architecture, and a search, scientific research, or mining problem a high-performance SQL engine cannot handle.

(2) You don’t have heavy legacy systems. Integrating (or migrating) existing business and relational code with a new-breed MR-enabled engine can’t be fun, quick or cheap. You might be one of the lucky few with pet projects on the table.

(3) You’re in Academia and have access to numerous cheap and competent programming resources, lots of metal, plenty of time, and limited pressure to succeed.

(4) Your organization has a track record of successful projects dependant on symbiotic working relationships between your DBAs and your programmers. In my experience, DBAs and programmers don’t work well together. They have different goals and approaches. And it seems intellectual and political integration of both resources would be a sine qua non condition to success with an MR database product.

Short of that, I can’t imagine too many people lining up at the MR-ADBMS vendors’ doors simply based on their MapReduce capabilities. And I don’t think vendors make that case either. In my opinion, supporting MR in the product simply says “Hey, look at me, I’m at the forefront of technology. See how smart I am.” But as a buyer, I’d be a little concerned about overreach.

In fact, I wonder how these vendors spread resources efficiently (and economically!) between database engine building, cloud provisioning (which Aster and Vertica now pitch), and MapReduce integration. I suppose marketing requires less focus than engineering as a discipline but still, that’s a lot on one’s plate.

SELECT SUM(blessings) FROM working_in_bi

2009-09-25T13:32:00.000-07:00

If you’re working in the business intelligence industry, you should really count your blessings. Mind you, you can do that if you’re employed anywhere these days. But there’s something special about what I call the “BI Family”. I’m not referring to a specific segment of BI, and I am not focusing on any particular job function. I’m talking about working in the BI industry as a whole.

I feel somewhat qualified to comment on this because, in the past twenty years, I’ve worked in numerous industries. To name a few: life sciences, research, semi-conductor, telecom, payroll, accounting, systems integration, consulting, audio-visual, online media, financials, accounting, and insurance. So I have a lot of background to compare from. And believe me, there are worse adoption outcomes than membership in the “BI Family”. Here’s my subjective top ten list of why working in this industry is really cool (no specific order).

#1. Brains

People are really smart. I’m not suggesting other industries spawn dummies, but the proportion of high IQs in the BI world always amazes me. It’s often humbling and always stimulating.

#2. Strength

As we all know, the BI market is not only huge, but getting larger and growing yearly to the tune of 8-10% a pop. Other industries are not so healthy to say the least. Simply put, BI is clearly not a fad, and no one questions its future. It’s one of the fuels of our free-market system by protecting businesses and providing them with competitive tools.

#3. Meaning

Guy Kawasaki says: “make meaning”. He’s right. Life’s too short to be in a meaningless industry. And if BI isn’t about making meaning then I don’t know what is. The whole purpose of the industry is to make meaning and support critical decision making. This industry yields real-life significant solutions to crucial sectors like health, research, medicine, and defense (to name a very few).

#4. Quality of life

In light of ominous HR predictions, news of recurring layoffs, and current employment trends in the IT industry, the BI sector has been relatively spared. Clearly I haven’t done any formal polling but I get the “vibe” that people are generally pretty happy to be in this game. And why not. Compensation is generally good. And location-wise, BI companies are clustered around national funding clusters namely Northern California and the New York/Boston areas. These comprise some of the most magnificent (and most expensive, granted) landscapes and vibrant urban areas in the country. Other industries have centers in, shall we say, less compelling geographic areas.

#5. Funding

I lamented the VC situation in my previous post, and clearly this doesn’t apply to all segments of BI, but if you have any sort of compelling BI proposition with the word “cloud” in your business plan, trust me you will get a VC’s attention. Maybe not an official invite to pitch in person, but most likely a phone call. In other industries, that opportunity is long gone.

#6. Gratification

BI projects used to take many months (sometimes years) to implement (when they even got completed). But nowadays the industry is in “agile” mode. And those who don’t embrace that won’t likely be in this business much longer. This means you get to build solutions and see results quickly. That’s gratifying.

#7. Globalism

BI is world-wide. True, so are most other industries, but from my experience, there is less of an “us versus them” attitude. It has a fraternal feel to it. Hands and minds seamlessly reach across continents in ways I have not experienced elsewhere. (I’ll go hug a tree now).

#8. Bozos

Most people are genuinely nice and unassuming. I know this sounds naïve at best but it’s true. I have not seen the level of ego, axe grinding, or personal animosity frequent in other industries. Every contact I’ve initiated from top analysts to CEOs in this industry has been followed up promptly with courteous, genuine and insightful discussion. I’ve found most people to be more generous with their time and advice than in many other industries. Maybe they fear less for their jobs or fancy titles. In either case, the BI industry is fairly low on the bozo scale.

#9. Passion

People in BI are passionate about their field. I’m not saying they get out of bed every morning to go save the world (onward BI soldiers), but overall they value their work and their contribution. Most people I’ve met in this business are workaholics. They know their stuff inside-out and boy do they love to talk about it. Passion signals a great, vibrant industry. Additionally (and this key), there seems to be better customer advocacy in this industry than others. It’s not perfect, but vendors often do listen and react accordingly.

#10. Innovation

I hate to reveal this well-kept secret (don’t tell anyone!), but there isn’t a lot of desire to innovate in the financial, accounting or insurance fields, for example. I’d be preaching to the choir by pointing out the myriad of new-breed ADBMS players out there, but also the multitude of new OLAP, data mining and analysis products, approaches and new (non-relational) ways of looking at data, cloud BI, EC2, etc. We’ve seen orders of magnitude of both hardware and software innovation in the BI world. It is a rich intellectual field teaming with innovation levels typical of a “new frontier” because it is.

So what’s the point of this apologist diatribe? Just to remind people in this field to count their blessings. There are many worse places and industries to be in. And it’s easy to take things for granted in the heat and excitement of daily business life.

In the past eighteen months I’ve met many challenges in this business. From coding to QA, to technical writing, from sales engineering to evangelism, from product management to market analysis. You name it. So I’ve seen a lot of the facets in a very intense, very short amount of time.

And I’ve also been lucky to interact with numerous players in the industry, many of which have generously spent time and resources supporting my self-education efforts with their insight, connections, and advice. You guys (and gals) know who you are and I thank you for the help. There are many mensches in this business.

For the first time in my life, I think I can say I’ve found a home here in the BI industry. I’ve never felt this way in the past twenty years, and I’m not exactly sure how to explain it, but like the old pair of shoes my wife keeps insisting I ditch, it just feels right and I’d like to keep it that way.

Please Stop Making More ADBMS Sausage

2009-09-22T13:20:00.000-07:00

If you’re thinking about building a new startup in the high-performance analytical database (ADBMS) market, hat’s off to you: kudos and respect my brother. I’ve been in the kitchen, and I’ve seen the sausage being made. But let me tell you something: you might be a day late and a dollar short to the party.

In the past years, I’ve often pondered where the new-breed high-performance analytical database industry was headed. Will the existing players manage to survive? And is there a chance in hell for new ones to succeed in this market? If you had asked me to peek into my newly minted BI crystal ball in early 2009, I would have said “no way”. Why? Because at the time I was predicting the demise of a majority of the twelve or so players in this space based on the observation that the field was too crowded and too expensive. I figured, in this cut-throat competitive space, and with tough economic times ahead, we’d be lucky to see two, maybe three survivors come 2010.

Since then, not only have several other significant players, technologies and business models popped up (for example, Groovy, XtremeData, VectorWise, Hadoop/MR and the OSS guys), but we have clearly not seen the level of attrition I was anticipating. Nobody has (officially) gone out of business save Dataupia as best I can tell, and Datallegro got a check from Steve Ballmer. Sure, some folks are experiencing tougher times than others (I dare say several are hanging by a thread) but overall, resilience has been the name of the game. So what gives?

From my point of view, there are two types of new-breeders really: those living in “comfortably numb” mode (quietly outliving peers may not be a bad strategy these days), and those kicking it into high-gear with a vengeance. In the latter category I can’t help but think of Vertica, Netezza, Aster Data and ParAccel. It takes a lot of “cojones“, cash and luck to build a new ADBMS company. But even blessed with all these, and given the proper planetary alignments, I would advise anyone considering a start from scratch nowadays to ponder the following points.

First, it is insanely expensive and complex to develop systems software to produce a complete analytical database engine (and I mean “complete” in a holistic Product Management sense). Tony Bain highlights some of the database startups challenges in his excellent series starting here (things are not significantly different between OLTP and OLAP in this respect). But systems software is a different animal than your run-of-the-mill corporate enterprise application. On average you’re looking at 200,000 man-hours and anywhere from $60M to $100M to fund such a venture to completion. This is just to get rolling. Additionally, you cannot just put out a database product and call it a day.

Maintaining (enhancing) a behemoth of Oracle or SQL Server stature runs hundreds of millions of dollars every single year. Everything from equipment to talent costs more when developing database software. Believe you me, the folks who will work on your SQL optimizer, inter-fabric communications, parallel or compression schemes better not be affordable newbies. Your development platforms won’t likely be the average oh-hum laptop attached to cheap storage. An efficient QA or Performance Group will cost a small fortune in payroll and redundant equipment. And seasoned performance architects don’t run the streets. You cannot assemble a database engine product by cobbling together open source bits and distributed talent like a new Web 2.0 RIA venture. You can’t take a Kia (even a dozen of them) to the Indy 500.

Second, I think it’s no longer possible to find sufficient levels of Venture Capital funding for such endeavors. My feelings on this issue are re-enforced when I read articles like this, or this. I think the writing was on the wall for several years now. Reports of the VC’s demise are greatly exaggerated, but the funds, the endurance, and the risk acceptance levels are gone. Small bets on small returns are in. Large bets on IPO-driven returns are out (for now). Even if you manage to score a major industry name like Mike Stonebraker on your Board, I think VCs in this space (those that are left and not now engaged in M&A) will say “talk to the hand”. Investors in numerous existing new-breeders are biting their nails to the bone (or looking for ways out). So to me, the train has left the station. And unless you can pony up your own seed money, trying to fund such a project via institutional money is currently, in my opinion, an exercise in futility.

Third, the field is already too crowded and spread out very thin. For a great overview of the major players out there, don’t miss Bloor Research’s competitive analysis paper. A lot of existing players do not have sufficient “boots on the ground” to make headway against larger ones, much less established powerhouses. Heck, even going against Vertica’s deep-pocketed marketing is no piece of cake. Worse yet, in this business, success is not guaranteed by technical superiority. I know it sounds heretic saying this about an industry dominated by performance claim testosterone, but it’s true.

Besides technical prowess, you need to get the word out louder than everyone else. Unfortunately, everyone has the same “word”. In a crowded space this means you have to yell “Fire!” pretty darn loud and relentlessly to get noticed. I think a lot of “database deals” are sealed on the golf course, more so than POC or bake-offs. Mind you, this is probably the case with most enterprise software. But to get your foot in the door, you need BOD and Primadona action. BOD are well-connected heavy hitters on your board. Primadonas are the star sales guys (or gals) currently working for your competition. Those you’ll have to poach with sweetheart deals to come work for you, a totally unproven new-breeder with a year of runway to go.

Fourth, pricing pressure in this business is relentlessly choking. This is a consequence of my previous point. A little over a year ago, word on the street was $100K/TB retail ($50K/TB street price) but now we’re seeing $20K/TB retail (TwinFin land), which probably means you can do $10K/TB on the street. Aster Data is pitching an appliance for $50K (1TB, includes Dell hardware), and Oracle’s new improved Exadata V2 (SATA storage) even touts $5,700/TB so I mean, at these margins, you’re basically talking about giving stuff away, and in a lot of cases, I suspect that’s what’s going on. So unless you’ve been around the block a bit and have some ammo in the bank, I don’t know how a newcomer can sustain this type of pricing “carpet bombing”. As if that weren’t enough, you have OSS and cloud players breathing down your neck. Customers expect more for less and perception of BI as a “commodity” is growing. In this pricing environment, survivability for a newbie is improbable at best.

Fifth, several windows of “technology opportunity” for ADBMS are closing. For example, if your great idea for a new ADBMS company involves a columnar approach, you might be too late to the party. If your “innovation” hinges on massive parallelism, compression, in-memory caching/cubing schemes, super-fast intra-nodal fabrics, hybrid MPP/SMP, hybrid row-column storage (PAX-like), or yet another SQL chip accelerator or super-duper FPGA, you might have missed the boat (on the other hand, if you figured out how to do analytics on compressed encrypted data, then you might be on to something).

I believe the top new-breeders did all the technical legwork in the past 4-5 years. It took Mike Stonebraker long enough, but Vertica pretty much put columnar on the map. And most of that engineering is now mature enough (and well proven) to warrant acquisition interest from the big boys. Initially, the big guys took a “wait and see” attitude (remember, OLTP butters their bread anyway, not analytical OLAP) but now, having seen results and traction on others’ dime, I think they’re ready to pony up some cash (classic buy vs. build decision) to absorb the bits and pieces suiting their marketing strategies. By doing so, the good ole boys re-invent themselves and say “hey look, we have columnar technology as well now!” (How Sybase didn’t corner this market with IQ is beyond me, especially having read Seth Grime’s excellent paper about it).

Better yet, by integrating new technologies into existing code bases, the big dogs can say “hey, we have the best of both worlds for OLTP and OLAP” (Oracle’s latest Exadata comes to mind). And perhaps “look, we have SMP on the processing side and MPP in storage layer”, or vice-versa, thereby returning to the old “one-size-fits-all” GP-RDBMS paradigm so criticized by Stonebraker (but so convenient for the corporate user).

And clearly, given the growing popularity of “operational analytics”, an OLTP+OLAP offering is compelling. So I think the “proof-of-concept” window for many new-breed technologies, specifically MPP columnar (but others as well, for instance, acceleration hardware, where Ingres is picking up VectorWise, or MPP where Microsoft snapped up Datallegro), has closed. The winners (and their results) are in and acquirers will likely make their move in 2010. This dovetails nicely with a recent TDWI survey claiming half the respondents plan to replace their DW platform between 2010 and 2012 (apparently this is presented here on October 7^th).

All this being said, is it possible that a brand new software endeavor currently in stealth-mode development in Nepal might suddenly dominate the analytical database scene within months? How about a revolutionary FPGA/SQL Chip/Flash/Optical hardware contraption that could blow the hinges off industry standards and benchmarks? Sure why not. Real innovation is usually unexpected, and often unintended. But I don’t see it being driven by the classic “VC funds startup makes big database scores big IPO” model much longer.

When I look at things like Hadoop and current developments in the OSS space for VLDB analytics, I still have trouble grasping the business model, but I clearly see “life force” innovation at work here. A year ago I would never have expected a place like Visa to stray from the “Big Threes” but nowadays these guys are messing with Hadoop! People are also doing amazing things with MapReduce implementations and BigTable KV types of massive data storage systems.

How does open source fare against the five points mentioned above? Pretty darn well if you ask me. Costs are significantly lower, venture capital is not needed or minimal, engineering is crowd-sourced, there’s more breathing room, market entry is viral and massive, distribution and testing self-fueled, and pricing (or lack thereof) better controlled.

Additionally, open source seems shielded from the “Borg Effect”. I don’t see how massive proprietary shops like Oracle, IBM or Microsoft can successfully “absorb” these entities. I don’t think Larry has a clue what to do with MySQL. He can’t really unload it, but he can’t really integrate it either. Darn Trojan horse! In 2008, Infobright went Open Source and raised $10M in the process. Looking at the results, I think these guys were smart!

So if you’re thinking about building yet another high-performance analytical database engine the classical way (and not going OSS), my advice to you is simple: unless you have $60M in the bank and technology significant (and new) enough to impress people like Daniel Abadi (good luck on that one), you might be climbing up a greased pole. I'm not saying it's impossible mind you, but there’s been a lot of cooks making the same sausage over the last five to six years to last us a while. Maybe it's time to look at the next curve.

Vaya Con Dios - The Day I left XSPRADA

2009-09-17T11:14:00.000-07:00

Effective today, I will no longer be working at XSPRADA. There, I said it. Catharsis, take me away! It was definitely a tough decision but this is par for the course in the startup world, and God knows I’ve been there done that, but in this case, my close long-time personal relationship with the founders and unwavering worship of the technology for the past ten years make this particularly bittersweet.

I was very lucky to work with some of the best, most resilient people this industry has to offer and I don’t claim this lightly. As you know, I am parsimonious with compliments. But "they" say that tough times never last, only tough people do. This is why I am still convinced that XSPRADA technology will someday take its proper place in the world come what may.

A failed endeavor is only one during which you have learned nothing. And in this case, the XSPRADA opportunity has enriched me in personal and professional ways beyond my wildest expectations. So I depart a richer, more experienced man for it. In an industry too often clouded by “smoke and mirrors”, I am always reminded of the following advice I received a while back:

Don't profess about things you are not sufficiently familiar with.
Don't assume something isn't true that might be, or is that isn't
Listen at least twice as long as you talk.
Ask twice as many questions as you answer and LISTEN to the answers.
When asked about something and you don't know, say "I don't know."
If there is even the slightest possibility that you could be wrong, acknowledge it....
and don't forget to thank people for their time and advice.

You combine that advice with the one found here and believe you me you’ve got yourself one kick-ass sales engineer there. They don’t run the streets. I’d say these points could (or should) constitute the Seven Commandments of the Sales Engineer (and probably any other profession, except perhaps politician). They were given to me when I started by Chris Piedmonte, founder of XSPRADA, and a guy whose courage, integrity and technical brilliance are, in my book, without equal.

Be that as it may, I must soldier on. Naturally, I will not be able to discuss topics pertaining to XSPRADA technology as an insider from now on, but Lord knows there is sufficient ADBMS/BI material out there that’s interesting enough to cover and discuss on a regular basis. Namely, the new “PAX Analytica” movement as originally brought up by Curt Monash and professionally laid-out by Daniel Abadi as usual in his excellent post.

Also, the old one-size fits all (OLAP+OLTP+whatever) versus dedicated engines (columnar OLAP) has been revived with the recent ORCL announcement touting the new improved Exadata V2 (exit HP, enter Sun). This deserves addressing in more detail. It leads to serious implications for current and potential customers.

Relevant as well is a discussion about the future. Will the “big boys” end up swallowing “new-breed” technology and integrating it (what I call the Borg effect, as discussed on Daniel Lemire's excellent post). Or will they become obsolete allowing the new-breeders to survive long-term as independent replacement entities?

And then there’s a recent thread about analytical speed which is very relevant at this point in time I believe. Finally, a little bird told me there’s about to be some really interesting rumble (again) pertaining to the infamous (or not, depending on which side you’re on) TPC organization. Indeed, there’s no lack of interesting topics out there!

Additionally, I think there’s a compelling story behind employment search and provisioning in the BI industry, so I’ll be penning some thoughts about that as I go along. In my experience, you can infer a lot about an industry’s state by checking its recruiting culture and pulse. Joy Chen claims in a recent blog posting that 54% of workers plan to resign after the recession. If this prediction is correct, the impact on our industry is sure to be felt and that, IMHO, is worth discussing.

So what’s really happening behind the employment scene in BI? I’ll be sharing some thoughts about that as I embark on the new path the BI Gods have charted for me. So thanks for sticking around, and as they say where I come from (well, ok maybe a little further South) Vaya con Dios!

Mole Whackers Need not Apply

2009-08-26T15:29:00.000-07:00

Two completely different events caught my attention lately. One of them is a post by Curt Monash called Bottleneck Whack-A-Mole, and the other is the much-publicized alliance for “BI in the Cloud” comprising RightScale, Talend, Jaspersoft and Vertica.

In the post, Curt describes software development (or developing a good software product) as “a process of incremental improvement”. Fair enough. The analogy he draws is between constantly fixing and improving performance bottlenecks and the annoying (if entertaining) arcade game of Whack-A-Mole where you have to be fast enough to clobber enough of the critters as they randomly pop up from below. He then makes the point that “Improving performance in, for example, a database management system has a lot in common with Whack-A-Mole.” Having spent most of my life designing, developing, improving and testing commercial and enterprise software applications, I have to say I don’t totally agree with his analogy for several reasons.

First, call me old-fashioned, but I’m an ardent believer in the fact that software building is deterministic. Whack-A-Mole engineering is not. The age-old controversy about software being more of an art than a science may never be resolved, but at the end of the day, I feel software is (should be) a scientific, engineering-driven, deterministic endeavor like any other engineering discipline. With Whack-A-Mole engineering, buildings and airplanes fall to the ground. That’s not good. In my experience, those who seek to “romanticize” software engineering are typically adverse to proper planning, design and testing as being too “dry” or unworthy an endeavor. That’s nonsense.

Second, there is a distinct difference in the way you develop “regular” software from “system software” and I’ve learned this from sitting in the front row the past several years at XSPRADA watching database software being built from the ground up. It’s a little bit like the difference between building a tree house and a major commercial skyscraper. And I believe that playing Whack-A-Mole games while trying to bring up a building is a scary proposition at best (especially for future tenants). And yet, the example Curt provides involves Oracle’s Exadata, of all products! He states: “When I spoke to Oracle’s development managers last fall, they didn’t really know how many development iterations would be needed to get the product truly unclogged” – This statement is mind-boggling to me.

Because for one thing, it suggests that Exadata is “clogged” (ouch) but worse, that their engineering people have no clue as to how they might eventually (if ever) snake the blockages out of it! So, basically it’s a trial and error approach to building a database. Notwithstanding their “professed optimism” that it wouldn’t take “many iterations at all” to finally figure things out, it certainly doesn’t give me (or any reasonable person) a warm feeling about a multi-million dollar product claiming to be the world's ultimate analytical machine.

I think there’s a lot to be said for sound engineering practices, proper planning and testing, setting expectations and deterministic engineering management practices in the world of system software. That Oracle (or Netezza for that matter, also referenced in the post) might just be going along whacking moles instead is a scary proposition indeed. Even if this little game is limited to “performance engineering” as Curt suggest (as if there was a more important endeavor in an ADBMS), that’s a serious allegation in my book. I say leave the arcade games to the kids, and let the real engineers design and implement database and system software please. There’s no room for amateurs in this game.

On to my next point of interest: the new Gang of Four in the Cloud (with apologies to design pattern aficionados) comprising RightScale, Talend, Vertica and Jaspersoft have recently promoted and demonstrated a “bundled” on-demand package for the cloud. I attended their webcast yesterday and was impressed, but with reservations.

Each of these vendors is impressive on its own, no doubt about it. But it seems to me the bundled proposition might be confusing at best to the unwary customer. This new offering is billed by the marketing folks as “Instant BI, just add water” which drives me nuts. Look, it might be simple in theory, and it might take a few minutes to setup the stack on your own (as Yves de Montcheuil from Talend claims) but it’s still a long way to actually accomplishing anything serious in a few simple clicks. Sorry, not going to happen anytime soon.

You still have to work your way through provisioning and instance management (RightScale), data integration and loading (Talend), feeding and configuring the database (Vertica), and setting up the reports/analytics you might need (Jaspersoft). All of which can be accomplished just as easily (or not) internally by the way. It’s true you’d still have to purchase or license Vertica internally, which may or may not match the SaaS pricing I’m not sure (and either way, Vertica has a SaaS offering as well) but the other components are open source so, I’m not sure I see the big advantage there.

An interesting thing I noticed as well is that some people didn’t seem to understand what RightScale’s role was in the whole offering. This tells me they don’t really grasp the intricacies of “the cloud” – because instance and infrastructure management for enterprise in the cloud is not trivial and you do need something like RightScale to grease the wheels (it’s an abstraction layer really), but I think many people assume moving to the cloud is “magic” and makes all these issues disappear. If that were the case, you wouldn’t need RightScale in the mix. Beware undermanaging expectations I'd say.

Additionally, the pricing model (which is supposed to be so much simpler in the cloud) is confusing at best as each vendor has its own menu. The best answer to that I can remember was “starting at $1,700 per month” – I’m not sure what to make of that. So I think from an engineering/technical standpoint, this endeavor is noble, but from a “let’s make things simpler and transparent for the user” perspective, there’s still a lot of work to be done. In other words, it's a nice play for the vendors holding hands, but I'm not sure how beneficial it might be to the average enterprise user.

As usual, caveat emptor – Beware promises of a holy grail in BI as there is no such thing. It’s all about work. Hard, detailed and careful work with proper planning and budgeting. In that respect, setting up successful BI solutions is a lot like running and implementing software projects. There are no shortcuts, and it’s not a job for mole whackers.

Oh yeah? Well my database is SMALLER than your database!

2009-08-17T13:27:00.000-07:00

Contrary to popular edict, smaller is not always better, unless of course you’re talking about analytical database engines. In that respect, it’s hard to find an ADBMS that can fit on hard media like a CD or a USB stick. For example, I don’t think SQL Server, Oracle, DB2, Greenplum, Aster, ParAccel, or the myriad of other ADBMS vendors can fit all their bits in a tight spot. Even in the open source realm, I doubt you can wedge InfoBright (MySQL) or IceBreaker (Ingres) onto a stick, much less shlep their bits around as an email attachment.

One exception to this is the V-stick from Vertica. When I first read about this, I initially thought it was a hoax but apparently not. It’s pretty cool too because it includes the O/S, web server, GUI and the engine all together on a 16GB thumb drive. How an engine like Vertica, designed around distributed MPP, can possibly operate representatively (using terabyte-size data) on a thumb drive is beyond me, and I’ve never heard of anyone actually using this gizmo but I’d sure love to get my hands on one and review it if it’s still available.

The other exception of course is RDM/x, the XSPRADA database engine. The reason is simple: its total deployment footprint is around 10MB. That includes the 32/64 ODBC drivers and a couple DLLs. The engine itself is currently around 6MB. Last I looked the installer clocked in at 16,760KB. This means you can actually deploy RDM/x onto a memory stick if you want to. I tried it, it works. It’s pretty cool. But after a while I wondered, why would anyone care about this?

The reason is two-fold. First, it’s really easy to try out software that is small and self-contained without expanding large amounts of time and resources. Yes, you can download RDM/x from our website but in many cases (like secured firewalled enterprises), that’s not an option.

Second, it means we’re a good candidate for embedded applications. Because if I can fit my database engine on a stick (or in an email), I can probably embed it in instruments and devices as well either as raw C++ code or libraries.

But for quick POCs, size and simplicity really does matter. Say you’re suddenly tasked with evaluating solutions to deploy a BI solution inside your company. Suppose you’re a Microsoft shop. Suppose additional capex is not an option, and suppose further you have a week to show results (namely a set of nicely formatted reports, pivot tables or dashboards). Now what? If you have significant in-house experience with SQL Server and associated SSAS, SSIS, SSRS, and Excel (and assuming you have a clear and deep understanding of the business scope and goals to begin with) you’re probably going to:

(1) Figure out where your source data is coming from (connection strategies)

(2) Model your DW (figure out grain on facts, dimensions etc, need to figure out BIDS and SSAS)

(3) Establish some preliminary ETL process (including incremental loads, need to figure out SSIS)

(4) Load your warehouse (if you screw it up, then need to drop and do it over)

(5) Setup an SSAS cube structure (figure out SSAS via SSMS or BIDS then publish the thing)

(6) Figure out what queries to generate (talk to DW DBA or learn MDX)

(7) Figure out what BI tool to use (Excel or browser, depends on policies and audience)

(8) Generate the reports (canned or ad-hoc)/dashboards/pivot tables for the POC

Now, if you have no prior experience with the Microsoft BI toolset, and you can whip this little project up in a week, guess what, you need to quit your job and start a consulting company because clearly, as a NYC recruiter once told me “you’re so money”. But if you’re a normal person with little prior BI experience (and the terms ROLAP, MOLAP, SCD and MDX don’t ring a bell), you’re in a bind.

So another thing you can do is download a tiny analytical database (say, the XSPRADA RDM/x engine, for example) and throw, say, 100GB of data at it (this is just a small POC remember?), then plop Excel on top of it and generate some really cool reports or pivot tables to show the boss (in under a week) it can be done. How hard is that to do? This hard:

Figure out where your source data is coming from.

Yup, that one is pretty universal in the BI world. Difference here is all your data sources will export as CSV to feed the XSPRADA engine. So at least that’s consistent across all sources (be they structured, semi-structured or not). CSV is data format lingua-franca so your connection "strategy" is this: get everything out as CSV. Plain and simple.

Model your data warehouse.

That’s always a smart thing to do for obvious reasons although the XSPRADA engine is schema-agnostic and you can feed it normalized or star/snowflake models at will. The secret phrase is: “we don’t care”! So for a quick POC, if you find yourself "forced" to feed RDM/x a 3NF model, no worries.

Establish some preliminary ETL process.

RDM/x runs against initial CSV data islands directly off disk. Point to the CSV files using the XSPRADA SQL extensions for DDL and you’re done. You’ll likely be doing this via script or code (C++, Java or .NET to the ODBC driver directly or via a JDBC-ODBC bridge). For incremental loads, just plop the new CSV files on disk and point RDM/x to them using the INSERT INTO…FROM extension. This process can be done in real time without disruption while other queries are running. No hassle there.

Load your warehouse.

That’s executing a single line of SQL DDL code such as

CREATE TABLE ….FROM “c:\file1.csv;c:\file2.csv…c:\file32.csv”; or INSERT INTO…FROM “c:\file1.csv;c:\file2.csv…c:\file32.csv”;

Made a mistake of want to modify the schema and “reload” real quick? Not a problem. Simply re-issue the same DDL command and the table/schema is instantly updated. From a trial and error perspective (which, in a POC situation, is fairly typical), that’s a high-five.

Setup an SSAS cube structure (figure out SSAS via SSMS or BIDS)

There is no concept of cubes inside the XSPRADA engine. RDM/x automatically slices and dices based on incoming queries in real time. So if you want to “cube” just feed the engine slicing OLAP queries. RDM/x automatically restructures and aggregates in real time. No need to pre-define or pre-load cubes, deal with hierarchies or materialized views. I blogged about this earlier. RDM/x is a lot like Luke 11:9 – Ask and you shall receive.

Figure out what queries to generate (talk to DW DBA)

That’s where an external tool using MDX (along with an MDX expert!) can come in handy (most people don’t roll their own SQL for OLAP, although it can certainly be done in POC mode). One cool thing about RDM/x is its ability to “withstand” poorly-formulated SQL because the queries are optimized against the internal mathematical model. RDM/x is typically more “SQL-forgiving” than most other engines. And a poorly formulated query is likely transformed internally to still yield optimal performance. So even if you’re no SQL guru, the RDM/x engine is still on your side.

Figure out what BI tool to use (Excel, no brainer)

Connect Excel to the XSPRADA engine directly via ODBC or connect Mondrian to RDM/x (via bridge) then connect Excel to Mondrian via the SimbaO2X ODBO/XMLA connector. Alternatively, make the argument that using OSS like Pentaho or Jaspersoft against RDM/x directly is more flexible and accessible (not to mention cheaper!) than messing with Excel. Depending on your user base and corporate standards, that argument may or may not hold water.

Generate the reports/dashboards/KPI/Pivot Table/ad-hoc queries required by management.

Exactly the same way you would using any other tool and/or SQL.

At the end of the day (or in our case, the week), it’s all about “time to results” and “pain to results”. In those types of situations, smaller and simpler clearly has a significant advantage over the rest. And speaking of smaller, I have run over my allocated space for this posting :)

Bits & Pieces Summer Posting

2009-08-14T13:40:00.000-07:00

I thought I would do a “freeform post” today to celebrate the lazy 2009 Summer and the fact that most of the planet (but not the BI world for some reason) seems to be on vacation at the moment.

As you know I’ve been following ParAccel with interest for a short while now wondering how they would deploy those $22M of Sales & Marketing greenbacks they just scored. I read this interesting article about them lately and it looks like “customer acquisition” might be part of their strategy. Good move. Unfortunately, I couldn’t determine who the “other database products” or the other “columnar-MPP database” vendor might refer to. I can only surmise it might be Vertica. If anyone knows who else OfficeMax looked at, please share the wealth.

For all my whining about missing TDWI in my own backyard (San Diego) lately, it seems I didn’t miss much after all according to Merv Adrian who posted about the conference shortly thereafter. From the looks of it, the highlight might have been a sunset ride on the Lyzasoft yacht in the San Diego bay. Talk about good PR!

Andy Heyler wrote a good piece about the demise of Dataupia called No Data Utopia. In it he refers to the “awkwardly named” Dataupia. Yes Dataupia is a weird name. But so are several others such as Kickfire, Tokutek, or Calpont, for example. And although XSPRADA is admittedly rather funky, at least it’s an acronym (Extended Set Processing for Rapid Algebraic Data Access). To me the money quote in there is “you need to have a clearly differentiated position in such a crowded market”. It’s pretty much what I’ve been saying for a while (and common sense if you ask me). At the moment, I don’t see any of the players in this market besides XSPRADA with a “clearly differentiated position” on anything. At the end of the day, it’s still all about the prisons that are columns and rows.

Netezza announced the long-awaited (not) TwinFin product line prompting a flurry of nasty posts from competitors like Kognitio, and an interesting Monash post about data warehouse pricing. Much like traditional software license pricing, ADBMS prices seem to be reaching for the bottom (stay tuned for $19.95 per terabyte while supplies last!). And as someone recently said, the bottom is open source. Should be interesting to see what happens. Personally, I think a lot of this stuff is going to become commoditized. The play looks a lot like printers or razors, where the actual hardware is sold dirt cheap, but the paper or blades cost a fortune to replenish. Caveat emptor.

A relatively new ADBMS vendor called XtremeData has emerged. It looks like they’re based in the US (Schaumburg, IL to be precise) but the actual brains of the operation are in India somewhere. They’ve certainly been vocal on several BI blogs (namely, DBMS2). To me the funniest thing is their “ChalkTalks with Faisal” screencasts. Faisal is apparently their India-based CTO. The entire presentation is like a Netmeeting whiteboard session where Faisal keeps talking while drawing stuff on a whiteboard in a Flintstonish manner. All that’s missing are the stick figures. It wouldn’t be so bad if they realized the audio is horrible, due to the incessant noise from the marker writing on the board. It sounds exactly like squealing puppies in the background. It’s totally distracting albeit very amusing.

SQL Server 2008 R2 is out but without the Gemini (or Madison née Datallegro) pieces I guess. That’s Office 2010, parts of which are going into the cloud if I understand correctly. This whole Madison/Gemini “revolution” in BI is starting to get a little, shall we say, boring for lack of materialization. Not sure what’s going on with Microsoft lately but I’m getting more and more concerned. Even .NET seems to be taking a backseat to Java/J2EE. It wasn’t like that a year ago. I am sensing a scary downward spiral. One thing’s for sure, I have yet to see anything remotely connected to .NET or C# in the BI programming world, save for those .NET C# extensions Aster provided in their engine recently for doing MapReduce (and of course the PushBI initiative but even there…). To me it seems the entire ADBMS/BI code stack is Java on Linux (SuSE, Red Hat and CetnOS). I’m talking about the supporting/ecosystem tools of course, not the underlying engines (those are mostly C/C++ I believe).

I could be wrong but, it’s not looking good for Microsoft. My prediction: this behemoth will eventually split off into a myriad of smaller entities, some of which will survive, some of which won’t. The sum of the parts may be worth more than the whole.

Ingres has actually managed to generate some buzz in the US lately by announcing it is teaming up with a company called VectorWise (well, it’s a research outfit actually) to develop a “project”. No customers yet but a lot of very fancy PhD types in Amsterdam (they did MoneyDB/X100) and a first option to buy VectorWise is rumored should the venture be successful. Time will tell. It always amazes me why Ingres isn’t more of a household name in the US. In Europe, they’re very popular (at least that’s what my French Ingres experts tell me ).

Finally, a recent article is claiming that BI is used by only 8% of employees in the enterprise and that’s, of course, only counting the shops that have implemented it to begin with. Actually I find that number high and would have guessed more like 5%. This is not surprising in light of the recent economic woes and scandals we’ve witnessed recently. In most of these cases, it wasn’t a lack of technology or resources at play but rather a conscious choice to ignore reality. The tools are there. The desire is not.

In my opinion, this disconnect is very apparent in numerous retail outfits. Places like Home Depot, Whole Foods, AT&T or Circuit City (RIP) for example, who clearly have resources and tools to perform and exploit top-notch business intelligence but still manage to provide mediocre service or product at best consistently.

At Home Depot, they can’t (or won't) keep track of inventory correctly. I was once told it was because there was too much theft (both internal and external) to update the databases frequently enough! Consequently, they can’t tell you if they have some items in the store or not. Not all items, just some of them. And when they can, they’re unable to locate them physically inside the store (as in what aisle and section). That’s shocking to me given what we constantly hear about RFID and data warehousing investments at these large box places. But Lowes does a much better job of this so clearly, it’s not a technology issue.

At Whole Foods (at least the one in Irvine where I live) the quality and quantity of their product is inconsistent at best. Some days the self-serve fish is fresh, and sometimes it is not (and has been sitting out for too long looking like a nice fat food poisoning lawsuit waiting to happen). Like the proverbial “box of chocolates”, you never know what you’re going to get. Similarly, their checkouts are never balanced. You’ll see huge lines at several of them while employees sit idle at empty others. Every time I go there some happy-go-lucky line manager with a bright idea of the week makes it harder for me to shop and enjoy it there which is why I never set foot in the place any more (instead, I go to an even more expensive supermarket). Apparently no one is keeping track of this. Least of all the department managers who seem to frequently act on impulse in trial-and-error fashion. Yet surely Whole Foods has an uber-BI stack running somewhere in Austin giving them a “big picture” on a store by store basis right? You’d think someone would be paying attention (or maybe even using it)? But clearly they are not, or someone would be fixing these problems (or at least genuinely addressing customer complaints, which they won’t because it's "inconvenient" for them, as they like to put it).

I don’t think it’s so much about the difficulty of implementing and leveraging BI as the article suggests. I think it’s about genuine laziness on the part of upper management. Because, at the end of the day, if you’re the CEO of a place like this, you need to get your highly compensated butt out on the floor to truly see, taste and smell what’s going on. You need to talk to your employees, your customers, and get in their shoes (incognito if possible). I’m always shocked to hear C-level people lamenting the fact that their CRM system isn’t giving them enough visibility into their customers. Or better yet, they need to “understand the customer” better. Who’s kidding who? Fact is they simply don’t give a hoot most of the time. And if they’re too lazy or too self-important to do that, they’re not likely to pay much attention to BI tools and warehouses either no matter how fancy or ubiquitous the software might be. That's really just the nature of what "service" has become in the US lately. In order to improve BI usage, we will have to improve the quality of Management first and put real folks back in charge.

How I Learned to Love Mondrian (confessions of a WISA guy)

2009-07-28T15:50:00.000-07:00

I’ve been playing with Pentaho’s Mondrian for almost a year now on and off. I have to say, everytime I mess with that stack I am more and more impressed by its richness and capabilities. And twelve months ago, when I started learning it, I was what you could call “severely LAMP-challenged”. I’ve sure made a lot of progress since then and wanted to talk about this as I figured it might help other Microsofties out there needing (or wanting) to put a toe in these mysterious LAMP/OSS waters.

The first thing I ever did with Mondrian was figure out how to install it on a Windows platform. The reason I did was twofold. First, we didn’t have appropriate Linux hardware/software in house at the time, and second, I have way more experience on Windows so it’s a lot easier for me, and third, I wanted to do it locally and avoid dealing with cross-platform bridging at the moment (our ODBC drivers are Windows only as well). Path of least resistance is an engineering mantra in my book.

Lucky for me I had worked with Java and Apache Tomcat in the dot-com days so I had no trouble pulling and installing the JRE/JDK and the web server itself (which comes as a Windows service). Next, I deployed the Mondrian WAR file into the Tomcat webapps folder which caused it to be automatically “deployed” as a web application. Way easier than deploying ASP.NET applications (but you didn’t hear this from me ).

Then, I fired up the Mondrian landing page, clicked on the Jpivot link and, of course, kaboom. Yes, without a JDBC driver, Mondrian is not a happy camper. It took me a little longer to figure out the Sun JDBC-ODBC bridge and how to plug corresponding connection string it into numerous Mondrian files to replace the default connections there (which are all for MySQL if I recall).

The Mondrian documentation isn’t great but if you Google long enough you can usually find some other poor slob with a similar problem and, with luck, published solutions online. [Side note: I once had the nerve to email Julian Hyde, their Chief Architect, about some technical question. He abruptly suggested I don’t bother him and use the “community” forums instead]. Unfortunately those forums are often useless for non-enterprise (read: non-paying) users. There are flurries of unanswered questions and problems up there. This is a generic OSS problem I suppose. You get what you pay for J

So finally I had my bridge setup, along with a DSN called MondrianFoodMart (the default) pointing at the default Access database (distributed with Mondrian). And now I was able to fire up Mondrian on top of the database and do a couple drills, run a couple MDX queries. Bliss.

Next, I took that Access database and exported it to CSV format with corresponding DDL. This is the way to feed the XSPRADA engine. Fired up our RDM/x server and ran the DDL scripts. Then re-defined the MondrianFoodMart DSN to point to us via our ODBC driver (32-bit only, 64-bit won't fly with the bridge - painful lessons learned...). Reloaded the Mondrian page, and voila! Mondrian was now talking to RDM/x and displaying the Sales cube.

One point of the exercise was being able to show Mondrian OLAP on top of our database. Another was being able to show our database’s behavior in time as more and more queries come in (hint: it gets faster). Now, with Mondrian, this is a little tricky because the platform is heavily cache-based. Mondrian shoots initial queries at a relational system and proceeds to cache heavily as it aggregates results. So the more you use it, the more it caches. Obviously the reason for this is that Mondrian is designed to run on top of relational databases, and not “OLAP-intelligent” engines such as ours. It has to translate the MDX into straight SQL queries every time, as it fills its caches initially. Nevertheless it does re-hit the database as needed when you start slicing and dicing on new dimensions or facts, as one would expect. So you can actually see our engine’s “dinosaur tail” behavior as I once described in a previous post.

Now, the MondrianFoodMart database is fine for setting up the stack, but it’s not particularly interesting in so far as data volume goes if you’re in the VLDB space. More recently, I attempted to setup the TPC-H/SSB sample data under Mondrian, meaning I tried to manually create the XML defining some SSB cube. There is a whole fairly complex XML language Mondrian uses to define and connect fact tables (measures) with associated dimensions. They have a UX driven tool called Workbench but I could never get it to work on my system (and didn’t have enough time to keep messing with it). With online help and using the FoodMart.xml sample file, I was able to get a basic cube up in about a day. Nothing fancy, but now I can OLAP into arbitrarily large data sets and that’s a good thing.

As cool as it is seeing our stuff run under Mondrian, I always dreamed of doing the same thing under Excel (as in 75% market share, yeah I want to support that please). Until recently, I thought this would not be possible until we implemented MDX in the engine but then I saw the light. It is called the SimbaO2X connector and it rocks!

[Start Commercial] Did I mention how much I love this company Simba? They pretty much wrote the book on data connectivity. Within a day they had me a 30-day trial version of their O2X offering, no questions asked. And follow-up to boot. Their stuff works, and they know how to take care of people. What a concept! [End Commercial]

This SimbaO2X puppy lets ODBO clients (say like Excel) talk to XML/A OLAP servers (say like Mondrian). Note, there is a similar offering from Pentaho called Pentaho Spreadsheet Services. It carries a small yearly license fee from what I understand. Supposedly you can email Pentaho sales for additional information and a local contact. I’m still waiting for their reply. Hey it’s OSS…Did I mention you get what you pay for?

Either way, the relevant fact is that the SimbaO2X connector works without a hitch. I am finally able to create and manage pivot tables from Excel, talking to Mondrian (via XML/A), talking to RDM/x (via ODBC)! This is the bomb! I need to really get a deeper understanding of MDX capabilities now. But the more I learn about it the more impressed I get, and the better demos I can do.

The Canary in the Gold Mine?

2009-07-28T11:30:00.000-07:00

I’ve been claiming for a while that data mining and predictive analytics (PA) were the new hills to conquer in BI and this morning the news came out that IBM had plopped down big money for SPSS. IBM is also investing R&D dollars in ways to manipulate data directly while encrypted and/or compressed. This particular research fascinates me because I believe it will be key to SaaS acceptance, where security is still a significant push-back for obvious reasons. This means analytics might actually have a future on the cloud. And this is important IMHO because this allows for significant progress in the UX systems required to use (drive) mining engines efficiently. The kind of improvements that cannot be generated and deployed quickly enough with fat client implementations. I’m thinking of really interesting things like www.spezify.com for example.

Another interesting trend is pushing analytical capabilities deep into the database engine either via stored procedures or user-defined functions in one or more programming languages (much like .NET inside SQL Server, for example). All this leads me to believe that insightful BI players have been turning their guns on solving the next big pain point of BI which is, IMHO, data mining and predictive analytics. This embedded capability relates to the deep kind of analytics I once blogged about in the context of Greenplum’s MAD paper.

So does this mean we’re all done with OLAP? Not likely, but I think a certain peak has been reached where OLAP has become “bearable”. I don’t really have a 3-5 year “future outlook” on OLAP at this point. Is it still hard to cube and do MDX? Yes. Is it still a pain in the behind to setup large SSAS analytics? You bet. Is setting up a production version of Pentaho’s Mondrian ROLAP for the faint of heart? Not exactly. But there are now multiple alternatives out there in both hardware (faster COTS components, FPGAs, GPUs, MPP) and software (columnar, ALGEBRAIX) realms.

Our own ADBMS at XSPRADA is designed and tuned specifically for OLAP workloads in its present form. Product such as ours have helped “commoditize” OLAP work by shifting design and pre-structuring efforts (cubing, slicing and dicing) from the user (DBA) to the software itself. This is done automatically and based on queries coming in. There is no need to configure cubes, mixed workloads are supported, and all the user really has to do is ask questions. It’s that simple really. Let the software worry about the darn cubes!

So I guess my point is, if there are people still struggling (read: losing time and money) with OLAP in the enterprise, I have to say it’s because they’re either poorly advised or simply not opening their eyes to new tools and techniques currently available. At this point OLAP pain is no longer a necessity. It’s an uneducated choice. From a technical standpoint, it has been addressed. Let’s move on to the next problem please. This is why I think the industry is poised to tackle another challenge now, namely data mining and predictive analytics. Even Curt Monash in a recent blog about the SPSS acquisition writes:

“So far business intelligence/predictive analytics integration has been pretty minor, because nobody’s figured out how to do it right, but some day that will change. Hmm — I feel another “Future of … ” post coming on”.

Sorry Curt, I beat you to it J

Mining is a totally different segment of the business intelligence endeavor. When you do OLAP, you’re asking “tell me what happened and why”. When you do mining, you have no clue what happened and much less why. In mining you’re asking “tell me what I should be looking at” or “tell me what’s interesting in this data?” And predictively, you’re asking “tell me what’s likely to happen” – as in, show me the crystal ball. Mining is not a pre-structured, pre-indexed kind of “cubing” world. It’s an ad-hoc discovery process. It’s iterative. Much like the way a human brain functions when discovering information, and trying to make sense of it. This “human-like” behavior is actually one of QlikView’s usability pitches. In mining, the relational model is a hindrance, not an asset, because relationships are not necessarily canned or static. Predictive analytics are more of an art than a science as well. These concepts don’t fit nicely in pre-structured, tabulated formats.

Additionally, mining and PA are creative endeavors (whereas OLAP is not). This is why it’s important to let users define their own “stuff” so they can trial-and-error through the problem. Conventional database engines don’t support this type of workload elegantly. It’s simply not “structured” nicely like OLTP or OLAP. You can’t easily (or cost-effectively) try, erase and re-start with conventional engines. They're not forgiving.

So what’s needed are systems that can first intelligently process data upstream in ELT mode because acquiring statistic on incoming data (at varying rates) is an important step for analytics. XSPRADA’s engine starts analyzing data statistically upon initial presentation. More importantly, it keeps doing so automatically in real time, and continuously via comprehensive optimization. This is a unique feature that causes the system to continuously re-evaluate system resources against queries and data to seek out additional or more effective optimizations.

Next, you need systems that can tell you where NOT to look. Because in this type of work, pertinent data is often clustered in very specific areas (as in 5% of 100TB perhaps). And user questions tend to hit within small percentages of those clusters. Yes there are always exceptions, but generally-speaking, that’s what happens. So what you DON’T want are systems that spend a lot of time scanning boatloads of data (needle in the haystack). What you need is intelligent software that can quickly eliminate vast areas of informational “no-man’s land” based on incoming queries. In such a problem space, throwing additional monies at ever more powerful metal is a self-defeating approach. It’s the software stupid! J

As it turns out, XSPRADA’s ALGEBRAIX technology is very good at eliminating "useless" (read: at a given time) data spaces. Not only that, but it also shines at inferring subtle relationships between different entities. The kind of relationships a human wouldn’t even think of asking on her own. It’s also very good at recognizing patterns (both in queries and targeted result sets).

In a way, you would expect that a system built on pure mathematical foundation would be particularly well suited to data mining workloads. And it sure is. This is the beauty of having a “wide” and rich enough technology that is as easily and readily applicable to a multitude of different BI problems. It means you don’t need to re-invent the wheel or re-architect your system every time a new problem space opens up. And that, in the business intelligence technology world is a rare find indeed.

Who leaves a country packed with ponies to come to a non-pony country?

2009-07-27T08:10:00.000-07:00

Usually I think of myself as a “hot-shot” when it comes to XSPRADA technology and its applications. This is because I’ve been involved with it for ten years, and that kind of history builds bonds. In a word, having lived and breathed it for so long, I’m severely biased, but at least, I’m aware of it. It’s a completely different story when you talk to a user who also happens to be biased from experience running the stuff to solve real problems. That’s when you hear praise that makes you step back and go “wow, we really do shine here above and beyond”. There is nothing sweeter than an adamant customer evangelist.

One such person we shall call Tim (why not, since it's his real name). He works for a major DOD contractor out here in California. Tim asked me to withhold his company’s name for obvious reasons. He’s been running POC projects using XSPRADA technology for years. As a matter of fact, Tim once ran a real-time CEP version of our engine (which can handle both real time and historical input) for a demo bid project he needed to put together where no other vendor could come close.

Tim is the ultimate engineer’s engineer and one of the smartest folks in the “information” field I’ve ever met. He’s got experience galore and has been around the block a few times. Currently there are other groups in Tim’s shop running the XSPRADA engine for other purposes, and he keeps abreast of those POCs as well. I could tell you what they entail but then I’d have to kill you . Suffice to say that the engineering being done there would blow most people’s minds (as in, holy cow, we're actually doing this?!?).

It turns out Tim is biased as well, but he’s biased from a user perspective. This is "been-there-done-that" advocacy. And that, in my book, is far more compelling than any argument coming from an insider like myself. Although I don’t know too many people who can explain and position the technology as well as I can (I’m so modest, no pictures please! ), Tim had the following analysis recently and I thought it was so “perfect” I had to reproduce it here (bolding my own):

“One of the distinctions I’ve been using lately to explain the difference between set based data processing and most everything else (row based, column based, partition based,…) is that most other DBs are based on defining somewhat arbitrary bins of data of fixed size or dimension (tables with fixed columns in RDBMs, column collections for things like Vertica ,et al, and chunks in Google’s BigTable designs, to name a few). Then there is significant overhead to partition the incoming/outgoing data to fit into these fixed containers. Inevitably, any operation on these artificial partitions will include wasted processing or I/O on irrelevant data that just “happens” to live in the affected partitions. This is a huge waste of time and resources. In addition, these bins are continually reused by means of destructive updates which require them to be locked during transactions to avoid data corruptions. This is the other main source of waste in that significant delays are now imposed not only on the relevant data involved in the operation, but also on collateral data that might be holding up other operations unnecessarily. These two effects are mutually opposed: larger partitions would help the I/O problem, but at the expense of exacerbating the locking problem. And vice-versa. By contrast set based data systems, like XSP, use completely variable sized containers (the sets) dynamically partitioned based on operational relevance and not on any predetermined partition sizes. In this way, the amount of irrelevant data moved across the I/O boundary for any data operation is significantly reduced. And because these sets are immutable, there is no locking interference with other concurrent data operations.”

To put it in Southern California speak, I was like, wow, this dude really gets it! And there isn’t much I can add to Tim’s conclusions. I bow to the completeness of his understanding; his analysis stands on its own.

The context of our exchange was about pre-structured or “canned” mechanisms used by other database engines versus the flexibility of our approach. It’s what I naively call “bucketizing”. And this topic is of course related to the ADR functionality I was discussing in my last post. But it also pertains to the “schema agnosticism”, parallelism, and ACID aspects of the database I’ve mentioned in the past.

The XSPRADA engine dynamically adapts to queries and data. It avoids the inherent rigidity prevalent in all other technologies. Yes it’s compelling to handle analytics via columns, but that’s only one of many ways you can address the problem. And if your technology is “columnar” in nature, it’s the ONLY way you can address the problem. You’ve put all your eggs in one basket. At the end of the day, you’re still looking at the world in a tabular format (which happens to be vertical). So it’s what I call the one-trick pony approach.

There’s nothing wrong with one-trick ponies if you need to solve a very specific business problem quickly and efficiently. But from a holistic business perspective, it’s a scary proposition. If you’re running an enterprise, you want flexibility. You want the ability to address problems as they come up using all available means at your disposal. You’re looking for a wide array of tools and methods to win battles, not a single weapon system. And this is what XSPRADA technology offers: “the ability to apply the right technique for any question for any data at any time”.

If Tim is concerned with waste and inefficiency, it’s because his shop deals with tera and petabyte volumes of data with limited shelf-life. In other words, waste is not an option. This isn’t about airline transactions messing up your trip or bank accounts being debited incorrectly by the way. This is about national security and people living or dying in real-life tactical situations.

In applications like this, one-trick ponies don’t cut the mustard. And this is why Tim and several other groups in his company have been looking at XSPRADA technology for years. There simply isn’t anything out there that can meet their requirements, and believe me they’ve tried all the usual suspects. To Tim and his colleagues, the unfair competitive advantage XSPRADA can deliver to their company (and clients) is well worth the risk of evaluating technology that’s a little out of the ordinary.

ADR and how I got kicked out by Kickfire

2009-07-17T12:18:00.000-07:00

I had the opportunity to brief Merv Adrian (major BI industry analyst) recently about XSPRADA and in discussing competitive technological differentiation, I highlighted the three major points that make XSPRADA uniquely stand out in a sea of “new-breed” ADBMS vendors, namely our algebraic engine (aka ALGEBRAIX), adaptive data restructuring (ADR), and temporal invariance. Those of you who have honored me with their readership in the past will no doubt be familiar with those terms and concepts. (side note: Merv is the one who coined this ADBMS term that I shamelessly borrow constantly now -- thank you Merv!)

I want to focus a little bit on ADR in this post because in the midst of our discussion, Merv asked me a really good question about it. He said “if you’re busy doing ADR, and more and more queries come in and more and more users get on board, what will the impact be on performance?”. Excellent point! Quite honestly, no one had ever asked me this before so after our call, I did a little more research and came up with a few more questions on my own. All of which I’d like to discuss here.

First of all, to recap, ADR is the process of adaptively restructuring data both logically (say the structures in RAM for example) and on storage (disk) based on the nature of queries coming in. What does this mean? Well for example if a query is clearly pulling only certain columns from a table (as is typically the case in OLAP), ADR will pull these columns out and optimally lay them out on disk for more efficient access. ADR could also include indexing (as in bitmap, in low-cardinality cases) and any other means at its disposal to optimize questions pertaining to these columns. Sound familiar? It should as this is basically the principle behind columnar systems. However ADR doesn’t stop there.

It may, for example, decide that sharding row blocks is a more efficient strategy given a particular query pattern and set out to do just that as well. It may decide that duplicating certain pieces of information on given disks is more I/O efficient. In memory, it may decide to implement different indexing schemes depending on the nature of the queries. In short, ADR has absolute “carte blanche” to take every means at its disposal to optimize the system in real time.

Unlike most other ADBMS out there, the XSPRADA engine is a living breathing entity constantly striving for optimal performance. And it has more than a few tools at its disposal (in other words, not a one-trick pony, compliments of the underlying mathematics) -- But so the question is indeed legitimate: how does this impact performance if at all?

The answer, unsurprisingly, is it depends. First, it’s important to realize the design principle behind ADR. It is called “crowdsourcing”. The philosophy is that the more people hit the database from all angles , the better chance there is of being able to optimize the database. From a technical perspective, overload is always a possibility, as with any other system. If too much is submitted at a given time, the system could theoretically run out of resources and impact performance. But when properly configured (balanced) with proper amounts of storage, the system should not degrade with additional users and queries coming in.

Note that concurrent queries are also beneficial to the system. This is because data streams can be shared between query processes in parallel, thereby reducing disk I/O. Additionally, ADR is not user or query-specific. So there is no risk of one ADR action resulting from Query1 from “clobbering” another ADR action from Query2. When sharing is possible, it is implemented and maintained in the mathematical model (algebraic integrity). But all queries from all users enter the “algebraic space” together in one big pool. This means everyone’s actions benefit everyone else. The XSPRADA engine is a very populist one J

On a completely different topic, I wanted to relate a funny incident that happened to me on the way to the Forum recently. Well okay, it wasn’t exactly a forum per say but rather the newly minted Kickfire on-demand trial process. They call it Cloud-based Trial.

I enthusiastically signed up for the trial and got an assigned time-slice for today. Within minutes, I get the following email from Kickfire’s Karl Van den Bergh, who is Kickfire’s Vice President of Marketing and Business Development (phew! Talk about long names to match long titles).

“Thanks for your interest in Kickfire. Our trialing system is reserved for prospects and partners. As our companies are somewhat competitive we are not able to give access to XSPRADA at this time.”

Wow. Cold man.

But next morning, I get this email from the company’s trial support team:

“Congratulations! This email confirms that your Kickfire On-Demand Trial will begin at 10:00am (PST) on 07/17/2009.”

Then thirty minutes later I get this:

“jerome, Your reservation has been deleted. Reservation #sc14a5eb4e82d0dc.”

By that time I’m thinking okay, somebody at Kickfire Trial Support finally got their behinds kicked (pun intended) for daring to confirm my trial session. Apologizing for this confusion is Karl again:

"Apologies for the registration mix up this morning - we have a number of system administrators who weren't in synch."

Yeah, I’d say. But wait, there’s more if you order now! Early this morning, after my session was supposed to end, I get a phone call from Kickfire asking me how my trial went! I couldn’t help but blurt out “you guys are really confused”. Then I explained that Karl had branded me persona-non-grata at which point I was promptly dropped like a bad case of H1N1.

Now, don’t get me wrong. I love this Kickfire appliance concept. And I have a positive informative relationship with technical folks there who are nothing if not extremely competent (and nice people to boot). And although I cannot play with the Kickfire appliance either locally or remotely, from what I’ve heard and read, these guys have a top notch team, great technology, a seasoned CEO and attractive market positioning.

In thinking about this, my initial reaction was that I was stupidly naïve. Honestly, it never occurred to me that Kickfire would ban any “competitor” (or anyone else for that matter) from remotely playing with their software. The reason is because we don’t do this at XSPRADA. As a matter of fact, anyone can pull our bits for free from our website, including Kickfire (which already has). So to me this has a strange, secretive, “we have something to hide” feel which, given what I know about Kickfire, really took me by surprise.

But it's true I have this weird concept about openness that many other vendors share, and I’m happy to put our stuff in competitive hands (after all, the customers certainly will!) anytime and get any feedback from the experience, negative or not. In my experience, few competitors will go out and trash another vendor’s offering, much less try to reverse-engineer it (at least not in this industry) -- Maybe I’m foolish.

However, it’s true I don’t have a “marketing” bone in my body. As an engineer and evangelist, I’ve always been adamant about transparency, peer review and feedback. So maybe it’s a good think I don’t handle XSPRADA Marketing, or Kickfire’s for that matter! J

Of Views, Cubes, Patents and Haystacks

2009-07-02T14:30:00.000-07:00

In keeping with a past commitments to do so, I plan on staying purely technical in this posting. The first topic I want to bring up relates to database views. The second one deals with OLAP “cubing”. I want to discuss those in the context of the XSPRADA analytical engine RDM/x.

Many conventional database engines support view and materialized views and I believe Oracle actually came up with the concept (I am not 100% sure of that) but numerous other vendors support them including Postgres, MySQL, DB2, SQL Server, etc. In either case, the idea is simple. A view is basically a logical rendition of a SQL query. It looks and behaves like a table. Views can be logical or materialized. A logical view is really just a “pointer” to the actual data. A materialized view actually contains data and is implemented as a full-fledged database object. There are several reasons to use views.

One is performance enhancement and convenience (caching a nasty JOIN or OLAP summaries dynamically, meaning original data changes trigger recalculations). One is abstraction. Another is security and access control (you can grant specific privileges to views while keeping underlying data protected from users). The important thing to remember is that views were conceived in response to a problem, namely that database performance can really suck without them (especially in warehousing applications).

If you didn’t have inherent performance issues in conventional database engines, you wouldn’t need views to begin with. And herein lies the reason why the XSPRADA engine doesn’t support classical views: it simply doesn’t need them. The engine caches and materializes query results dynamically on the fly. As such, “materialized views” are generated on an as-needed basis. But this is true for everything else going on inside the XSPRADA engine because the adaptive data restructuring (ADR) feature dictates that all forms of optimizations be materialized for optimal logical and I/O performance on the fly as queries come in.

Who determines when this happens? The optimizer and the algebraic engine (now officially known as XSPRADA ALGEBRAIX under US Patent 11/383,477). If and when the system detects more than one query involving a particular set of joins, it materializes (and likely caches) the results for subsequent use, assuming that a request pattern has emerged. Similarly, it is always possible to “store” a complex query into a named result set using the INTO extension. For example, you can write something like:

Select p.product_name, s.store_city, sum(f.store_sales) tots from sales_facts f

Join product p on p.product_id = f.product_id

Join store s on s.store_id = f.store_id

Group by s.store_city, p.product_name

Order by tots desc INTO ds;

At that point, the ‘ds’ table will contain the results and can be used independently and updated at will. Currently, the XSPRADA engine will not automatically update ‘ds’ should any change occur to any of the underlying tables. Regardless, ‘ds’ now becomes a regular RDM/x table like any other. There is no distinction between views and tables in RDM/x. And even though views are defined in RDM/x, this is done for sheer syntactic convenience, and using them yields no technical advantage.

On the OLAP front, the system behaves in a similar way. This is why I always say, with RDM/x you don’t need to pre-structure or “cube” your data. The engine does that automatically internally for the user based on his/her query patterns!

When you feed RDM/x a query like this:

SELECT s.store_city, sum(f.total_sales) from store s, sales_fact f

Where s.store_id = f.store_id

GROUP BY s.store_city

RDM/x automatically starts building a “cube” for it. What does that mean? The system creates a set of mathematical expressions relating the detail data to the aggregates you’ve just requested. In the process, instances of the data in intermediate and final form are realized. This city-based slice is “remembered” by the system and any subsequent query against that slice or variation thereof will benefit from immense optimization (and performance increase). This is how subsequent queries just get faster and faster in the XSPRADA system. And if you slice along another dimension next, say products, for example, the same optimization and ADR is performed for that slice. If you ask enough questions involving enough dimensions, you get closer and closer to instantaneous response time. Adding a dimension may cause a slight performance dip on the first shot, but the penalty is nowhere near as painful as having to add another slice or recompute aggregates (cells) in a conventional “manual cubing” system (say like Mondrian or SSAS to name a few).

People have asked (namely Chris Webb, an MDX guru who runs an excellent blog) if there is some sort of internal structure that makes the query run faster over time. The answer is that the re-structuring of the data, combined with the algebraic system and the methods for accessing the data internally (in memory and storage) are what makes these queries get faster with time. Another excellent question was “can you edit these structures?” And the answer is no, as the system is entirely automated and requires no user intervention. Importantly, DML operations (INSERT, UPDATE, DELETE) on the original detail data cause automatic updates of the aggregated data, as long as the aggregates continue to be expressed in terms of the original data. Simply put, the system maintains internal integrity automatically unless, of course, you change the original query.

MDX and dynamic “view” updates are features likely to be implemented by RDM/x in the near future. Without MDX, you lack the convenience of being able to talk to RDM/x directly from clients such as Excel (or SAP for that matter, who has apparently adopted MDX now), although you can clearly question RDM/x via Excel using OLAP queries. MDX is obviously more powerful and convenient. This doesn’t take away from the internal “magic” occurring inside RDM/x when handling OLAP workloads.

And I have just a few more lines to make one last point. Technically speaking, RDM/x is currently heavily biased in favor of OLAP workloads. This is not a consequence of the technology being used per say but rather reflects marketing priorities. I sometimes meet people who try out our product to do search. For example, they’ll try to grab a few select rows from 100,000,000 rows of data. This is not where RDM/x currently shines! I always tell them: if you want to do this, you’re better off using a conventional OLTP system like SQL Server or Oracle. RDM/x is not about finding the proverbial needle in the haystack. It’s about telling you what the haystack looks like.

ParAccel's Big Ad-Venture

2009-06-29T14:20:00.000-07:00

I was just about to write a piece about the sad state of venture funding in the analytical database and BI space over the past years when BAM! Out of nowhere comes the announcement that ParAccel just scored $22M in a C-round investment. More importantly, as Merv Adrian points out, “ParAccel’s previous investors participated as well.”

This last bit of information is even more telling than their ability to raise new money in this zero-point-zero economy. Because if you look at past events surrounding Dataupia (and even Lucidera, in a way), it’s clear to me that there is a lot of “investor fatigue” in the BI industry at the moment. In my book, this is partly due to the fact that a lot, if not most, of these investors had no clue about the data management market or BI as a whole (but at the time, it sounded cool, and the guys next door were touting it so…) or how much it takes to actually build a full-fledged high-performance analytical engine from scratch (which is about 200,000 man-hours just on the engineering and maybe $20-60M if you’re lucky). So of course when the going gets tough, these guys get jittery and bail out. No surprise there.

My prediction is that, by year end, there’s going to be a lot of bodies left “on the carpet” as we say in French (I think it's an old boxing term). ParAccel’s new money, combined with their recent marketing coup involving a 30TB TPC-H benchmark, will surely help them survive these tough times provided they don’t squander the funds and from what I’ve seen so far, these guys tread lightly and wisely.

Not to say this makes the BI capital markets suddenly look better. The speculation “out there” (meaning in the “connected” unofficial circles) is that the average VC returns for the 2000-2009 decade (which won’t be reported before April 2010) will likely be close to zero or negative. The spectacular returns achieved in 1999 will drop out. Clearly VCs would not take kindly to this information being published and the NVCA has not made it public. Nevertheless if you follow VC performance sites like MoneyTree and the NVCA you can see the writing on the wall and it isn’t pretty. There are a lot of angel and VC investors looking for a way out nowadays at any cost. And to paraphrase Dr. Evil, when this happens, “people DIE!”

According to this article, venture capital is getting depleted. It’s hard to raise serious cash these days, imagine that! I’ll go one further and speculate that the venture capital system as we know it in the US may not be around another ten years. At least not in its present form. By then, the major source of funding will likely be big government (European-style) as risk will become more and more demonized and “regulated” (and you can’t regulate risk by definition, or it isn’t risk!). But what do I know? I’m not a VC.

For full-service SEs, this product is for you.

2009-06-29T13:51:00.000-07:00

I have not written specifically about my SE job before here at XSPRADA because I try to keep this blog fairly technical, or at least directly related to database and BI issues, but I want to make an exception this time. Reason being I recently discovered a very interesting SaaS offering called TeamSupport that makes my job easier. And I figure, if it does that for me, chances are it can also do it for others in the profession. So I felt compelled to share the wealth.

First, to put things in context, I discovered TeamSupport completely by chance interacting with their COO/VP Sales Eric Harrington on a LinkedIn group. Eric offered to give a quick and dirty online demo and five minutes later, he produced. That in itself was impressive to me.

TeamSupport is a cross between an issue/bug management tracking system (not unlike JIRA, FogBugz, Team System or Bugzilla, all of which I have used in the past) and a CRM system (not unlike Salesforce.com or RightNow.com), although they don’t bill themselves as being CRM per say. The TeamSupport pitch is about “bridging the gap” between customer service/support, product development, engineering and QA. Given the nature of my work as full-service SE , this seemed like a pretty compelling tool to me.

Now, typical sales engineers in larger shops ride in tandem with Sales people (AEs) on most accounts and are mandated with “greasing the rails” of sales, as I like to put it. But I typically don’t go out with AEs (nothing personal, we just don’t have any) and always pretty much face clients and prospects (both technical and executive) on my own. By necessity, I have a very close and tight relationship with our engineering group, and can typically remember specific tracking issues by JIRA number. Similarly, as I also handle a lot of the sales/support side of things, I’m often up on Salesforce.com managing and supporting accounts, and mining opportunities and what have you.

As great as JIRA is and as useful as Salesforce can be, they don’t play nicely together out of the box for this purpose. TeamSupport integrates functionality from both sides of the house. As it is on-demand software, I was able to create an account and log on in no time. I just love the simple clean UX of this product. On the left pane menu are all the entities I can create, edit and manage such as issues, features, tasks, bugs, users, customers and products. In the middle is the workspace corresponding to the selected menu item. But make no mistake this software is rich, rich, rich.

So, for example, I use Features to enter feature requests from user and prospects’ wish lists. If I click on Features, I immediately see my list by ticket number. I can therefore track those with Engineering but also product management (as in, when can we accomplish this and should we?) and give customers feedback on progress (or at least estimates). You can associate Features with one or more customers. This is useful when more than one customer makes a similar request (guess what, that’s common).

I use Tasks to keep track of what I need to handle or resolve on a daily basis. Those too can be associated with multiple customers. That’s cool. You can subscribe to tasks and get email notifications when they are modified by other users (TeamSupport is free for up to three users, by the way).

In the Bugs section, I can enter pertinent items from our internal JIRA tracking system. I can assign or link those to various corporate groups like Engineering, QA or Sales Engineering.

In the Knowledge section, I like to enter resolutions to past issues or problems. These are likely to come up for other customers or prospects so it’s a great way to keep track of those for future reference.

In Customers, I entered all my current customers or prospects. I don’t differentiate on that. Paying customer or evaluating customer, whether you’ve paid your money or just kicking the tires, you get the same high level of service from me. Priority to existing customers, clearly, but same level of service

In Products, I can enter all versions of existing product lines and link those with customers and prospects (as in who bought what or who is currently trying what version and which maintenance release). You can of course track issues and features by product. We do maintenance releases fairly often so this is a great way for me to track critical feature enhancements/bug fixes on a per-version level.

If I click on Dashboard, I get an immediate 30,000 foot picture of where I’m at this point in time. I can then drill into my tickets or customers at will. I now have all I need on hand to help me cover everything from the most minute technical details to the most important pain point gleaned from my last interaction with a prospect.

TeamSupport can also ingest our JIRA database. All you have to do is export your database and they help you get it loaded. This is great on the ticketing side. They are also working on a future API to do this automatically. On the CRM side, they are integrating with Salesforce as well, and I will be beta-testing that effort shortly to bring in accounts from that side into TeamSupport.

Last but not least, TeamSupport lets you setup support portals for each customer. This is important (and unique) for several reasons. First, it makes us look a lot more polished than just handling everything via email (or Twitter). Second, it creates a history of interaction that can be mined and referenced at will (this in itself is very valuable business intelligence!). Third it facilitates a push model of interaction with the customer because each side gets change notifications. This means I can respond in real time to questions and issues. I like real time.

As most great software, it’s sort of hard to do it justice in a quick write-up. You kind of just “get it” as soon as you start using it. It’s elegant, simple, fast, and easy to use. It just flows and it’s intuitive. I actually enjoy using the darn thing! Go figure. I forget what the exact subscription costs are but it’s dirt cheap (ahem, I mean cost-effective) considering the value provided out of the box. I should conclude by saying that I am in no way connected to this company. I had never heard of them before last month, and have zero affiliations with anyone there. I can tell you they built a really valuable product if you’re in technical sales. Did I mention their support is stellar? Enough said: if you want to try this product out, shoot my friend Eric a quick email or twit him up at TeamSupport. Tell him I sent you J

Pawn to King IV

2009-06-25T23:07:00.000-07:00

I recently read (sorry it's in French ) that SAP and Jive had partnered up to offer BI in an Enterprise 2.0 context (aka enterprise social networking). The same happened in France with Dassault Systems and blueKiwi Software. I’ve also been following the new “BI for the masses” trend exemplified by offerings like PushBI and RoamBI. Then several weeks ago I saw the Google Wave presentation at I/O and thought to myself, geez wouldn’t it be cool if people could share and collaborate using data wavelets under management from a BI Wave “bot” dedicated to a specific community of users.

I think BI is on its way to becoming commoditized. I see it moving up from the technologists directly into the hands (and mobile devices) of the users. Years from now, people will look back on monolithic enterprise data warehouses and their infrastructures and wonder “how could people ever have lived like this?”

I think it’s really crucial for progressive companies in the BI space to ask themselves what their users will look like ten or twenty years down the line. It’s obviously important to understand current users, but doing so is relatively easy. Anticipating what today’s young people will purchase in 10-20 years as senior corporate buyers (and decision makers) is not so straightforward. But as today’s “kids” are the people you’ll be selling corporate BI to in the future, doesn’t it make sense to (1) analyze their mindset and (2) start reaching out to them now?

Those of you who read this blog regularly know that I worship Guy Kawasaki. In his book Reality Check, there’s a chapter called “Get a Clue: The Global Youth Market”. In it, he interviews Kathleen Gasperini, the cofounder of Label Networks. These folks analyze “global youth culture” for major corporations. It’s a fascinating read (as is most of the book).

You’re thinking great, what does selling Nike shoes or Levi Strauss jeans to kids have to do with pitching enterprise software to business intelligence users? I say a lot. Because if you understand the behavior and expectations of upcoming generations of BI buyers, you’ll understand your future BI customer and gain unfair competitive advantage in the process. Now, I don’t purport to understand current youth behavioral traits, but I do make the following subjective observations based on past experience.

Instant gratification. Immediacy has become a birth right. Expectations of “just-in-time” are prevalent in everything they do, purchase or share. I don’t think these people are the kind who will sit around the office waiting six months for a $4M BI project and associated resources to get provisioned, staffed, configured, and maybe then approved. These folks are going to want something up and running in days. Anyone not operating in the same timeframes will be left in the dust.

Social consciousness. Business is business, but “good” business wins points. A “good” business does not exist exclusively for pecuniary purposes. Making meaning (as Guy K. would say), as opposed to just making a quick buck, will matter. Being “green” will matter (not sure what that means but it’s a hot button). I know this sounds awfully naïve, but young people are experiencing the ability to “make a difference in the world” (something past generations may have missed out on) and respecting those who strive to do so. They’ll take that to the corporate world as well. Help them do so. They mean it.

24/7 Connectivity. Young people are constantly connected to the internet “matrix”. It’s a 24/7 world for them. Like an addiction, this isn’t a habit you casually kick with age. Accessing future BI buyers outside the realm of the “matrix” will be futile. Accessing or supporting users during “regular business hours” will get you laughed out of the market. Not reaching out to or monitoring social networks will be foolish at best. The thought of instant, always-on BI following users around on mobile devices 24/7 may make some people laugh, but this is how young people already live. There’s no reason they’ll leave this behind in the context of “the office” in years to come.

Do-it-yourself is the new mantra. Self-empowerment is alive and well. As Bob Marley used to sing, “when one door is closed, don’t you know, another is open”. Unless you provide the tools and infrastructure to “DIY” you probably won’t get much traction (we see this happening now with open source already). Empower users because they’ll be savvy and used to it.

The current economic turmoil and its endless revelations, combined with the technological advances that have shifted the way young people see, communicate with, and gauge the world around them, will translate into a new type of enterprise BI buyer in the coming decades. If I were running a BI company, I’d want to bring the “good word” into the schools now, and start cultivating my future clients in the dorms, in the research centers, in the classrooms and on their mobile devices.

I could be wrong on this, but my money is on the bet that the upcoming generation of BI buyers is an entirely different animal. In this business, like chess, it pays to think several moves ahead.

Leave the Gun, Take the Cannoli

2009-06-24T14:40:00.000-07:00

I grew up in New York City and spent enough time in Jersey back East to have witnessed a couple interesting brawls in my life (I even had neighbors who dug holes for a living or worked in waste management) but it’s been a while since I’ve seen anything like the recent scuffle among industry analysts and vendors regarding the recently published ParAccel TPC-H benchmark. Maron!

It all started innocently enough two days ago when Merv Adrian, BI industry analyst emeritus, published the news in his blog titled “ParAccel Rocks the TPC-H – Will See Added Momentum”.

Now, it’s not every day that a vendor publishes audited TPC-H benchmarks (“audited” being the key word, as the process runs around $100K from what I understand). Very few companies besides the Big Three have the deep pockets and technology prowess to accomplish that. Furthermore, ParAccel did its benchmark based on 30TB which isn’t exactly a small chunk of data. And so Merv made the point that, at the very least, the news should certainly help put ParAccel on the map. To quote him: “This is a coup for ParAccel, whose timing turns out to be impeccable”.

Immediately, this was picked up by no other than Curt Monash, BI analyst to the stars (and I say that quite seriously), who happens to despise the very concept of TPC benchmarks for reasons he clearly outlines in a recent post entitled “The TPC-H benchmark is a blight upon the industry”. To pull a couple of money-quotes from the site:

“...the TPC-H is irrelevant to judging an analytic DBMS’ real world performance.

“In my opinion, this independent yardstick [the TPC-H] is too warped to be worth the trouble of measuring with.”

“I was suggesting that buyers don’t pay the TPC-H much heed. (CAM)”

“TPC-Hs waste hours of my time every year. I generally am scathing whenever they come up”

Now, notwithstanding the TPC-H issues, I think Curt will concede that he doesn’t particularly appreciate or trust ParAccel as a company either as the following statements will show:

“I would not advise anybody to consider ParAccel’s product, for any use, except after a proof-of-concept in which ParAccel was not given the time and opportunity to perform extensive off-site tuning. I tend to feel that way about all analytic DBMS, but it’s a particular concern in the case of ParAccel.”

“I’d categorically advise against including ParAccel on a short list unless the company confirms it is willing to do a POC at the prospect’s location.”

“The system built and run in that benchmark — as in almost all TPC-Hs — is ludicrous. Hence it should be of interest only to ludicrously spendthrift organizations.”

“Based on past experience, I’d be very skeptical of ParAccel’s competitive claims, even more than I would be of most other vendors’.”

The combination of published TPC benchmarks and the originator of the benchmark seem to have created what Curt himself refers to as “the perfect storm”. To say he doesn’t like either would be a gross understatement :)

Both blogs immediately started getting “opinionated” comments from the public at large, including ParAccel’s VP of Marketing Kim Stanick and a gentleman named Richard Gostanian who may or may not be connected to Sun Microsystems (depending on which Twits you read). Sun supplied the hardware for the ParAccel benchmark. To cite a couple quotes from the comments, Richard Gostanian responds:

“Perusing your website, I detect a certain hostility towards ParAccel.” – (No kidding!)

“Indeed you do more to harm your own credibility than raise doubts about ParAccel.”

“…TPC-H is the only industry standard, objective, benchmark that attempts to measure the performance, and price-performance, of combined hardware and software solutions for data warehousing.”

“So Curt, pray tell, if ParAccel’s 30 TB result wasn’t “much of an accomplishment”, how is it that no other vendor has published anything even remotely close?”

Then Kim Stanick says: “It [TPC-H] is the most credible general benchmark to-date.”

And an anonymous reader chimes in:

“After reading Curt’s post about ParAccel and Kim this is obviously personal…I wonder why the little fella didn’t have a fit over Oracles 1TB TPC-H? Check his bio. He consults for Oracle.”

To which Curt replies (among other things):

“As for your question as to why other vendors don’t do TPC-Hs — perhaps they’re too busy doing POCs for real customers and prospects to bother.”

Ouch! This nasty sudden melee took me by surprise at a time when I was considering blogging about the whole TPC-H system for analytical engines anyway. I’ve wondered for quite a while whether or not publishing such metrics actually helped “new breed” startups like ourselves from a marketing and sales standpoint. Given the high cost and resource drain, what’s the return on this investment? What’s more, I have yet to meet a prospect or user who either cares of knows about TPC-H benchmarks. So far, the only people I’ve ever seen show any interest in the matter are venture capitalists and investors which tells me right there that something is amiss (or, maybe that’s why the small players take the plunge, I don’t know).

As some of you may know, XSPRADA is a recent member of TPC.org alongside other industry startups like Kickfire, Vertica, ParAccel and Greenplum. Numerous other startups in the same category are not members. They don’t seem to fare any worse. Furthermore, as best I can tell, even some existing members (namely Greenplum or Vertica) don’t publish audited benchmarks. Yet clearly these two vendors don’t seem negatively affected by the lack thereof.

Although we at XSPRADA have conducted TPC-H benchmarks (and continue to do so) internally, we have never attempted to get them audited and published. If a prospect asked me about it, I would recommend we help him run those benchmarks in-house on his own hardware anyway! Even if we had $100K to blow on getting audited benchmarks, I’m not sure it would make sense to pursue.

I’m usually a pretty opinionated black & white guy, but with respect to this TPC-H business, I tend to centerline. Strangely enough, I identify with both sides of the argument. One the one hand, I don’t believe the benchmarks to be totally useless. Having been involved in generating our internal results, I can vouch for the fact that it takes a lot of tedious work and kick-ass engineering to even complete the list. By any stretch of the imagination, this is not a small inconsequential feat. Doing so on anything above 10TB is, in my opinion, nothing to sneeze at. If nothing else, being able to handle the SQL for all 22 queries is a decent achievement. And then of course, there’s the notion that even “trying” to do it is noble in itself. In that sense I tip my hat to the small guys who pulled it off.

On the other hand, I don’t feel the benchmarks are holistically useful for evaluation purposes. As a prospect looking at several vendors, they might figure in my check-list but not more significantly than others I consider more important. Namely: how easy is the product to work with, what resources does it consume (human and metal), how does it play in the BI ecosystem as a whole (connectivity), and last but not least, what kind of support and viability will the vendor provide? I’m a little weird that way. I tend to evaluate companies based on their people over most everything else. But that’s just me.

At the end of the day (and everyone does seem to agree on that), what matters are onsite POCs. Nothing can beat running your own data on your own metal. I want a vendor to hand me the keys and go “ok, have a good ride, call me if you need anything” and mean it. BMW sells cars this way. Enough said. This is what I drive to when helping people evaluate our offering.

It remains to be seen how much of this brouhaha will benefit ParAccel in the long run. They say there’s no such thing as bad publicity. If they end up getting recognition and sales from it, then they have chosen wisely, and no one can take that away from them. Personally, I wish them the best. I believe the more numerous we are in this upstart game, the better it is for us, and more importantly, for our customers. So I say leave the guns, and take the cannoli.

In-House or SaaS? How About Both?

2009-06-23T16:48:00.000-07:00

Chuck Hollis just penned another interested blog post about the economics behind private clouds for the enterprise. There is a lot of talk about on-premise versus on-demand SaaS these days in the BI community (and when I say SaaS I mean either private or public).

From a financial standpoint, the two models are fairly well established. Basically, on-premise is budgeted as capital expenditure, while on-demand is budgeted as operational expenditure. Much like the difference between buying your TV and paying for your electric bill on a monthly basis, or the difference between buying and leasing a car.

With on-premise you buy a lot of expensive stuff and it depreciates (and loses value) over time. With on-demand you rent a service for a short critical amount of time as needed. In many ways, the parallels are strikingly close to hiring in-house software developers versus outsourcing to consultants. The business case for either direction is easily conceived.

In-house developers are a long-term investment. They will learn the business and be allocated as needed on a per-project basis. The project is likely to be long-term. Much like capital equipment, they also need to be “upgraded” periodically – namely allowed and encouraged to keep up with technology so they remain productive and far-sighted. They also need to be provisioned with tools and resources to do their jobs effectively. The most progressive shops understand that. Not unlike expensive heavy metal and software licenses, they also tend to get worn out or outdated with time. And they’re expensive to replace and renew.

Consultants (either remote or in-house) are a quick-fix solution, usually applied to a pressing problem when in-house resources are either not available or not competent to handle the pressing business need. They’re expensive, but they (hopefully) get the job done quickly, get you the answers you need when you need them, and ride out into the sunset. Like on-demand offerings, they can be shutdown at will, but they also carry vendor-lock risk.

Having been in the software business for two decades, I’ve been on both sides of this fence numerous times. In my experience, the best shops implement a hybrid approach with strong internal cores supplemented as needed by top-notch “gun slingers”. In the best of cases, I’ve seen synergy and knowledge transfer occur between the two entities (when the politics were right) with significant benefit to the enterprise.

I think the same thing will probably occur in the BI space. I would imagine large shops will probably have both in-house staff and equipment, backed up by quickly ramped up SaaS offerings dedicated to what I call “transient data mart needs”. I could be wrong about this, but hybrid approaches (in business and technology) are usually more flexible and not necessarily conflicting. They can also feed off each other in positive ways.

I don’t believe the choices are going to be 100% on-premise or 100% on-demand. The trick for the CIOs out there is going to be determining which projects and which needs are better served by internal (strategic) or external (tactical) solutions in an agile way. In that sense there is really no “battle” between the two approaches. They should be considered complementary parts of an intelligent BI strategy toolbox.

This ACID Leaves No Bitter Taste

2009-06-23T08:51:00.000-07:00

In my previous post below I received a comment (question) from Swany about inserts in the XSPRADA database engine RDM/x. Specifically, he (or she) asked:

“What happens if there is an error during the SELECT .. INTO? Are such inserts ACID or will I get partial data in a table if the system crashes?”

This is of course an excellent question, and I thought it was worth addressing on a wider level beyond just incremental loads. To place this in context, recall that ACID properties are defined as a set of rules pertaining to transactional database management systems defined as Atomicity, Consistency, Isolation and Durability. If a transactional database does not meet these conditions, it is not considered “reliable”. I won’t bore the reader with yet another ACID definition. Suffice to hit Wikipedia for a reasonable description.

Now the first question of course is whether analytical database engines supporting OLAP style work can or should meet the same criteria as classical transactional OLTP systems. By definition, analytical systems are biased for read access and updates are supposed to be rare, but inserts certainly occur as incremental loads are performed on warehouses (and data marts) at various intervals (from hours to weeks typically). In either case, an analytical engine clearly needs to handle data changes in an ACID way or data loss and corruption can occur. Similarly, data value and integrity need to be protected (locked) from concurrent (possibly conflicting) access patterns. Internal database structures are vulnerable to corruption during these transactions. So how does XSPRADA technology handle these issues?

The answer, not surprisingly, lies in XSPRADA’s “magic sauce”, namely, the mathematics of Extended Set Processing (XSP). To appreciate this, one needs to understand that all entities inside the XSPRADA mathematical model (ie: tables, rows, fields etc.) are represented as extended sets. And all extended sets by definition are immutable. This means updates to the system are implemented by creating additional extended sets, and the original ones are never mutated or deleted by subsequent processing. This ensures that data sets in the system are never corrupted or worse, deleted by mistake.

Internally, the XSPRADA data model is fully contained in a “universe” of extended sets. This is the set of all sets. Sets in this universe are related to each other via algebraic relations (hence the “relational” part of Relational Data Miner or RDM/x). Depending on the state of the system at a specific point in time, sets are either “realized” (materialized) to disk, or “virtual”, meaning they have an internal mathematical representation defined by algebraic expressions involving other sets, but no physical existence. (This has repercussions concerning “materialized views” which I’ll attempt to discuss in a future post).

So when sets are modified by internal operations, both new and old sets remain in existence within the universe. This means updates and inserts never actually change information, but only add to it. To complete the transaction, RDM updates the universe metadata to include knowledge of the newly created sets (if any) along with the algebraic relations linking them to their original brethren. Genesis is maintained. This is crucial because, unlike in conventional DBMS systems, the original data never needs to be re-generated (or created) to achieve rollback. The universe is only updated once all operations have completed successfully. If an error occurs, no harm no foul, as the previous state of the universe was maintained and still exists. In essence, the INSERT, UPDATE, DELETE functionality of the XSPRADA database is merely a logical emulation of conventional DBMS DML. Each of these statements internally results in an additive activity. In fact, UPDATE is internally implemented as DELETE+INSERT. So to recover from a failed set of operations (a transaction gone south) the system simply deletes any incomplete sets and does not update the universe metadata! This mechanism enforces atomicity and consistency natively without any need for additional programming or complexity.

On the isolation front, lists of in-process and pending operations are maintained in dynamic pipelines for each extended set. The system examines these pipelines and algebraically identifies any potential conflicts between operands. Again, the mathematics allows this to occur natively. So if the results of pending operations do not affect in-process operations, the system executes them concurrently and immediately. Conversely, if the mathematics identify a potential conflict or deadlock, pending operations are queued until conflicting running operations have completed.

Durability is the last remaining condition. The system maintains all realized sets in persistent storage (disk drives). Although sets or parts thereof can be (and often are) kept in cache, any complete set also has an image on disk. This prevents system failures from affecting the durability of realized sets. When the system restarts, it also restarts any operations that were executing at the time of failure.

For all these reasons, XSPRADA technology is actually superior to conventional database mechanisms for enforcing ACID, as the enforcement is inherently “built-in” via the mathematics underlying the system at all times. As I mentioned in the last post, the engine is also time-invariant, meaning it can always be queried at a given time point in the past. The ability to do this without any external programming or internal modeling is significant. One use case that immediately comes to mind (to me anyway) in wake of the recent Wall Street disasters is being able to ask a financial database to yield answers as if it were being queried months or years ago. Imagine being able to roll back time to analyze or audit results that supported past decisions and the people who signed off on them. What a concept!

If you'd like to take the XSPRADA database out for a spin (and pull the plug in the middle of using it just to see what happens ) feel free to get the trial bits from our website.

Can Rover Roll Over Too?

2009-06-22T13:29:00.000-07:00

In this post, I want to stay in gear-head mode and "demo" a couple of neat tricks from the XSPRADA RDM/x analytical engine puppy. I’m going to address quick prototyping capabilities (thanks to schema agnosticism), incremental inserts, and time invariance.

We’re going to do this with some really simple tables and data structures just to highlight the concepts behind the engine’s functionality. So suppose we start with a CSV table called ‘demo.csv’ consisting of the following rows (NOTE: the file must be terminated by CR-LF):

1000,12,"now is the time"

1001,45,"for all good men"

1002,76,"to come to the aid"

Now we want to load this into RDM/x so we use the XSPRADA SQL extension CREATE TABLE FROM as follows:

create table demo(id int,c1 int,c2 char(128)) from "C:\ demo.csv"

Then we do a SELECT * on the table just to make sure all is well (for example, using QTODBC or any other ODBC-compliant SQL front-end tool) and we see:

1000 12 now is the time

1001 45 for all good men

1002 76 to come to the aid

And if I look at my schema in the QTOBDC object browser I see this as expected:

So this is fine and dandy when you know the actual schema of your data but now what if you don’t or what if you’re not sure or what if you're out to determine what the best schema might be given your application?

Ok well what we can do it simply load everything as text types. So now we do:

create table demo(id char(128),c1 char(128),c2 char(128)) from "C:\ demo.csv"

The original ‘demo’ table is overwritten with the new schema which now becomes:

And now we can play around with that until we’re satisfied. Assume we suspect the optimal way to work with this data might be using INT for the index, a double precision type for the c1 column, and an 80-char VARCHAR for the last field. We might take our existing table and “flip” it into a new schema (called flipped) as such:

select cast(id as int), cast(c1 as double precision), cast(c2 as varchar(80)) from demo into flipped

Note how we convert the schema into a new one directly into a new table on the fly. Now if we select from flipped we see:

1000 12.0 now is the time

1001 45.0 for all good men

1002 76.0 to come to the aid

And the schema for the new table, as expected, is:

Now if we needed to do some querying on numerics for C1, we could. Note we didn’t even have to explicitly create or schema the ‘flipped’ table. RDM/x took care of that automatically, making these types of operations ideal for quick prototyping. Incidentally, we could have done this directly on the existing table as well. This type of flexibility is pretty cool and allows you to “play” with the data in a trial and error mode.

Now suppose we come across some additional data and we wish to INSERT this information into our existing database. This is typically what happens with daily incremental into a data warehouse. On a periodic basis, chunks of data are added to (typically) fact tables. Our incremental CSV looks as such:

2000,132,"to be "

2001,465,"or not to be"

2002,786,"that is the question"

There are several ways to handle this using RDM/x. The most direct one is to simply use the XSPRADA “INSERT INTO FROM” SQL extension as follows:

INSERT INTO demo FROM “c:\demo_062209.csv”

In this case RDM/x is told “hey, there is more data being presented to you so, algebraically, union it with the existing data”. As RDM/x does not “load” data in the conventional sense of the term or support bulk loading (as it doesn’t need to), additional data can be presented virtually in real-time with little or no effect on the ability to query the system simultaneously.

This is what’s called “non-disruptive live updates” or concurrent load and query capability. Initially, users are a little surprised that bulk inserts are not supported. But in fact, all that’s needed is “dropping” the incremental data somewhere on disk and telling RDM/x of its existence. RDM/x can ingest this information as fast as it can be written to disk.

Another way to do this is by loading the incremental “chunk” into its own table as such:

create table demo_062209(id int,c1 double precision,c2 varchar(80)) from "C:\ demo_062209.csv"

And then doing

select * from demo union select * from demo_062209 into demo

A bit of subtlety there: when you do this, you’re effectively “updating” the existing demo table with the unioned results of the incremental. So you take the old demo table, add the incremental, then flip that back into the original demo table. You could have saved the “old” demo table first before doing this as such:

select * from demo into demo_current

Notice how these “dynamic” tables really behave like variables in a loosely-typed programming language. They are conceptually related to conventional database “views” (although RDM/x does support view objects in the database but from a purely semantic way). As a matter of fact, all tables and views in RDM/x are essentially the same thing and materialized (to disk and/or memory) on a JIT basis anyway.

Of interest here is that you can always recover past instances of any data RDM/x via a nifty little feature called “time invariance”. This feature is unique in the world of databases to the best of my knowledge. Essentially, you can query RDM/x much like a “time machine”, asking it to yield results as if the question were being asked in the past.

So suppose you had inserted your incremental into the ‘demo’ table by mistake and suppose your insert had occurred at a given time such as '2009-06-22 15:13:52.775000000' (and you can tell this if you are logging your SQL statements to the database using the usrtrace option of the RDM/x ODBC driver). You can always recover the state of ‘demo’ at the time by issuing:

SELECT * FROM demo AT TIMESTAMP '2009-06-22 15:13:52.775000000' into recover

Now your ‘recover’ table will contain ‘demo’ exactly the way it was at that time.

The implications are far-reaching because you can essentially always query the RDM/x database at any point in the past. RDM/x never deletes information UNLESS the information has been explicitly deleted using a DROP TABLE or DELETE type of DML statement AND the garbage collector kicks in. Short of that, the database maintains information and integrity through time.

These are just a couple of nifty tricks from one original database engine called RDM/x.

Choose Wisely Grasshopper

2009-06-19T14:28:00.000-07:00

If you’re been tasked with implementing or supplementing business intelligence offerings in your organization lately (read: a data mart, for example), the choices have gotten quite a bit more complicated. A while ago, you would have your pick of three or four vendors. If one of those vendors was already in-house handling your transactional systems, guess what, he was likely to also handle your warehousing needs. Or at least try to.

Nowadays, the decision path is a little more involved because the BI world is no longer ruled by a small oligarchy (namely Oracle, Microsoft and IBM). The proliferation of “new-breed” analytical engine vendors has greatly expanded a buyer’s options. There are around twenty-some players in this field now. Not only that, but delivery options have expanded as well.

Nowadays, you can get BI delivered and running in-house sitting on commodity or custom hardware. You can buy canned appliances. You can go proprietary bits. You can go Open Source. Or you can tap the “cloud” with an on-demand subscription-based model. You can pick a columnar vendor, or a row-based one. You can choose an MPP architecture, or an SMP implementation. You can even stick with the “gorillas” if that makes you feel better (and money is no object). These different paths are all strategically different both technically and economically. The modern BI strategist is compelled to choose wisely in an unforgiving economy where failure is no longer an option (this time they mean it).

So for the sake of argument, I’m going to assume the following buyer profile:

Money is _definitely_ an object.
IT resources are non-existent, limited, not available, or not inclined to help.
A lengthy proof-of-concept (POC) cycle is not an option.
Project timeline is measured in weeks not months or years.
C-level people want to see incremental results starting today.
A single DBA is left standing in your organization (but next week, maybe zero).
Your ass is on the line.

I think this describes a fairly common scenario these days. Faced with such odds, I think most people opt for the path of least economic and implementation resistance. Most people don’t have $2-6M hanging around including staff and a six-twelve month window to implement a full-blown Oracle, Microsoft or IBM solution. These days are simply gone (good riddance on that). It would appear at first glance that the only remaining alternatives would be open source (OSS) or on-demand SaaS software.

Now, OSS is attractive from a cost basis, as most freebies tend to be. Yes, you do pay for support if pulling down “enterprise” versions of the software but in many cases, some buyers get away using the free versions for a while, at least for quick POCs. In some cases, the free versions are either limited or incomplete in functionality (like InfoBright, for example) so that could be a “gotcha” depending on your application needs. Similarly, non-enterprise versions usually depend on community for support. If you’re in a jam and need serious dedicated support on a moment’s notice, you’ll need to pony up for an enterprise version or wait until “the community” comes up with an adequate answer to your problem (if ever).

Additionally, OSS does have hidden costs, not the least of which are installation, setup, configuration, and maintenance. But, if you happen to be a Linux shop and have enough LAMP developers on staff with sufficient expertise and time, and your management happens to be accepting of the whole OSS concept, it might just be a viable option.

Another quick way to get up and running quickly is to go the Cloud (SaaS) service route. In that scenario, you pay a monthly fee to access a BI platform in the cloud. This hands-off approach is certainly attractive in many cases provided data volumes and security restrictions do not get in the way. Shlepping 10-100TB of data offsite is not something most people consider yet. But, for smaller data sizes, SaaS is certainly an option. Vertica, Kognitio and Aster Data come to mind as the latest new-breeders to provide cloud-based services to customers (either on proprietary or public cloud platforms like EC2). There is a flurry of other on-demand BI players as described in several of my past posts. Of course, the downsides there tend to be upload time, limited functionality and vendor lock-in.

Now if I can get on my soapbox for a minute, I’m going to pitch a third option. At XSPRADA, we’ve developed a 5MB high-performance analytical database running as a Windows service on commodity hardware and Server 2003 or 2008 x64 operating systems. You can install this puppy internally (your data stays nice and safe in-house) in about 2.5 minutes including ODBC drivers. You then point it at your CSV data on disk and start firing off queries immediately. The more you ask, the faster it gets. And you can show results in minutes, not weeks or months. No cubing, no indexing, no pre-structuring, no partitioning, none of that nonsense. That’s the bottom line in a nutshell. I promise you business intelligence is not the rocket science so many folks make it out to be! For those who might be hesitating between open source, SaaS or much more expensive in-house solutions, I think this is a pretty unique proposition. There really is nothing like it anywhere else on the market, and it just so happens we have a 30 day trial going on right now if you visit our website J

Yabadabadoo and off to Austin!

2009-06-04T16:00:00.000-07:00

I’m headed out to Austin again tomorrow. Last I checked it was 94F over there with 60% humidity. Guess I won’t be going on my daily power walk for a little while lest I keel over. That’s OK though, I’ll get my workouts at Bone Daddy’s. I’m jazzed up about the trip as usual. I love that town. It has a relaxed, genuine, homey feel I have not found in many other US cities. When people ask “how ya doing?” they actually mean it and expect an answer!

I have prospects, clients and partners to visit. Now that our pre-release RDM/x software is available, I’m getting to see how people actually use these bits in real-life situations and that’s fairly exciting for many reasons.

Product claims are one thing. But it’s quite another to actually see people’s reaction when they discover the software. When folks install our software the usual reaction is “huh? Is this all there is to it? What did I miss?” – The answer is “you missed nothing” – RDM/x deploys and installs as a 5 Meg Windows Service. Read my lips: a 5 megabyte Windows Service EXE is all you need to query TB-level data sets faster and easier than you’re used to. Install the bits, present data, connect to ODBC driver, and ask questions. Done. Naturally, given users’ past experience and expectations with classic analytical engines, it’s hard to swallow initially. And speaking of shockers, here a doozy of a benchmark you can find on the TPC website.

That’s right folks, $6 Million dollars with Godzilla metal and full-blown enterprise software will buy you queries into a mere 1TB of data. Or, you could install and start a 5MB Windows Service on a $20,000 server.

What else am I picking up? People are incredulous about the fact that there’s no need for partitioning, indexing, pre-structuring, worrying about rigid schemas and data models. In one case, I was actually able to load 18GB of data by pulling everything in as strings. These particular data sets were CSV clickstreams for a web analytics application (conveniently, they’re often text log files which suits us well as CSV is the only format we currently ingest). The ETL attempts had become tedious due to poor data quality and uncertain schemas. It wasn’t clear initially what the best schema might be for the type of data at hand so experimentation was needed. So we took more of an ELT approach and just DDL’d everything in as text. We then quickly flipped internally to another schema, casting columns as needed on the fly. And if that schema turned out to be less efficient or interesting than originally thought, we just flipped the whole dataset to another one in one line of SQL. This type of just-in-time flexibility in presenting and transforming data is seldom (if ever) found in more ponderous solutions. For web analytics type of applications (where DQ is typically poor and schemas fairly dynamic) this is a huge competitive advantage.

Another sweet spot is OLAP cubes. Or lack thereof I should say. In typical OLAP applications, building cubes is lengthy, complicated and expensive (as anyone who ever footed an analytical consultant’s bill will attest to). Re-aggregating or adding dimensions is also painfully slow and tedious as data volumes increase (and they always do). A lot of times these processes are setup to run nightly as batch processes. If you’re lucky and if you’ve done your engineering properly (lots of ifs), you show up in the morning and it’s done. But you can’t add a dimension in real-time without impact on the entire system. And you need to plan for your questions up-front, as in “what shall I slice & dice this year?” With RDM/x, you don’t need to setup dimensions explicitly. The very act of querying in a dimensional way alerts the engine to that effect and it starts building “cubes” internally and aggregating as needed, optimizing for specified dimensions. Then adding a dimension is painless – all you need to do is actually query against it more than once. With RDM/x, once is an event, twice is a pattern.

This “anticipatory” self-adjusting behavior is actually at the heart of RDM/x. This is one area where the product distinguishes itself from the competition. Because the software continuously re-structures both data and queries as more and more questions come in. This means you see response time decrease as quantity and complexity of queries increase. This is fundamentally different from conventional database behavior where query response time “flat-lines” up to a given number (not including load time) and pretty much hovers there until and unless an “expert” can optimize either code or configuration (if possible at all). And when you add to or complicate your query patterns, performance suffers.

The RDM/x response-time profile is more of what I call the “dinosaur tail” effect: big initial hump, then exponential taper down over time. Here’s my Fred Flintstone rendition of the effect using MS Paint (nostalgia?):

It is always a blast observing people as they first witness this unique behavior. It’s like discovering a new life form J -- And I’m looking forward to repeating the experience in Austin next week and throughout the country soon enough. If you care for a head-start, pull down the bits from our website and give it a whirl!

Data Warehousing for Dummies

2009-05-25T17:55:00.000-07:00

Every so often I’ll be talking to someone implementing our solution at a customer or POC site and the question comes up “what’s so special about your database anyway?” and “So, is it really different from MySQL, SQL Server or Oracle?” or “I don’t understand why your database talks SQL since it’s not a normal database like Oracle”. Better yet: “What else do I need to do after installation to get this working?” Usually these questions come from non-LOB folks who are tasked with implementing a particular solution using our product. Typically these people tend to be experienced software developers or DBAs. People who, as Joel Spolsky likes to say, are “smart and get shit done”. The type of folks you can throw a problem at and say “Ok, go solve it using this new tool.”

But as often happens in large organizations, they may not have been briefed fully by management on the features/functionality of the new tool needing evaluation. Or maybe this is their first exposure to BI. They may also never have encountered or worked with an analytical database product. I know that, several years ago, if you’d asked me what the difference was between OLTP and OLAP I would have blurted something like “one if for transactional stuff, the other for reporting” and been in the right ballpark but no cigar.

So when these questions come up, I am always ecstatic to be able to share what I’ve learned with the guys in the trenches doing the real work! The first thing I do is give I very general view of the differences between transactional (operational) and analytical use cases. Then I’ll try and give a 30,000 foot picture of data warehousing and its history. I‘ll mention Kimball and Inmon, of course, then several books and a series of blogs, websites and youtube videos for further exploration.

But this weekend, I discovered the holy grail of data warehousing 101. I was hanging out at my local Borders looking to trade my 40% off coupon in exchange for yet another good data warehousing/BI book when I noticed the yellow “Data Warehousing for Dummies” on the bottom shelf. Obviously I couldn’t resist picking it up, especially since I have yet to meet anyone remotely “dumb” in this business.

To my great surprise, I noticed the author was none other than Tom Hammergren, the owner of Balanced Insight, one of the top BI Software Innovator firms in the country. To say that Tom is a warehousing and BI guru is an understatement. This much I knew. But I had NO idea he was also an accomplished writer who could present this complicated subject in clear, simple terms anyone can understand and relate to! From now on, whenever someone asks me for the quick low-down about BI and data warehousing, I’ll be referring him or her to Tom’s book.

Now to answer the above questions about our own product RDM/x. There is nothing magical about our analytical database, at least from a usage standpoint. RDM/x talks and walks like any other database product out there on the market using ODBC. The magic is on the inside, certainly not in the interface (thankfully) as it supports a significant subset of the SQL-92 standard (minus TCL and DCL). Is RDM/x “really” different from SQL Server, Oracle, MySQL or DB2 though? You bet.

RDM/x is designed for data analysis, not transactional processing. As a matter of fact, RDM/x is the smallest, nimblest, on-premise solution available on the market that will let you query terabytes of data in minutes from installation. And that’s why I believe most people are a little confused from the get-go. Because they’re used to large footprint multi-module database clients with 500 page installation and setup manuals, followed by complicated tuning and optimization techniques involving indexing, partitioning, and all that “good” stuff. When they see a 5 megabyte piece of software installing as a Windows Service, ready willing and able to handle queries on giga or terabytes of data within minutes, they think they’re missing something. How can BI be this simple? Well it can. The proof is in the pudding and since we’re allowing you to download a fully functional 30-day evaluation from our website effective now, the best I can do is recommend you take me up on that assertion by visiting our website.

Running RDM/x on Amazon EC2 is DICEE!

2009-05-21T12:03:00.000-07:00

I want to keep this post fairly brief because there is so much stuff going on at XSPRADA lately that I find myself pressed for time from 6AM to midnight on a typical day which usually also includes weekends, but that’s the price you pay for building a revolution. Ask Fidel, he knows.

First of all, I finally had time last Sunday to record a screencast explaining how to install and setup our RDM/x software. In the process, I discovered that Camstudio and Microsoft Windows x64 decoder were my friends, reducing a 1.2GB video to 35Megs (phew!).

Second, I want to talk about my recent epiphany with EC2. Early this week I decided to see if I could install and run our Windows-based database engine RDM/x on some sort of cloud platform because I don’t think anyone in their right minds in enterprise software can afford to ignore this trend any longer. My purpose was certainly not to setup a full-fledged production system up there, but rather to setup a quick and dirty demonstration system so people could either duplicate or use it on the fly to test-drive our software, for example.

After poking around a bit I settled on Amazon’s EC2. They seemed like the only “big-time” player supporting WinTel boxes (our software runs on 64-bit Windows Server 2003 and 2008) and they have enough credibility and market “karma” at this point to alleviate most basic concerns about reliability and security. So after checking out possible configuration tools and hitting our CEO up for some plastic, I signed up for EC2 and started exploring this brave new world.

It turns out EC2 is really several “platform” components comprising: the actual EC2 O/S instance (a VM blade) known as an AMI (Amazon Machine Instance), persistent storage called EBS (Elastic Block Store) which presents as “volumes” you mount onto the AMI, and persistent (hot/cold) storage (also used for EBS snapshots) called S3. There’s also a queuing system called SQS but that didn’t enter my mix. Confused yet? It’s not that bad once you get used to it J

For the configuration tooling, you can use command line tools (which I suspect most *NIX/LAMP people prefer), a FireFox pluggin, or the web-based AWS (Amazon Web Services) Console. I used both of the latter to compare.

I brought up one of the standard Windows AMI as a Windows 2003 R2 64-bit datacenter server. I actually tried two different instance types. One extra-large standard with 15GB of RAM and 4 cores, and one large high-CPU with 7GB of RAM and 8 cores. I found better performance on the 4 core box with twice the RAM so I ended up sticking to that one.

For credentials, you need to generate a key-pair, and then plug in the private part into a dialog box which then spits out an admin password for the new instance. You then connect remotely to your instance using Remote Desktop Connection (or SSH if you’re talking to a *nix instance).

Right off the bat my instance came with four attached 500GB “hard drives”. These volumes are more like flash drives I think. This is not persistent storage but it’s pretty darn fast. For “real” storage you need to create Elastic Block Storage (EBS) “volumes” and attach them to your instance. So I did just that and slapped 4 additional 500GB drives to my box, and then converted each drive to an NTFS mount point (because this is best practice for our particular application). Unfortunately, I extracted a maximum 22-25MB/sec I/O to and from these volumes. I had read somewhere that these dynamic “block devices” were more like instant SANs but in fact, Amazon Silver Support (another pay-for service but well worth it if you ask me) stated the following to me in an email:

“Even though it presents a block interface, EBS isn't intended to be equivalent to fibre-channel SAN storage. The performance you should expect from EBS would more closely align with a NAS device. You can stripe several EBS devices together for higher I/O rates, but your rates will be limited by various shared components in the system, including the network between your instance and the storage servers. Larger instance types will typically see better performance than smaller instance types.”

“More closely align with a NAS device”. Huh oh. That means gating at the NIC level. The systems are clearly not setup for intense I/O data processing needs, at least not using the standard EC2 configuration models currently available. Nevertheless, I was still able to do sufficient work with sufficient data to build a reasonable “functional demo” machine. And that was my goal from the onset. Given this took me about 1.5 days to figure out, at an average cost of around $16/day (not including the additional Silver support fees) I am very impressed with this cloud platform to say the least and I’m sure Jeff Bezos is basking in the bliss of my endorsement :)

Quite honestly, this cloud business is no joke. I haven’t seen, heard and felt such a buzz around a new “platform” in the industry since I got my hands on Windows 3.0 in the early nineties. It was the same “oh my God” emotional feeling at the time, or DICEE as Guy Kawasaki likes to put it (Deep, Intelligent, Complete, Elegant and Emotive).

Software that Sucks

2009-05-16T11:57:00.001-07:00

Here’s a classic from the tech press that really caught my attention recently:

“In the context of software, the word “Enterprise” has now officially come to mean software that sucks. Enterprise Software hit the nadir of suckitude (sic) at the launch of “Enjoy SAP”. This is like the American Dental Association launching “Enjoy Root Canal”. SAP is certainly an easy target, but let’s face it, “Enterprise Software” is generally a poorly integrated mess. Working with Enterprise Software feels a bit like walking through an industrial landfill or an airport hangar. Nothing is built to human scale.”

This was written on the SOA Center blog by no other than Software AG’s Chief Strategist Miko Matsumura. His use of the techo-political term “suckitude” is one for the annals of our new post-TARP technology world. If nothing else, the current situation seems to be facilitating proverbial “paradigm shifts” (namely, on-demand software) while encouraging more anti-status-quo “frank-speak” from industry figureheads. I’m all for that.

Because, notwithstanding all the pain, suffering and incertitude in the economy lately, one of the really brilliant consequences of this world-wide mess is that people are starting to say out loud what everyone’s been thinking silently for years. Even in the sacrosanct enterprise software glass mansions, people who matter are starting to throw stones. When major industry players start talking straight and using technical terms like “sucking”, you know the BS gloves are off. I think established players, platforms and ways of doing business and thinking about customers are all up for questioning at this point. Sunshine is the best disinfectant.

And speaking of gloves off, SAP and industry shifts, this old article from April 2008 refers to a slug match between two industry titans at the Churchill Club. One is Marc Benioff from Salesforce.com and the other Dr. Hasso Plattner of SAP fame. There’s a video of the exchange on Youtube. I know it’s a long one, but I assure you it’s worth watching entirely if you care anything about the on-demand versus on-premise religious wars of late.

I am not going to comment at length on the video as anyone can draw their own conclusions, but I did want to point out what I consider some key points, and throw in a few gold nuggets.

First, the body language between those two guys is simply priceless. It is more than obvious from the get-go that they can’t stand each other. You can catch the vibe even in that one picture in the article (and throughout the video). Benioff’s looking away from Dr. Plattner constantly (he fidgets with his wedding band incessantly), and Dr. Plattner is reflective in his own world as in “why the hell am I here”. To my amazement, at the end of the video, they both reveal that this is their very first in-person meeting! Incidentally, one audience member does ask Dr. Plattner at the end why he accepted to do this. His answer: “for the challenge”. Not sure what that means.

Second, the verbal jousting between the two is fairly aggressive. I don’t think these guys have much respect for each other notwithstanding their pseudo-polite claims to the contrary. If you asked me whether Benioff hates Microsoft or SAP more, I’d be tempted to say SAP.

At one point Benioff states: "We have been passionate about moving obstacles out of the way of the old enterprise software companies.” I guess this is one major tenet of the on-demand adepts. Power to the users! In my opnion, Dr. Plattner really does buy the on-demand proposition but not “religiously”, and either way, he can’t say it in public. He knows SAP screwed it up in the past. I’m not sure he believes in SAP’s ability to execute such a shift internally. And I bet he wouldn’t mind buying Salesforce outright with one check. He implies as much several times but then claims he doesn’t want to get into a bidding war with Oracle. Hogwash.

Throughout the video, both contestants score evenly, in my opinion, on the arrogance meter. I guess they can both afford to be that way, but it does take a certain piece of the “human” side away from each. For Dr. Plattner, I think the Germanic personality comes through more than genuine arrogance. After all, he doesn’t need it at this point. The guy built and ran a $40B company. Enough said. Benioff often has this “do the right thing” Google-ish “morality” in several other interviews and videos. But when you watch him in action here, the only thing that comes out is ruthless self-convinced warrior (it’s no coincidence his favorite read is Sun Tzu’s The Art of War). Although conviction and the ability to back it up is noble (and key to business success), I’ve always feared people immutably driven by their own dogma (mind you, I actually buy into on-demand big time). But as my high-school math teacher used to say “you can never shelter yourself from a surprise”.

Finally, as I was lauding “frank-speak” earlier, I did want to point out that Dr. Plattner uses the term “shit” several times during the exchange. Initially, referring to Salesforce grabbing Dupont from them he states “Why did he win DuPont? Because we had a shitty CRM system, and he had a much better one.” Then later, referring to a customer still using code Dr. Plattner himself wrote: “…Shit! There is a customer in America still using the code I wrote.” Then referring to SAP’s earlier attempt at on-demand CRM: “…Shit, yeah! It was better than our CRM on-demand.” I find that endearing.

To conclude, if you truly want to understand the ongoing (and upcoming) battles between the SaaS and on-premise proponents of the enterprise software industry, you owe it to yourself to watch this video or, at the very least, pull down the transcript. And bring some popcorn!

Tidbits and Check this Guy out

2009-05-07T09:18:00.000-07:00

I read this article a couple of weeks ago and thought about one of our field test partners (telecom) because they had some political issues shipping us some data due to (very legitimate) privacy concerns – as in their CSO going “are you guys out of your f$##$ing minds?!?”.

As it turns out, there are several data obfuscation tools out there on the market, including DMSuite’s offering as described in this article. I’m curious if most companies’ privacy policies make an exception for data that’s been altered by such a tool and if so, is there some sort of standard or certification these tools must meet? If you know anything about that, I’d appreciate some insight.

I didn’t know until last night that there actually is a CIQP Certification. What is CIQP you ask? Come on, get with the program! Everyone knows what a Certified Information Quality Professional is! There is a whole website dedicated to DQ as well. I had never heard about this professional category. If anyone reading this happens to be in that category and/or CIQP Certified, I’d love to chat with you and learn more about it.

For those of you who think the economy really sucks, your deduction is likely valid. Nevertheless, BI and on-demand software market indexes seem pretty healthy to me as this article demonstrates. My conclusion: I’d rather be in the “avant-garde” BI enterprise software sector than working for SAP or Oracle at this point J

I discovered Guy Kawasaki’s Entrepreneurial Lectures delivered at Stanford in 2003-2004 via this videocast series and sat there mesmerized listening to every single clip for hours.

Guy (who now runs this blog and this company) successfully evangelized the Mac in the mid-80s and now runs a VC firm called Garage Technology Ventures. His reputation and track record are legendary. In the clips, he lectures young Stanford engineers-to-be on how to become successful entrepreneurs, change the world, and keep their soul in the process. These are the points (or quotes) from his lectures that were etched on my mind:

Make meaning and make the world a better place.
Don’t write a mission statement, write a Mantra.
If your product is not unique and adds no value, you’re doing something stupid.
Don’t ask people [customers] to do things you wouldn’t do.
Be a Mensch.
Hire infected people.
Suck down. The higher you go in the enterprise, the thinner the oxygen.
A milestone is something that increases the valuation of your company.
The valuation formula for a startup is: add $500,000 per engineer and subtract $250,000 per MBA.

All these points are perfectly in line with my personal experience. But I’d never heard anyone formalize them in such an entertaining way before! And I don’t know that any comment can properly decorate any of these either. It’s one of those “you either get it or you don’t” kind of things. It can’t be taught or inculcated by anything else than passion-driven experience.

MAD About You.

2009-05-05T09:32:00.000-07:00

I haven’t had a minute to sit down and blog lately. Our upcoming “pre-release” software slated for Cinco de Mayo has absorbed all my efforts. I just flew back from Austin, TX last week to participate in final engineering and release touches. At the same time, one of our major Defense prospects just re-activated a huge project so it’s all hands on deck at XSPRADA these days. I love it when a plan comes together.

Nevertheless, I recently picked up on a paper called “MAD Skills: New Analysis Practices for Big Data” referenced on Curt Monash’s blog by one of its authors Joe Hellerstein. Joe is a CS professor at UC Berkeley, if I understand correctly, and a consultant for Greenplum. I expected to read a lot of pro-Greenplum content in there but I don’t feel the major arguments presented are specifically tied to this vendor’s architecture or features per say. What makes this paper really valuable, in my opinion, is that it “resulted from a fairly quick, iterative discussion among data-centric people with varying job descriptions and training.” – In other words, it is user-driven (and not just a bunch of theoretical PhD musings) and as such, probably an accurate reflection of what I consider to be a significant shift in the world of BI lately.

Namely, that the “user” base is shifting from the IT type to an analyst and business person type. Note I didn’t say “end-user”, because the end user is still the business consumer. But what’s really changing is the desire (and ability) to cut out the middle man (read: the IT “gurus”), in essence, at every step of the way from data ingestion to results production. To put it in terms Marx would have liked, the means of production are shifting from dedicated technical resources to actual consumers J -- This is particularly true in the on-demand world, but probably pervasive throughout as well. I think the paper outlines and defines this change.

“The conceptual and computational centrality of the EDW makes it a mission-critical, expensive resource, used for serving data-intensive reports targeted at executive decision-makers. It is traditionally controlled by a dedicated IT staff that not only maintains the system, but jealously controls access to ensure that executives can rely on a high quality of service.”

The starting premise of the paper is that the industry is changing in its concept and approach to EDW. Whereas in the past an “Inmonish” view of the EDW was predicated on one central repository containing a single comprehensive “true” view of all enterprise data, the new model (and practical reality) is really more “Kimbalish” in the sense that the whole of EDW comprises the sum of its parts. And its parts are disparate, needing to be integrated in real time, without expectations of “perfect data” (DQ) in an instant-gratification world. This new model is premised on a MAD model: Magnetic, Agile and Deep.

Magnetic because the new architecture needs to “attract” a multitude of data sources naturally and dynamically. Agile because it needs to adapt flexibly to fast-shifting business requirements and technical challenges without losing a beat. And deep because it needs to support exploratory and analytical endeavors at all altitudes (detail/big picture) and for numerous user types simultaneously.

“Traditional Data Warehouse philosophy revolves around a disciplined approach to modeling information and rocesses in an enterprise. In the words of warehousing advocate Bill Inmon, it is an “architected environment" [12]. This view of warehousing is at odds with the magnetism and agility desired in many new analysis settings.”

“The EDW is expected to support very disparate users, from sales account managers to research scientists. These users' needs are very different, and a variety of reporting and statistical software tools are leveraged against the warehouse every day.”

Fulfilling this new reality requires a new type of analytical engine, in my opinion, because the old ones are premised on a model which, apparently, has not delivered successfully or sufficiently over time for BI. So the question beckons, what set of architectural features does a new analytical engine need to have in order to support the MAD model as described in this paper? And more importantly (from where I stand), is the XSPRADA analytical engine MAD enough?

To support magnetism, an engine should make it easy to ingest any type of data.

“Given the ubiquity of data in modern organizations, a data warehouse can keep pace today only by being “magnetic": attracting all the data sources that crop up within an organization regardless of data quality niceties.”

The “physical” data presented to the XSPRADA engine must consist of CSV text files. Currently CSV is the only possible way of presenting source data to the database. CSV is the least common data format denominator so this means the engine can handle pretty much any data source provided it can be morphed to CSV. Since the vast majority of enterprise data lends itself to CSV export, that covers a fairly wide array of data sources. How easy is it to “load” data into the engine? Very easy, especially since the engine doesn’t “load” data per say but works off the bits on disk directly. All it takes is a CSV file (one per table) and a SQL DDL statement such as “CREATE TABLE…FROM ” to “present” data to the engine. “The central philosophy in MAD data modeling is to get the organization's data into the warehouse as soon as possible.” – I’d say this fulfills that philosophy.

To support agility, an engine shouldn’t dictate means and methods of usage and must be flexible:

“Given growing numbers of data sources and increasingly sophisticated and mission-critical data analyses, a modern warehouse must instead allow analysts to easily ingest, digest, produce and adapt data at a rapid pace. This requires a database whose physical and logical contents can be in continuous rapid evolution… we take the view that it is much more important to provide agility to analysts than to aspire to an elusive ideal of full integration”

The external representation of data on disk and the internal “logical” modeling of the data changes dynamically based on incoming queries. The XSPRADA engine is “adaptive” in that sense and constantly looks at incoming queries and data on disk to determine the most optimal way of storing and rendering it internally. This feature is called Adaptive Data Restructuring (ADR). On the flexibility side, the XSPRADA engine is schema-agnostic. This is a fairly unique feature that allows users to “flip” schemas on the fly.

For example, it’s possible to present entire data sets to the engine based on an all VARCHAR schema (ie: make every column VARCHAR). Maybe you don’t know the real schema at the time, or perhaps you don’t care about it or perhaps the optimal schema can only be determined after some analysis is performed. Or perhaps there are inconsistencies in the data or DQ issues preventing a valid “load” based on a rigid schema. Or maybe it was just easier and quicker to export all the data as string types in the CSV. In either case, the XSPRADA engine will happily ingest that data. Later on, you can CAST each field as needed into a new table on the fly and run queries against the new model, or try others as needed. Similarly, ingestion validation is kept to a minimum by design. For example, it’s quite possible to load an 80-char string into a CHAR(3) field. This is not possible with conventional databases. The implications of this from a performance and flexibility angle are impressive. The XSPRADA database lends itself to internal transformation; hence it favors an ELT model, minus the “L”.

“In recent years, there is increasing pressure to push the work of transformation into the DBMS, to enable parallel execution via SQL transformation scripts. This approach has been dubbed ELT since transformation is done after loading.”

And finally, to support depth, an engine should allow rich analytics, provide an ability to “focus” in and out on the data, and provide a holistic un-segmented view of the entire data set:

“Modern data analyses involve increasingly sophisticated statistical methods… analysts often need to see both the forest and the trees in running these algorithms… The modern data warehouse should serve both as a deep data repository and as a sophisticated algorithmic runtime engine.”

A salient feature of the XSPRADA engine is its ability to handle multiple “types” of BI work at the same time. For example, it’s possible to mix OLAP, data mining, reporting and ad-hoc workloads simultaneously on the same data (and all of it) without resorting to “optimization” tricks for each mode. Similarly, the need for logical database partitioning doesn’t exist in the XSPRADA engine. Duplicating and re-modeling data islands on separate databases (physical or logical) for use by different departments is neither necessary nor recommended.

In an OLAP use case, there is no need to pre-define or load multidimensional cubes. The very act of querying consistently (meaning more than once) based on fact and dimension axes causes the engine to realize that this particular section of data is being accessed “multi-dimensionally”. It then starts cubing information internally, aggregating as indicated (if needed) by incoming queries. In this mode, perhaps the engine will decide a columnar storage approach is optimal and will re-structure the data accordingly. In a data mining use case, the approach is likely different because incoming queries are “incremental” (often pinpointed) and results are used to generate new queries without pre-determined patterns. The engine will likely start by eliminating vast “wasteland” areas of the data (the forest) from consideration as needed, then proceed to optimize specific islands of interest (the trees) as they become more relevant in the queries.

So overall, I think the XSPRADA analytical engine was indeed designed with “MAD-ness” from the get-go, even if the term didn’t exist years ago. It’s the approach and the philosophy that really matters. In that respect, we’re definitely headed for the MAD-house :)

On-demand BI beyond SMB

2009-04-13T13:28:00.000-07:00

The more I read, test and learn about on-demand BI every week, the more surprised I am to realize how many players there are on the market. And this isn’t even just a small & medium size (SMB) market anymore as I originally assumed. There’s a whole slew of on-demand BI companies out there targeting serious size enterprises with giga and terabyte size warehouses.

It seems one of the main arguments against the performance (latency) issues for BaaS is that the penalty is imposed only once at load time. In other words, yes it’s time-consuming and fairly slow to upload warehouse data (sometimes up to weeks) due to current network and pipeline bandwidth, but it’s something that is done only once and subsequent pushes are essentially incremental and consequently much quicker. I guess I can buy that argument provided the “done once” endeavor is resilient enough to resist catastrophic errors. For example, I better be able to just pick up and continue where I left off if I spent 9 of 10 days waiting for my upload to finish before my network connection dropped or my server exploded.

But even the “big guys” are dipping toes in the SaaS pool lately. As this article points out, SAS is investing in the cloud big time.

Vertica seems to have tacked from an “appliance” model to a hosted or cloud-based one (Vertica for the Cloud) as evidenced in their latest webinar as well, The Cloud and the Future of DBMSs in which they pretty much repeat their usual marketing litany.

Kognitio, who claims to have put the “D” in DaaS, just announced a cloud deal with Kelkoo.com, "Europe’s largest e-commerce website after Amazon and eBay”. They also have a cloud-based DaaS implementation with British Telecom (BT), who is about to lay off 10,000 people incidentally. Kognitio has been one of the most cloud-aggressive companies out there.

And of course behemoth Microsoft is breathing down everyone’s neck (discretely at the moment) via things like www.sqlserverdatamining.com/cloud and the entire Madison and Azure platforms.

Pretty much all the big players have some sort of stake in the “cloud” one way or another. No one wants to be left out, just in case. But even beyond these well-known players you also have folks like these here, some of which address specific analytical niches:

www.deciphertech.com – sales analytics with Salesforce.com

www.hostanalytics.com – financial analytics (budgeting/revenue planning) niche. I initially thought these guys might be connected to www.i-lluminate.com by the nature of the audio on their website.

www.adaptiveplanning.com - Budgeting, forecasting and reporting analytics.

www.quantivo.com – Customer behavior analytics.

www.1010data.com - I believe they use tenbase on the backend, are columnar in architecture and have an ODBC as well as an Excel plug-in connector.

www.shsna.com/ Nutricia North America (owned by Danon) makes baby food and now also runs Pentaho over MySQL in the EC2 cloud for its internal BI needs as described here.

www.kpiweb.com – Some French startup focusing on (I’m willing to bet) KPI metrics.

www.limtree.com – Another French startup. In fact, just a QlikView integrator.

Speaking of French sites, if you can read French then check out this BI blog at www.legrandbi.com – If you can’t read French, head over to translate.google.com and read it anyway because it’s a precious resource full of interesting “in-your-face” BI insight and informational tidbits I have not found elsewhere.

www.pivotlink.com which I mentioned in a previous post, is geared exclusively at large enterprise warehouses in the cloud (small players need not apply) and backed by Trident Capital from what I gather.

Even data integration seems to have made some inroads in the cloud realm. I often read that ETL and integration consume 70-80% of a typical BI project. I don’t know if the actual proportion is this huge, but I _do_ know from experience that integration tends to get grossly underestimated. That being said, it’s clearly a huge BI pain point and I was initially surprised to see anyone trying to do this on a hosted/cloud basis but the guys at www.boomi.com are pitching just that.

And speaking of boomi, I have to hand it to them for trying this approach which I believe has merit but they have about the worse webinar I have _ever_ attended. I think they’ve managed to do absolutely everything that a company should avoid doing in a webinar namely:

Advertize a webinar lasting more than 60 minutes. Right off the bat, that doesn’t give me a warm feeling. How could they possibly need that much time when everyone else is managing to stay at or under an hour?
Make it an expensive pain in the ass for people to connect. There’s a toll-full number to call in the US. Fine. Then there’s also a 9 digit access code. Then there’s an audio pin, then another 9-digit webinar code. What the heck?
Display what seems like an interactive question-answer widget during the session but in fact have no one managing it on the other end. Hate when that happens.
Hire a consultant presenter who is obviously quite astute technically but sporting a monotonically depressive voice.
Take 35 minutes to explain how to take an FTP input, transform some rows, and output it to a folder, all on a local machine. Fascinating, but I think most people could have groked this rocket science in say, five minutes. No wonder they need 90 minutes to get through all this!

The point being, I couldn’t stand more than 35 minutes of this treatment and decided to bail out and try it out later on my own. Basically these guys have an agent-based architecture that allows you to connect to their “AtomSphere” (get it? Like atmosphere. Yeah) and have agents manipulate your data via connectors and transformations. You can setup and “send” agents onto other boxes and platforms. It did sound interesting technically, and I tried to pull their demo from the site to no avail. I got a message saying they welcomed me. Great. Where’s the bits at? I emailed back to support. I was assigned a “request for assistance” case number. Wow. Finally I got another message saying my account was active and I could login from the main page (which contradicts the initial instructions claiming you’ll get some email with a link in it) – Oh, and as for the suggestion that since I was new to Boomi, I should register for one of their training webinars, thanks but no thanks. I will definitely try it out though. It’s too compelling to pass up based on a few minor glitches from weak marketing or customer support departments.

Moving right along, I did want to mention www.lyzasoft.com, even though their offering is not “on-demand” per say, it kind of is in a “local” way. Basically you download their java thick client application. You then plop a bunch of connectors onto a workbook, bring in some data, and start graphing or analyzing it within minutes. All point-and-click, drag and drop. Yes I know this sounds like a “so-what” scenario but you don’t understand: I actually had _fun_ using this thing, yet it’s far from a toy! I wasn’t planning on spending more than 30 minutes with the product initially but ended up messing around with it for a couple hours. You can pull in anything from ODBC to flat-file connections, then you graphically describe relationships among tables (ie: joins) then you can merge that output with other data sources into a graph or statistical “component” where you then drag measures and attributes into corresponding axis “boxes” (much like the Excel pivot table designer interface). Amazingly enough (to me) this stuff just worked. It’s really cool how you can try stuff out and then back out or delete, then start from scratch or add/delete relationships and data at will. It’s very intuitive. There’s some performance and UI quirks (not surprising running Java UI code on Windows) and I doubt you can bring in significant (read terabytes) amounts of data at this stage. Yet Lyzasoft claims 175-200 maximum number of data input sources with largest customer databases at a “few million rows” of around 250-300 columns (I think that’s probably around 30-50GB of data roughly?). But overall it’s an impressive beginning and, quite honestly, probably easy enough to adapt to a hosted model. Add to that excellent and efficient real-time customer support, and you have a winner worth looking at here that could, in my opinion, pose a serious challenge to someone like QlikTek, with the right analytical engine behind them.

And up through the ground came a bubblin' crude

2009-04-07T14:05:00.000-07:00

Anyone not convinced that “doing BI” is both very hard and very expensive need only consult this link. Yes it’s from 2006, but have things really changed since then? Probably the opposite. In his postings, SSAS guru Dave Rodabaugh describes the hiring process for BI/DW architects in a really compelling five-part story. He starts off by stating “I admit to being a real hammer when conducting an interview.” – Quite honestly, I don’t blame him given these folks are charging anywhere north of $250/hr these days. At that price, you _better_ damn well know your stuff!

Here are a couple other posts on the trials and tribulations of hiring good DBAs: http://tinyurl.com/ct75os and http://tinyurl.com/dksywb

And finally, here’s what I thought was an interesting set of SQL Server interview questions (understanding them alone is challenging).

What’s the point of all this? DW/BI is indeed rocket science. Its High Priests are DBA and DW Architects. And with time, they become irreplaceable in the enterprise. If a programmer, a software architect, or even an enterprise architect makes a mistake, you can usually catch it in time and if not, there is usually an opportunity to fix the problem downstream – it’s like losing one of several engines in flight. It’s not fun, it can get bumpy, but chances are no one is going to die. On the other hand, a database BI/DW management mistake can be fatal because data can get whacked, irreversibly. Or it can get stolen, or corrupted for example, with dramatic legal repercussions for the business. This is like an explosion in flight. It’s not something you can fix in time, and survival is not likely. That’s why competent DBA and DW architects rule the BI/DW world. If you want to see what these guys really do for a living and why they commend the big bucks, check out the following for starters:

http://dbscience.blogspot.com/

http://prodlife.wordpress.com/

http://www.fulltablescan.com/

http://jesseorosz.spaces.live.com/

http://optimaldba.blogspot.com/

http://www.petefinnigan.com/weblog/entries/

http://www.sqlservercentral.com/

So I have two questions about the existing state of BI. First, why did it get so complicated and second, is it good or bad for business?

I think the first question is simple enough. Things are like this because the darn products are simply too hard to use. You shouldn’t need a PhD to properly setup, configure, deploy, tune and question a database engine. Forgive me for liking simplicity, but last I checked, operations manuals for behemoths like DB2, Oracle and SQL Server were in the thousands of pages combined. To me, when a set of product manuals have more pages than the Federal Budget, that’s cause for concern. I’m an avid follower of the KISS principle.

Recently I purchased this book which is essentially a Microsoft SQL Server Analysis “bible” covering every part of that ecosystem including SSAS, SSIS, SSRS and Excel. It runs 624 pages. And quite honestly, unless you have significant mastery of each subject matter, you’re not going to be very effective doing BI at the enterprise level (at least on the Microsoft stack, but the others are typically hairier anyway, let’s face it). In my opinion, this book represents the _minimum_ one needs to know to be effective in this business. And believe me that’s not for the casual weekend DBA.

The second question is a bit more subtle. How do you define “bad for business”? For the past forty years, business leaders (CIOs) and bean counters (CFOs) have computed the total costs of owning and exploiting these databases (salaries, hardware, software, licenses, power, time wasted, failure rates etc.) and determined that the rewards justified the costs. But do they really? In light of the well-documented failure rates among business intelligence projects in the past years, can they possibly be right? It isn’t too challenging to pull up a myriad of articles such as these decrying and documenting the dismal success rates of BI project all over the planet: http://tinyurl.com/cr8lt8 or http://tinyurl.com/cnc36h

And in some cases, you also need a PhD to even figure out _how_ to calculate ROI on a BI endeavor as this 2002 44-slide PowerPoint presentation from Jonathan Wu will attest to.

So from the looks of it, countless millions of dollars and hours of intellectual effort have been spent in the past decades achieving mostly failure. The same argument could be made (and has often times) about software development in general, but it looks like BI is faring even worse! When’s the last time you heard about a successful enterprise BI project that wasn’t over budget, over schedule or over-complicated? We know what happens when complexity is used to hide questionable assumptions, processes and results. Ask anyone at AIG. If it’s too complex, too obfuscated, and too inaccessible, then chances are it’s bad business.

And so it is in light of this that I have recently started to think differently about this “cloud” offering in the BI world sometimes referred to as BaaS, BIaaS or even DaaS. Having been in software technology for twenty years, I’ve seen my share “new improved”, world-changing, and “revolutionary” concepts come and go. But people are missing the point about this one as it applies to BI.

The BI “cloud” isn’t about technology, delivery, security or governance hurdles per say. It’s about abstraction. And abstraction is the antidote to complexity. This is why I feel many players in today’s BI world are threatened by this concept, and why there’s so much gold in them thar hills when applied to enterprise business intelligence in judicious ways.

Mind your own business [intelligence] Part II

2009-04-01T09:10:00.000-07:00

First, I want to thank Curt Monash (www.dbms2.com) for mentioning my blog in his postings this morning -- that's really nice of him. He is also reviewing several BI players at the moment. God knows there's a lot of them. Go read his reviews.

Since my last post about that market, I emailed with Gooddata's CEO and Founder Roman Stanek who was kind enough to put up with my questioning back & forth emails. He suggested several other names I might be interested in exploring so I set out to do just that. In the process, I discovered Indicee who totally blew me away. They have a clean, fast, operational UX. Not only was I able to upload data immediately, but they were nice enough to kick up my account size so I could push up more data. Next, I have to say their customer support response time is just awesome. For me, Mr. instant gratification, this says a lot about a company. And finally, believe it or not, you can ask questions of the data in English! This has been one of my "dreams" for a long time and I had actually envioned the same kind of interface Outlook has to setup mail rules, but theirs is even cooler. As you probe the data, they show you what the question is like _IN ENGLISH_ -- Nirvana. Their stuff just works. These Canadians can write software I tell you!

Not so impressive was PivotLink. Their website clearly displays a link marked "Start Free Trial". Stupid me, I thought I might be able to get a free trial. No such luck. Instead, I was contacted by a Sales Development person who offered to schedule a "demo/webinar" for me and "go over my requirements" and determine if their "solution is a fit for the company". Are they kidding? Why don't you let ME the customer decide for myself? Thanks but no thanks. Then, they offered to "send me over" to their Chief Marketing Office. Needless to say, by that time I had already scratched off their name from my list of potentials. Disingenous and condescending.

Next in line was Lucidera. I contacted the company to see if a trial version was available. I'm not even going to write one word about this little experience. Instead, I'm going to post the email response I received -- you decide:

We actually have somewhat of a different "trial" experience than most
organizations. Since we are an on-demand model and analytics tends to be
pretty extensive, we have what is called a Pipeline Healthcheck. What we
do with the Pipeline Healthcheck after having a call with you to go over
your analytic requirements is we access your Salesforce.com data and
bring that into our application via a read-only log in and we analyze
your sales data, focusing on your sales people, sales processes and your
pipeline while highlighting areas of opportunity or potential risks that
we find based on our best practice analytics. Customers and prospects
are typically finding hundreds of thousands to millions of dollars in
revenue opportunity. The Pipeline Healthcheck also serves as a business
case with quantifiable value based on your sales data if you think the
application makes sense for your business. If you'd like to set up a
call to discuss further and your organization is a Salesforce.com
customer, please let me know of a few good times and a number to reach
you and I'd be happy to go over this further as well as discuss your
interest in sales analytics. Thank you!

Huh?!? I don't know about you, but the last thing I need these days is to be "pipeline checked". It just doesn't sound appropriate on a first date.

2 = 2

2009-03-31T15:13:00.000-07:00

Trawling through various “sales engineering” blogs recently, I happened to come across this one http://tinyurl.com/dz8nma by Xavier Petit who is apparently a fellow “paisan” from France. Xavier seems like a sharp guy with an interesting manipulation of written English but nevertheless full of excellent insight, in my opinion. Not the least of which is his reference to this other site here http://www.morenewmath.com. Part of my job sometimes involves trying to explain or demo esoteric mathematical concepts to people who want to “drill-down” into our product’s technology (rare, but it happens) so you can imagine I was eager to learn more about this “new math” approach to performing successful demos. Little did I realize initially that these were just mathematically-expressed (terse) philosophical principles (equations about life) – some of which are really pertinent to what XSPRADA is about. For instance:

I wish I had a dollar (ok, in this economy, make that a Jackson) for every time I’d heard someone say “you guys can’t do this, it’s impossible”. Now we have working software to prove them wrong.

A lot of folks initially said we were crazy for even attempting to disrupt the relational “status-quo”. Five years later, here we are, and in good company I might add.

The concept of uniqueness is very hard to define mathematically. Our ability to do just that is one pillar of our math-based technology.

Why disappoint people (and the market) by making overhyped promises and under-delivering. We went the other route.

When perception matches reality over time, you have a successful product (think Rolex or BMW). People often ask me “what’s your value proposition”? I say “You tell me, here’s the software, here’s what it does, but don’t take my word, try it out.” They’ve all heard the “better, faster, cheaper” pitch before, so why waste their time when it’s better spent actually using the software?

And my personal favorite, which has nothing to do with software, databases, or XSPRADA but fulfills me as a wine lover. A votre santé! J

Mind your own business (intelligence)

2009-03-29T16:44:00.000-07:00

As I mentioned previously, I’m not a BI power user. Until twelve months ago, I couldn’t have told you the difference between HOLAP, ROLAP or MOLAP, much less explained what a slowly-changing dimension was, or what the difference was between star and snowflake schemas.

Nevertheless, I do have a technical background and I’ve learned a boatload about BI in the past year. In the past months, I’ve been trying to put myself in the mid-market LOB user’s shoe by using all and any available tool I could find to “do BI” using our ODBC-talking analytical database as the backend (I wanted to do this from a non-enterprise perspective).

I tried to get my hands on every product I could get from front-end analysis packages (either thick client or web-based) to ETL tools, both proprietary or open source. Those are part of what I call the BI “ecosystem”. My testing had several goals:

How easy is the tool to install, setup and use without direction, manuals or training.
How easy is it to connect to our database engine via ODBC.
How easy is it to extract data from our engine and do some very basic analysis.

So I loaded the small Pentaho Mondrian sample database (FoodMart) into the XSPRADA engine and went to work.

Microsoft Excel -- www.microsoft.com/excel

With Excel 2007, I generated an ODBC connection and quickly pulled in a data subset using Microsoft Query. Then, I inserted a pivot table based on that data island in a new sheet, and did basic slice/dice analysis on the data. Total time spent: all of maybe 5 minutes. There’s a small instructional video on how to do this at http://tinyurl.com/ckfwp6. Not rocket science.

QlikView -- www.qliktech.com

You can download evaluation copies of QlikView from the website. When I first ran it, I thought, hey, this is MDX for dummies with an Excel twist (workbook paradigm). QlickTech actually bills itself as “Excel on steroids”. You can get results with QlikView fairly quickly out of the box. Yes it takes a bit of training and practice to do anything really significant, as the product is very rich (and can do some serious ETL in the process) but overall it’s very visual and intuitive. And if you run into any issues, their sales engineers are readily available and quite effective.

Tableau -- www.tableausoftware.com

I pulled Tableau evaluation bits from their website and also attended one of their webinars. Unfortunately, Tableau doesn’t talk to ODBC. I’m not sure why. That pretty much ended my evaluation right there.

AltoSoft -- www.altosoft.com

I tried Insight Studio but I couldn’t extract table schemas from our system via ODBC. Quite honestly, I think it’s because our driver probably doesn’t support some level 1 API the tool is using. So then I tried pulling from my FoodMart database inside SQL Server 2008. Even then, I never “got” the tool. It requires some training and/or patience I simply don’t have. It’s too complicated right off the bat. I see I can create some PKIs, but then the UI isn’t really obvious or smooth. It just doesn’t “flow” and there’s some web-based dashboard stuff too which itself isn’t exactly self-explanatory and pops up tons of little Cassini web server on my system.

Panorama -- www.panoramasoftware.com

For obvious reasons, these folks only talk to Microsoft SQL Server and SAP back ends. Sharp UI and good usability, but, in my case, of little use.

Talend -- www.talend.com

Talend is a French open source (OSS) company. (Incidentally, in this economy, they managed to recently score a $12M round -- not too shabby). I tried their Open Studio tool and was able to do some very basic ETL to generate and convert CSV files to XSPRADA CVS specifications. This is important because at the moment, this is the only format you can present to our engine. So I wanted to see how easily someone could extract data from say Oracle or SQL and convert it to our specs. The answer is: easily. That tool has a lot of knobs and switches and deep functionality but I was able to do simple stuff fairly quickly without diving too deep into Java code. Given my OSS biases, I was impressed.

Pentaho -- www.pentaho.com

Pentaho distributes an open source OLAP platform called Mondrian that talks SQL via MDX to numerous underlying databases, including ours. I’ve been using Mondrian on Windows with the built-in JDBC-ODBC bridge to our database for months now without a problem. With Mondrian’s web-based UI you can slice and dice FoodMart data any which way you like. Where it gets a little hairy is actually setting up the cubing structures for your own data. That does take expertise and is not for the casual LOB user (think: lots of trial and error) . But with enough effort and community support, numerous people and organizations have apparently done just that.

Birst -- www.birst.com

I was very excited to hear about this new company because it seemed they had a really new approach to hosted on-demand BI. They have a subscription-based service. So I quickly created an account and tried to upload some simple data (SQL’s from my ODBC connection) and see immediate results. Unfortunately that was about a week ago and since then, I still haven’t been able to either load or analyze data on their site due to various technical glitches. This is really unfortunate because Birst’s claim to fame is “ease of use” but if they can work the quirks out (and find some really good UX people) I think there’s huge potential in their “hands-off” approach which is basically what I was looking for, namely “here’s some data, here’s a question, show me something quickly without my having to become a cube expert”.

Gooddata -- www.gooddata.com

This company is another “on-demand” hosted BI offering. The application is in beta mode and it shows. Nevertheless, I was able to upload data and run slice/dice charts in a very intuitive manner within minutes. One of the major issues I see is performance. Even with small data sets, upload and processing operations are very slow (I believe they run off of Amazon cloud services). I am hopeful the UI/performance/usability quirks will be resolved in a production version because this is the only on-demand player I’ve seen that seems to “get it” so far. Ironically enough, they offer the Mondrian sample data set as a “default” project to play with when you create an account. Felt right at home.

I draw the following conclusions/recommendations from this little experiment:

· Doing BI is not easy (duh). Although some of these tools are more intuitive than others, we’re still a long way from “easy” BI. Even with Excel you still have to know something about pivot tables, cubes (if you’re connecting to SSAS) and connections. With Mondrian, it really helps to know MDX. In that context, the on-demand solutions are simpler. And of course with hosted approaches, you don’t need a license key for Office.

LOB users can’t use OSS tools without “expert” help. There is no way a typical business person can suffer the slings and arrows of Mondrian or Talend quirks and “gotchas” on his or her own.

A lot of vendors are really very confused about what constitutes good UX. In some cases, I truly wonder if these folks ever run usability studies. Please take the app home to a spouse or parent and have them use it. I swear that’s helpful. My mom is an Excel wizard yet can barely plug in a USB wireless mouse.

Unless you have a tool that is easier to setup and use than Excel, don’t bother! I mean what’s the point? Excel has massive market penetration. Nobody is going to care about a new application unless it can beat Excel hands down in usability and performance. Also, you can share Excel spreadsheets and data in the “cloud” nowadays using various providers (like google) so again, unless you have some compelling collaboration features to beat that, think again.

On the enterprise side, people are pegged into lock-in situation driven by licensing discounts and vendor pressure. If you’re an Oracle, IBM or Microsoft shop, chances are you will be using a certain set of BI tools no matter what. And I don’t think a Microsoft shop will be running Cognos, or an IBM shop SSAS. Companie like QlikTech have a really tough time penetrating the large enterprise I believe.

But everywhere else, new approaches are not only available, but also politically possible. So I think the commoditization of BI will have to come from the bottom up. In that respect, the foundation currently being laid by on-demand and hosted BI services is much more important to the future of BI than meets the eye.

There are only two types of people I cannot stand: people who are intolerant of other peoples' cultures, and the dutch.

2009-03-12T16:09:00.000-07:00

In my quest to mine the business intelligence market to the last drop, when I’m not busy doing other things like talking to people, setting up POCs, coding, looking at product features or evaluating what I call “ecosystem” products like BI and ETL platforms, I spend a heck of a lot of time online reading blogs and scanning websites.

This market is very dynamic and it’s important to keep up and spot trends and pain points early on. This isn’t unlike an NSA type of endeavor really. There are gazillions of “signals” out there and it’s up to me to pick up the pieces and make sense out of them as best I can.

When I do this successfully, I can not only enhance our competitive edge, but also discuss intelligently how we fit into the business intelligence “big picture”. In this business, someone who is not constantly educating himself is in a world of hurt.

To do this, I scan several dozen blogs and websites an almost daily basis (see my partial list following this posting). This activity, combined with talking to folks in the BI world, product specialists, consultants, being involved in coding and testing a full-fledged database product, and attending webinars for the past year has given me a certain outlook on the market that carries “no baggage”. Let me explain.

I don’t come at this with the perspective of someone who has been immersed exclusively in the BI world for decades, or someone who has the hands-on background designing and implementing data warehouses his entire career. I’m just a software engineer who happens to have worked for a multitude of different businesses and in numerous domains, not just BI. So I feel this gives me a more “detached” view of the DW/BI market than an industry expert with BI-specific experience might have. That being said, the following are nothing more than observation-based personal opinions about the warehousing and BI market developed over the past 12 months. I’d love to hear your thoughts as well – at the risk of being branded a mere neophyte.

And then there was light…
Acceptance of “non-standard” (read: not from Microsoft, IBM or Oracle) approaches to analytical databases (including non-relational approaches) is going mainstream in the warehousing realm. Four to six years ago, if you talked to anyone in the enterprise about trying a “non-relational” database, they’d flip the bozo bit on you instantly. Nowadays, although relational “bigots” still (and always will) exist, BI people are more open-minded, mostly thanks to dozens of “new-breed” players in the market with well-documented successful implementations (not the least of which are Netezza, Teradata, Vertica and SybaseIQ). To use a phrase I detest, people are finally “thinking outside the Big Three box”.

Gimme a little OLTP with that OLAP would you?
The distinction between operational (transactional) and warehousing (analytical) business activity is blurring. Operational and analytical business efforts are often integrated. Warehousing and analytics is no longer “the crazy uncle in the attic” project. Another sign of this is the recent desire to enable ever more frequent insert/updates to warehouse stores than in the past. Not content with infrequent batch updates, people are now looking at efficiently pushing update/inserts real time into their warehouse. The updates often come from both an ODS and external data sources. Unfortunately most analytical databases are designed around the assumption that warehousing is mostly read-only (InfoBright ICE doesn’t even support DML) and optimize as such. I think they’re up against hard times unless they can “transactionalize” quickly.

Open source: cheaper than free?
Open source is having a large impact in analytics for both economic and practical reasons. Nowadays, businesses can set up data marts using engines like InfoBright (MySQL), for example, and use BI/Integration tools like Talend’s Open Studio or Pentaho’s Mondrian. Clearly a lot of this is driven by current economics, but open source deployment and licensing models are competing head-on with proprietary solutions on many levels.

Hey buddy, want some good BI?
Everyone’s talking about BI. Microsoft is running TV ad spots about it during prime-time and pushing the concept in a very public manner. After buying up Datallegro last year, they announced the Madison project, put out best-practice configurations for Dell and HP “appliances” hosting SQL Server, (http://www.intelligententerprise.com/channels/information_management/showArticle.jhtml;jsessionid=GSPUKPDEIPLJCQSNDLPSKHSCJUNN2JVN?articleID=214502509) and have been discretely adding warehouse-oriented features to SQL since 2007 (http://msdn.microsoft.com/en-us/library/cc278097.aspx). What’s more, they own the BI desktop with Excel and Office 14 promises a significant OLAP functional push. Vaporware? Maybe, but I think they’re gunning to eat a lot of BI folks’ lunches out there in the coming months. If anyone can commoditize DW and BI technology, it’s Microsoft.

Performance, shperformance
“Performance” is a big mystery. No one really knows how to define it in the DW/BI space. Is it load time? Is it query response time? Is it data presentation to result time? Does it include backup time? How about data recovery? Opinions differ. Service level expectations are often misguided or unrealistic. People don’t seem to care much about TPC-H or TPC-DS performance metrics.

The proof is in the concept
POC POC POC! People want to see numbers on their data on their hardware up close and personal. That’s the way it should be. There’s too much hype in the industry, and people tune out the bullshit http://www.biblogs.com/2009/03/12/rejecting-stale-tech-marketing-words/ -- Who can blame them. These folks have been abused and lied to for decades now. As I’ve been on the other side of the fence many times, I can relate. That’s why I always keep it real simple when talking about our product. Everyone’s heard the “better, faster, simpler” shpiel a million times, so why insult their intelligence. Give them a set of keys and let them test-drive the darn product! POCs should be run as described at http://www.altosoftcommunity.com/ and http://www.altosoftcommunity.com/?p=61 and http://www.dbms2.com/2009/03/02/ideas-for-bi-pocs/. There isn’t a single solution in this market that applies perfectly to every customer across the board and any vendor claiming otherwise is either naïve or disingenuous.

Cloud cool-aid
“Cloud computing” madness has taken hold in the DW/BI industry as well. Vertica and Kognitio are big pushers of warehouse cloud hosting solutions. The buzzword now is DaaS (data as a service). I find it hard to get excited about this recent trend. I don’t mean to Andy Rooney the whole concept, but in enterprise warehousing and BI, given the amounts of data involved and the security, governance, availability, and SLA issues, I just don’t “grok” it. Maybe for small segments of “cold” data? I don’t know. I see the craze, I feel the buzz, I get the marketing upside, and I notice the traction but…I am not of the body. The power of clouds doesn’t compel me.

Here’s my list of DW/BI industry blogs:
http://www.altosoftcommunity.com/?feed=rss2
http://www.b-eye-network.com/rss/content.php
http://www.biblogs.com/
http://bibrain.blogspot.com/
http://www.byteandswitch.com/
http://chucksblog.emc.com/chucks_blog/
http://customerexperiencematrix.blogspot.com/
http://www.dbms2.com/
http://prodlife.wordpress.com/
http://www.informationweek.com/
http://www.intelligententerprise.com/movabletype/blog/msmith.html/blog/
http://www.intelligententerprise.com/movabletype/blog/skemsley.html/blog/
http://www.intelligententerprise.com/movabletype/blog/dhensche.html/blog/
http://www.intelligententerprise.com/movabletype/blog/nraden.html/blog/
http://kevinclosson.wordpress.com/
http://www.petefinnigan.com/weblog/entries
http://www.sqlservercentral.com/
http://storagezilla.typepad.com/storagezilla/
http://www.databasecolumn.com/
http://thestorageanarchist.typepad.com/weblog/
http://andyonsoftware.com/
http://www.b-eye-network.com/blogs/imhoff/
http://www.b-eye-network.com/blogs/business_integration/
http://bisqlserver.blogspot.com/
http://biforbusinesspeople.blogspot.com/
http://kognitio.blogspot.com/
http://dbscience.blogspot.com/
http://www.fulltablescan.com/
http://optimaldba.blogspot.com/
http://optimaldba.blogspot.com/
http://www.dwoptimize.com/
http://oraclestorageguy.typepad.com/oraclestorageguy/
http://qlikviewmaven.blogspot.com/
http://ianalyze.net/
http://www.gillespol.nl/
http://talkdig.blogspot.com/
http://biguru.wordpress.com/
http://360degreeinsight.com/

"Out of nothing I have created a strange new universe". [Janos Bolyai]

2009-03-10T13:20:00.000-07:00

Right off the bat, let me refer you to an interesting discussion of extended set theory on the web at http://www.xprogramming.com/xpmag/xstSomeThoughts.htm written by by Ron Jeffries. It's not a bad conceptual introduction but I'm going to try to do it one better in simplicity below.

In our last post we discussed the notion of “distinguishability” among classical set theory members, and how XSPRADA extended set theory allowed one to clearly differentiate between and uniquely identify extended set members. These extended set members become values in extended set “couplets” and an XSPRADA “couplet” is an XSPRADA “ordered pair”. Each member is defined by a scope and a constituent piece, either of which is interchangeable at will, but neither of which can exist without the other. Now, in cases where order and identity are irrelevant, classical set theory can be used to manipulate these members. But in cases where we do need to identify, discern or order set members, we apply extended set theory accordingly as such:

Suppose you have a set of elements {A, B, C}. We want to represent both mathematically and physically (in a computer) so we can do calculations on it using the mathematics to answer questions, and then bring answers back to the computer. For example, we may want to count members in a set (cardinality), or calculate the intersection of two sets. So typically you store this information on the computer in one way or another. Then you “read” back the physical data (via disk I/O), map it to your mathematical ontology (in the software), do your calculations (count or intersect operation), then bring results back to the “real world” namely as a SQL result. The faster you can do this, the better off everyone is.

In a computer, you can’t simply store {A,B,C} because all possible permutations of the triplet {A,B,C} can be represented in many different ways (well, six actually, or factorial of 3) in memory (or storage). So which one do you pick? Clearly you can’t store every possible combination efficiently. Even if you wanted to, you’d still have to somehow indicate which combination you meant in a given context. So engineers came up with a hack called “indexing”. But now you also have to store the index values along with the data, even though the index itself is not native to the information. Back on the math side, when you load this record back into the model, there is still no inherent notion of order. To the model, it makes no difference if you load back A B C or C B A or B A C because they are all equivalent. So you have to load the index as well. Then you need to do some math “magic” to tie the two, and promulgate the magic all the way back to a result, which you must then move back to the physical world as well. It’s the quintessential anvil to the ankle problem.

Extended set theory models physical storage (RAM or disk) using newly defined set members and so right off the bat we would express our initial set as {1.A, 2.B, 3.C}. This cannot be confused with an entirely different set known as {1.B, 2.C, 3.A}. When “materializing” this abstraction to disk or RAM, the representation is clearly as follows:

ADDRESS VALUE
0x002200 A
0x002201 B
0x002202 C

In other words, we have now modeled computer addressing using extended set scopes. This is one of the many possible applications of extended set theory for information management.

Now take a different arrangement, for example:

ADDRESS VALUE
0x002200 C
0x002201 A
0x002202 B

When you move this back to the extended math model (in the software), you end up with {1.C, 2.A, 3.B}. No need for slight-of-hand indexing or other “unnatural” steps. There is zero ambiguity.

Now suppose you have 2 different sets and wish to determine if they are “equivalent” as follows:

ADDRESS VALUE
0x002200 A
0x002201 B
0x002202 C

ADDRESS VALUE
0x002200 C
0x002201 A
0x002202 B

Using classical set theory, you don’t have enough information (without indexing hacks) to mathematically make the call. In extended set theory, you end up comparing two sets {1.A, 2.B, 3.C} and {1.C, 2.A, 3.B}. You can see 1.A and 1.C are clearly distinct, as are 2.B and 2.A and 3.C and 3.B. Why? Because extended set theory says two members are equal only if both their scopes and constituents are equal! Consequently, you can say with certainty “these two sets are different”.

Let’s apply this to finding intersection between these two sets. In classical set theory intersecting sets {A, B, C} and {C, B, A} yields {A, B, C}. Intuitively, both sets seem to contain the same elements. But do they really? It depends. In extended set theory, the question is: do {1.A, 2.B, 3.C} and {1.C, 2.B, 3.A} have anything in common? And the answer is a resounding no as none of the elements has matching scope and constituents. In this example, if order matters, classical set theory cannot help us.

If you’ve been following me so far and thinking, this is really stupidly simple, you’re catching on. Simple in concept? Yes. As Einstein once said, “Make everything as simple as possible, but not simpler”. Simple in implementation? That’s another story. But the fact is, the implications for information management are mind-boggling. Why? Six reasons:

First, since the mathematics formally models order and relationships, and can manipulate these internally via a formal algebra, humans don’t need to pre-index or model information before exploiting it. The software extracts that information from the data directly. More interestingly, it does so by examining the bits and bytes right off storage with no loading or pre-canned processing logic needed.

Second, every piece of information presented to the system is internalized in the math engine. This means the original format of the information (be it relational or unstructured, CSV, XML, or video) becomes moot once inside the math engine. Once modeled, any information (bits and bytes) becomes sets, which can be manipulated and operated on consistently.

Third, because the mathematical model being maintained is complete and closed, the integrity of the “universe” is maintained at all times. This means information is immutable. It cannot be changed or deleted. This feature is called “temporal invariance”. Consequently, you can actually query the database as if you were asking questions weeks or months ago, not unlike a time machine.

Fourth, every query (question) fed to the system is transformed into its mathematically equivalent set of algebraic expressions. As such it can be optimized mathematically. Expressions and relations are also cached and exploited in real time. This lets the system discover existing relationships that a human user could not have foreseen (and much less asked about). It also lets the system adapt to “poor SQL” or poorly formulated questions without impacting results or performance.

Fifth, the math engine partitions queries. As a matter of fact, it can be shown mathematically that the number of possible query “types” in a given business intelligence domain is finite. No matter how complex or convoluted a user or machine-generated query is, the engine can determine without ambiguity which partition “bucket” it belongs to. Because this taxonomy is well-defined and finite, spectacular optimization techniques can be applied.

Sixth, it can be mathematically demonstrated that a finite number of operations suffice to handle any possible workload presented to the engine. The mathematics prove that any query can be answered using a small set of well-defined operations such as union, intersect and complement, for example. Consequently, the XSPRADA math engine need only implement a fairly small number of operations, and no more. As these operations can be threaded and parallelized in the software, this makes for a very highly optimizable high-performance system.

So from a technical perspective, XSPRADA technology and its application to database software is a genuine breakthrough. But more importantly, for business intelligence purposes (reporting, OLAP, decision support, or data mining) the rigorous application of formal mathematics to the software implementation of an analytical database engine offers unequaled competitive advantage. I’ll describe those in real-world terms in a subsequent post, and I'll also dig up a couple places and instances where this technology was actually succesfully applied in the past as well!