The Next Evolutionary Step - Analytical Connectivity
By KLaker on Jul 12, 2010
Within data warehousing there is a growing trend, created and heavily promoted by certain vendors, to pull data out of the enterprise data warehouse platform and spread that data across specialized platforms according to how it needs to be analyzed. This process is seen as the next evolutionary step in data warehousing
Many of these vendors imply that business users are becoming more demanding in their analysis. For example, sales teams need access to data mining tools to help identify customers who may be considering switching to another brand/vendor. Planning teams need access to spatial information to help them understand where to locate new stores. Even IT teams, in conjunction with marketing teams, need access to advanced analytics to help them to design and evaluate designs for websites to determine which pages generate the most traffic and the most click-throughs. Financial users need to be able to analyze changes to the organization of teams within the business, and possibly the structure of the business itself. Each of these tasks or projects requires, it is claimed, some form of specialized functionality which goes beyond the "normal boundaries" of the enterprise data warehouse. More importantly, the analytic appliance vendors will convince the business users within your company that the data warehouse is already over-loaded with the everyday business requests from users and applications and that adding a more analytically sophisticated workload will only increase the performance problems. These vendors position their analytical appliances as the next evolutionary step in the life of the data warehouse.
The general perception of today's data warehouse is that business users see their needs escalating while IT struggles to keep the data flowing. Consequently, the business users struggle to run the types of analysis they need to make informed decisions about targeted marketing campaigns, up-sell opportunities, market catchment areas etc etc. Business users, we are told, are looking for a solution to this problem and we all know the answer is not Excel. Fortunately, every week another analytic appliance vendor pops up offering a new type of solution that is "carefully designed to truly revolutionize your business". The general message is that each business team needs their own specialized analytic appliance to manage their unique requirements and, as we know, business users are not shy at demanding what they think they need. By slicing up corporate requirements into departmental requirements vendors can sell lots of different appliances to the same customer, which is a very nice business model.
If you sit through a sales presentation on data warehousing today and you are not sold an analytic appliance at the end, you feel as if you have been cheated, or as if the sales team has not correctly understood your requirements. This means that many businesses believe that the only way they will be able to run analytic queries is if they have a whole range of departmental analytical appliances sitting in their data center, each delivering its own tightly focused set of answers. Many customers are now so convinced that this is the only answer that their RFPs, RFQS and even POCs are structured in such a way that the only answer can be an "analytic appliance" (more on this "loading the dice" issue in another post).
For IT the situation is similar to the old detergent commercials, where if you were not using Brand X you were letting the whole family down. The analytical appliance vendors are pushing the same message, as an IT group you are simply not looking after the best interests of your business users if your data center is not stuffed full of analytical appliances.
What everyone is missing is that at best an analytic appliance is a short term solution. In fact I would argue that these appliances will have disappeared within the next 2-3 year. The reason is very simple: the complexity of analysis is growing rapidly but not in a single direction. What I am now seeing from a lot of customers is that analysis within the context of a single analytical domain is no longer sufficient. Business users have, in a very short space of time, made the jump from needing to simply run a data mining model to needing to run a data mining model, link that result set to results from an OLAP cube, overlay all that information on a geographical map and understand how that view of the data changes over time, or changes as a result of a change to the basic business model. This evolution from needing a specialized appliance to needing analytical connectivity has happened in a very short time. Analytical connectivity is the next stage in the natural evolution of data warehouse platforms I would argue that the "analytic appliance" should now be considered as dead as a dodo.
For customers who have invested heavily in analytic appliances moving to this new level of evolution is simply not possible. Once you functionally isolate your data it is very difficult to start stitching result sets back together to perform higher levels of analysis. The more isolated data sets you have the more difficult the problem becomes and the bigger the strain on your network as users revert to downloading data into Excel to try and make the connections they need.
Many customers are probably looking at the various functional analytical areas of data mining, text mining, OLAP, spatial, time series, text, images, video analysis etc in isolation and the vendors selling analytical appliance have a vested interest in maintaining this specific status quo. The problem for these vendors is the level of analytical connectivity is definitely going to increase rapidly. Some business users are probably already looking for all sorts of new ways to analyze and bring together data sets. This means the end of the road for the specialized "analytical appliance".
In my view if you are planning any new data warehouse related projects or simply refreshing your data warehouse platform, you must consider the impact that analytical connectivity will have on your system. Planning for analytical connectivity now will ensure that you do not create an environment of isolated and disconnected analytical islands. A platform that supports analytical connectivity is the only way forward.