Appliance Data warehouses: Is this where we are headed?
For the datawarehouse fraternity Appliance Datawarehouse vendors are basically the Teradatas and the Netezzas of the world who primarily load their proprietary RDBMS software on massive parallel processing (MPP) servers in order to get the necessary scale and the speed
Without going into the details of the technological superiority of an Appliance Data warehousing (which I guess is a given) I would want to discuss on whether it makes business sense in adopting this as a technology imperative for your BI/data warehousing needs. As I see it, the USPs of an Appliance Datawarehouse is in aiding your loading speed and increasing your computing speed by use of its special storage algorithms.
Now today if your DWH is on a traditional RDBMS (non-Appliance based) and are looking at moving into an Appliance based system, one of your business drivers would typically be an increase in your data volumes. While I agree that increasing data volume is something every business has to cope with but I am of the opinion that the future of BI does not lie in processing large volumes of data but processing data on a more real time basis and on propagation of the data to the right audience in the enterprise. Here I am not taking into consideration the data that you may need to store due to the regulatory compulsions, where the loading speed may not be a critical factor.
I would suggest an analysis be done on the overall BI strategy that your organization has adopted in conjunction with the above paradigm and then make a decision on adopting an Appliance Datawarehouse. Adopting an Appliance DWH would be a life changing decision for organizations of any size and would be difficult to come out of given the magnitude of investments that go behind setting up the whole framework. Hence the strategic imperatives need to be very clear. The next key factor to be considered would be the speed of computing that is requited and here what matters is how quickly would you want to carry out your analytics? How is your data based on which you carry out your analysis stored physically? In my experience I have seen speed to be always a bit on the perception of the end user and needs some degree of perception management. However here as well I would see a lot of our traditional alternatives in improving speed like partitioning, purging and also looking at Column based databases to give that speed. My approach to handle this would as well be to explore a good physical database design as well as a robust logical design to exploit the inherent speed of the underlying RDBMS and then look at going the Appliance route if it fails to meet the desired speed.
In a nutshell I believe that the BI world is not headed towards speed, scale or volumes but more towards an approach which enables the appropriate user community to have current and real time data. At the same time it could still be justified to adopt an Appliance based DWH based on the specific business imperatives in your enterprise.
Comments
As pointed out by Somnath, the major drivers for investment into an Appliance DWH for an organization are increasing data volumes, need for increased computing speed to process data on a more real time basis and most importantly the returns/benefits to the business gained from adopting Appliance DWH in comparison to optimizing the performance of traditional DWH (non-Appliance based).
Some of the industries where the performance & utilization of BI applications can be significantly improved by adopting Appliance DWH are Telecommunications, Retail and Banking industry.
In the telecommunications industry, as the number of subscribers is growing exponentially across the world, so is the number of calls. This creates an enormous amount of data making it extremely challenging for the companies to quickly and efficiently analyze the call details of their customers. In fact, traditional approaches have been inefficient in processing queries on even one month's data, seriously hampering an organization's ability to perform trend analysis to reduce customer churn, better tackle revenue assurance issues and generate timely reports. Appliance DWH approach helps in yielding appropriate information efficiently for customer satisfaction & network management.
In Retail industry, traditional organized retailers as well as on-line retailers capture huge amounts of data relating to customer transactions, operational and supply chain information, which hampers efficient processing to provide an optimized BI solution. Appliance DWH could enable retailers to manage and analyze these terabytes of information in near-real time.
In the Banking sector, the growth in the retail operations world-wide has lead to newer channels of banking like internet, call-center, mobile telephony etc. apart from the branch operations. Banks are today capturing imposing amounts of data relating to customer transactions creating a data explosion that threatens to overwhelm an average banking organization and its current IT infrastructure. Appliance DWH could help in enabling banks to manage and analyze the transactional data more efficiently, for example empowering them to offer customized products & services to the customers based on their lifetime value (based on historical transactions analysis).
Posted by: Sumit Sharma | February 23, 2009 6:26 AM
Good article stressing the strategy needed for an enterprise.
The growth in the DWH appliance adoption points to the increasing business scenarios which are in need of large volume data processing. One specific scenario in Telecom I have come across is: there are few countries in which the Government mandates the telecom service providers to advise the best tariff plan for their customers based on their usage. This requires the analysis of the call details of the customer against the multiple pricing plans available with the telecom service provider.
Posted by: Sankara Narayanan | April 21, 2009 11:56 AM