Adopting Big Data - Challenges and Success Factors
Big Data comprises of 3-V factors; namely Volume, Velocity and Variety. However, considering well understood benefits reaped on adopting Big Data in enterprises, one could be tempted to club another -V (Value) to the existing troika. Mckinsey Global Institute Big data study says that 'The total amount of data created and replicated in 2009 was 800 exabytes -- enough to fill a stack of DVDs reaching to the moon and back' (source: Mckinsey global institute. Big Data: The next frontier for innovation, competition, and productivity. May 2011). Though Big Data adoption is well established in industry segments like Retail, Financial and Insurance, and Manufacturing etc, there is still a need to continuously innovate and implement factors that guarantee success and enable rising returns. The primary challenge in Big Data implementation is the need to handle high voluminous data (that exist in multiple storage sources and multiple data formats ) at the speed it is received and processing to generate intelligent business insights. This leads to an even more complex problem to solve - that of management aspects, where the enterprise structure and processes will need to change in response to findings from Big Data analysis to enable the enterprise to evolve and reap business benefits.
Some of the technical challenges to Big Data adoption include the following :
- Choosing the right technology fit for processing and analyzing Big Data and providing in time analytics and intelligence.
- Coming up with extensible and scalable architecture to support current and future needs economically
- Focusing on operational aspects
- Focusing on ease of use
Similarly challenges from the management aspects include:
- Consolidation of data at organizational level rather than limiting it to internal business unit level
- Defining and categorizing short term goals and long term goals to aid in business decisions
- Identifying and involving right skilled data scientists who can look deeper into data and provide business intelligence and provide valuable recommendations.
The above challenges are generic and are applicable in almost all industry segments though not at the same level. In spite of these challenges, leveraging Big Data for better business insights become a key basis for any firms to cope up with competition and growth. While those challenges have to be addressed by adopting right strategies and involving right stake holders and expertise etc, it is important to understand critical success factors that realize the real potential of Big Data. These success factors vary depending upon industry segment, business objectives ,enterprise structure etc and hence have to be addressed accordingly.
In the manufacturing context, data is available in various formats and across various business units. Consider data in terms of a Product Life Cycle - there is data related to product engineering, product design, product quality, product manufacturing, partners and vendors specific data etc. This data could be stored in various formats and at various destinations. One needs to extract value from a large volume of such data by getting a holistic view of this data available from these various sources, analyzing as per the business objectives and obtain fine insights that will improve product quality, optimize product design/manufacturing cost, understand various trends to improve product value etc. Looking ahead, all the data related to product engineering and manufacturing, when clubbed with sales and support data, can give important insights that would increase sales and help beat competition. So, it is important to get visibility of all available data and understand various trends to achieve additional business insights.
Some factors that enable extracting real value out of large volumes of data are as follows :
- Understanding data and mining it efficiently: Identify and consolidate scattered data to enable processing that enables Business intelligence. Compared to hitherto existing data warehouse and BI solutions and approaches that enterprises adopted for business intelligence, Big Data solutions can contribute much better value to improving an enterprise's business by analyzing data in both structured and unstructured forms to extract meaning. But, it is critical to know how to use this data, what to analyze to come up with real values and right decision support system.
- Choosing right technologies and architecture: Have right technologies for handling multiple types of data, ranging from structured to unstructured to semi structured and to support the scale and speed at which data is received. Though technologies like Hadoop and NoSQL data bases like Mongo DB, CassandraDB, CouchDB are synonymous at present with Big Data analytics, there are number of other technologies from commercial vendors like SAP, IBM , Oracle, and Microsoft etc and many other open source technologies such as R, Cascading, Scribe, Elastic Search etc as well. As we know, one size doesn't fit all, and hence technology evaluation for specific needs has to be done before choosing the right tools and technologies. Apart from this, coming up with right solution architecture becomes critical as the data volume, data types and rate at which data is received is very high. It needs thorough knowledge on capabilities and limitations of NoSQL data bases, distributed computing technologies and cloud computing. Solution should be scalable and extensible to support future demands. Factors on maintainability and support should be considered as well.
- Data Scientist Expertise: As compared to traditional expertise on technical and project business side, for reaping the real benefits of Big Data, enterprises have to have right skill sets. Data scientists can help understand the breadth and depth of the data from the business/domain aspects and examine the same to derive various trends and come up with innovative perspectives, which would support newer business opportunities and growth potentials. It is critical to identify and involve the right combination of skills on domain, data, technical and management side to make the Big data initiative successful.
- Flexibility for Cultural change: There is a need for cultural change in the organization for making Big Data initiatives successful. Leaders have to educate business group on challenges and put a strong process and governance for enabling better collaborations. Specific issues like data security and data ownership have to be addressed convincingly. CEOs have to focus on value of Big Data and identify right IT executives who can execute the strategies well. Depending upon the available opportunities, a flexible common platform which enables easier access for infrastructure setup, data analysis, work load monitoring and getting business insights should be envisioned and implemented at enterprise level. Each business units can leverage such platform to analyze specific data and get required advantages more economically and efficiently at business unit level as well.
Above success factors are applicable to any industry, but it is important to understand all these with relevance to corresponding industries and objectives and put a robust process as per enterprise strategies.