« May 2015 | Main | August 2015 »

July 6, 2015

Three Points of View on Big Data

Big Data is all the rage. Every company has a Big Data strategy (or is working on one) and every supplier has a Big Data line of products and services (or rebranded existing product under this banner). You would be surprised how many times the word Hadoop is mentioned in a business conversation, but you probably would not be surprised that many people who use the term have no idea what they are talking about. Turn the conversation to 'structure on query' versus 'structure at load' and you will get a room full of blank stares (unless you are in a room full of techies).

Let me share with you three different 'points of view' about Big Data that coexist in a large company. Each perspective is right and adds value to its audience, but each perspective has a different focus and goals.

Imagine the blind men and the elephant story. If Big Data is the elephant, each blind man sees a very different attribute. If one assumes that everyone is speaking about the same thing, they will be very confused and disappointed. Taking that analogy further, each has different needs based on his perspective. It might help to understand that Big Data is needed for three different reasons (at least). It would also help to understand the technology and service solution that fits the unique requirements of each. 

The first will be the IT function perspective, followed by senior management perspective, and finally the digital engineering and business analyst point of view.
For the IT function-Big Data means large and growing volumes of data that has to be managed while trying to gain insight into the impact of all this has on their computing and communications infrastructures. They look at the growing volumes of data and the different types of data and translate the Big Data challenge into the number of storage devices needed and the amount of bandwidth they have to provide to accommodate Big Data. When you add cloud computing to this story, for IT is means IaaS (Infrastructure-as-a-Service) and provisioning for Big Data by renting servers and deploying data, storage, and computing resource virtualization instead of the procurement of physical devices in their data centers. Big Data is a budgeting and provisioning problem from this perspective. Try selling data appliances and unified storage solutions. Don't' bother them with questions about data and decision quality, this is a bottoms- up point of view. To this blind man who is touching the leg of the elephant, they see a tree.

Senior management looks at Big Data and sees KPI and performance metrics scorecards, dashboards, and cockpits to gain better insight into the performance of their investments and the health of cash flow. They are the captains of the financial ship looking at the control panels of their organizations. Big Data means the ability to get a more updated view of how things are going, improving the Velocity and responsiveness of their ability to control the ship. Moving from month end close data to daily sales figures is a transformation. Seeing early results from new investments allows them to act and intervene much quicker. This guy needs scorecards and analytics (but only the high-level type). Don't bother him with the details of how to get the real-time performance metrics, this is a top-down point of view. Some analytically inclined managers (those that tend to the micro-manager end of the scale) may inquire about drill down capability, so be ready for that question with some kind of federated search tool. To this blind man who is touching the elephant's trunk, they see a snake. 

My favorite audience is the business analysts and Digital Engineers. They live in the world of operations IT not the traditional corporate IT. While positioned in the middle of the organization chart, they actually have a broader systems view of Big Data.  They span from field instrumentation, to control systems, to historians, to modeling and simulations applications. They do tend to be more data consumers than data managers. Their world has far too many overstuffed spreadsheets and Visual Basic routines that used to take a few minutes to run and now take hours to days due to the higher volumes now involved. Their goal is trying to get operational insight into assets to make more money, to operate more efficiently, and safely.  These folks are more technical, so tend to like the detail, unlike their senior managers. They may have a na├»ve view of the IT infrastructure but a better view of the business process. You can sell these folks almost anything if they have the budget authority. They love new toys and have a higher tolerance for the rough edges of new technology. In tough budget times, you can have a very interesting conversation with them but end up with no sales. To this partially-sighted individual, they touch the body of the elephant and can almost make out the total animal but end up missing a few critical details, they see a living wall. 

So it is often an external perspective that sees the total elephant and the full system architecture. There is a role for external consultants after all. Big Data is large volumes and variety requiring a different kind of infrastructure and very different information plumbing (service oriented, data virtualization, etc.). Big Data is about velocity and the ability to see the organization moving and reacting to the market, but the integration of the parts must be designed to follow the anticipated business process and make sure that quality data drives decisions. Big Data is about understanding the lifecycle of the data flow from the sensor to model and back again. It requires a careful disciplined design of the total system and the full lifecycle of the asset. Big Data is about all of the above.


In other words, this is really hard to get right. Just buying a new kit for any of the components doesn't ensure a better system. Technology advances are ripe to upgrade the capability of operations and business functions. Big Data technology (Hadoop distributed file management, MapReduce, NoSQL, federated search, semantic understanding of data, ontologies, graph databases, service-oriented architecture, data appliances, real-time streaming analytics, in-memory analytics algorithms, statistics and physics based simulators, etc.) is really cool and potentially transformational. However, before you throw out the old technology (enterprise data warehouses, operational data warehouses, data marts, ETL, ERP transactional platforms, SQL, RDMS, historians, process control systems, business intelligence reporting tools, taxonomies, OLAP, OLTP) make sure you see the full picture, the whole elephant, or your Big Data investments will miss the mark on achieving the full potential of this technology revolution.