« People and Data: Two Important Assets | Main | Thoughts From a Wandering Mind at PNEC 2015 »

Big Data, The Other Velocity Problem

The attributes of the Big Data revolution are often described as the three Vs (volume, variety, and velocity) and sometimes the author throws in another V (veracity) for good measure. It is pretty well accepted that,

  1. the amount of data being collected by our business is growing,
  2. the different types of data that are important to understand our business are growing, and
  3. the speed of data collection and the need to reduce our response time to incidents and opportunities is growing.
So, the three Vs are a pretty good definition of this emerging technology trend.

However, I would like to talk about a hidden V, another form of velocity, that is becoming a significant factor in the adoption of these new technologies, as well as a critical aspect of dealing with the fourth V (I will use the term data quality instead of veracity in this article). The hidden element of velocity deals with how fast internal IT departments can react to the three Vs driving this revolution. If an IT department responds too slowly, the alignment and partnership between the business and IT is strained. If it gets too serious, the business is tempted to go it alone and shadow IT. Otherwise known as "spreadsheet hell," where external solutions become favored over internal ones. In this scenario, the internal IT department is relegated to back office and infrastructure services and is left out of the party. Technology vendors are keen to sense this divide and are prepared with proposals for the business to by-passing traditional IT service offerings.

Let me define the service response velocity in a little more detail. Here are the questions I would ask your organization to measure this hidden velocity metric:

•             How long does it take to find good data?

•             How long does it take IT to fix data quality problems?

•             How long does it take IT to produce my report, load my data? Build my scorecard?

•             If I do not know what I am looking for, can IT help? (data discovery)

•          Does IT have the budget to acquire new technology this year? (versus renting it from a SaaS vendor)

•             How long does it take IT to develop new skills?


Your IT department probably has a mature process for managing data where there are defined systems of record for each function, even if there is less standardized than preferred, at least there is a home for most data. There is likely a master data management system with the "gold version" of each critical information object and ETL routines for transferring data from source to your enterprise data warehouse (at least for structured data). You probably also have standard reporting or BI (business intelligence) tools for routine reports. There are likely additional, but separate, data management solutions for documents and transactions, for models and sensor data. As well as, a data retention and data security policy in place for protecting your most critical data.

If all this is not yet in place, there are probably projects in your IT investment portfolio that intend to build out this traditional data management infrastructure and support these technologies and processes up to the service level agreements that they have negotiated. That is one reason the internal IT budget is so large, and unfortunately growing.

This environment was built to solve the business challenges from yesterday's perspective, but how are they coping with today's world? With the Big Data three Vs changing the playing field, can IT respond to higher expectations with traditional architecture? How fast can internal IT respond to data loading, data mastering, data quality checks, data modeling when the volume is rising, the variety is growing, the pace is accelerating and the need for trusted data, integrated into a more holistic view is now the minimum requirement?

Sadly, I think that the traditional solutions are coming up short. That is not to say that they do not have an important role still to play. The business requirements that drove the traditional solutions are still there, especially from the traditional drivers for IT, finance, HR, procurement. Most of the new requirements are coming from Operations, which has not been a large IT customer before. However, despite internal IT services improving the gap it is still growing as the three Vs are becoming a powerful force in business. Moving at the pace of business is a growing challenge. When IT cannot keep up, their budgets are cut, making it even harder to keep pace. They are not invited to planning meetings when important business projects are discussed or vendor presentation for the new technology alternatives, often at the vendor's request and the manager's approval.

When business reaches the point where they have given up on the expectations that IT can move at their accelerated pace, there are consequences, not just for IT but for the success of the Big Data initiative. Despite the sometimes slow pace of internal IT, they do have important controls for data quality, data governance, high-availability, reliability, information protection, and master data and metadata management. Leave IT behind and most likely these processes are left behind also.

Does data quality no longer matter when you have many statistics to compensate? Actually, there are activities when relevant patterns can be identified even with poor data quality outliers. However,  there are other activities when your data needs to be right in order for you to make the correct business decisions.

In summary, your internal IT department is under pressure. Their best practices are necessary, but no longer sufficient, for the current "internet of things" world your business is now swimming in. However leaving them out is not the answer either. The partnership is strained. The CIO is losing sleep and starting to worry about job security. The vendors are coming up with ways, such as cloud-based SaaS (software as a service) solutions that only require a credit card and bypass the IT budget process. Also, low-cost infrastructure options like Hadoop to by-pass the "IT Police," so that they can make their sales commissions this year even if the IT budget is cut.

The right way to solve this dilemma is for IT to find a way to speed up, for business to listen to IT's issues, for vendors to partner, and for consultants to facilitate the alliance. However, will the urgent bypass the important, will the temptation to jump overshadow the need for caution, will the marketing hype leap over the proven? Tune in next time to find out, same bat time, same bat channel (sorry about the Batman metaphor, I could not help myself).

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.