« Imagine Shopping for Your Data... | Main | Too Many Choices »

Slow Motion Data Crisis

Most of us respond to an alarm pretty quickly. A smoke alarm in your house, fire alarm in a mall, lightning alarm on the soccer field, or a tornado alarm or siren. We take these alarms seriously and get everyone to a safe place as fast as we can.

There are, however, other kinds of alarms we tend to ignore allowing a crisis to creep up on us. Let's call these slow-motion crises. These work along the same premise as the old Cajun story of how to cook a frog. The best way to cook one is to put the frog in a pot of cool water and turn up the temperature slowly so that the frog gets used to the rising temperature until it's too late. Turn the temperature up too quickly and the frog realizes his situation and jumps out. Sadly, we are the frog in this metaphor.

Businesses have alarms for almost everything. Operations boasts the ones we ordinarily think of - high pressure, low pressure, or failed systems alerts triggered by SCADA systems. However, one could consider the corporate monthly, or even weekly, management dashboard reports, where there are alarms for key metrics that impact the profitability of your business. Field personnel and corporate executives take these alarms seriously and do something to get to a safe (or profitable) place as fast as they can.

Slow-motion crises happen when phenomena or processes have potentially substantial or even devastating consequences that build and become problematic only after the passage of considerable time. They are unlike the fire alarm or the failed pressure gauge alarm. If this article was about geopolitics, I might quote the national debt, or falling education scores, or climate change as examples of slow-motion crises. But since we're discussing information management, and I think this is the way many, if not most companies treat data management, it is a classic example of a slow-motion crisis.

I don't think many companies have a key performance indicator (KPI) for data on their management dashboards. It is difficult to even know what the right trends in data management are that deserve management attention. Is it growth in data volume? Is it something around data quality? Would it be something about user productivity in finding the right data? Or having data stewards for every critical information object or effective data governance councils? For companies focused on data-driven decisions, would it be "time to insight" or a change in decision quality for key investment decisions? Would it even be an estimate of the value of the information that your company owns?

From my experience, management, even the IT leadership teams, doesn't monitor any of these trends on a regular basis. They may look at data management periodically, but there are no alarms to call them to action. We slowly get used to the growing volume, variety and velocity of data being generated by our operations and being imported from external sources. We know that it takes our analysts and data scientists too long to find, check and reformat the data needed for their reports and models, but we are getting used to that as well.

We know that there is a lot of data in "shadow IT systems" that never finds its way to our official "systems of record", but we tried to tackle that problem and didn't make much progress. We know the data quality for some key information is not as good as we would like, but when we really need that data, we spend the time to make corrections before we analyze and use it. We are just like the frog and we are getting used to the rising temperature of the business demand for better access to data while trying to improve their business processes and operate more efficiently.

There are alternatives to this situation besides just ignoring the alarms. Likely, the answer dearest to our heart is to step back and focus on data management, update our information management strategies, focus on standard definitions and standard ways of exchanging data with partners and suppliers. And we can work with the business to establish effective data governance teams and recognize the value of good data stewards.

Also, we can benchmark our information management practices with our industry peers and look at what they are doing. We can bring in new technology, fill data lakes stocked with data catalogs to make data management a little easier for both the data managers and the data analysts. We can work with consultants and technology vendors to evaluate our current situation, get a second opinion on the data and integration architecture we are using and build a road map for a practical pace of improvement in data management practices and capabilities. Any of these steps helps to put the state of your data management practices in focus against the rising temperature of Big Data trends.

I realize this isn't the really exciting work that emerging technology offers and it isn't the highest priority when budgets are squeezed. And it is rarely a frequently asked question when we meet with business leaders or attend high tech conferences full of high-priced thought leaders. Whether the consequences kick in gradually (when productivity grinds to a halt) or hits us suddenly (when a critical decision is missed because a project couldn't find the right data to see the alternatives), the slow-motion crisis of the inadequate response to Big Data trends will hit us. The water is already heating up.

What temperature will it take in your organization before you react?


Jim Crompton is a thought leader for Noah Consulting, an Infosys Company, helping pioneer the relationships between complex Upstream processes and enterprises with automation to create competitive advantage.  His experience over numerous decades combined with the development capability of Infosys is working to ensure successful alignment of man and machine.

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.