« Five Stages of Digital Transformation in the Oil and Gas Industry | Main | Is Traditional Data Management Still Relevant? »

Suffering in Silence

As a consultant, it isn't very often I get to spend much time with end-users - the ultimate audience for most digital solutions projects. They are busy, often distracted and usually suspicious of why a third-party is poking around their work processes. I might get a few time-limited interviews on specific questions and requirements gathering, but it doesn't often get beyond that.

Usually, the sponsors are in the IT department and give a good picture of what is going on technically, with the computing infrastructure, with the applications and the support perspective. But that is only one side of the story.

One of the subjects I am always interested in is the adoption of technology. The long-term success rate in many industries is not very high, but the problem is rarely due to technical issues. New technologies work, so why don't we want to use them?

While there are well-documented challenges in introduction and adoption methods for new technologies, one silent-but-deadly killer is the lack of trusted data. One of the biggest and most obvious symptoms is to find that critical data is missing or incorrect. Like a slow-moving cancer, the insidious data quality issues don't take long to surface leading to users "self-treating" with spreadsheets. 

But if data quality is that poor, why aren't the business users speaking up and demanding better support? From my experience, there are a few reasons companies are content to allow serious data quality problems to persist and why there isn't a user revolt.


To start, experience has taught users to have low expectations of enterprise systems (such as the ERP platform, the engineering standards data base or other discipline related systems) and at the first hiccup, they revert to their work-arounds and spreadsheets. They know how to cope with the current state - regardless of how bad it is. Requests for IT to fix the data issues generally take longer than the deadline to make a decision. Add in that the definition of good data is often different depending on your location in the business and lack of clear ownership responsibility. The perspective and approach to resolution can be wildly different, therefore, strong communications are required to make any real progress.

Whenever a new application is deployed, standard practice is often that IT will perform a one-off spring cleaning of data along with migration of (some) historic data. Data quality improves for a while, but poor data management processes soon pollute the new system and trust declines over time. Since not all legacy systems are incorporated into new solutions, access to older systems are often migrated into the shadow IT world along with personal spreadsheets.

Most users are good at scraping together the data they need and storing it in spreadsheets. They have their own version of the truth so they don't worry much about the data quality in the official system of record. Depending on skill levels, it isn't surprising to find some basic SQL database programming and simpler structured databases such as Microsoft Access. There are even those using Visual Basic language and building sophisticated algorithms as spreadsheet macros to manipulate data in applications like Tibco Spotfire.

When senior managers are asked about data quality, the usual response is that they don't have any data quality issues and they have all the data needed to make decisions. Pressed a little further and the story is one of heroic efforts by a few individuals behind the scenes who gather data from diverse sources, clean the critical data elements in their spreadsheets and make sure that management is always presented with the most accurate and timely data. These guys are eager to impress so working late in the day or over the weekend before a big meeting is just part of the job for a young staffer.

Often the last quality check uses the rarest of tools - the Experience Filter (otherwise known as the "I- know-bad-data-just-by-looking-at-it" filter). It is true - an experienced manager can look at data and identify the questionable entries quickly, however, it can provide space for the data cancer to grow. Because the bad data is filtered out, it isn't corrected leaving the poor data in place. Experience can also narrow down the data that is deemed relevant - searches are focused on a critical data elements and as soon as those are found, a decision is made.

All these coping mechanisms help prop up current data foundations. A top-down data clean-up is daunting, time and resource intensive, therefore, not attempted. The attraction of new solutions seems irresistible, applications are added to the current inventory, complexity grows without a serious review of existing standards or strategic planning.

The best quality and primary data source should be from the official foundation with spreadsheets having a smaller role to play. Enterprises need tools to explore data collections, put together analysis, discover new patterns and gain new insights. Users and IT will need to work to find agreement on the approach to data quality, for starters. This can build trust - between teams and of the data sources. And if it doesn't, we will all be better off if there are more complaints and less suffering in silence. 

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.