« December 2014 | Main | March 2015 »

February 4, 2015

Ten (New) Things I have Learned about Information Management

A few years ago, I was invited to talk about what I had learned about information management. The audience was a working group that got together to address enterprise data challenges for a large oil company. They were working on an update of their IM strategy and kindly invited me to share my thoughts on the direction and challenges for information management. I organized my thoughts into ten lessons. The discussion went pretty well, so I thought I would share those observations that still hold true and some new things I have noticed.

Lesson #1: We have to look at data differently. The traditional approach for data management is to segment the type of data with the specific technology used to manage and analyze it. Data comes in many flavors, from structured data, documents, transactions, sensor or measurement readings and maps to models, trends, interpretations, and forecasts. We usually segment data, and our capabilities, by the type of data format we are handling. However, consumers of information management systems do not really care about the format, so we have to begin to see the problem from their perspective. Yes, the specific technologies are important. However, our vision has to be more than just deploying new technology (i.e. SharePoint, a data appliance, or Hadoop) to manage more data. Lesson #2: We need to better understand how data is being used. This follows from the first point. Knowledge work is becoming more interdisciplinary and virtual. Increasingly, joint venture partners or regulators are interested in the data we collect. Data that is captured in one part of the organization may produce more value when used in another part of the organization. Data needed for some of these workflows comes from sensors, historical data, and documents. Solutions for complex workflows connect workers from different locations, functions, and even different organizations ─ all to produce a more accurate and holistic view of the asset and the decisions needed to optimize value of that asset.

Lesson #3: We need a data foundation beneath business intelligence and advanced analytics efforts. Far too many projects to deploy sophisticated analytics solutions come up short because of the difficulty in finding and cleaning up the data needed to fuel these innovative solutions. Too many projects focus on the top of the food chain and neglect the data foundation. As important as they can be, do not be fooled by the sexy analytics. You still have to do the dirty work of getting your data in adequate shape to make these investments pay out.

Lesson #4: We need more focus on how the consumer accesses data. Whether they are called portals, dashboards, cockpits or advanced control panels, there are a lot of cool new information visualization tools coming out. These tools ─ tabular, map-based or based on stop signs or speedometers ─ are most useful when designed to help a data consumer get a task done or visualize certain patterns. The challenge is understanding the user experience. Do not get carried away with the capabilities of new technologies. Develop something someone can use to make a quicker analysis and reach a better decision.

Lesson #5: We need to put more emphasis on data and less on systems of record. We have done a lot of work in the context of standard enterprise database programs developing and deploying new systems of record. However, some early returns are in, and they suggest we have not really solved many of the problems yet. Many applications are very configurable, so standard designs are soon modified to fit local interests. Many deployments skimp on the data cleanup work needed for migration of data from a legacy system. Many systems assume that common definitions are well understood between functions. We can easily get our heads around a new database applications project and bypass the important work of information architecture. A new application can trick us into thinking the problem is solved. However, if we have not done the data design right, with the right data definitions, appropriate master or metadata, and right business and technical rules in place to sustain good data quality, then we may have just deployed a new shell that will soon produce the same questionable results the last applications did.

Lesson #6: We need to pay more attention to IM organizational capability. Just about every strategic staffing study I have seen recently, whether inside the company or by an external consultant, lists a number of information management roles as critical skill sets. Many of these roles are in the functional and operational groups as well as in IT. Some skills need to be better developed in the end consumer (the knowledge worker) and not in a formal support position. Wherever the role sits in an organization, we need to put more attention on developing these people to play the bigger role now being asked of them.

Lesson #7: The data explosion is real and will not stop anytime soon. Just about every report or magazine I pick up contains an article about the explosion of data. Recently, I read about the billion cell reservoir simulation models from Saudi Aramco, the 80,000 channel seismic crew from WesternGeco that can record one petabyte of data an hour and the 20-30,000 sensors on the Transocean Clear Leader drilling ship. Data from field operations is growing seemingly without end. We better be prepared. The challenge is not just to collect and store it. The challenge is to find ways to use it to make better decisions.

Lesson #8:
The three IPs (Information Protection, Intellectual Property and Individual Privacy). As we are improving data access, we cannot forget the legitimate restrictions we have to build. Data access should be improved for those who have the business and legal right to use it (and not others). Some data will carry company classified status (or other designations), and this data requires special handling. Certain data about individuals is subject to country laws about privacy, access, and processing rules. Read and understand the appropriate company policies on the three IPs!

Lesson #9: Do not forget to consider the future.
There are some exciting developments coming around the corner, such as data appliances, "big data" platforms (Apache Hadoop), streaming analytics, in-memory processing, self-service BI and semantic web. Companies such as IBM, SAP, Microsoft, Oracle, Netezza, NetApps, and SAS are already trying to get us interested in these emerging products. Moreover, there are some new kids on the block like Cloudera, Hortonworks, Platfora, Tableau Software, Ayadsi, and Datameer with an interesting story to tell. Two things you can count on, like death and taxes, are there will be more sensors capturing more data and faster computers that can process more transactions and simulate larger models. The challenge is to connect this capability to effective decision-making.

Lesson #10: The ultimate value of better information management is better decision-making.
The one thing all the stakeholders (business, functional and IT) can agree on is that the most important result of collecting, processing, managing, reporting, analyzing and modeling activities, is making better investment and operational decisions.

Please share what lessons you have learned about information management.