Yes I am referring to upcoming stream called “ediscovery”. This means process through which we can make things appear which are stored in computer or network for purpose of reproducing them as evidence in court of law. Law Instructed or Government backed ethical hacking also comes in preview of this domain

With new nature of frauds coming into our way and most of state governments working on laws to protect cyber crimes and ensure quality of output it becomes all the more important to focus on this stream.

Although “eDiscovery” has recently caught my attention and I am still thinking about this. To me it is more than technology. It deinately has some element of law, security and compliance all built into it. Though I am not a law expert but I can certainly pay attention to IT related issues where technology can be used

Few Scenarios which can boost this stream of work are:

1.       With search coming into shape and latest technologies coming our way to perform advanced searches. It makes it possible to search digital content using advanced algorithms

2.       Digital Data once written over network cannot be lost easily. Retention and Archiving Policies make sure that we can retrieve desired information back

As part of ediscovery all kind of data can be searched ranging from images, movies or text content. Emails and Chat Messages are another important source of information which can be required to search and reproduce.

This whole stream can be serviced with following services:

1.       Data Analysis

2.       Data Retrieval and Conversion

3.       Scanning

4.       Metatagging data for easy retrieval

5.        Indexing and Processing of data

6.       Exporting

7.       Printing

8.       Quality Control

9.       Recognition of data

With lot of legality and complexity involved in this topic I intend to DISCOVER (and invent) more and write about it.

Any thoughts from your side?

Let me make an attempt to define this Avatar: A coordinated effort and initiative to bring together the Enterprise wide Information for driving the Organizational goals at Strategic, Operational and Tactical levels. If I were to list down the key terms comprising EIM,I would choose following:
- Data from Source Systems (Structured and un-structured)
- Interfaces
- Data Integration
- Data Transformation & Aggregation
- Information Dissemination (Portals, Extracts, Reporting Applications, OLAP Cubes, Feeds etc)
- Aligning Data (drilling to Tactical and Operational levels) to Strategic Objectives
- Standardize the Technology and tools across the Organization
- Best practices in Industry on Data Integration, ETL, Data Warehouse/Data Marts/Data Store, Reporting, Metadata, MDM

Few of the business initiatives which where EIM becomes important and strategic arm for any organization, and without a firm EIM in place achieving those is like a nightmare
- Regulatory Compliance (BASEL-II, HIPAA etc)
- Data Governance and Data Quality Enterprise Wide
- Master Data Management, CDI/PDI
- Align Business with IT or vice versa
- Leveraging Pooling of resources and cross departmental data sharing
- Information Risk management and data Security

Today the data that gets integrated from multiple data sources with organization & bigger challenge is to bring together the structured and un-structured data (comprises of over 70% untapped potential) together for decisioning. This data ventually flows to various users at different levels of organizational hierarchy. If we pick Retail business to illustrate -
At Operational levels: Right from POS (for right pricing, discounts etc), to store owner managing the floor spacing and food stocks on shelf, to Warehouse managers integrating orders to supply chain and maintaining optimal inventory, to logistics manager for deliveries or warehouse replenishments to  performance reporting on daily basis.
At Tactical and Analysis levels : Business Analysts, Marketing specialist doing analysis on operational level data gathered and identifying issues, problems before they scale large. Do reporting to higher levels on Departmental performances, financial performances
At Strategic Levels : All the above rolls on to strategic level aiding decision points, refocus on strategies or go aggressive in specific geographies, check on the profitability and organization financial performances.

With that level of decision makers involved and accessing information, the need of the hour is a solution that holds all of those pieces together under a common data Governance Umbrella and a common language spoken across the organization - This is how i view EIM as.

In continuation of this I am planning to elaborate on pros, cons and myths on EIM, and how should one approach towards a successful EIM which can showcase a transparent ROI with equipping one with effective decisioning power.

Do share your views and experiences with EIM world.

I guess I wasn’t the first one to arrive at this conclusion. After a little more digging, I came across the concept of Attention Economy . Basically information on the web has exploded and the choice available to the end user is making it more and difficult for them to be able to filter out what they are really interested in. This affects both users looking for information they need as well as or organizations who are creating the information for you. So organizations have started to realize that your attention is becoming a scarce commodity and are willing to do anything and everything possible to grab it, maybe even pay for it. So what impact will this trend have on the space of Information management, especially portals, content management and search? Here are some thoughts:

Personalization has been one of the core features of portal products. But it is rarely used to its full-extent. It is just not considered worth the money. But as the attention market heats up, personalization will become the differentiating factor. The better the personalization, the sooner the end user can find the information they need.

How personalized is personalized enough?

Most people think that personalization based on demographic information like location, language and age or organization specific information like membership, groups and department is enough. But these factors rarely help you understand the user intimately.  One technique which has gathered momentum is analytics based personalization. User behavior is tracked and used to predict what they might like. This is more effective and it lets you get closer to the subconscious mind of the user, even if they do not explicitly call it out.

But is this personalized enough. Not nearly. Web 2.0 changed everything again. Collaborative filtering is another technique which has emerged which helps leverage crowd intelligence. Analytics are gathered from multiple users and used to make recommendations on what you will like. This goes closer to the subconscious of the user as it analysis not only what you have done in the past, but also what other similar users did in the past.

End of Information Portals?

So I get thinking, If someone can really personalize the information and deliver it to me, do I really need a Rich, attractive Portal with heap loads of content. Maybe not. What I need is an Attention Management system which knows and learns what I like, which goes and fetches the information I am interested in , which lets me configure that information and view it the way I want to.

Putting it all together – An Attention Management System (AMS)

So how will personalization change from what exists today in the form of personalization engines and portals? I believe the entire machinery of information generation and delivery will be tuned to deliver personalized content. An Attention management System might have the following components :

a)      Personalization Information source – This will be the component which provides the information required for executing the personalization rules. This could further be divided into

1.       Implicit information source -  This includes well know information about the user like location, department, groups etc. usually stored in directories.

2.      Explicit information source – This includes information provided by the user in terms of their areas of interest, likes and dislikes, opt-in/opt-outs etc. which is stored in systems like portals or social networking sites.

3.     Behavioral information  - Information about the browsing behavior of the user e.g. sites visited, time spent online.

4.     Collaborative information – Collective information about browsing habits of numerous users , and their interests. This could be gathered

There are concepts like Attention Trusts which might emerge as the source of majority of this information.

b)      Content source:  This will be the source of content that needs to be personalized. It could be content management systems, feeds or applications. But the change will be that it will no longer be enough to just create the entire page and tag it. Content will have to be created to be consumable in a semantic way. This is because instead of it being directly consumed by end users, it will be read and filtered by Personalization engines. So the simpler you make it for the personalization engines to find, the better.

c)       Content categorization engine : Personalization engine may not be able to pick up raw content, as they will not be able to read and categorize content. They can only provide a recommendation on the kind of information to pickup. Categorization will need to be done by another specialized component, and what better solution than to use a search engine. Search engines can read both the metadata as well as contents and categorize the content into different buckets e.g. news, United states, athletics, home improvement etc. When the recommendation engine throws out a recommendation, it can be matched with these categories to pick up the content. Ofcourse, Web 2.0 also plays a role here. Collaborative tagging can help categorize the content better through collective intelligence.

d)      Personalization decision system: This will be very similar to the personalization engines of today. This is where the rules for personalization will be written and executed. This engine will utilize the rules and the information from the personalization sources to provide a recommendation. These engines might evolve as standards like JSR94 and collaborative filtering algorithms mature. This will be the heart of the system, and the vendor who gets this right will win the game.

e)      Content Delivery system: Structured websites (read portals) might be a thing of the past. Instead, a Attention Management system might give you a completely configurable front-end (read Mashups). You will be able to assemble your portal by puling together attention blocks and configuring them to show information of a particular kind e.g. news, quotes, Tweets, shopping,  books, music, etc. The Attention management system will use its personalization engine to deliver the best content in that block. The content Delivery system could also become the source of behavioral information as it can track what you are clicking on.

The Attention Management Cycle: Spinning the Web

Well, what I mean by the Attention management cycle is that soon the web will go into a spin. This will be a continuous cycle of improvement where user analytics will feed the personalization information sources, which will lead to better content and better tagging, which in turn leads to better personalization, which in turn will lead to a satisfied end user who can give better analytics. This cycle will continuously strive towards a more satisfied customer. Content producers will win if they make their content easily consumable by personalization engines. Portal providers and search providers will win if they can get the personalization game right. 

The core of the attention management cycle will be the accuracy of personalization. This accuracy will rely on analytics information. As time goes by, information about you will not be free. You can protect this information and chose to share it or not. Some ideas like Attention Trust are a strong indication of a move in that direction. What this means is that if you can protect information about you, both explicit and implicit (analytics), you could also monetize it by charging direct money or points which you can redeem at the site you share your attention with.

 Thoughts?

Based on above brief explanation, I dare to comment that it is mostly (not completely) analogous to BI/DW architecture of Federated Architecture. So if you assume our body to be an enterprise with regional spread, then peripheral nervous system is the regional DW managing its local requirements independently and sometimes with HQ help. While central nervous system is the Corporate HQ that controls all parts of organization with input from regions. And together through their activities, they display the organization behavior.

Do you find it Interesting? – please comment.

In next blog, I plan to write on how ‘Data capture and integration offerings of product companies’ are still way behind the way we human capture and integrate information. There is a great learning and can be applied in next steps in EIM.

Until recently, predictive analytics solutions were serving niche business domains and within that certain specific business processes, however the intensity of investments planned by the BIG IT service providers hints at another “commoditized” IT service in the pipeline. Another exciting side of the story is that the industry is looking forward to explore the benefits for daily & intra-day operational decision making. Traditionally analytics applications have supported strategic & tactical business decision making, which did not put ‘low latency’ pressure on data integration but operational decision making needs data refresh as close to real time as possible. As far as operational analytics solutions are related, debates are already on embedded and stream analytics but the real challenge would be obviously on data feed. And it would be interesting to watch the trend, how solution providers align with this demand – whether it is much advanced data integration through typical Operational Data Store (ODS)/Data warehouse/Data mart route or Analytics Solution embedded in Business Process Workflow itself.   

1.       The objectives have to be listed first, like: the business drivers, what should attract the attention of senior management, speed of availability, and done with lowest possible cost

2.       Then we need to understand the important business and technology hurdles.  Business hurdles are usually faced: standardisation of definitions, clarity on information / data ownerships, agreements on master data hierarchies, and achieve common understanding on business case.  Technology hurdles are like: data standardisation, data cleansing, enterprise data model building, manage metadata, and other important items like scalability, reliability and availability

3.       Since the IM programs are usually extensive, it is essential to give Business Centricity to establish quick wins, and continue the goodwill gained to further the program

4.       Look at ways and means of reducing the effort wastage, through re-use of methodologies, frameworks and documentation templates

Finally, the IM is supposed to have and provide:

1.       A pretty good base has to be built to serve as a foundation.  This can be multi-layered data bases, data marts etc., but should have proper rules controlling the basic data management

2.       To improve the standardisation, master data management (MDM) has to find a place in here.  Business needs should dictate whether this should be a centralised MDM or decentralised MDM

3.       Data privacy and information access is paramount, in addition to the information availability.  Hence, access controls have to be properly set for the different business users, based on their span of control and information life cycle exposed to them

4.       The last but one is the availability of play area(s) for business users.  This could be either in the form of departmental data marts, just sand boxes, or some play pens.  This area could be controlled in the form of persistent areas, aggregate areas, and non-persistent areas.  This should also allow business users to bring in all their tools and expertise to explore data, fit the statistical modelling, and bring out information for informed decision making

5.       To make all this work in tandem, and still provide freedom to the business users, we need better IM Governance. 

Allow me to spend some more time on the important aspects that should be considered as part of IM Governance.  First of all, IM Governance should be a body of some permanent folks, and some semi-permanent folks.  The permanent folks are the ones who are more associated with the technology aspects and the data modelling aspects.  The semi-permanent folks are the ones who are business users, who act as business analysts for the whole build.  The semi-permanent folks are constantly replaced with a new set of people, depending on which business function is being addressed at any point of time.  The permanent folks will also take care of the overall program budgeting, spend, re-usability aspects etc.  They will also take care of the complete information lifecycle, its Governance and Stewardship.  The Governance should handle Corporate Goals and Principles, define Roles and responsibilities, establish a working structure and along with it processes.  The Information stewardship focuses mostly on the: Identifying vital / sensitive data, exposing metadata models for business users, define business rules on data manipulation and sharing, data rules on transformations, cleansing, creation of applicable metrics at different management hierarchies. 

More on the IM Guidelines later....

However, for these Advanced Analytics to be made possible, we need the have the appropriate Business Domain Knowledge and the Frameworks to quickly deploy and make them adoptable.  Hence, we may say that customers can easily leverage their existing BI installations for Advanced Analytics, as they would have the necessary cleansed, meaningful and well arranged data.

Most of the BI usage, at the most, is around reporting only.  Very few have ventured into AA, although we are seeing some such stuff in the Performance Management or CPM areas.  Deploying Advanced Analytics will be easy if the customer already have some kind of BI DW in place, as this becomes the very foundation of it.  In that case, deployment of AA in the absence of established BI environment, will be challenging in timelines and efforts.  Quicker deployment of BI environment can be achieved through deployment of ready-made BI solution frameworks, which are best addressed through mature domain BI data models. 

Some vendors are already offering AA applications, especially in the areas of Procurement Spend Analytics, Value Chain, Activity Based Costing, Fraud, Production Planning, Customer Churn, ALM, Warranty Budgeting etc.  There is another school of thought to include the real time analytics / decision making also as part of AA.

AA is very popular nowadays, but will be interesting to note the percentage of revenues, compared to the regular development & maintenance revenues.  Will be continued in my future blogs.

Most of the big companies have already invested in BI in a big way.  It is quite evident that big companies thrive on multiple BI tools and technologies.  Small companies have started plans to invest in BI in big way. Credit Crunch and economic downturn is causing heartburns for the CIOs who are interested in investing, but not having enough funds on hand.  BI Programs being obviously high expenditure programs suffer the most. 

Take heart that innovations are happening in BI space too.  These new innovations let you play hard in the game, even though your hands are tied on expenditure.  Some people call them as BI Mashups, others call them as BI Killer Tools.  These have different flavours too.  Some are most IT users oriented and others are business user oriented.

I don't want to talk about a new tool in the form of BI Mashups or BI Killer tools that are going to replace the existing BI / Reporting tools.  Then what is this?

Consider this: More than 70% of the BI is involved in standard reports.  Rest of the BI is spread across ad hoc reporting, analytics, OLAP, Dashboards etc.  So, savings can be brought if there is a good answer to address the 70% of the problem. 

In case of large organisations, where multiplicity of BI tools exist, there is always a need to bring the entire organisation under one umbrella of Enterprise Reporting.  This can be achieved through different means.  Either by standardising the reporting tool or by introducing a new thin umbrella on top of all these multiple reporting tools.  Standardising the reporting tool will be a huge investment, instead the second seems more suitable.  Tools like InfoFlow are making waves in this space.  These tools provide single user interface for most of the standard report users.  It can leverage the existing reports that are developed on different tools for its own reports.  It can also schedule the existing reporting tools.  This drastically reduces the need for expensive extra licenses for different reporting tools. 

But for smaller organisations, where they have just embarked on BI programs, or contemplating for the same, it is definitely a good idea to put more power in the hands of business users.  IT Team size will be small in these organisations.  Reporting / BI Applications can be quickly developed by the business users themselves using tools like QlikView, Tibco Spotfire etc.

I can see a great future for both these types of tools in the market.

Corporates feel that their best data sources are their agreements and the mail server databases, where the entire correspondence and details of their third party arrangements are available.  But at the same time they cannot let their transactional information attain any lesser importance.  Providing information, trends and insights in the transactional information is also necessary.  Business users are also interested in building small but quick applications to understand aberrations in their data, and take fast decisions.  All these things call for data parsing, data arrangement, data transformation, and data analysis techniques using Search, BI and Interactive Visualisations tools / platforms.  Sometimes, Corporates also need to access external / streaming-in data to understand the public opinion about their products / brands and services.  All these play an important role in shaping the decision making processes within corporates.

Managing all these technologies to help the decision makers to make informed decisions is important.  Most importantly, getting the users access the right information / data is very important, in view of plethora of data sources and tools available.  Information Access platforms should take care of parameters like Relationship Effectiveness, Operational Effectivenss, Risk Mitigation and Technology Efficiency in view to drive the Customer Centricity for business improvements. 

The challenges to the new information management are: "Findability" of relevant information, Contextual integration of fragmented information, Extraction of intelligence from unstructured data, leveraging and integrating traditional structured data. 

Search Platform is fast emerging as a viable Information Access platform: as it easily combines public domain data, unstructured and structured data.  It comes across as a Insight Driven Info Platform, by providing Guided Access, Unified Information Access, Knowledge Platform, and sometimes as a Virtual Data Warehouse. 

VSA offers some of these best procurement strategies, as it focuses on the biggest cost area.  Looking at procurement function, we can think of multiple facets like maintaining relationship with suppliers, understand more on their performance: what quality of the products they are supplying, how punctual they are in their delivery timelines, what is their turn-around time from the purchase order to delivery etc. 

On the other hand, VSA also enables to understand the purchasing behaviour our Procurement executives, like, who are the preferred suppliers for some of the executives, how much amount of business we are doing with each supplier, are we availing the discounts and fully exploiting the payment terms, check on utilisation of freebies offered by suppliers in terms of goods or service, check on the long term vendor agreements signed & utilisation of the same, leverage big sources for more discounts etc.

VSA also acts as a good platform to integrate with the procurement processes with the sales forecasting, production planning, and raw material planning engines.

We can clearly see that VSA focusses on External, Internal and Integration Fronts.  The first one is to understand the relationship with the suppliers, second one is more on the purchase practices and the last one is alignment with the company goals (short term and long term).

Probably, we can categorise VSA into Tactical and Strategic pieces.  Tactical piece will be required to streamline the vendor relationships, vendor consolidation, discount maximisation.  The Strategic piece will be required to integrate the sales and production planning, and also in arriving at the Economic Order Quantities (EOQs). 

For a good implementation of VSA, we need some sort of Master Data Management to cleanse and maintain the vendor and product codes.