Winning Manufacturing Strategies

« Show me some Green | Main | Electric, Hybrid or Gasoline: Illusion, Fusion or Confusion »

How does Search technology boost Decision Intelligence?

Guest Post by

Ketan Chinchalkar, Senior Project Manager, MFG-ADT Online, Infosys


Decision Intelligence leveraging search engine technologies and text analytics is the revolutionary and new approach of decision intelligence by exploiting the enterprise information assets and the unstructured web content. It started with the vision of addressing Business Intelligence platform limitations, but then it started redefining the field of decision intelligence by merging Business Intelligence (BI), Competitive Intelligence (CI), and innovative and new technologies derived from the search engine market. We all know that, BI encompasses a range of IT tools, usually accessible through a common BI portal that aids decision-making for enterprise analysts and managers, including financial performance reporting, monitoring of current operations, performance benchmarking, marketing analytics and sales trends identification etc. CI enables the enterprise to understand their competitors' strategies, and those of other important market players, and is particularly intended to identify trends and new and potential opportunities for growth, mergers and diversification.


BI platforms have evolved from a simple operational reporting to multi-dimensional analytical capabilities platform, but still face significant financial and technical challenges. BI platforms only process the structured enterprise information assets and do not process the large volumes of unstructured enterprise assets like emails, presentations, office documents, web pages, chat transcripts, logs etc. which have a very large potential in critical decision making for the business. The platforms have multiple interfaces for accessing various functions, needs to integrate with multiple data sources and hence make it overly complex for the business users, limiting the adoption and use. The efforts to scale and integrate the BI platforms with multiple data sources within the enterprise is highly complex, very costly, time consuming and difficult to manage within an enterprise. Also, as the ETL tools doing the input to data warehouses and data marts operate in batch mode, the decision making is not effective as it does not happen on the fresh data. Like BI, CI systems also have some inherent limitations arising from CI's dependence on huge volumes of unstructured data as an essential data source to derive competition information and trends. The web in particular has been a challenging CI data source, because of the strong dilution of the information relevance and very weak signal to noise ratio. Hence because of this automated CI systems remain fairly limited in scope and till now focus has been on customized manual research.

Technologies derived from the enterprise search market and text analytics are proving to be an ideal complements to the BI and CI technologies, as they are capable of eliminating the above mentioned technical and financial challenges and opening the door to their integration within a unified decision intelligence platform. The enterprise search engines in the market today can handle large volumes of structured as well as unstructured data and they also give a meaningful structure to the unstructured data indexed by the engines. The semantic technologies at the core of search engines are specifically designed to analyze and process textual data (unstructured information assets) and provide a meaning to the unstructured data and create relationships within the structured and the unstructured data indexed within the platform. The search platforms are scalable to handle hundreds of millions of data due to the search engine Index and Distributed architecture, which, even the strongest of RDBMS cannot handle, and that too it does with a very low TCO. These search platforms also support SOA architecture which enables rapid deployment and easy integration within the complete enterprise information ecosystem like enterprise databases, BI back ends, file shares, email systems, web content like websites, microsites, blogs, forums, communities etc. and enterprise systems like ERP,CRM, CMS and legacy systems etc. Along with the advanced technology, search platforms also provide simple and user friendly interfaces with natural language querying, intelligent querying and browsing options, collaboration and rich graphical tools capable of doing multi-faceted navigation. 

Unstructured data sources are heterogeneous and widely dispersed inside and outside the enterprise firewall and hence the challenge is to locate the exact and meaningful resources, and to extract, classify, and exploit the useful information they contain from the point of view of decision systems, e.g., effectively detecting named entities, identifying patterns, enriching thesauri, performing semantic analysis, dynamically creating content summaries etc. The unstructured analytics in the search engines work by autonomically transforming non-structured textual data into structured information using semantic analysis techniques drawn from artificial intelligence, including:


  • Automatic language detection
  • Lemmatization, or the intelligent recognition of the form and variations of words from a language, i.e., feminine or masculine, plural, conjugation state, adjectival usage etc.
  • Advanced phonetic recognition to manage typographical errors, inversion of letters or letter groups, homonyms, synonyms etc.
  • Personalized semantic filtering, adapted to the ontological context of each enterprise
  • Semantic analysis, detection of lexical forms (patterns), recognition of named entities (people, places, times, dates etc.), integration of business thesauri (ontologies), and creation of specific business context semantic rules 

The traditional decision intelligence systems can answer the question like "Which product was the top seller for the last quarter" but would not be able to answer prognostic questions such as "Which product is likely to attract the highest number of consumers in New York two months from now". The second question will need analysis of data within the enterprise boundaries and also a deep analysis of data outside the enterprise boundaries to detect the latest trends and sentiments of people to yours and competitors' products in New York and also recent trends which can influence the buying process. The search technology exploits new data collection, processing, indexing and access technologies that remove the existing technical and financial barriers associated with leveraging unstructured data in traditional BI systems. This will enable the enrichment of BI platforms with emotive analyses and product or brand popularity using the information available in an unstructured format in blogs, forums, press sites, dedicated web sites, testimonials, buyer comments, communities etc. 

Let's consider, another example, the case of a business analyst, studying factors that may explain a marked decrease in maintenance contract renewals for a specific product.

With the decision intelligence platform leveraging search technology, the analyst can:

  • First of all, locate the period in which there is a sharp decline
  • Search customers that have chosen not to renew the contracts
  • Look at e-mails containing requests or complaints sent to the company by these customers, and, if desired, view text transcripts of telephone conversations between these same customers and the company's customer service representatives
  •  Search for the names of competitors possibly contained in these documents
  • Seek out data on the company website blogs and forums, where customers have talked positive or negative about that product
  • Seek out data on the Web containing special offers or promotions offered by these same competitors
  • Seek out data on the Web containing blogs, forums, communities where people are talking about that product and the competitor's similar product. The platform is capable of doing the sentiment analysis of yours and competitors' products and this emotive analysis will definitely help the business to understand the factors of declining sales
  • Create an overall report for the top management, sales and marketing, maintenance departments with the data collected from the platform in terms of graphical visualization and data exports.


Some specific examples of this platform in Manufacturing domain can be, to detect the maintenance and manufacturing problems by processing unstructured data such as repair logs and service notes. Manufacturing organizations can also leverage this platform for global data aggregation, dashboard presentation and analysis for the Production Control Organization, analyzing data pertaining to Inventory Management, analyzing data pertaining to the relationship between various entities like Suppliers, customers, automotive parts etc. thereby enabling more effective crisis management. Organizations can also use the platform to analyze and study the market sentiments for their products and services by aggregating the web and the social media data and can have a single unified dashboard which provides them with a 360 view of their data within the enterprise and data related to their and competitor products on the social web.

This market of decision intelligence leveraging enterprise search and text analytics is massive and many traditional BI vendors have still not fully switched on to the value of this market and hence we see many fairly unknown vendors (at least unknown to the traditional BI professionals) pushing into the BI market and claiming market share. Vendors such as Endeca, Fast, Exalead, InXight, Attivio, and ClearForest etc. are all doing well in this space of decision Intelligence using Enterprise Search and Text Analytics. It is no surprise that BI vendors are starting to introduce partnerships with these relatively new kids on the BI space and we could see more acquisitions possibly occurring in the next 12-18 months. We already know that FAST got acquired by Microsoft around 2 years back and lately we have seen Endeca being acquired by Oracle as a major game changer in this space. Watch out how this market grows in leaps and bounds in the coming months & years and how major players like Oracle, IBM and Microsoft etc. strategize to acquire the huge potential of unstructured data. 

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.

Subscribe to this blog's feed

Follow us on

Blogger Profiles

Infosys on Twitter