Enterprise Information Management

Cloud BI Scenarios:

2012-01-27T16:27:07Z

Given high data volumes & their movements in BI and DWH space, ever wondered on where you will deply Cloud in BI? I came up with some quick reference guide on the areas where we can apply Cloud. Here are some examples (in the RCLL space):

External Data Analytics: Data from AC Nielson and IMS (medical POS data) is analysed by most of the organisations already, but mostly in silos at country level. This way the metrics and other parameters they measure are usually country-specific, leaving very little information for sharing across countries. Organisation-wide implementation of the external data analytics is gaining ground mainly in increasing the productivity, optimise costs and increase information sharing. Since there is a lot of hesitation for org-wide adoption, cloud plays a very important role in providing scalable infrastructure. Cloud also provides anywhere-access, thus making information sharing even more easy

]]> 1. Pricing Analytics: Pricing analytics can be offered on SaaS model. Retailers can easily avail these analytics by providing their pricing information to an externally hosted pricing engine. This engine operates on data provided by many other retailers, and sourced by hosting agency. This SaaS model helps in competitively pricing the products. This works well because of 2 accounts, one reason is the application is less used and expensive. The second reason is we need external pricing information from competitors, which is possible only on SaaS model

2. Package booking & Delivery information: Logistic organisations can use Cloud packages successfully, to understand how many deliveries are being booked, and how many are delivered on time, without having to resort to use of expensive WAN. If this data is available on Cloud, BI can be leveraged to understand the delivery bottlenecks and optimise the route planning etc. Many other standard metrics can be calculated and circulated across functions and countries

3. SFA on Cloud: Sales Force Automation is already available on Cloud. Packaged analytics can provide appropriate feedback to Sales Force on field about their performance, so as to assist in making amends to their plans

4. Planning, Budgeting & Forecasting applications: Many organisations still use rudimentary and time consuming applications or mere spread sheets to do this activity. PBF on cloud can tremendously reduce the budgeting cycle times, improve their productivity, and reduce costs

5. Secondary Sales information: CPG organisations can leverage Cloud to understand secondary sales from their distributors easily. In addition to the sales, they can understand other information like inventory positions etc. This helps to understand distribution & region performance

I will add more to the list later. Please keep watching.

BI on Cloud

2012-01-26T09:20:11Z

You must be aware that Cloud has been on the Top of CIOs Agenda these days. Business Intelligence and Analytics were on the top 3 of CIOs Agenda till last year, but now they appear within Top 10. If you separate them as infrastructure and applications, then BI and Analytics would still occupy the top slots.

Business Intelligence always evolves at an organisation, hence is best described as a journey, rather than a reaching a destination. The side effect for this evolution is that it accumulates lots of data over a period of time, and generates attention across the organisation. Our experience shows that data accumulation has serious consequences on server scalability and slows down future developments. Its popularity among the other functions across the organisation attracts more data usage and building extra data marts/ data pens etc.

]]> The above paragraph discusses only the server volumes and the consequences of exploding data volumes. Imagine the yearly license payments for the tools, version upgrades and migrations involved. All these are going to cost a lot of money. At times, some tools are used very rarely, but they are very important. How to handle these situations without making exorbitant license payments?

Another dimension to this complex situation is the availability of the qualified, but scarce, BI & Analytics Consultants to work both on continuous and on-demand basis for the ever growing appetite on BI. Sometimes, the organisations need highly qualified & expensive consultants for short spurts only. Multi-national Corporations usually work on new developments in "develop-once, but deploy/ adopt-many" mode. Adoption of these sorts of applications sometimes takes years. So, investments made for global applications get under-utilised for many years.

In the above paragraphs, we discussed about lack of server scalability, under-used expensive tool license payments, lack of qualified consultants & skills, under-utilisation of global program resources etc.

Economic conditions demand that Retail & CPG Organisations need to constantly engage with their customers and generate interest to purchase products. But at the same time, the organisations should be able to keep their costs low. Cloud can redress all the above-mentioned issues. One can start with a small hardware, and then expand it seamlessly as their needs increase. Similarly, it allows re-sizing the development & testing environments, when required. Rarely used expensive tools can be engaged on pay-per-use basis. Expensive consultants can be moved to off-site and engaged on-demand basis. Additional benefits offered will be the accompanying security, cheaper disaster recovery and backup facilities. This also brings in anywhere access, rather than relying on organisation WAN. Cloud BI and Analytics is most applicable in applications dealing with external data like from IMS, AC Nielson etc., where international trends can be studied, leveraging successful campaigns, and easy exchange of global information.

Cloud may face some hosting challenges in the area of Data Privacy regulations. However, these can be overcome with the help of data masking applications for offsite development.

Start up to MDM (3).....

2011-12-09T16:41:39Z

Understanding the organisation's information / reporting needs and categorisation of the same into Subject Areas / Information Areas gives a good start into MDM structuring. Next we need to label the subject areas with respect to the organisation parameters like business units, geography, and the hierarchy of people it is supposed to service.

]]> An important flag such as "Local Master Data Standardisation" and "Global Master Data Standardisation" is also required for the subject areas. This flag indicates if the master data required in that subject area has to be standardised across the organisation or just at the parameter level indicated on the flag.

A BI Roadmap (Framework Definition) exercise would give the prioritisation to the subject areas for build and deployment.

Next, we should create a conceptual data model for every subject area indicating the important master / reference data entities.

To have a visual idea in terms of which subject areas are supposed to service which hierarchy level, it is better to arrange them in two levels. The top level indicates organisation wide subject area, whereas, the bottom level indicates the local organisation. If there is any subject area that spans both the levels, then we need a master / reference data translation layer between them.

Since the topic is "Start up to MDM", I am ending this blog here. Please connect with me for more information.

Start up to MDM (2)...

2011-11-28T12:35:46Z

Before we jump on to the MDM bandwagon on a full-scale, we need to actually take a step back and think. If you look at the inners of any MDM application, what do we get to see? We see data extractions, data integrations, transformations, cleansing, standardisation, and data loading...., right? Where do we see such stuff? We see all these in the larger picture of Business Intelligence or Information Management. So, MDM has to fit in the overall picture of Information Management or Business Intelligence. All these BI or IM help in Decision Support for Business Users, right? So, why not look at it from the top down and bottom up approaches in terms of information requirements at various levels or across the organisation. You will have information / decision making needs at various levels of the organisation like: Top management, Business Unit level, Analytical level, and Operational level. I added Operational level included here, as BI / IM is becoming omnipresent in most of the organisations.

]]> But vertically slicing up the organisation is not good enough, as it doesn't give the flavour of what is required at each level. We only know that a different data granularity is required at different levels. We need to use these slices to understand the different data granularity as well as hierarchy needs of the organisation.

If we are able to do horizontal slicing of the organisation into different departments / functions and other subject areas, it gives clarity on what master data is required for different subject areas.

By extending the horizontal slicing to the vertical slices already done, we can understand the different parts of the organisation that are using different versions of master data. Then we need to map the existing source systems / OLPT systems associated with the generation of data within every subject area.

After that we need to examine the need to either correct the data at the source, or just bring in de-duplication only there. We also need to work out a business case associated with every source system for either de-duplication or complete standardisation of master data. The first statement is associated with the Operational ease; the second is associated with major transformation.

Please watch this space for more ......

Start up to MDM

2011-11-23T11:30:08Z

When an organisation takes up a Master Data Management exercise, we expect the whole organisation to adopt the new Master Data / Reference Data Codes. How correct is this? What happens to the part of the organisation which is still dealing with day-to-day operational activities & related challenges? What happens to the part of the organisation that is still operating Mainframes and other legacy systems?

]]> What happens to the people who are used to in dealing with existing set of item / SKU codes? Do the operational level people have to be re-trained at considerable cost? Do the mainframes and legacy systems have to start using the new SKU codes? Well, all these involve considerable expenditure. Software is supposed to improve the productivity of the organisation, and not discomfort the operational level people. All these questions have to be answered fully before embarking on the MDM exercise or while including Operations People.

Does it call for different sets of Master data for different parts of the organisation, depending on whether we are able to bring in changes in OLTP systems or not? When we maintain different sets of Master data, do we still need to a bit of cleansing and building a proper hierarchy among them?

We deliberated on multiple questions so far. But have we looked at the business need for doing this. I think we need to envelop the technical solutions with business needs, then only we get proper answers, and also sponsorship to take up IT projects.

Things have to be answered in business speak first. Please watch this space for more information on what we delivered .....

BI & MDM in CPG Demand Forecasting (3). . .

2011-09-07T18:08:53Z

This is a continuation to the blogs dated 18^th and 22^nd Aug2011. We talked about the business requirements in the first blog. Then we talked about how we plan to address the Master data hierarchy requirements in the second blog. In this blog, I am going to cover the most important aspect of the business that is margins / profit.

]]> When it comes to margins and profit analysis, we first needed data on the costs and sales prices. The costs associated with this organisation vary with different production plants, as their raw material procurement is done from regions closer to their production plants. Their operating costs also vary depending on the plant age and the types of equipment used. This organisation uses Standard costs in their main ERP, which remain more or constant for a good part of the fiscal year. This has reduced the complexity a lot. Sales prices are captured in various price lists in their main ERP. Here promotional price lists are very important, as the volumes change drastically when price promotions are introduced. We automated the data extraction of SKU level Standard costs, Sales prices and Promotional prices using ETL with CDC feature.

We provided only the sales prices to the Forecasting application, as we need to know what kind of SKU volumes are required whenever price promotions are run. We decided to hold the cost prices within the BI Data Mart, as they are not going to influence any forecasting figures. Margin calculations are important to perform, but we kept these calculations within the BI Data Mart. This way we have partitioned the work between Forecasting application and the BI data mart. We took the feed from the forecasting application into the BI data mart to arrive at the forecasted margins across customers, regions, depots etc.

Customer was using invoices and credit notes information to arrive at their revenues. On detailed enquiry, we found that invoices are used to bill the customers based on the put up demand, and credit notes are used to return the excess money to the customers as the deliveries fell short of invoice quantities. Credit notes are also used as a goodwill gesture, but these are not quite often. 90% of the credit note amounts are for delivery shortfalls. Hence, the invoice data worked well for us, as it represented the customer demand, and also represented the adjusted sales prices, as these sales prices are sometimes different from the price lists. It is a good situation to be in as the demand is always higher than supply.

Please see this space for more on the remaining solution.........

BI & MDM in Demand Forecasting . . . .

2011-08-22T13:03:41Z

This is the continuation of the blog on "BI & MDM in CPG Demand Forecasting "dated 18Aug2011. I am going to address the problems we saw and the solutions we offered, one by one.

Let me start with the multiple hierarchy requirements. The SKUs are supposed to be rolled up as per the Product Group (main one: based on type of material), Brand (different names for different customers, regions), Sub Brand (based on grade of the material), Packaging size (based on the quantity packed), Business Unit (whether selling directly to big retailers, retail consumers etc.), Category (manufactured / bought), Recipe (to indicate main ingredients), and SKU Case (trollies, packs, SKU etc.). A huge number of hierarchies as you can see.... These are the ones recognised so far. There are many to be defined yet.

]]> We understood that most of the OLTP applications store only basic details of SKU and their case sizes only. For the same SKU, the product codes are different across different OLTP systems. We also noticed that some of the hierarchies are non-existent in any of the OLTP systems. After speaking to the business users we came to know that only 3000 of the 4000 SKUs are active. This led to many unknowns, but there was a silver lining!!! That is they all need to be defined afresh.

Now, who can define the hierarchies? We turned to the business users to start defining the various hierarchies, which are just 3 or 4 levels only. This proved to be easy and was also done fast, as it was almost like defining from scratch. Then then mapping of the SKUs to the hierarchies was left. This proved to be a slightly difficult affair, as there has to be a single set of standard SKUs that needs to get into the BI data mart before going to the demand forecasting application. We took the latest application in the client organisation that had all the SKUs, and called them standard SKUs. Distributed the responsibility of mapping the standard SKUs and their hierarchies, based on their area of operation & business unit they work for. We created multiple templates using spread sheets and cleverly linked them to capture the hierarchies first, then link the SKU to them, and used the same to fill in the important attributes for the SKUs. Remaining attributes were completed by the concerned business users.

The client was not interested in maintaining the chronological changes taking place to the attributes, because any changes to the attributes is considered to be a new SKU. There is also not a huge team to create new SKUs, and maintain them. This led us to consider a simple Master Data Maintenance solution. The spread sheet templates were loaded into the BI Data Store using ETL scripts. The ETL scripts were built with intelligence to recognise the changes to the existing data and warn the business users to accept them or reject them. Data for the new SKUs went thru after basic validations. The whole solution of Master Data Maintenance was maintained using a relational schema, as this offers a flexible way of defining multiple hierarchies and also help in defining new ones easily.

This way we avoided implementing an expensive Master Data Management application. This is a very simple MDM (maintenance) solution accepted by our client, as it involved spread sheets and ETLs only, and no data extraction from any OLTP system. A small Data governance team was set up to monitor, maintain the MDM, and also coordinate with all the concerned stakeholders.

Please see this space for more on the remaining solution.......

BI & MDM in Demand Forecasting

2011-08-18T17:51:01Z

Demand Forecasting in CPG organisations is gaining more importance these days to forecast their customers' product off-take. This is about one of our Retail CPG customer's decision to adopt a Demand Forecasting application to meet all their forecasting needs.

This is about the same customer about whom I described earlier as:

]]> "This organisation does a bit of process manufacturing, supplies products to retail customers, supermarkets, mom & pop corner shops and other high street coffee shops. Retail customers can purchase their products either online or offline, based on their convenience. They also resell lot of 3^rd party products. Based on the products they supply and the customers they service, it has been divided into multiple business units. Accordingly, they have numerous applications in place, from AS400 systems to Oracle e-Biz Suite to handle their business processes. It is interesting to note that there is a common raw material / ingredients used in all of their products."

I am giving the complexity of the requirements here:

· This customer is interested in understanding the product off-take at different customer levels and use this information in arriving at actionable forecasts. Since the retail customers are in excess of a million, they grouped them under their regional servicing areas (or) depots, and reduced the forecasting complexity

· They have about 4000 products / SKUs, and are adding more products / SKUs frequently. They are grouped under different hierarchies, like Product Group, Brand, Packaging type, Package size etc.,. Different hierarchies are used for reporting by Finance, Marketing, Sales and Operations. Despite multiple existing hierarchies, they want to keep the option of adding new hierarchies in future

· Customer is interested in understanding the present off-take, forecasted off-take, gross & net margins, customer margins, region level off-take, region level production requirements, distribution fleet utilisation, hand held devices' utilisation, people utilisation, product shrinkage, through various reporting

· Different price-lists are used for different customers, adding to the complexity. Even the raw material standard costs are different across their production sites

· At depots, to avoid distribution shortfalls, they resort to emergency purchases and product substitutions. They are interested in understanding the impact of substitution products on their customers' purchase behaviour, and ultimately reduce the expensive emergency purchases

To address the above mentioned complex requirements we actually devised a simple solution. We have consciously avoided implementation of MDM (Master Data Management), heavy duty BI, even more complex configuration in the Demand Forecasting Application. We also brought in their business users participation easily.

Please see this space for our next blog on the solution .....

Where is the missing order data?

2011-05-08T23:39:58Z

How will you react when the business complains that there are few million dollars order information missing in the DWH?

In a typical data warehouse scenario, the data from the operational systems is loaded into the data warehouse (DWH) system on a regular basis. The data load is usually incremental, particularly when the data volume involved is huge. The DWH provides the data to reporting systems.

In a large retail enterprise, there is an enterprise DWH system, which is architected in the above manner. ]]>
The business users run a reconciliation process at the end of the month, against the source system and the reporting system. During the reconciliation process, there was a huge mismatch to the tune of couple of million dollars. The business team gave the report to the IT team asking 'where are the missing orders?'.

The reconciliation process was tweaked to query on daily data and surprisingly, all the mismatches were on a single day. Hence it was concluded that the mismatch was not a continuous daily issue, but isolated to a single day. All the DWH load jobs for that day were checked and all of them had completed successfully. There were no issues found in the job logs. So there was no operational issue with the DWH load jobs.

The next step was to analyze the data sent by the source operational system. The missing order data was indeed not pulled from the source system. It turned out that the order was created in the source system with a back-dated entry. The data pull was designed in a way to pull only the last 5 days data, while the backdated entry was made for a 7 day earlier date and hence the data was missed.

Thankfully, the issue was found during the reconciliation process. If not, the consequences could have been lot more complex. This issue became a good opportunity to persuade the business team to fund for a data quality audit process between the source operational system and the DWH system, which will audit a fortnightly data. The data audit will be auditing on fortnightly data, so as to account for the backdated corrections. The process for backdated corrections were also revisited and redesigned to include a notification to the downstream systems including the DWH.

Enterprise 2.0 - A Promise yet to be delivered?

2011-05-05T05:43:42Z

With the risk of sounding repetitive and stating the obvious...let me begin...by suggesting that this may NOT be a closed and sealed topic...And my recent interactions with business and IT leaders of some leading global organizations confirmed this point.

Right...we are living in an age where social networks play a big role in the rise and fall of political leaders, ideologies and institutions...we all vouch by the significance of this phenomenon in our daily life...we all have our virtual identity as our alter ego. But when posed with the question of Enterprise Social networking, Shakespearian style soliloquies and musings continue to live on in the minds of Senior Management..."To B(log) or Not to B(log)...that is the question..."

Is this too powerful to be contained within the enterprise? (Is 'contained' an old economy expression...That is yet another discussion). Does it appear to be too destructive (to the corporate fabric) and distractive (to the corporate citizens)? Is Enterprise 2.0 trying to wish away Social Networking - its very own bedrock?

May be it is time to take a relook into the whole phenomenon....

There was a time when Enterprise Social networking was all about throwing in a set of the so called "Enterprise 2.0 applications". We all have gone past those ages of infancy. We have started to appreciate the deeper facets and cultural hues of Social Networking and that is all to do with the way people communicate.

In this new paradigm of collaboration, an organization's "information fire hoses", transforms into "Channels of Expression". The seemingly casual, huddle formation but very relevant and useful exchange of information that normally happen at the water fountains now becomes the norm. Likeminded people flock together and start to express freely and Interesting topics attract user attention and people pour their hearts out. Discussions change course through the conversations...really interactive and participative. In this construct 'Organization' became incidental and the organizational hierarchy breakdown. Wear your creative hat for a second and try to visualize a day in this 2.0 Enterprise - You meet the CEO at the water fountain almost every day and even the newest employee get an opportunity to walk the corporate corridors. Right...in this world 'Personalities', 'Topics' and 'Need for Self Expression' take the center stage.

So let me go back to the earlier question...Does this appear to be too destructive (to the corporate fabric) and distractive (to the corporate citizens)? With all the perceived benefits around the "breaking down of Information asymmetries", "Employee Engagement" and "Knowledge Sharing" it IS in a way triggering off '"too many changes too soon". And this IS the cause of concerns for the wannabe Enterprise 2.0 CIOs. The organization needs to be primed to handle this cultural shift. The organization need to transition though a set of changes based on its state of affairs and priorities.

If we look at Social Networking or Enterprise 2.0 as a dynamic and synergic co-existence of 3 key entities - People, Purpose and Information (People, Intent and Content), on a framework that enables 'Self Organization' and 'Participation' the jigsaws suddenly falls in place.

This forces the CIO to ask the question, so what are we trying to achieve? What is my priority?

In a globally distributed organization, the primary need may be to keep the employees connected...
In a family run business, the priority may be to drive participation and employee engagement...
In a knowledge based organization, guiding the people to the right information or to the right experts is what is required..

Thus a road map based on the organiational priorities would make this transition and transformation into a true Enterprise 2.0 organization more seamless and fruitful.

This transition does not happen overnight. The tranformation is more cultural and as we always say technology is an enabler. And with the right Enterprise Social networking strategy and roadmap, the organization seamlessly matures into a Smart enterprise

Search & Insight Driven Computing - Celebrating the Joy of Creation

2011-04-26T09:00:22Z

3 years and a ton of learning & insights...a good milestone to take a pause, reflect on our experience, relook at the vision and rationale behind our venture and...get ready for the next dash. Yes..."Search & Insight Driven Computing" one of our prized incubations within the Information Management space turned 3 this year.

Now for the curious and the new entrants into this space, let me explain what this "animal" is

Search as a "technology" is revolutionizing the way Information would be Organized Managed and Used within an enterprise.
Search & Insight Driven Computing team is focused on leveraging this technology to build the Next Generation information Management Solutions.
We are one of the very first System Integration firms to have a set up full-fledged service line in this space and that way we are ahead of the curve in this area.

We started this incubation almost 3 ½ years back and now this Service line has successfully moved out of its incubation phase with a bouquet of case studies, multiple account openings and very strong GTM relationship with the leading product vendors in this space

Now, you may ask what led us into this...A few trends that we saw in the Information Management space, almost 4 - 5 years back, caught our attention and those and, in a way, formed the key rationale for the formation of the Search Practice.

Let me try to recollect them for you...

Trend 1: Organizations had started to realize the value hidden in the unstructured information

It is a known fact that 85% of the information within the firewalls of an enterprise is in the unstructured form. It is more interesting to note that most of the IT investments over the last 4 decades and the entire information management industry had been focused on building solutions to address the structured information. The opportunity and possibility to open the vault of unstructured information and unlock the insights from that excited us and we were thrilled to envision the possibilities that would open up for an organization like ours.

Trend 2: The enterprise boundaries had started to blur

We saw the "post dot com" world slowly crystallizing around "consumers". Organizations were (and are) trying to be more "consumer-centric" or outward focused. The concepts of Web 2.0 and Enterprise 2.0 were emerging and with all these the information that is outside the firewall was becoming significant and there was a need to access and leverage that.

While these were going on there were some interesting development within the Enterprise as well

Trend 3: "Formation of Mini Internets within the Firewall"

Within the enterprise space, due to obvious reasons, we were witnessing the growing heterogeneity, replication and fragmentation of information. The enterprise information domain had started to behave like a "mini internets" within the firewall. For any information architect the challenge would be to see how the most relevant information can be provided to the business users comprehensively and at the right time.

And successful information management frameworks would be ones those are "aware of this chaotic situation and can address that"

Meanwhile on the Technology front a new kid had entered the block, changing the way we looked, accessed and managed information

Trend 4: Technology Trend: Search has transformed from a keyword lookup mechanism into a contextual information integration engine.

We see this as a journey that would involve

Seamless Access to Information
Enrichment of Information and User Experience
Extraction of Insights &
Applications on top of that

Based on our perspective the potential impact this approach would have on the Enterprise Information Management space is very similar to what RDBMS created 4 decades back.

We look at Search as a unified information access platform that would lead the users to the right information
Search as a virtual data warehouse will help users slice and dice information in a rather contextual manner
Search can be your Knowledge Platform that would enrich information through contextual content association
Search will eventually be the gateway for all user interactions and will have a pivotal role to play in user experience management

It is quite heartening to see how valid these thoughts are...even now and to move on with our onward journey

Complex Retail Architecture . . .Our search continued

2011-04-11T16:04:26Z

Continuation of the blog published on 25 Feb'11.

Our search for solution / salvation continued. . . .

We once again looked at the Option 1. This option is very much valid given a green field implementation. This follows a classical DWH implementation, but the reconciliation mechanism will be like quids chasing pennies. Then we looked around for the work that is already done, by going through our Minutes of Meetings and other customer supplied documents.

We realised that the customer had already implemented Vendor Spend Analytics, Customer Web Analytics, Spare Parts Inventory Analytics and Profitability calculations using some old / proprietary technologies. Repeating these in the Option 1 is not going to be helpful, other than consuming the precious budget.

]]> We looked at the organisation set up, and realised that the organisation is divided into multiple divisions based on the customers they service. We also have to leverage the existing implementations on old / proprietary technologies. We hit upon a new solution, called it Option 4. This option is a ground-up decentralised DWH, servicing different business functions. Data belonging to different business divisions can be stored by sub-typing. We called these as Federated Data Marts servicing different business functions, but not business processes as in an ERP.

We needed standardised transactional data and aggregate data for analytical purposes. We also needed integrated understanding on the data held in different data marts. Integrated understanding on multiple data marts is possible through standardised master data. Well, I cannot divulge the whole solution here. Please contact me if you need more details....

Making the Right Decisions in this Digital World

2011-03-21T04:07:13Z

"How can we help people make the right decisions - choose the right options in this constantly changing world?" Sounds like a noble dream... a philosophical question...A good case for a modern age Sidharta to leave the royal pleasures in search of the answer...Well, almost...Last week 500 technologists, business leaders, analysts and psychologists converged at the fringes of the Arizonian desert to reflect on the same question...sharing their experiences, observations, opportunities and insights

The event - Discover 2011 - the annual User Conference hosted by Endeca Technologies, Inc., a search and business intelligence (BI) company featured dozens of customer-led sessions, that deliberated on how the simplicity of Search combined with the power of BI can transform their organizations by enabling business users, partners and end users to make better decisions.

The sessions opened with a bang with Paul Sonderegger (Chief Strategist & Ex Principal Analyst at Forrester Research) who set the right context to the event...He painted a picture of

A world that is getting more digitized every day
"Abundance of choices" and the resulting "Scarcity of attention" faced by fellow human beings in this Digital world
The importance of sifting through the ever growing, abundance of data and to "Guide" the users in their daily decisions

This was followed by and that too quite seamlessly by another insightful talk on "How we take decisions". Jonah Leher author of best seller "How we decide" took the audience into the realm of neuroscience to explain the complex events happening in our own 3 pound supercomputer - Brain - during our mundane day to day decisions, like picking up a box of cereal from the store shelf... How the cognitive centers and centers of emotions work together in guiding us to our choices...

Let me take a pause here...Shouldn't the "Information workbenches" mimic or facilitate this process? Every application that we build thus becomes an information exchange that will help users to experience predictability in their daily transactions.

On the same note, it was quite heartening to see a world that we visualized...a strategy that we charted out almost 4 years back, turn out into reality. This approach called for a fresh look at information management and attempted to perceive next gen information management as

An art of intelligently delivering the most "relevant" information to the user for that specific context

An infrastructure that can

Connect with the User,
Sense the potential intentions
Predict and Persuade the users to the desired outcomes

A realm of complete user empowerment, where the users interactively discover their needs and choices

Interesting isn't it...Now the questions in front of us will be, "Will this impact the IT fabric of an organization?" or "Will this change that way we design and develop applications?" The short answer is "Yes"

The locus of control will shift from Information to the User and User experience will have to be addressed and managed comprehensively
Analytics and Digital Channels will converge - The technology domains, those were kept in administrative silos rather artificially will come together to address the need of the Digital User, whether it is an employee or a consumer
Next Generation Information management initiatives will call for even closer functioning of Business and IT
Explosion of customer touch points will mandate organizations to develop "Seamless and Consistent" Customer Experience Management as a core capability

Complex Retail Architecture

2011-02-25T10:51:39Z

I am giving an interesting anecdote here, based on my recent BI roadmap experience at a Hybrid Retail Organisation. This organisation does a bit of process manufacturing, supplies products to retail customers, supermarkets, mom & pop corner shops and other high street coffee shops. Retail customers can purchase their products either online or offline, based on their convenience. They also resell lot of 3^rd party products. It is quite an interesting portfolio of products and services from this organisation. Based on the products they supply and the customers they service, it has been divided into multiple business units. It is interesting to note that there is a common raw material / ingredients used in all of their products.

This organisation has dozens of depots to service far flung customers, many central warehouses to supply the depots, major customers, and also store low shelf life and high shelf life products. It also maintains a huge fleet of vehicles to help in raw material collection and FG distribution.

]]> As it is an old organisation, they have numerous applications in place, from AS400 systems to Oracle e-Biz Suite. All major Business processes like "Procurement to Pay", "Manufacturing to Stock", and "Order to collect" are executed across multiple applications. It is like, individual steps in each business process are carried out in a variety of applications, and then they are all pooled together in financial application for inter-unit financial reconciliation. For example, purchase orders are placed by an application, material (GRNs) is received in another application, and vendor payment is made through another application. For each business unit, they maintain different AR, AP and GL modules. At the group level, they have kept the Oracle Financials. They transfer all the AR and AP transactions from individual unit to the central Oracle Financials' respective AR and AP modules.

The entire organisation is stitched together by an army of Accountants, as the units supply products to other units through transfer price mechanism. They work on Standard cost basis, and the variances are allocated to the various products, at a later point of time.

As any other retail organisation, this organisation also works on wafer thin margins.

We observed many issues on reporting requirements. They are given below:

· Reports are produced manually, without any drill downs in them. Most of the reports are done using spread sheets

· Reports, KPIs and Metrics are designed and created in a hierarchical manner, starting from raw material procurement, process plants, FG stocking areas, distribution, order management and collections

· After that the financial & management accounting reports are produced at the group level, by circulating a set of spread sheets to collect inputs from the various business functions

· They also calculate margins for different products, by using a series of Essbase applications, MS SQL server databases and spread sheets

· They are also doing financial planning and budgeting exercise on a series of Essbase and OFA applications

· With all these set of manual reporting activities, it is quite difficult for the senior management to obtain useful information quickly, and be in a position to understand the root causes from the high level consolidated reports. They need to call the concerned people across the organisation for more details

· Their SKUs / product codes are different across the different units, hence FG reconciliation gets difficult for this geographically spread organisation

We came up with multiple BI architecture options:

· Option 1: Unified DWH with layer of OLAPs to provide analytics, Planning & Budgeting and MDM applications. This has to pick up data from multiple systems across different business units, then string them together with a single ODS, and then aggregate the data into analytical OLAPs.

· Option 2: Decentralised DWH as per their business units, then with a layer of OLAPs to provide analytics, Planning & Budgeting and MDM applications. This option offers multiple ODSs, one for each business unit, then aggregate the data into analytical OLAPs.

· Option 3: Hybrid DWH, combining Option 1 and Option 2, so as to offer operational reports as well. Then include Planning & Budgeting and MDM applications. This starts with individual ODS for different business unit, followed by a unified ODS, then followed by aggregated analytical OLAPs.

We were not happy with all these options, as Option 1 involves an expensive proposition, as one needs to build it in a natural data generation sequence. More than that, it would involve an expensive reconciliation mechanism to manage all the small retail transactions. We felt that this would be unnecessarily complex and expensive for the requirements.

For the Option 2, we were not happy again, as we will be increasing the complexity of data movements from one ODS to another, as there are common production sites and warehouses. We felt this is going to be even more complex and expensive.

Option 3 is likely to exhaust all the budgets by the time the individual ODSs are built, even before the unified ODS is built.

Our team was not happy with any of the above mentioned options. We wanted to break out from the old BI thinking. We thought a lot about the solution, went thru our minutes of meetings to better understand the DNA of organisation, then came up with a better, inexpensive, easy to build solution.

Please see this space for our next blog on the same topic.....

Design considerations for a Data Quality Framework

2010-12-08T06:11:34Z

When an organization decides to implement Data Quality strategy, there are several design patterns that can be considered. The data quality goals should be the key driving factor in choosing the design factor.

]]> The first step would be to capture the Business Rules that are critical for each of the key systems. Each of the business rules may need to be implemented as multiple data quality rules. The below diagram provides a list of the different types of the data quality rules that can be defined. You could notice that each type of rule can be of different technology and can be used for different type of systems including operational, data warehouse and data marts.

Multiple types of the same Data quality rule: The data quality rule needs to be defined in multiple variations at the design stage itself. The different types include the multiple variations for batch version and online version. There need to be multiple batch versions which can perform data quality checks on the complete data set, incremental data or randomized sampling set. The complete data set validation would be the preferred approach. The randomized sampling variation can be used in data warehouse environments whenever there are performance constraints or the time window available for the data quality check is low.

Data Quality as a Service: The on demand Data quality rules can be used by the Operational systems. These rules can be exposed as web-services and can be offered to the data suppliers. For e.g. the order capture information from the resellers can be validated even before the resellers send the data. These rules can be used to monitor the data available in the websites of the customer organizations. For e.g. the product catalogue information, which is sourced from the enterprise, available in the website can be randomly or regularly monitored using the data quality web services. By extending the data quality rules even beyond the enterprise, the quality of the data flowing into the enterprise and its usage even beyond can be assured.