The Infosys Labs research blog tracks trends in technology with a focus on applied research in Information and Communication Technology (ICT)

December 16, 2013

Data Virus Guard

Clients are, or soon will be, ingesting all sorts of data thanks to information brokerages and the Internet of Things (IoT) and processing that data in novel ways thanks to the Big Data movement and Advanced Analytics.  Decisions made through business intelligence systems require that the data being used is trusted and of good quality.  How will companies ensure that the data being ingested and acted upon is untainted?  This has been an interest of mine as I work to protect the integrity of my clients' decision making processes and systems.

Last year I shared a forward looking concern about the concept of a data virus: data that has been purposefully manipulated to render operations on an entire data set flawed, and it perpetuates its induced error. As noted in the original What will you do about a Data Virus? blog, a tricky situation arises when data fed into the enterprise is determined to be corrupted.  How do you unroll all the down stream systems that have made decisions based on the bad data?  Maintaining this data contamination is tricky.  Many legacy enterprise systems simply don't have the ability to "roll back" or "undo" decisions and/or persisted synthetic information.  So, the first and obvious line of defense is blocking, or sequestering, suspect data before it enters the enterprise.  Much as a network Firewall blocks suspect requests to ports or machines in your network, a similar concept can be employed..... a Data Virus Guard if you will .... in many situations as a first line of defense.

Please keep in mind that my focus has been on streaming sources of data, which are typically sensor based (maybe a velocity reading, or temperature, or humidity, or ambient light, ...) and associated with a thing (a car, train, or airplane for example) and comes in for processing in a streaming manner.  What I'm sharing in this blog could be applied to other kinds of "streaming" things such as feeds from Social Web systems, for example.

What is a Data Virus Guard? 
A Data Virus Guard is a logical unit that has the responsibility of identifying, annotating, and dealing with suspicious data. 

Where should a Data Virus Guard be deployed?
A Data Virus Guard should be deployed at the initial ingestion edge of your data processing system, within the data capture construct.  The data capture sub-system normally has the responsibility of filtering for missing data, tagging, and/or annotating anyway so it is the perfect location to deploy the Data Virus Guard capability.  If you identify and contain data at the "edge", then you run less risk of it containing your enterprise.

How do you Identify a Data Virus?
This area of the Data Virus Guard is what drew my research interest .... how do you go about discerning between normal data and data that has been manipulated in some way?  The approach that I've been taking is focusing on steady state data flows because I'm interested in a generalized solution, one that can work in most cases.  If one can discern what constitutes steady state, then deviations to steady state can be used as a trigger for action.   More elaborate, and case specific, identification approaches can be created and placed easily with the framework I'm proposing.

What kind of Annotation do you do?
As data enters into an enterprise, ideally there is meta-data that helps with maintaining data lineage.  That is, what was the source system that produced the data, what is the "quality" of the data, when was the data generated, when did the data enter the enterprise, is it synthetic (computed versus a sensor reading), etc. etc.  Added to this could be an annotation that indicates which Data Virus Guard algorithm was applied (model, version), and the resulting score of likely suspicion. 

How would the Data Virus Guard deal with suspect data?
Based on the rules of your data policies, the data judged as suspect may be set free to flow into your enterprise, discarded as if it never existed, or kept in containment ponds for further inspection and handling.  In the former case, if you let it in the enterprise and it was annotated as suspect, when data scientists work with the data, they will see that it is suspect.  If you have automated algorithms that make decisions, they could use the suspect score to bias the thresholds of making a choice. 

What are characteristics of a Data Virus Guard?
In the search for "the best ways" to guard against a data virus, a few criteria have popped out to make the system practical.  Firstly, it has to work on all common types of data.  To be truly useful in an enterprise setting, the Data Virus Guard can't work with only strings or only integers, it must work on all common types to provide true utility.  Secondly, its determination of suspicious or not data must be very fast.  How fast?  As fast as practically possible as the half-life of data value is short. This is a classic "risk vs reward" scenario, however, and can be done on a scenario by scenario basis.  Thirdly, it must have the ability to learn and adjust on its own of what constitutes normal, or not-suspicious, data.  Without this last capability, I suspect enterprises would start strong with a Data Virus Guard, but then it would find itself out of date as other pressing matters would trump updating the Data Virus Guard with the latest Data Virus identification models.  In summary, it must work with all types of data, it must be fast, and it must learn on its own.

How would you implement a Data Virus Guard?
Putting together a Data Virus Guard can be a straight forward endeavor.  By blending a stream processing framework with a self-tuning "normal" state algorithm, it would be possible to identify, and annotate, data flows that deviate from some norm (be it values, range of values, patterns of values, times of arrival, etc.).  One could envision a solution coming to life by using, for example, Storm, the open source streaming technology that powers Twitter, and a frequency histogram implemented as a Storm "bolt" (the processing unit of a Storm network) to discern out of norm conditions.

Admittedly, the usage of a frequency histogram would create a weak Data Virus Guard, but it would get the Data Virus Guard framework off the ground and be easy to put in place.  However, by using Storm as the underlying stream processing framework, swapping in a more powerful "out of norm" algorithm would be relatively easy.  Do you go with a Markov chain, a Boltzmann machine, or even the very interesting Hierarchical Temporal Memory approach of Numenta? This would all depend upon your system, the characteristics of the data you're ingesting and the amount of false-positives (and false-negatives) your enterprise can withstand.  Of course, you even go further and apply all three of the approaches and come up with some weighted average for discerning if some piece of data is suspicious. 

Summary
This is a forward looking post about what we can expect to be issues in Enterprises as all companies embrace the concepts of Big Data, Advanced Analytics, the Internet of Things, and true Business Intelligence: a Data Virus, and what we can do about it: a Data Virus Guard.  My work in this area is still evolving, and is intended to keep our clients a few steps ahead of what's coming.  Bad data plagues all enterprises.  It can be incomplete, malformed, incorrect, unknown, or all of these.  Unfortunately, we now also have to watch for malicious data.  Putting in safeguards for this condition now before the malicious data issue becomes rampant is a much cheaper proposition than re-hydrating your enterprise data stores once a contamination occurs. If nothing else, if you don't implement a Data Virus Guard, be sure you have your data policies in place for addressing this coming issue.

October 11, 2013

Can formal requirement methods work for agile?

By Shobha Rangasamy Somasundaram & Amol Sharma

Formal methods adapted and applied to agile, provide clear and complete requirement, which is fundamental to the successful build of any product. The product might be developed by following Methodology-A or Methodology-B, which changes very few things as far as knowing what to build goes. So we could safely state that the development methodology used by the project team could be any, but good requirements are absolutely necessary. The manner in which we go about eliciting and gathering requirements would differ, and needless to say, this holds true for agile development too.

Continue reading "Can formal requirement methods work for agile?" »

September 30, 2013

Ten Challenges of a Chief Innovation Officer

It is very easy to allocate funds and develop a strategy for innovation within an organization, however it is very difficult to make it successful and keep the interest of innovators. If innovation program is not managed well then it loses its importance in very short period of time. While implementing innovation program, there are many challenges such as operational, financial and resources, etc. However, ten main challenges for sustainable innovation management are discussed as following.    

1. Innovation Governance is the key for major decision making, such as acceptance of an idea for incubation, funding of an idea and marketing of finished product or service. Innovation governance should have decision makers from various functions of an organization as well as from outside of the organization. It should also have subject matter experts. This varied group helps to select the best ideas for incubation. They also decide the funding required and helps to calculate the risk associated with the innovation idea. The challenge for innovation officer is to define most effective governance structure and proactive members for various roles of governance.

2. Innovation Team is essential to execute the innovation program. This team must have entrepreneurial mindset. Thus, they have to be psychologically assessed on their entrepreneurial skills before they are selected for various roles of innovation office. These team members should have varied experience and skills. Identifying such team is a serious challenge for chief innovation office as entrepreneurial quality people are few.

3. Innovation Process has to be very simple and transparent. There should not be delay/gap between two connecting processes. For example, after submission of an idea by innovator, its result has to be declared in very short time. Otherwise, innovator usually loses interest and rarely pursues idea to the next level. Also, chances of innovator to participate again are rare. Thus, innovation processes should speed up the procedure and not to hurdle the innovation program. Also, numbers of processes have to be minimal. Innovation office has to work like a startup. Thus, has to be more informal in processes and quick in action. Therefore, innovation officer has to manage the balance between organizational formal set of processes and informal set up of innovation office. Automated innovation program management portal is very useful for managing innovation processes.

4. Innovation Tools are essential for training of innovators. Very first time, innovators mostly fail to realize the concept of innovative idea. Thus, their ideas are unrealistic or less valuable to the organization. Many participants are enthusiastic to contribute in innovation program but may not be able to understand how to participate because they face challenges in generating ideas. Thus, innovation tools come handy for innovators. These tools help them to make their idea more powerful and useful to the organization. There are many tools available. But, the real challenge for innovation officer is how to train innovators on these tools. It is very difficult to train every interested person in class room. Therefore, best way is to use online learning.   

5. Idea Selection for incubation is the process after idea submission by innovators. In this step, every idea is thoroughly investigated for its possible business potential and investment required. As investment is required in incubation and commercialization of an idea, every idea has to be analyzed on various risky parameters. Usually idea selection model is used to quantify the potential of an idea. Thus, innovation officer has to see that idea selection model is well defined and customized according to the need of domain and organizational business needs.

6. Innovation Portfolio has to be well balanced with combination of ideas having low (L) risk, medium (M) risk and high (H) risk. More the risk better is the rewards. However, more risk also leads to chances of investment loss. Thus, innovation officer has to define initial portfolio with lower risk and has to transform it to moderate risk portfolio over a period. Thus, first innovation portfolio should look like 10-20-70; it mean that 10% resources allocation for high risk projects, 20% for medium risk and 70% on low risk projects. After transformation, new portfolio should have more projects from both medium risk and high risk. Thus, new innovation portfolio should be 15-35-50 or 15-45-40 on H-M-L risk. The challenge for innovation office is to select the best proportion of projects of various risks, which suits the organization.

7. Rapid Prototype is the key in assessing the idea quickly and also reduces time required to take the idea to market. Agile methodology has to be adopted to develop the quick prototype of an idea. It reduces the investment required and gives better picture of an idea in a short period of time. It also reduces the risk because the idea can be judged before going into full-fledged production. Survey can be conducted on prototype to understand an acceptance of idea by customers. If the idea is well accepted by customers then prototype is transferred for production otherwise idea can be scrapped, thus it reduces further investment. The challenge for innovation officer is to select tool(s) and methodology for quick development of prototype with less investment.

8. Rewards and Recognitions is the integral part of innovation program. It attracts innovators for participation as well as it is the way to recognize their contribution. Though, the overall structure of rewards and recognition has to be in parallel with an organization policy, but considering the importance of innovation and psyche of participants, it has to be tweaked to match their interest. It is essential to give rewards or recognitions at each stage of innovation process. This keeps participants involvement very high. The challenge for innovation officer is to decide the rewards and the recognition given which will appeal the innovators. Officer also has to balance the organization's rewards and recognition policy and rewards given to innovators.

9. Innovator Engagement tactics keep innovator's participation high in the innovation program. Better engagement is possible only by transparent and automated processes. Innovators also have to be supported for idea elaboration and should be allowed to lead their ideas if selected for incubation. The innovation officer has to identify and implement all possible means to keep innovators engaged. Rewards and recognition also helps keep them engaged.

10. Success Measurement of innovation program is necessary as organizations are investing into program.  Hence, they are interested to know how innovation program is effective within the organization. Challenge for innovation officer is to define the best measurement method. While defining measurement method, innovation officer has to decide the combination of non-financial and financial parameters for measurement. The non-financial parameters are very important during the early stages of innovation program. Also, it is essential to note that success measurement has to be aligned with the purpose of innovation program. Linking success only with the return on investment is most likely to showcase as failure of innovation program, which is not always true because many in-tangible benefits can be realized through innovation program.

How should an Innovation Portfolio be?

Deciding an innovation portfolio is one of the big challenges for chief innovation office. It involves deciding about which innovation ideas have to be incubated, considering the investment required and risk involved in every idea.

Every potential idea may look like a gold mine which may change the fate of the company. However, it may not be always true because there is a risk of failure in every idea. Thus, before investing into an idea, it is required to assess an idea to manage the investment risk. Adoption of portfolio approach for incubating ideas is a better way to manage the investment risk.

Innovation portfolio should typically look like a combination of multiple ideas of low, medium and high risk. We can use the analogy of A-B-C rule where, A stands for bucket of projects of high risk, B stands for bucket of projects of medium risk and C stands bucket of projects of low risk.

Organizations should have their initial innovation portfolio based on A-B-C- or H-M-L Rule. Thus, initial innovation portfolio should look like:

  1. A or H Bucket of Projects: This is a bucket of high risk innovation projects. These projects are usually radical/transformational and mostly new to industry. These projects are focused on developing new generation products or services, and usually use emerging or very new technologies. Being high risky projects, the numbers of projects are limited to few so that sufficient attention can be provided to control the risk and maximize return. Return on investment on this bucket of projects can give up to 60%, while which resources allocation can be up to 10% of total resources/efforts used in innovation. 
  2. B or M Bucket of Projects: This is a bucket of medium risk projects. These innovation projects are mostly new to organization and focuses on development of products or services those are new to the organization but known to industry. These projects usually leverage existing technologies. Considerable return can be expected from this bucket which may rise up to 25% and resources allocation can be up to 20% of total resources/efforts used in innovation. 
  3. C or L Bucket of Projects: This is a bucket of low risk projects. Resources allocation can be 70% of total resources/efforts used in innovation. These projects are usually incremental changes in existing products or services of the organization. Thus, risk is very low; hence, return on investment is also not much and may be up to 15%. Many times benefits may be non-financial or in-tangible. The use of technologies is restricted to mature and well accepted technologies.

How to allocate the project into L, M or H bucket is a critical task. Organization can have their own model for allocation of the project into their respective buckets. While developing the model, following parameters and questions can be used:

  • Novelty: What is novelty of an idea?
  • Market: What is market size of an idea?
  • Customer: What is the benefit to a customer?
  • Growth: How is it supporting the growth of the organization?
  • Allocation of resources: What are the resources required?
  • Investment: How much is the investment required?
  • Return on investment: How much is expected return on investment? Is it tangible or intangible?
  • Competition pressure: Is there any pressure because the competitor has adopted it?
    Risk: How much risk is involved?
  • Technologies Used: What kind of technologies are required to develop an idea and, are those mature?

Aggregated reply to the above questions on certain measurement scale will help to allocate project to any one of the buckets H,M, or L.

During initial phase of innovation, typically innovation portfolio may have maximum number of projects of incremental changes or low risk. These types of projects will never generate expected growth from the innovation portfolio, thus portfolio has to be transformed over a period from low risk to medium risk.

As organization matures on innovation, it should systematically analyze innovation portfolios and manage risk to increase the proportion of innovation projects having more risk and more rewards. Thus, H-M-L portfolio should not look like 10-20-70 or low risk portfolio. The new portfolio should have more projects from medium risk and from high risk. Thus, new innovation portfolio should be 15-35-50 or 15-45-40 on H-M-L risk.

Periodically or every year organization should evaluate their innovation portfolio to see the ratio of H-M-L. It should neither be in low risk zone nor be in very high risk zone. Having low risk or very high risk in innovation portfolio may always lead to failure of innovation program.

References:

  • Day, G. S., (2007). Is It Real? Can We Win? Is It Worth Doing? Managing Risk and Reward in an Innovation Portfolio. Harvard Business Review, December, pp. 110-120.
  • Nagji B., & Tuff G., (2012).Managing Your Innovation Portfolio. Harvard Business Review, May, pp. 68-74.

Disruption in TV industry

Ever since the first TV broadcast happened in late 1930s, the penetration, choice of content offered, and average viewing hours per day has gone up and so has the cable subscription fees. The turning point in the industry came in 1995 with the delivery of first ever TV program via internet. Nearly two decades after that, internet TV, VOD (video on demand), YouTube, Netflix, Hulu etc. have penetrated the market but the TV industry is still doing its business in the same manner. However, some recent developments in the market with the entry of players like Aereo suggest that it's time for the long impending change in business models.

Let's just briefly check who the players are and what the flow of money presently is-

Multi video programming distributors- MVPDs are cable companies and DTH (direct to home) satellite TV providers like Dish TV. They earn their monies through the subscription fees paid by the consumers and local advertising revenue. They buy programming content or channel bundles from the networks.

Networks- Networks like Viacom provide programming content in the form of channels with TV shows. Usually networks give a "bundle" of channels to MVPDs for which MVPD pays affiliate fees. So it's the networks which make channel bundles not the Pay TV companies.

Over the top services (OTT) - OTT is the term used for web based or broadband delivery of video and audio without a set top box. It includes Netflix, Hulu, Itunes, Amazon etc.

Then there are 3rd party OTTs which include the likes of Google and Apple TV. These are on demand boxes which enable delivery of content on your TV or computer, tablet, phone etc.  The OTTs either charge a subscription fee (e.g. Netflix) or pay per view (e.g. iTunes).

OTTs have been trying to cut licensing deals with networks in order to populate their libraries with desired content. The challenge here is that the networks want to charge retransmission fee and sell "bundles" rather than a-la-carte.

Networks have long been the most powerful players in TV industry. On the other hand the OTTs are struggling to come up with a business model that would generate revenues for them and provide customers more choice (a-la-carte) at lesser cost along with mobility to consume content anywhere, anytime.

The company that looks like the game changer here is Indian innovator Chaitanya Kanojia's Aereo backed by media baron Barry Diller. It provides tiny antennas on rent which can pick up broadcasters free over-the-air signal and transmit them to an Apple TV or Roku device for as low as $8 a month. Consumers can watch live or record on a cloud based DVR (digital video recorder). The channels can also be streamed on computer, Ipad or Iphone. It doesn't carry all channels, but only those that are broadcast over the air (does not carry "cable only" channels) and it does not pay any retransmission fees to networks like cable companies and other OTTs do.

Major broadcasters, including NBC and Fox, filed a case against Aereo in New York, alleging that Aereo's technology is copyright infringement, but court ruled in favor of Aereo, while in Los Angeles Aereo lost a similar case. There are chances that the litigation might end up in Supreme Court or the congress in which case the broadcasting laws will have to be changed. As of now Aereo's services are available in New York and Aereo is planning to reach 20 cities in the first expansion phase.

Aereo's future is not clear but its entry sure has disrupted the status quo. While the big broadcasters like CBS are considering pulling their signals off the air and go cable only, cable companies like Time Warner who were paying retransmission fees to the networks till now are considering using similar technology as Aereo. All this can lead to huge revenue losses for broadcasters.

Customers want mobility in consuming content and to pay only for content they actually consume, Aereo gives them this choice. Early adopters have already picked it up and seems like the industry is on the verge of the next big change which will make the broadcasters rethink on new ways to distribute content.

 

September 27, 2013

Dilemma of Innovative Idea

Innovator usually has two questions - Is my idea innovative? The answer is simple- YES, but at the same time organization may or may not accept it for incubation. Thus, innovator has next question- How organization select the idea for incubation.

Dictionary meaning of innovation is something new, may be a new method, process or a product, etc. Thus, innovative idea has to be new. When we say an innovative idea or a new idea, we always think about a path breaking idea or a very unique idea which never existed in this universe. However, this common perception is untrue. Are we looking for innovation or invention? In an invention, we look for something new which never existed, while in an innovation newness has an importance but it may exist somewhere else and not in the current context of idea. Innovation can't be an invention rather it is smart adoption of invention. One of the best and the most commercially successful nature inspired invention- Velcro is biommicry of burdock burrs. This invention has innovatively used in various products and we always see new innovations of it. Thus, Velcro is a single invention and has used in multiple innovations.

There are many such examples of innovations which are derived from inventions. Most of us believe that touchscreen based mobile is the best innovation. Is touch-screen an innovation? No, touch screen technology was invented in 1965, and it is in the use since then for various purposes. Some of us might have seen touch screen based Kiosk before the launch of touch screen based mobile. Then, what is the innovation in touch screen based mobile; and the answer is - replacing physical keypad by touch screen to improve user experience. Thus adoption of technology for better experience is the real innovation. Another good example is LED. They are in the market since 1968 and since then we have seen many innovations around them. The recent one is adoption for house/office and car lighting. Thus, innovation necessarily need not be invention.

Innovative idea can also be the smart implementation of things from the other fields. Use of camera and social media in a mobile is the best example of innovation which integrated two known different technologies. It helped user to take a picture and upload immediately on a social media site. E-commerce is the good example of better implementation of the same idea. Amazon used it successfully and made it popular though it was in use before Amazon. Some innovations are also adopted from the nature. Most innovative aerodynamic designs of automobiles are inspired by nature.

Therefore, apart from the dictionary definition of an innovative idea, it can also be defined as any existing idea which is new in the context of its use.

By this definition, organization may not always accept and invest in new ideas. Organizations will always look for some qualifier to nominate an innovative idea for incubation. This qualifier is an opportunity in pursuing the idea. The opportunities are mostly financial or nonfinancial. After using touch screen into mobiles, the handset manufactures earned a lot of money. This is a clear case of financial benefits. However, improving the processes within the organization for the speedy approval of employee's welfares may be nonfinancial benefit for the organization but positive impact on the productivity because of the employee's happiness is the real benefit for the organization.
When an organization is looking for a new idea, it is mostly judged by the size of opportunity or impact of an idea over a business and a customer, and infrastructure readiness of the organization. Every organization has its own model for calculating size of opportunity or impact of an idea over a business.

If organization is ready for accepting the innovative idea then the next question is, how and who will pursue it in the organization. The answer is simple- it depends on the size of opportunity or impact of an idea over a business. If opportunity or impact is huge then organization usually pursues it on the high priority and may be incubated centrally or at unit level. If opportunity or impact is not so great but organization has advantage if implemented and also investment required is also manageable, then idea is implemented as a routine work. Mostly incremental changes are implemented through regular channel rather than innovative idea.  Sometimes, idea may be implemented without considering an opportunity or impact because competitor has implemented it or it is pursued as an edge over the competitors.

Thus, innovator has to understand the two set of questions to overcome the dilemma

  1. Idea: Is the idea new? Is it completely new or adopted from other areas? What is it the best way to adopt a new idea?
  2. Idea Buying by Organization: Why organization should adopt the idea? Who will be benefited from the implementation of your idea? Why is this idea important for the organization? What is the size of opportunity? What is the impact on business? Is it financial or nonfinancial gain to organization?

The last and most important question is: Will you start your startup with this idea? If answer is yes, then there is high probability that your idea may be accepted for incubation by the organization.

 

August 26, 2013

Achieving Consistency and Agility through Design Accelerator

Couple of years back, I was involved in a debate with my friend about a possible design to a particular problem. Even after a lengthy discussion, we were not able to reach a conclusion. We differed on quite a few aspects. His principles were centred on high modularity, while mine was centred on performance and scalability. Another point of contention was between maintainability of code and faster delivery. Now when I look back at that discussion we had, we both were right considering our thought process that led to our own design. This is a typical problem designers face. Since design involves lot of art, it is bound to be more subjective. There isn't anything called "The Best Design". However there are designs that follow the best practices and those that don't. But why shouldn't there be a Best Design to a particular problem. Let us look at some of the reasons.

Continue reading "Achieving Consistency and Agility through Design Accelerator" »

August 19, 2013

Mobilization : Required Business Orientation and Associated Technology Options

Mobile computing has become the business transforming factor for many of the enterprises. With the advent of cheaper and affordable smartphones, phablets and tablets getting released every now and then, the mobile market is on the rise and hence the consumption of the services via those mobiles are also on the high leading to the increased demand for the mobilization of consumable services.

Continue reading "Mobilization : Required Business Orientation and Associated Technology Options" »

August 12, 2013

Choice of Channel for mobile application development and launch

With the rapid growth of mobility channels, developers and enterprises are faced with the challenge and choice of catering to all the channels. However catering to all channels in one go is not the right approach. We explore the ways to select the channel for mobile application development and roll out based on various parameters including the business angle. 

Continue reading "Choice of Channel for mobile application development and launch" »

July 23, 2013

Planning Future Strategies with the aid of History

"Whoever wishes to foresee the future must consult the past" wrote Niccolò Machiavelli, the renaissance era Italian strategist. What does that mean for corporate strategists? Does this mean that one should conduct a historical study of the industry, of the company or of a particular strategic area where one wants to take a decision? There would be many who rubbish this. They would say what good is studying strategies that won yesterday's clients, markets and businesses worth in today's supersonic world of businesses. However, I would argue that for leaders and strategists to seize the zeitgeist, they need to take the effort to study the past. It is important to understand why certain strategies were adopted, actions taken and the business environment that prevailed when such decisions were taken. The glass for a crystal ball must come from the mirror that has seen the past.

Continue reading "Planning Future Strategies with the aid of History" »