Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« January 2018 | Main | June 2018 »

March 14, 2018

Future is Intelligent systems: Are you ready to test them?

Author: Harleen Bedi, Industry Principal, Infosys Validation Solutions

Artificial Intelligence (AI) is frequently making headlines with various possibilities that it offers to make our lives easier and how it is driving innovation in all spheres of our lives. There are opportunities for AI applications in almost all domains/ fields - right from home automation, personal virtual assistants, automated service agents, fraud detection, preventive maintenance, personalized experiences, financial advisory, healthcare recommendations and many more.  Multiple reports or studies predict huge market potential for AI and associated technologies.

Like most emerging areas, AI has its own share of failures - Self driving car's deadly accident, racist response by AI system, inappropriate tagging by image tagging software, some of these as extreme as when Microsoft has to shut down Tay Bot as it started posting abusive messages on twitter.  This poses a vital question - how were these systems tested? And more importantly - Are we ready to test them and roll them out with confidence to make important decisions on our behalf?

But why we need to worry about testing strategy? Software testing is an advanced field and has been around for decades now. What makes testing an AI application different than any other conventional app?

There are few basic differences -

  • AI applications are non-deterministic and probabilistic
  • Dependency on humungous amount of data
  • There is no defined input / output
  • Difficult to predict all scenarios
  • Continuous learning from past behavior

Traditional approach and testing tools will not meet the needs of testing these systems.

Before getting to what needs to be tested, let's first understand basics components of any AI based application -

  1. All AI systems are driven by huge and diverse data that it needs to process before making any decision. This data could be in any format - text, speech, images, videos... and from different sources. The data may get ingested in one go or it might be a continuous process. Data is extremely critical and forms base of any predictions, decisions or actions taken by the AI systems.
  2. Next is, machine learning system or core algorithms - this forms the heart of AI system as the system is trained here to perform certain actions. Depending on the need and context these algorithms could vary in kind and complexity. Multiple algorithms can be combined to give a certain desired output.
  3. Finally, there are interfaces with 3rd party systems. AI systems gets infused with a larger application to meet end business goal. Eg: A recommender system may not work in silos in real time, rather it will integrate with an e-com application to provide real time product recommendations to users.

Now, let's see what testing needs to be performed at each of these layers-

  1. At Data ingestion layer - it's important to validate that system is able to accept data from various sources and in various formats. Secondly, there need to validate that the ingested data gets converted in the format expected or required by the target system for consumption. Few queries to validate the right ingestion of data and data types will be helpful and can prevent anomalies showing up later due to data.
  2. Next step is to verifying accuracy of system once the algorithm is selected (selection of which algorithm to use is considered out of scope for this discussion). As a first step, split the ground truth data into 70-30 for training and testing. It uses training data set to understand and model the system behavior and test data is used to validate the system accuracy or response. For classification algorithms, metrics like classification accuracy or confusion metrics can be calculated to get accuracy of model. Metrics like Mean Squared error (MSE), Root Mean Squared Error (or RMSE) or R square can give used to evaluate prediction on regression algorithms.
  3. Finally, the 3rd party interfaces need to be tested. And it's important to test how the AI system fits in fulfilling the overall business objective. End to end business driven tests are needed to validate this.

These are the few guiding principles that will help testers to get started in this direction. A lot of effort needs to be put to generate right test data, build appropriate test models and identify right set of metrics to contextualize it to the system under test. Opportunities are immense and how well we are prepared makes all the difference!