Author: Rohini Mukund Sathaye, Group Project Manager
There are several myths about non-functional testing such as only highly technical testers can carry out such type of testing, cost is very high and ROI is not favorable.This blog talks about the simple techniques which are cost effective yet help validate non-functional requirements.
Any software system is bounded by functional and non-functional requirements. Non-functional requirements define 'ability' characteristics such as 'Scalability, Reliability, Availability, Usability' etc. along with other 'Quality' properties like security, performance, exception handling capabilities etc.
In this blog, a set of testing techniques are proposed that enable cost-effective non-functional requirements validation for web based applications. The key requirements which are considered are Performance/Scalability, Availability, Usability, Accessibility and Security.
1. Scalability/Performance Validation -
Performance Validation is done to check if the system performance in terms of application response time, server utilization metrics is acceptable under normal, peak and projected workloads. Scalability indicates if the applications/systems can scale up to the workload that is anticipated due to business growth in next 5-10 years without QoS degradation.
Cost Effective Techniques to measure performance and validate scalability:
Option 1 - Measuring single user performance with open-source tools such as HTTPWatch, Yslow etc. If the application is not performing well for one user, it will never give good performance under load. Identify the pages which demand optimization; drill down to individual objects/queries for optimization
Option 2 - Simple incremental load test by selecting open-source tools such as Jmeter, LoadUI for load testing rather than COTS tools which are very expensive. There are multiple options available for web based applications, web services/REST API, mobile performance testing. Analyze an incremental load test results to identify the problematic transactions and components.
Additional cost reduction can be achieved by further reducing the scripting efforts using additional utilities (E.g. Blazemeter Chrome Add-in along with Jmeter) and data setup efforts using freeware utilties like Databene Benerator
2. Availability Testing
Availability is the percentage of time when system is operational. MTBF and MTTR are two important aspects of availability. Mean Time between Failures (MTBF) is the average duration for which the application will remain up till a failure occurs. Mean Time to Repair (MTTR) is mean time to restore an application once it goes down. (Has components such as Mean Time to Discover, Mean Time to Isolate & Mean Time to Repair) MTTR does not include planned downtimes such as upgrades, maintenance and deployment activities.
Availability = MTBF / (MTBF + MTTR)
Availabilty, MTBF & MTTR can be calculated by measuring the application uptime and downtime over a period of time.
Cost Effective Techniques to measure Availability:
Option 1 - Writing a simple macro to parse HTTP/Application logs. There are several open-source utilities available like IIS log parser to get this data.
Option 2 - Using monitoring tools with agents running on individual application boxes. This way it is not only possible to measure application downtime but one can measure other hardware failures as well
Option 3 - Writing a simple CRON job to ping the webservice at regular intervals, and record the http responses
3. Usability Validation
Usability validation is carried out to find out if the built application/product is user friendly. It also indicates if the users are comfortable with the application/product based on certain parameters like layout, navigation, content. With the advent of mobility and responsive web design (RWD), it is imperative to validate usability.
Cost Effective Techniques to validate usability:
Option 1 - There are several open-source/freeware utilities available for usability testing e.g. UserPlus, Usabilla etc. These tools give readymade recommendations to fix UX issues
Option 2 - Manually checking some of the key aspects. In case of eCommerce website, following are the simple checks which any tester can carry out:
- Check the complexity of the checkout process
- Check if enough security checks are there while entering the credit card/payment information
- Check the final cost of the product and compare with expected cost
- Check the currency of the price
- Check if information related to product return policy, shipment process, contact information is readily available and easily accessible
- Check if the associated images provide enough information about the product so that purchase of the same is encouraged
4. Accessibility Validation
Accessibility validation is a sub-set of usability validation. There are multiple accessibility guidelines available specific to regions, some of the examples are 508 standard, W3C standard etc. There are multiple freeware tools available for this which includes browser extensions as well. http://usabilitygeek.com/10-free-web-based-web-site-accessibility-evaluation-tools/
Key manual checks are:
- Keyboard Shortcuts for every button on the screen (including up, down arrows, standard windows shortcuts)
- Compatibility with Screen Readers like JAWs
- High contrast setting for application
5. Security Validation
Security testing is performed to assess the sensitivity of the system against unauthorized internal or external access. There are several black-box manual techniques for security validation:
- Brute-force: a trial and error mechanism employed to crack passwords
- Insufficient authentication: To check if the anonymous user is able to access sensitive information without appropriate access
- Session prediction: To impersonate a session or user by predicting the session value
- Cross site scripting: To check if it is possible to execute an embedded malicious script on client machine
- Buffer overflow: To identify invalid memory referencing through input validation
- SQL Injection: To check if unauthorized users get access to database through input validations
- Directory indexing: To check if directory listing is forbidden
There are several myths about non-functional testing and one of them is higher cost for carrying out non-functional testing. However with afore-mentioned simple yet effective steps/methods, non-functional testing can be carried out in frugal way for any web based application throughout out the lifecycle.