Testing Services provides a platform for QA professionals to discuss and gain insights in to the business value delivered by testing, the best practices and processes that drive it and the emergence of new technologies that will shape the future of this profession.

« Transforming Paradigms - From Quality Assurance to Business Assurance | Main | Automation - A new measurement of client experience »

Automated Performance Engineering Framework for CI/CD

Author: Aftab Alam, Senior Project Manager, Independent Validation and Testing Services

with contribution from Shweta Dubey

Continuous Integration is an important part of agile based development process. It's getting huge attention in every phase of Software Development Cycle (SDLC) to deliver business feature faster and confidently.

Most of the times it's easy to catch functional bugs using test framework but for performance testing, it requires scripting knowledge along with load testing and analysis tools.

The key to success in CI/CD is

  •          Find issues at early stage.
  •         Automate anything that is done more than once
  •          Automated build , various quality gates and release process
  •          Automated metrics collection and correlate metric - data based automate quality gates

 There are many test driven framework which enable software development and test engineers (SDE/SDETs) to write automated test cases to verify functional behavior of the code but what about performance of the software? It is important to have automated performance tests as well to get early feedback as well.

 

CI-Performance engineering framework (CI-PEF) is built to provide performance validation capability to devs as part of build phase itself.

CI-PEF enables SDE/SDET to run performance test plan as part of CI pipeline on a click of a button and get performance feedback without spending time on test scripts, load infrastructure, data  and test execution.

Performance engineering framework can be built using various open source and/or license tools. Keys features of framework should be

1)     Ability to schedule and execute performance test on demand or base on certain condition e.g. time based, dependencies on success of functional test.

 Any CI build server can be used (Jenkins, Bamboo, Puppet, Go, Ansible etc) to trigger load test plan(Jmeter, SOASTA ,HP LoadRunner , NeoLoad or any load test tool that supports command like or APIs based execution)

2)     Ability to execute same test scripts in different environment

Maven/Ant can be used to manipulate target environment in test scripts at run time

3)     Ability to change data based on targeted environment

Maven/Ant can be used to manipulate data set provide by test data management(TDM) solution based on target environment

4)     Ability to trend and compare performance KPIs( response time, errors) and mark performance execution as PASS/FAIL based on pre-defined performance threshold and send alert/email

5)     Ability to pull and store performance report from other tools like dynatraceappdynamicsnewrelic to provide deep dive analysis at one place

6)     Ability version control scripts and test configurations

Traditional way vs automate way of performance testing-small-final.pngView image

Traditional way of performance testing and key challenges in agile SDLC

  1. When teams are moving towards agile model (fast deployments, short sprints) to achieve business requirement, Performance test cycle become elephant in the room
  2. Long wait to validation performance or get feedback of fix done. (dev env-> test env->perf env->Developer)
  3. No continuous integration or version control support to load testing. 
  4. Manual build status comparison and result analysis
  5. Login into individual APM or monitoring tools to check status
  6. Dev just waits for test report from performance engineers
  7. End up turning off features due to performance issues as no time to fix in on -going release(increased time to market)


Performance testing using CI-PEF and key benefits

  1.         Performance testing now takes a shift left approach into development phase to make process agile and flexible to track some perf issues as early as possible.
  2.     Code can be perf tested as part of CI build cycle in any lower environment using CI-PEF (for existing scripts) and dev can get the feedback compared to previous code.
  3.     If there is small code change and no time to wait for performance engineers, code can be deployed to PROD if CI-PEF report is good.
  4.     Continuous Integration with Jenkins and test scripts version controlled with git embodies agile practices in TDD environment.
  5.     Jenkins provides comprehensive reports on continuous builds and present performance trends in graphical format, so one can get instant feedback.
  6.     Integrated auto email notification to team with customized content.
  7.     Allocate the available load agents (from the pool) for the test

Please note that automated performance engineering framework does not replace the need of a full scale performance testing completely. This framework enables team to get performance feedback of new features or bug fixes without full scale performance test and left in SDLC(e.g dev sandbox, DIT or in SIT). Full scale performance test is still required for capacity planning, infrastructure changes or upgrades. 


Frame work provides quick view of build performance health


Response time and errors comparison by build e.g. over all response time increases in build #5. 


Jenkin response time.PNGTransaction level details e.g checkout transaction has caused overall spike in response time.


Jmeter-report-1.png

Transaction report

If framework is integrated APM tool, code profiling and design/architecture validation can be done using APM tool reports. e.g below report from apm tool (dynatrace) shows that high response time is due to an Update statement. Due to high response time from JDBC layer, DB connection pool was 100% utilized and there were connection pool exception as well.


Dynatrace report
Dynatrace-Image-2.png
WIN-WIN situation for both developer and performance engineers:

With help of performance engineering framework, all these can be found and fixed in dev sandbox or system environment. Hence reducing development efforts and overall mean time to release (MTTR) feature in production. At the same time performance engineering team is able to spend time on framework enhancement or other high
priority tasks 


Comments

Nice to see this shift left approch in performance testing.

Very nice framework Aftab.

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Please key in the two words you see in the box to validate your identity as an authentic user and reduce spam.