Automated Performance Engineering Framework for CI/CD
Author: Aftab Alam, Senior Project Manager, Independent Validation and Testing Services
with contribution from Shweta Dubey
Continuous Integration is an important part of agile based development process. It's getting huge attention in every phase of Software Development Cycle (SDLC) to deliver business feature faster and confidently.
Most of the times it's easy to catch functional bugs using test framework but for performance testing, it requires scripting knowledge along with load testing and analysis tools.
The key to success in CI/CD is
- Find issues at early stage.
- Automate anything that is done more than once
- Automated build , various quality gates and release process
- Automated metrics collection and correlate metric - data based automate quality gates
There are many test driven framework which enable software development and test engineers (SDE/SDETs) to write automated test cases to verify functional behavior of the code but what about performance of the software? It is important to have automated performance tests as well to get early feedback as well.
CI-Performance engineering framework (CI-PEF) is built to provide performance validation capability to devs as part of build phase itself.
CI-PEF enables SDE/SDET to run performance test plan as part of CI pipeline on a click of a button and get performance feedback without spending time on test scripts, load infrastructure, data and test execution.
Performance engineering framework can be built using various open source and/or license tools. Keys features of framework should be
1) Ability to schedule and execute performance test on demand or base on certain condition e.g. time based, dependencies on success of functional test.
2) Ability to execute same test scripts in different environment
Maven/Ant can be used to manipulate target environment in test scripts at run time
3) Ability to change data based on targeted environment
Maven/Ant can be used to manipulate data set provide by test data management(TDM) solution based on target environment
4) Ability to trend and compare performance KPIs( response time, errors) and mark performance execution as PASS/FAIL based on pre-defined performance threshold and send alert/email
6) Ability version control scripts and test configurations
Traditional way of performance testing and key challenges in agile SDLC
- When teams are moving towards agile model (fast deployments, short sprints) to achieve business requirement, Performance test cycle become elephant in the room
- Long wait to validation performance or get feedback of fix done. (dev env-> test env->perf env->Developer)
- No continuous integration or version control support to load testing.
- Manual build status comparison and result analysis
- Login into individual APM or monitoring tools to check status
- Dev just waits for test report from performance engineers
- End up turning off features due to performance issues as no time to fix in on -going release(increased time to market)
Performance testing using CI-PEF and key benefits
- Performance testing now takes a shift left approach into development phase to make process agile and flexible to track some perf issues as early as possible.
- Code can be perf tested as part of CI build cycle in any lower environment using CI-PEF (for existing scripts) and dev can get the feedback compared to previous code.
- If there is small code change and no time to wait for performance engineers, code can be deployed to PROD if CI-PEF report is good.
- Continuous Integration with Jenkins and test scripts version controlled with git embodies agile practices in TDD environment.
- Jenkins provides comprehensive reports on continuous builds and present performance trends in graphical format, so one can get instant feedback.
- Integrated auto email notification to team with customized content.
- Allocate the available load agents (from the pool) for the test
Please note that automated performance engineering framework does not replace the need of a full scale performance testing completely. This framework enables team to get performance feedback of new features or bug fixes without full scale performance test and left in SDLC(e.g dev sandbox, DIT or in SIT). Full scale performance test is still required for capacity planning, infrastructure changes or upgrades.
Frame work provides quick view of build performance health
Response time and errors comparison by build e.g. over all response time increases in build #5.
Transaction level details e.g checkout transaction has caused overall spike in response time.
If framework is integrated APM tool, code profiling and design/architecture validation can be done using APM tool reports. e.g below report from apm tool (dynatrace) shows that high response time is due to an Update statement. Due to high response time from JDBC layer, DB connection pool was 100% utilized and there were connection pool exception as well.
WIN-WIN situation for both developer and performance engineers:
With help of performance engineering framework, all these can be found and fixed in dev sandbox or system environment. Hence reducing development efforts and overall mean time to release (MTTR) feature in production. At the same time performance engineering team is able to spend time on framework enhancement or other high