Is the environment spoiling your party?
I frequently come across performance testing projects entangled in cost and time overruns - the culprit usually being environment issues. Since we can't wish away the environment and the issues that come with it, the next best thing is to be better prepared by figuring out the pitfalls and addressing them proactively.
Since a stable test environment is critical for test script development, load simulation and bottleneck analysis in the performance testing life cycle stages, let's take a good look at what to watch out for when we prepare for a testing cycle.
Know thy application: Any environment issue during the test execution phase is like the proverbial spanner in the works. It should come as no surprise if the performance testing team has to spend significant effort in debugging and analyzing it and, of course, following up with support teams for resolution. To be effective in test environment risk assessment and issue resolution, we must well know the application architecture, functionalities, workflows, and the interconnecting components. Stubs and virtualization techniques can be handy during test execution when one is familiar with the component level details and how to use them. While investigating environment issues, the development and infrastructure teams often seek the testing team's input - so lending a hand with specifics will mean a faster turnaround.
Dependencies - wheels within wheels: Another party-pooper can be the dependencies that may impact testing way before we even run the performance test. Multi-tiered enterprise computing systems have these dependencies on each tier and layer, within and outside the enterprise boundaries. In addition to the functionalities, there are other factors at play that may impact the performance test results. These could include high resource consumption by another process hosted on the same infrastructure, execution of batch jobs, or parallel test runs by another team. An outage in the environment during the test run can force you to reschedule the test. That's why, it is all important to gather information about all the possible dependencies that may impact the test execution during the planning stage itself. It is always good to document these issues as one comes across them for reference in future test cycles.
Stay in touch: The performance testing team cannot operate in a vacuum. Team members must establish proper communication with the development, infrastructure, functional and integration testing, and release management teams right from the strategy phase to synchronize test preparation as well as execution activities. The test schedule should be published well in advance in case you are using a shared test environment. Notifications prior to running a test must be sent out to the teams concerned to bring up the servers, mount monitors, clear logs and keep the environment stable during the test execution. A calendar that blocks shared computing resources can keep all stakeholders posted on the date with the test execution and reduce retest efforts significantly. A small tip: Keep your contact information handy and up-to-date.
Think ahead: Being well-prepared is half the battle won. For testing, this means to think about possible environment failures and look for workarounds well before the actual test execution. While preparing test estimates, don't forget to factor in unknown environment issues that could adversely impact the effort and the schedule. Keeping some buffers as a percentage of the overall estimate can save you a lot of grief later. It is also important to prioritize critical business transactions for the performance test so that, if some functionalities flop during the planned test window, a test run on a subset of the transactions can provide meaningful insights into application performance. Finally, remember that time is your most precious resource. So, if the test environment becomes completely unavailable during the planned test window, utilize that time effectively in activities such as offline reporting or knowledge management and promptly schedule the test in the next window available.
So these are a few best practices to keep in mind while preparing for a performance test cycle. You can use these to build your own set of rules specific to the challenges and constraints of your set-up. The bottom line for a successful testing cycle is to keep tabs on incidents and work through issues smartly.