Performance Modeling - Implementation Challenges and Benefits
To continue my previous blog "Performance modeling - Implementation Know-Hows", I would like to highlight the implications of not doing performance modeling, the implementation challenges and the benefits.
"Ok so what will happen if I do not implement Performance Modeling in Design phase?" is one of the most frequent questions that I come across from the business stakeholders and application managers especially when planning for application performance management.
Well, here are my views with regards to implications of not doing performance modeling:
1. The predictability of application's performance will be low.
· No numbers to back the choices made in Design and Architecture - one needs to wait till Test phase to figure out if the system really meets performance and scalability SLAs.
2. The individual product specific benchmarks & industry standard benchmarks (SPEC4j, TCP, TPH to name a few) are not helpful because:
· The workload used for the industry-standard benchmarks typically refers to basic operations and is not representative of production-like workloads (in general, these benchmark are better off when choosing between two different hardware platforms).
· Most of the enterprise applications are custom-built using multiple products/stacks from different vendors which will have different performance characteristics when compared with individual benchmarks.
3. In general, Design and Architecture reviews are qualitative in nature and hence need to be supported by numbers, especially when planning to implement systems that are business critical and have financial penalties associated with performance SLA non-compliance.
So, to make informed decisions and avoid any performance and scalability risks that arise from Design and Architecture of a software system, it's a good idea to go for Performance Modeling exercise.
At this point, I would like to reveal few of the challenges while implementing performance modeling:
1. In design phase, sometimes it's not possible to build a prototype that can represent critical user operations such as an application heavily depending on external systems.
2. The accuracy of performance models depends upon the accuracy of the service demand measurements and the layers that are considered to be modeled:
· Especially in shared hardware systems, service demand measurement is a tough task, unless coordinated well with other application teams.
· Modeling few resources such as Network is not straightforward - one has to go with certain assumptions with respect to network bandwidth, network congestion and model them as 'Delay Centers' with fixed delay which is not a correct representation of real-world network.
· It is difficult to model few aspects of applications such as Caching and Application level Queuing. Experience with different systems and applications in performance modeling will help decide which component to 'model' and which 'not-to'.
3. Tendency to be 'as-much-accurate-as-possible' makes model heavy, complex and time-consuming from implementation standpoint - for instance, with server infrastructure advancements, Physical Memory has become inexpensive and hence modeling 'memory' resource might not be required.
However, the benefits of implementing performance modeling are significant and it's worthwhile to deal with some of the challenges. Let's take a look.
1. One can assess performance of Design & Architecture early in SDLC and predict performance & scalability for current and future workloads.
2. Surprises with respect to application performance can be minimized due to early performance validation.
3. Models built in Design phase can be re-used with the data captured from Test phase which can be further used to plan for hardware capacity needs.
4. It saves Load Simulation licensing costs by avoiding execution of Load Tests with very high virtual user load (5x, 10x etc.) which is rare but application teams would like to have the data.
To conclude, performance modeling helps understand IT systems' performance with certain accuracy levels right from the Design phase. However, the trade-off between the implementation challenges, and the benefits should be thoroughly reviewed before actually planning for Performance Modeling exercise.