Performance Extrapolation that Uses Industry Benchmarks with Performance Models
A white paper written by me and my colleague Kiran C Nair was presented in SPECTS'10 conference (http://atc.udg.edu/SPECTS2010/program.php). This conference targeted professionals involved in performance evaluation of computer and telecommunication systems. The paper described an approach to predict performance of applications using Industry standard benchmarks and Queuing Network Models. Multiple requests for extrapolating applications performance to different hardware was the key motivation for coming up with this paper.
The industry benchmarks like SPEC and TPC provide a standard way to compare system performances and also act as pointers for capacity planning. Some analytical methods use these benchmarks for linearly projecting the system utilizations and throughputs. However when it comes to prediction of performance metrics (Utilization, Response Time, Throughput etc) for a multi-layered application like an OLTP application, distributed across multiple resources - - this approach of extrapolation does not provide holistic results.
Then there are performance modeling techniques like QNM, QPN, LQN, etc. that provide detailed understanding of application performance for varying scenarios to enterprise architects. These models are created using the measurements from existing system, and any change in hardware would change these measured values and the performance model itself.
So industry benchmarks were useful to compare hardware performances, but they could not provide much insight on performance metrics for distributed applications. For what-if analysis of such multi-tiered applications, in horizontal scaling and varying workload scenarios, performance models were used. But independently they could not predict vertical scaling impact. Thus to get the overall performance prediction for scaling of hardware and other changes, a hybrid of these two approaches was experimented. The approach and the findings are detailed in this paper.