How to Measure Forecast Accuracy?
But, if it is just the question of comparison of Forecast Vs Actual comparison then why we have so many Forecast Accuracy measures? To name a few, almost every Statistical Forecasting Tool has following error measure as a part of their standard offerings - Mean Absolute Deviation (MAD), Mean Absolute Percentage Error (MAPE), Root Mean Square Error (RMSE), Mean Percentage Error (MPE), Error Total (ET) and Mean Square Error (MSE)so on. What does all of these measures mean? And in what context which measure should be used?
Answers to above questions can be found by answering following 4 basic questions in your Business Context.
- Error Sign - Do you want to treat Positive and Negative Forecast Error same or different? - In simple words it means, will it make any difference to you if Forecast is more than Actual Sales or less than Actual Sales. This depends on what business you are in. If you are in business of say perishable products then you will always prefer Forecast to be less than Actual as surplus production is as good as loss. On the contrary if you are in a business of selling products which have reasonable shelf life then Forecast should be always greater than Actual as loss of sales is more costly than holding inventory.
- Reference to Base - Do you want reference to base value while measuring Forecasting Accuracy? - In simple words it means, whether a forecast error of 2 over a forecast value of 100 and an error or 2 over a forecast value of 10 mean the same to you. Answer to this question depends of what you are forecasting. If you are forecasting dispatches from factory where KPIs are "fulfillment" related like On Time In Full (OTIF), Order Fulfill Rate, Dollar Value Fulfilled on time etc then reference to base is important since you know 100% perfection is never possible. On the other hand if you are forecasting for Point of Sales (POS) then every unit stock out cost you same. 100% shelf availability is a must and there cannot be any compromise on this count. In such case reference to base does not mean anything. For example no point in achieving shelf availability of 98% and missing 2% availability during pick times.
- Error Spread - Does it matter to you if forecast error is concentrated on few points or do you want forecast error to be spread evenly?- 100% forecast accuracy is utopian. Some error is bound to happen. Question is what is your preference for distribution of error? Do we want it to be evenly distributed over all points or you don`t mind if you go horribly wrong on few points and look good if calculated for entire horizon. Answer to this depends upon type of products you are dealing in. If you are dealing with product that have very long shelf life which means unsold inventory from one period can be effectively put into use in subsequent periods then we don`t mind going horribly wrong with the forecast in one period as long as we make up for that in the horizon. Forecast Accuracy of the complete horizon is more important in such case than accuracy of an individual month or a week.
- Weightage to History Data Points - Do you want to treat error on every point equally or you want to differentiate between individual data points based on their place in history? - For example, when you are analyzing forecast accuracy do you care if total forecast error is contributed more by recent past than by distant past. In some fashion business it does make lot of difference. Fitment over recent past is more important than fitment over distant past. Again choice of this decision depends on what you are forecasting. If forecast is meant for long term investment planning in planning for facility then all data points means same. However if forecast is for POS and influenced greatly by consumer behavior then close fitment over recent history is more important than fitment over distant history.
Now let us map commonly offered Forecast Accuracy measures to each of the above criterions.
Measure Sensitive to Sign of Error?
|Measure has reference to Base?||No||No||Yes||No||No||Yes|
|Measure gives equal importance to all data points?||No||Yes||Yes||Yes||Yes||Yes|
|Measure sensitive to distribution of over data points?||No||No||No||Yes||Yes||No|
Going forward if you face dilemma in selecting proper Forecast Accuracy Measure to suit your business context then make quick reference to above table and your problem will be solved.