TY - JOUR

T1 - Measuring forecasting accuracy

T2 - the case of judgmental adjustments to SKU-level demand forecasts

AU - Davydenko, Andrey

AU - Fildes, Robert

PY - 2013/7

Y1 - 2013/7

N2 - Forecast adjustment commonly occurs when organizational forecasters adjust a statistical forecast of demand to take into account factors excluded from the statistical calculation. This paper addresses the question of how to measure the accuracy of such adjustments. We show that many existing error measures are generally not suited to the task due to specific features of the demand data. Alongside well-known weaknesses of existing measures a number of additional effects are demonstrated that complicate the interpretation of measurement results and even can lead to false conclusions being drawn. To ensure an interpretable and unambiguous evaluation we recommend the use of a metric based on aggregating performance ratios across time series using the weighted geometric mean. We illustrate that this measure has the advantage of treating over and under-forecasting even-handedly, a more symmetric distribution and is robust.Empirical analysis using the recommended metric showed that on average adjustments yielded improvements under symmetric linear loss, but in terms of some traditional measures adjustments harmed accuracy. As a consequence, further support is given to the critical importance of selecting appropriate error measures when evaluating forecasting accuracy.

AB - Forecast adjustment commonly occurs when organizational forecasters adjust a statistical forecast of demand to take into account factors excluded from the statistical calculation. This paper addresses the question of how to measure the accuracy of such adjustments. We show that many existing error measures are generally not suited to the task due to specific features of the demand data. Alongside well-known weaknesses of existing measures a number of additional effects are demonstrated that complicate the interpretation of measurement results and even can lead to false conclusions being drawn. To ensure an interpretable and unambiguous evaluation we recommend the use of a metric based on aggregating performance ratios across time series using the weighted geometric mean. We illustrate that this measure has the advantage of treating over and under-forecasting even-handedly, a more symmetric distribution and is robust.Empirical analysis using the recommended metric showed that on average adjustments yielded improvements under symmetric linear loss, but in terms of some traditional measures adjustments harmed accuracy. As a consequence, further support is given to the critical importance of selecting appropriate error measures when evaluating forecasting accuracy.

KW - judgmental adjustments

KW - forecasting support systems

KW - Forecast accuracy

KW - Forecast evaluation

KW - Forecast error measures

U2 - 10.1016/j.ijforecast.2012.09.002

DO - 10.1016/j.ijforecast.2012.09.002

M3 - Journal article

VL - 29

SP - 510

EP - 522

JO - International Journal of Forecasting

JF - International Journal of Forecasting

SN - 0169-2070

IS - 3

ER -