Forecast adjustment commonly occurs when organizational forecasters adjust a statistical forecast of demand to take into account factors excluded from the statistical calculation. This paper addresses the question of how to measure the accuracy of such adjustments. We show that many existing error measures are generally not suited to the task due to specific features of the demand data. Alongside well-known weaknesses of existing measures a number of additional effects are demonstrated that complicate the interpretation of measurement results and even can lead to false conclusions being drawn. To ensure an interpretable and unambiguous evaluation we recommend the use of a metric based on aggregating performance ratios across time series using the weighted geometric mean. We illustrate that this measure has the advantage of treating over and under-forecasting even-handedly, a more symmetric distribution and is robust.
Empirical analysis using the recommended metric showed that on average adjustments yielded improvements under symmetric linear loss, but in terms of some traditional measures adjustments harmed accuracy. As a consequence, further support is given to the critical importance of selecting appropriate error measures when evaluating forecasting accuracy.