Source-linked AI summary
A better measure of relative prediction accuracy for model selection and model estimation
Chris Tofallis
TL;DR
MAPE is problematic because it favors forecasts that are too low. The paper investigates the log accuracy ratio, Ln(Q), and reports that regression using this measure predicts the geometric mean.
Problem
MAPE is problematic because it favors models whose forecasts are too low, reflecting asymmetry problems associated with percentages.
Method
The paper investigates an alternative relative accuracy measure based on the log of the accuracy ratio, Ln(Q), and compares metrics' capabilities.
Results
Regression to Ln(Q) led to predictions of the geometric mean, while Figures 4 and 5 display log accuracy ratios.
Takeaways & Limitations
Using Ln(Q) connects regression predictions to the geometric mean, providing a basis for analyzing the resulting model properties.
Takeaways & Limitations
Σ (ln Q)2 has a bias toward under-prediction when the noise is [condition unspecified in the passage].
Abstract
from arXiv · showhide
Surveys show that the mean absolute percentage error (MAPE) is the most widely used measure of forecast accuracy in businesses and organizations. It is however, biased: When used to select among competing prediction methods it systematically selects those whose predictions are too low. This is not widely discussed and so is not generally known among practitioners. We explain why this happens. We investigate an alternative relative accuracy measure which avoids this bias: the log of the accuracy ratio: log (prediction / actual). Relative accuracy is particularly relevant if the scatter in the data grows as the value of the variable grows (heteroscedasticity). We demonstrate using simulations that for heteroscedastic data (modelled by a multiplicative error factor) the proposed metric is far superior to MAPE for model selection. Another use for accuracy measures is in fitting parameters to prediction models. Minimum MAPE models do not predict a simple statistic and so theoretical analysis is limited. We prove that when the proposed metric is used instead, the resulting least squares regression model predicts the geometric mean. This important property allows its theoretical properties to be understood.
2. Measuring relative accuracy using the log ‘forecast to actual’ ratio: Ln Q
The paper develops Ln Q, the logarithm of the prediction-to-actual ratio, as a relative accuracy measure for comparing methods and fitting regression models. It addresses asymmetry in Q and provides theoretical support for the measure’s practical use.
- Q denotes the prediction-to-actual quotient, or accuracy ratio, with 1.0 as the ideal value.Q is also described as the complement of relative error.
- Q is asymmetric because it is bounded below by zero but unbounded above, despite having a theoretical mean of 1.
- Ln Q can compare competing methods by comparing the sums of squared log accuracy ratios.
- Applying least squares to Ln Q produces regression models that predict the geometric mean.
- Earlier work found roughly half of Ln(Q) regression errors positive and half negative, but the approach lacked theoretical support.
3. Theoretical support for Ln Q
The paper adapts theoretical results on relative-change measures to forecasting, identifying Ln Q as a symmetric relative-accuracy measure. It shows that least-squares fitting with Ln Q predicts the geometric mean and provides balanced relative predictions, especially for multiplicative-error models.
- Theoretical basis: Ln Q is the remaining measure when relative-change indicators are required to be symmetric, additive, and normed.It is defined as ln(g/f), which changes sign when the two values are interchanged and combines ratio and difference properties.
- Least squares and location: Least-squares fitting of squared log accuracy ratios estimates the geometric mean of positive observations.The estimator is Y = exp[1/n Σ ln(yi)] = [Π(yi)]1/n, and the result generalizes to functions of x and continuous positive distributions.
- Least squares and location: Minimizing MAPE produces under-predicting models and does not predict a simple statistic or measure of location.This motivates using the proposed metric for parameter estimation instead.
- Relative unbiasedness: With a constant coefficient, the geometric mean of predicted-to-actual accuracy ratios equals unity, paralleling zero-mean residuals in OLS with a constant term.Equivalently, the arithmetic mean of log accuracy ratios is zero.
- Multiplicative errors: For multiplicative-error models, least squares on Ln Q is equivalent to least squares on the logarithm of the multiplicative error, yielding error factors with geometric mean unity.Multiplicative error models are suited to heteroscedastic data.
4. Model fitting: Empirical applications
Empirical applications show that MAPE-fitted models systematically sit too low and produce mostly under-predictions, whereas least squares on log accuracy ratios yields more symmetric predictions around zero.
- Data characteristics: Project effort scatter increases with project size, providing clear evidence of heteroscedasticity in software-project data.This pattern is reported as typical for this type of data.
- Straight-line models: MAPE fitting produced the model Predicted effort = 10.05 + 3.8 FP, while the proposed approach produced Predicted effort = 52.93 + 7.525 FP.The MAPE line was positioned too low relative to the observed points.
- Straight-line models: For the Finnish data, MAPE-based results mostly under-predicted with few over-predictions, whereas the proposed method showed greater symmetry.The proposed method's plotted statistic had mean zero, equivalent to a geometric mean accuracy ratio of unity.
- Power-function models: Power-function fits showed the same pattern: minimizing MAPE yielded mostly under-predictions, while least squares on Ln Q produced more symmetry.The paper attributes the resolution of the asymmetry issue to using logarithms.
- Desharnais dataset: In the Desharnais data, the MAPE line under-predicted 61 of 81 projects, compared with 49 under-predictions for the other line.The other line was computed using the proposed method.
5. Model selection: Simulation experiments
Simulation experiments compare accuracy measures for selecting the true model under multiplicative and additive noise. The log-ratio-based measure performs especially well for multiplicative noise, while MAPE is consistently biased toward under-prediction.
- Multiplicative noise: As multiplicative noise increases, MAPE becomes the worst performer in identifying the true underlying model.The generated data used a multiplicative error factor and a power-function model.
- Multiplicative noise: Σ (ln Q)2 performs best across all multiplicative noise levels, followed by SMAPE and LSD.LSD improves on MAPE but does not perform as well as the simpler Σ (ln Q)2 criterion.
- Selection bias: MAPE favors models that predict too low, whereas LSD favors over-prediction and SMAPE and Σ (ln Q)2 are fairly neutral.The reported bias comparison concerns how often criteria select models predicting too low or too high.
- Additive noise: For additive error, LSD identifies the true model best, while Σ (ln Q)2 and SMAPE remain much better than MAPE.The latter two measures are almost as good at low noise but gradually worsen as additive noise increases.
- Conclusions: For strictly positive data, the proposed metric is presented as superior to MAPE for relative prediction accuracy and avoids MAPE’s under-prediction bias.The paper recommends the log of the accuracy ratio, Ln(Q), and connects it to geometric-mean prediction.
APPENDIX
The appendix shows that least squares applied to log accuracy ratios predicts the geometric mean. This parallels the arithmetic-mean result for ordinary mean squared error.
- APPENDIX: For any continuous distribution with y>0, least squares applied to log Q gives the geometric mean as the expected value.The derivation identifies the resulting Y with the geometric mean.
- APPENDIX: The appendix frames loss functions as selecting different measures of location or expected value.It contrasts the geometric-mean result for log accuracy ratios with the arithmetic-mean result for ordinary squared error.
- APPENDIX: The distribution f(y) may represent the conditional distribution of y for a given x-value.This extends the location-measure argument to regression settings.