Source-linked AI summary
Mean Absolute Percentage Error for regression models
Arnaud De Myttenaere, Boris Golden, Bénédicte Le Grand, Fabrice Rossi
TL;DR
The paper examines the use of MAPE for evaluating regression models and asks whether optimal MAPE models exist. It establishes theoretical guarantees, characterizes MAPE optimization through weighted MAE regression, and illustrates the approach with kernel regression on simulated data.
Problem
The paper studies the consequences of using MAPE as a regression-quality measure and whether an optimal MAPE model exists.
Method
The paper analyzes MAPE regression theoretically, relates it to weighted MAE regression, and applies the weighting strategy to kernel regression.
Results
The paper proves the existence of an optimal MAPE model and universal consistency for MAPE-based Empirical Risk Minimization, with kernel-regression behavior illustrated on simulated data.
Takeaways & Limitations
MAPE optimization can be approached through weighted MAE regression, providing a basis for applying MAPE-oriented weighting to kernel regression.
Takeaways & Limitations
The minimum need not be unique because the objective function J is not strictly convex; the theoretical analysis also considers a lower-bound assumption on |Y|.
Abstract
from arXiv · showhide
We study in this paper the consequences of using the Mean Absolute Percentage Error (MAPE) as a measure of quality for regression models. We prove the existence of an optimal MAPE model and we show the universal consistency of Empirical Risk Minimization based on the MAPE. We also show that finding the best model under the MAPE is equivalent to doing weighted Mean Absolute Error (MAE) regression, and we apply this weighting strategy to kernel regression. The behavior of the MAPE kernel regression is illustrated on simulated data.
1. Introduction
The introduction motivates studying MAPE for regression because relative error is useful in applications and poses distinct questions about optimal models, consistency, and practical regression methods.
- MAPE is used because it provides an intuitive interpretation in terms of relative error.
- Relative error is relevant for measuring financial gains and losses and for calibrating product prices when customers respond more to relative than absolute variations.
- The paper studies the consequences of using MAPE as a regression-model quality measure.
- An optimal regression model can be defined for MAPE, paralleling the role of the regression function under MSE.
- MAE-based capacity measures can upper bound MAPE-based ones, supporting universal consistency results for Empirical Risk Minimization under MAPE.
- The paper adapts quantile kernel regression to MAPE regression and studies the resulting model on simulated data.
2. General setting and notations
The paper formulates regression through measurable functions that predict a target from explanatory variables and evaluates them with loss-based risks, including MSE, MAPE, and MAE.
- Regression uses a measurable function g from explanatory variables X to predictions intended to be close to target variable Y.
- A model’s quality is defined through a loss function applied pointwise to g(X) and Y, then averaged as a risk.
- MSE uses squared loss, l2(p, y) = (p − y)^2, and its optimal model is the conditional expectation m(X) = E(Y|X).
- MAPE is presented as an alternative to MSE for situations where squared-error risk is not a suitable measure of closeness.
- Finite MAPE risk implies finite E(|g(X)|), so relevant models belong to L1(PX).
- MAE uses absolute-error loss, lMAE(p, y) = |p − y|, with its corresponding MAE risk.
503. Existence of the MAPE-regression function
The paper studies whether the point-wise MAPE optimization problem has finite values and an optimizer, establishing conditions under which an optimal MAPE-regression function exists and is globally optimal.
- The point-wise problem asks whether a function m_MAPE minimizes the MAPE risk among all models.
- A point-wise approximation minimizes the conditional MAPE expectation separately for each value of x.
- The analysis gives necessary and sufficient conditions for the point-wise objective to have finite values; otherwise, J(m) is infinite for every m ≠ 0.
- When T can be arbitrarily close to zero, the objective may be infinite, as illustrated by the uniform distribution on [−1, 1].
- Under Proposition 1's conditions, J is convex and has at least one global minimum, although the minimum need not be unique.
- Because m_MAPE is point-wise optimal, it is also globally optimal; when multiple minima form a bounded interval, the paper selects its mean by convention.
4. Effects of the MAPE on complexity control
The paper analyzes how MAPE changes the complexity controls used for ERM, focusing on covering numbers, VC-dimension, and uniform convergence. Because MAPE is not uniformly Lipschitz, its complexity analysis requires assumptions keeping |Y| away from zero.
- ERM selects the model minimizing empirical risk, and its consistency depends on controlling deviations between empirical and expected risk.
- MAPE requires specialized analysis because the loss generally lacks the uniform Lipschitz property assumed by standard ERM results.
- Covering numbers: Covering numbers measure function-class complexity through a dissimilarity, and sufficiently slow growth supports uniform laws of large numbers.
- Supremum covering numbers: For MAPE, a lower bound YL on |Y| makes the relevant supremum norm well defined, paralleling the upper-bound assumption YU used for MAE and MSE.
- Supremum covering numbers: MAPE covering numbers are related to MAE covering numbers when |Y| stays above a positive threshold.
- Lp covering numbers: The section extends the comparison to data-dependent Lp covering numbers and uses these complexity controls to derive uniform convergence conditions.
that ∀i, Yi̸ = 0, then for all p ≥1,
The paper connects MAPE complexity to VC-dimension and covering-number bounds. Replacing MAE with MAPE does not increase the relevant VC-dimension, while lower and upper response bounds govern convergence conditions symmetrically.
- VC-dimension: Replacing MAE with MAPE does not increase the VC-dimension of the relevant function class.
- VC-dimension: The MAPE VC-dimension is bounded above by the corresponding MAE VC-dimension.
- Complexity bounds: Consequently, MAPE covering numbers and VC-dimension can be bounded using the corresponding MAE-based complexity quantities.
- Uniform convergence: For MAPE, YL plays a role analogous to YU for MAE and MSE in the resulting complexity and convergence bounds.
- Uniform convergence: Almost-sure uniform convergence requires the bound terms controlling empirical deviations to be summable.
- Uniform convergence: If YU grows faster than √n or YL approaches zero too quickly, the relevant series may fail to be summable.
5. Consistency and the MAPE
The paper establishes strong universal consistency for ERM under MAPE under a positive-response-bound assumption and structural conditions on the model classes.
- ERM can produce a strongly consistent estimator of the optimal MAPE risk under minimal assumptions on the data.
- The consistency theorem assumes |Y| ≥ YL > 0 almost surely.
- The theorem uses independent observations and a sequence of measurable model classes whose union is dense in L1(µ).
- Each model class is required to have finite VC-dimension for the MAPE-derived function class.
then LMAP E(bglMAP E,Gn,Dn) converges almost surely to L∗
The consistency proof separates approximation and estimation error, controlling the latter through MAPE-specific complexity bounds. The argument must account for small response values in both components.
- The proof decomposes MAPE excess risk into approximation error and estimation error.
- Approximation error: Approximation is handled by using L1(PX) approximation because finite MAPE risk implies membership in L1(PX).
- Estimation error: Estimation error is controlled with the preceding covering-number and VC-dimension results specialized to MAPE.
- Combining approximation and estimation arguments yields almost-sure convergence of empirical MAPE risk to the optimal MAPE risk.
- Scope of the proof: The proof specifically handles the possibility of small Y values in its approximation and estimation components.
6. MAPE kernel regression
The section develops practical MAPE regression by recasting MAPE minimization as weighted quantile or MAE-style regression and extending the approach to kernel methods. Kernel regression adds regularization, so it is not direct empirical risk minimization, and theoretical consistency remains open.
- MAPE regression: MAPE regression minimizes empirical MAPE over a model class, which is equivalent to median or quantile regression with weights based on 1/|y_i|.The weighting follows because the denominator can be treated as a fixed instance weight.
- Implementation: Instance weighting can be implemented directly for some models, including linear models and software such as the quantreg R package.For linear models, the resulting optimization is a linear programming problem solvable by interior-point methods.
- Kernel formulation: Kernel quantile regression is transformed into kernel MAPE regression by modifying the loss while retaining the RKHS regularization framework.The kernel formulation uses a regularized empirical-loss optimization problem with a quantile check function.
- Kernel formulation: The kernel construction is not a direct form of empirical risk minimization because regularization is introduced.The section distinguishes the practical kernel method from the unregularized ERM setting.
- Open question: Extending the theoretical results to the kernel case remains an open question.The section presents the kernel method computationally without establishing the corresponding theoretical extension.
6.2. MAPE primal problem
The primal MAPE problem is obtained by replacing absolute error with absolute percentage error in a regularized kernel quantile formulation. Its dual reparameterization changes coefficient bounds according to the target magnitude, giving smaller responses a larger optimization range.
- MAPE formulation: MAPE kernelization replaces the standard regularized quantile objective with a relative-quantile form, without fixing the quantile parameter τ.The authors note that the practical relevance of this relative quantile remains to be assessed.
- Primal problem: The primal problem is rewritten using f(x)=⟨φ(x),w⟩ and auxiliary variables, with C=1/(nλ) controlling the regularization scale.The derivation introduces a variable vector and standard absolute-value constraints before obtaining the dual.
- Dual interpretation: For y_i>1 the MAPE dual optimization set shrinks, whereas for y_i<1 it expands, increasing the range available to α_i for smaller targets.This weighting gives smaller y_i values greater influence in the optimization.
- MAE comparison: As C becomes very large and regularization vanishes, MAPE and MAE can have the same optimal α_i because fitted values approach y_i.The authors connect this equivalence to potential overfitting when λ approaches zero.
6.3. A simulation study
The simulation study evaluates kernel MAPE regression against kernel median regression on translated sinus cardinal data. MAPE generally performs better in MAPE terms near zero and produces estimates that differ from the conditional median.
- Design: The experiments compare kernel MAPE regression with kernel median regression on simulated translated sinus cardinal data.The study uses a translation parameter a to vary proximity to zero and selects C by 5-fold cross-validation.
- Design: 1000 training points and 1000 test points are generated for each experimental setting.Test-set MAPE is reported for the MAPE and MAE estimators together with their regularization parameters.
- Results: In most cases, the MAPE of the MAPE estimator is lower than that of the MAE estimator, especially when y values are close to zero.This is the principal comparison reported in Table 1.
- Results: When conditional outcomes can be negative or positive, the MAPE estimator is near or equal to zero to avoid 100% error, whereas the MAE estimator is closer to the conditional median.The median-based estimate therefore has substantially higher MAPE in these cases.
- Results: The MAPE estimator changes shape strongly with the translation parameter, while the MAE estimator remains roughly the same up to translation.The MAPE changes under common translation of observed and predicted values, unlike the absolute error criterion.
- Results: The MAPE-optimal estimator is below the median, and the two estimates appear to converge for high translation values.The figure describes the MAPE estimate as the red curve and the median estimate as the blue curve.
7. Conclusion
The paper establishes an optimal MAPE model and consistency of MAPE-based empirical risk minimization, then demonstrates kernel-regression implementation on simulated data. Its practical scope is tied to targets that are positive by design and remain away from zero, while several theoretical questions remain open.
- Contributions: The paper shows the existence of an optimal MAPE model and consistency of empirical risk minimization based on the MAPE.These are the principal theoretical conclusions stated in the conclusion.
- Contributions: Simulated experiments illustrate efficient MAPE minimization through kernel regressions.The conclusion presents the kernel results as evidence from simulated data.
- Scope: The approach is presented as applicable when the target variable is positive by design and remains sufficiently far from zero, such as price prediction for expensive goods.This defines the practical context in which the authors describe the error measure as adapted.
- Open questions: The lower-bound assumption on |Y| remains a theoretical issue because upper-bound clipping strategies for MSE do not immediately transfer to MAPE.The lower bound is important in the approximation part of Theorem 2.
- Open questions: Consistency of empirical regularized risk minimization for MAPE remains an open question.The conclusion identifies this separately from the established ERM consistency result.