Source-linked AI summary
Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology
Alexei Botchkarev
TL;DR
Performance-metric research lacks a universally best measure and includes numerous classifications across regression, forecasting, and prognostics. This paper reviews the literature and analyzes metric structure to develop a four-category framework and a component-based typology for primary metrics. It identifies point distance, normalization, and aggregation as the three components determining primary-metric structure and properties.
Problem
The paper addresses the need to organize and understand a wide variety of performance metrics and classifications to facilitate their selection in regression, forecasting, and prognostics.
Method
The study identifies peer-reviewed papers, critically reviews the literature, and uses critical thinking and inductive reasoning to analyze metric structure and classifications.
Results
The proposed framework has four categories, while primary metrics are structured by point distance, normalization, and aggregation and the typology covers over 40 commonly used primary metrics.
Takeaways & Limitations
The typology supports metric selection by structuring decisions around point distance, normalization, and aggregation methods suited to the task’s purpose.
Takeaways & Limitations
Not all existing metrics directly match the typology, including MAAPE, and further conceptual and empirical research is proposed for other metric categories and numerical behavior.
Abstract
from arXiv · showhide
Performance metrics (error measures) are vital components of the evaluation frameworks in various fields. The intention of this study was to overview of a variety of performance metrics and approaches to their classification. The main goal of the study was to develop a typology that will help to improve our knowledge and understanding of metrics and facilitate their selection in machine learning regression, forecasting and prognostics. Based on the analysis of the structure of numerous performance metrics, we propose a framework of metrics which includes four (4) categories: primary metrics, extended metrics, composite metrics, and hybrid sets of metrics. The paper identified three (3) key components (dimensions) that determine the structure and properties of primary metrics: method of determining point distance, method of normalization, method of aggregation of point distances over a data set.
Introduction
Performance metrics support evaluation across machine learning regression, forecasting, and prognostics by comparing predictions with observed data or measuring forecast deviations. The paper reviews existing classifications and develops a typology to improve metric understanding and selection.
- Performance metrics compare trained-model predictions with actual testing data in machine learning regression experiments.
- Forecasting metrics measure how much forecasts deviate from observations to assess quality and choose forecasting methods.
- Prognostics for condition-based maintenance of critical systems relies heavily on performance metrics.
- Typologies organize knowledge by common resemblances or differences, helping define scope, simplify studies, retrieve information, and detect duplication.
- The paper’s typology aims to improve understanding of performance metrics and facilitate their use in regression, forecasting, and prognostics.
- The proposed framework contains primary, extended, composite, and hybrid sets of metrics, with primary-metric properties receiving most attention.
Methodology
The study combines literature identification and critical analysis to examine performance metrics in an interdisciplinary setting. It standardizes its terminology, metric notation, and actual-versus-predicted data representation for the analysis.
- The study’s objectives were to overview performance metrics and classifications and develop a typology of their structure and properties.
- Researchers identified peer-reviewed papers, critically reviewed the literature, applied critical thinking, and used inductive reasoning.
- The search retrieved and previewed around 500 papers, while over 80 were selected for detailed review and citation.
- Because the field is interdisciplinary, terminology may vary across domains.
- The paper uses typology for its own construct while retaining source authors’ terminology during the literature review.
- Metric abbreviations commonly use M for arithmetic mean, Md for median, and GM for geometric mean.
- Mathematical definitions appear in Appendix 2, implementations are available in R Studio and Azure Machine Learning Studio, and alternative definitions appear in Appendix 3.
- Metrics compare actual data A=(A_1,...,A_j) with predicted data P=(P_1,...,P_j), using field-specific synonyms for both.
Literature Review
The literature contains many performance metrics, competing classifications, and no universally best measure. The review highlights popularity, criticism, dimensional distinctions, and the need for more coherent organization.
- The literature includes many metrics and reviews, including 24 measures studied for air-quality forecasting.
- The most frequently identified metrics across surveys were MSE or RMSE, MAE, and MAPE.
- Metric preferences changed over time: MSE/RMSE led in the 1980s with 34 percent, while MAPE led in the 1990s and 2000s with over 40 percent.
- New metrics are developed by modifying existing measures for task-specific conditions or combining information from several measures.
- Researchers have not reached consensus on a single best metric because each measure emphasizes a particular aspect of model-error characteristics.
- Evaluation error is a random variable whose complete description requires a probability density function or moments, if they exist.
- Scale-dependent metrics have the variable’s physical units, whereas scale-free metrics commonly divide quantities with the same units, as in MAPE.
- MSE and other squared-error metrics can have changed dimensions, such as squared dollars rather than dollars.
Performance Metrics Framework
The paper organizes performance metrics into four categories based on how metrics are constructed and used. Primary, extended, composite, and hybrid metrics differ in normalization, combination, and output structure.
- The proposed framework contains primary, extended, composite, and hybrid sets of performance metrics.
- Primary metrics calculate point distance, normalize it, and aggregate point results over a data set.Examples include MAE, MSE, and sMAPE.
- Extended metrics add normalization after aggregation and include variants such as NRMSE_sd, NRMSE_max-min, and NRMSE_m.The normalizer may be the standard deviation, actual-data range, or actual-data mean.
- Composite metrics combine two or more primary metrics to produce a single result.Examples include MASE, RelMAE, and RelRMSE.
- Hybrid sets use two or more metrics separately to provide mutually complementary performance properties rather than one combined number.Examples address combinations such as bias and accuracy, including suites for solar-power forecasts and radiation-belt modeling.
- The paper identifies development of nonredundant, sufficiently informative hybrid metric sets as a research priority.Suggested topics include informational relationships, redundancy avoidance, and minimum adequate sets.
Primary Metrics Typology
The primary-metric typology represents each metric through point-distance, normalization, and aggregation choices. These components shape metric properties, while the surveyed options and applications have important scope boundaries.
- Three dimensions determine primary-metric structure: point-distance method, normalization method, and aggregation method.
- A generic primary metric sequentially determines point distance, optionally normalizes it, and aggregates the results across the complete data set.The formula uses actual values A_j, predicted values P_j, data-set size n, and method index z.
- Subtraction-based point distances include error, absolute error, and squared error, corresponding respectively to signed, nonnegative, and squared deviations.Subtraction distances are the most widely used point-distance type in the literature reviewed.
- Division-based distances include quotient and logarithmic quotient errors, but division distances are undefined when actual values are zero.Logarithmic quotient error is used in metrics such as MdLAR, while quotient distance also appears in entropy-type metrics.
- Signed error preserves direction and units but can cancel positive and negative deviations during aggregation, producing a falsely high accuracy.Its direction can nevertheless help assess forecasting bias.
- Absolute error prevents cancellation and retains data units, but it cannot determine skewness or bias and may complicate gradient calculation.It gives individual errors equal weights when aggregated by an arithmetic mean.
- Median aggregation is more resistant to outliers than the mean, although its lack of a clear mathematical formula complicates theoretical analysis.
Visualising Typology
The typology is visualized in tables that organize primary metrics by their components. The presentation is intentionally selective rather than comprehensive.
- Table 3 organizes 40 primary metrics by their components to clarify metric properties.
- The table omits uncommon components for visualization, placing metrics such as MNFB in an appendix instead.
Discussion
The paper’s two-level typology organizes performance metrics through their component structure and supports task-oriented selection, while covering over 40 commonly used primary metrics. Its conceptual scope is numerical regression, forecasting, and prognostics, and some metrics do not directly match the typology.
- Over 40 commonly used primary metrics are covered by the suggested typology.
- The typology distinguishes metrics through components that determine their properties, rather than treating each metric as an indivisible structure.
- Point distance, normalization, and aggregation methods provide a structured basis for selecting metrics according to task requirements.Squared error with arithmetic mean emphasizes outliers, whereas absolute error with geometric mean isolates them.
- The framework can also support creation of new metrics by selecting unoccupied combinations of existing point distances, normalizers, and aggregators.The study did not identify newly conceived types of these components.
- The study is conceptual and qualitative, focuses on numerical data and regression, and does not directly accommodate every existing metric.MAAPE is given as an example of a metric that does not directly match the typology.
Concluding Remarks
The paper organizes performance metrics into a four-category framework and characterizes primary metrics through three structural dimensions. It proposes a typology, generic formula, and visualization intended to improve understanding and practical selection of metrics.
- The framework contains primary metrics, extended metrics, composite metrics, and hybrid sets of metrics.
- Three dimensions determine primary-metric structure: point-distance calculation, normalization, and aggregation across the data set.
- The primary-metrics typology covers more than 40 commonly used metrics.
- The generic formula sequentially determines point distance, normalizes it, and aggregates the results over the complete data set.
- A visualization chart supports assessing existing metrics and creating new ones.
- The findings can facilitate teaching, expedite practitioners’ metric selection, and streamline academics’ development of new metrics.
- Future work includes analyzing extended, composite, and hybrid metrics and empirically testing associations between conceptual properties and numerical behavior.
Appendix 1. List of metrics abbreviations Metric Abbreviation Metric Name
The appendix lists abbreviations and names for performance metrics spanning distances, errors, normalized measures, relative measures, and aggregation variants.
- ED denotes Euclidean Distance (L2-norm), while NCSD and SquD denote Neyman Chi-Square Distance and Squared Chi-square Distance.
- Other listed measures use geometric, harmonic, logarithmic, factor, inner-product, symmetric, or divergence-based constructions.
- Common absolute-error measures include MAD, MAGE, MAE-related variants, MaxAE, and MdAE.
- Percentage-error measures include MAPE, MPE, MSPE, RMSPE, sMAPE, SMdAPE, and related median or root variants.
- Relative and scaled measures include MARE, MRAE, RAE, RelRMSE, RMAE, RSE, RRSE, MASE, and RMSSE.
- Normalized RMSE variants use the actual-data mean, range, or standard deviation, while NMSE is normalized by variance.
Appendix 2. Metrics mathematical definitions
The appendix introduces notation for metric definitions and explains that metrics are organized by framework category and error type.
- The notation defines actual values A_j, their mean A̅, predicted values P_j, errors e_j = A_j − P_j, and data-set size n.
- Metrics are listed by category—primary, extended, composite, or hybrid sets—and then by error type.
- The appendix labels its main column “Metric Name,” with alternative names supplied in brackets.
PRIMARY METRICS
Primary metrics differ through their distance, normalization, and aggregation choices, producing absolute, relative, percentage, normalized, and alternative aggregation forms.
- MPE and fractional absolute error represent primary metrics based on percentage or fractional error forms.
- MAE and its listed alternatives include Mean Absolute Deviation, Mean Absolute Gross Error, and Mean Character Difference.
- MARE, MRAE, MdRAE, GMAE, and GMRAE provide relative, median, or geometric absolute-error forms.
- SMdAPE, RMSPE, and RMdSPE apply median or root-mean-square aggregation to percentage errors.
- RMSE, MD, and NMSE use squared, Manhattan, or variance-normalized distance constructions.
- MdLAR and MNAFE use median log accuracy-ratio and normalized absolute-factor-error formulations.
- NRMSE_m, NRMSE_sd, and NRMSE_mm normalize RMSE by the actual-data mean, standard deviation, or maximum–minimum difference.
Appendix 3. Performance metrics alternative mathematical definitions Metric Abbreviation
The appendix lists alternative mathematical definitions for two relative squared-error metrics: RSE and RRSE.
- RSE stands for Relative Squared Error.
- RRSE stands for Root Relative Squared Error.