Source-linked AI summary
A Comprehensive Survey of Regression Based Loss Functions for Time Series Forecasting
Aryan Jadon, Avinash Patil, Shruti Jadon
TL;DR
Time series forecasting requires suitable objectives for varied data conditions, but the appropriate loss function remains uncertain. The paper surveys 14 regression losses, organizes their useful circumstances, and evaluates them across four datasets. Its results show that the best-performing losses vary by dataset, while quantile loss performs well in each reported dataset.
Problem
Time series forecasting involves varied conditions such as biased data, outliers, skewness, and unknown distributions, making loss-function selection uncertain.
Method
The paper summarizes 14 regression loss functions, identifies circumstances for their use, and compares them across four datasets with Temporal Fusion Transformers.
Results
Quantile loss performed well on the Electricity, traffic, Favorita, and volatility datasets, while other top-performing losses differed by dataset.
Takeaways & Limitations
Loss functions should be selected according to dataset properties and objectives rather than treated as universally interchangeable.
Abstract
from arXiv · showhide
Time Series Forecasting has been an active area of research due to its many applications ranging from network usage prediction, resource allocation, anomaly detection, and predictive maintenance. Numerous publications published in the last five years have proposed diverse sets of objective loss functions to address cases such as biased data, long-term forecasting, multicollinear features, etc. In this paper, we have summarized 14 well-known regression loss functions commonly used for time series forecasting and listed out the circumstances where their application can aid in faster and better model convergence. We have also demonstrated how certain categories of loss functions perform well across all data sets and can be considered as a baseline objective function in circumstances where the distribution of the data is unknown. Our code is available at GitHub: https://github.com/aryan-jadon/Regression-Loss-Functions-in-Time-Series-Forecasting-Tensorflow.
I. INTRODUCTION
The paper surveys regression loss functions for time series forecasting, where temporal structure and data characteristics complicate objective selection. It summarizes 14 losses and compares them across forecasting datasets using Temporal Fusion Transformers.
- Time series forecasting uses historical time-stamped data to learn patterns that inform future decisions.
- The paper summarizes 14 regression loss functions and describes circumstances where each may support forecasting optimization.
- The study compares all loss functions on Electricity, traffic, grocery-sales, and volatility datasets using Temporal Fusion Transformers.
- Loss-function choice remains uncertain because forecasting data and objectives vary in distribution, boundaries, outliers, and computational requirements.
- Forecasting models must learn level, cyclicity, trend, and seasonality while avoiding random noise.
B. Mean Squared Error (MSE)
MSE squares prediction errors, giving larger mistakes greater weight and producing a nonnegative objective that optimization algorithms can use. The supplied passages also contrast MSE with bias error, whose signed errors may cancel.
- MSE is the mean of squared differences between predictions and actual values, also known as quadratic loss.
- Squaring errors gives outliers more weight and creates a smooth gradient for minor errors.
- Optimization algorithms can benefit from MSE’s strong penalization of large errors when estimating model parameters.
- MSE is nonnegative, ranges from 0 to infinity, and grows exponentially as mistakes increase.
- MBE differs from absolute-error measures because positive and negative prediction errors can cancel one another.
D. Relative Absolute Error (RAE)
Relative Absolute Error (RAE) normalizes total absolute prediction error by the absolute deviation of actual values from their mean. It ranges from 0 to 1, with values near zero indicating better models.
- RAE divides total absolute error by the absolute difference between actual values and their mean.
- RAE is a ratio-based metric for assessing predictive-model efficacy.
- RAE ranges from 0 to 1, with zero as the best value and values near zero characteristic of good models.
- Relative squared error normalizes total squared error by the squared error of a predictor that outputs the mean actual value.
F. Mean Absolute Percentage Error (MAPE)
Mean Absolute Percentage Error (MAPE) expresses forecast error as an average absolute percentage, preventing positive and negative errors from canceling. It is widely used when data contain no outliers.
- MAPE calculates the average absolute percentage error for each time period.
- Using absolute errors prevents positive and negative percentage errors from canceling each other out.
- MAPE is widely used because forecast errors are expressed in percent units.
- MAPE works well when the data contain no outliers and is often used in regression analysis and model evaluation.
H. Mean Squared Logarithmic Error (MSLE)
Mean Squared Logarithmic Error (MSLE) measures squared differences after logarithmically transforming actual and predicted values. The transformation reduces emphasis on percentage differences involving large values and requires adding one because log(0) is undefined.
- MSLE measures the difference between actual and expected values after logarithmic transformation.
- The logarithm makes MSLE roughly treat small discrepancies between small values and large disparities between large values similarly.
- MSLE can be interpreted as a measure of the ratio between true and predicted values.
- One is added to actual and predicted values because log(0) is undefined.
I. Root Mean Squared Logarithmic Error (RMSLE)
Root Mean Squared Logarithmic Error (RMSLE) applies logarithms to actual and predicted values before computing their difference. It is resistant to outliers and penalizes underestimation more than overestimation.
- RMSLE is computed by applying logarithms to actual and predicted numbers and subtracting them.
- RMSLE penalizes predictions below the actual value more than predictions above it.
- RMSLE is useful when overestimation is acceptable but underestimation is unacceptable.
- Normalized RMSE connects RMSE to the observed range, enabling comparisons between models with different scales.
K. Relative Root Mean Squared Error (RRMSE)
RRMSE is a dimensionless RMSE variant that normalizes error against actual values, enabling comparisons across measurement approaches. Its reported score bands classify model accuracy from excellent to poor.
- Definition: RRMSE scales and normalizes root mean square error against actual values, allowing comparison across measurement approaches.Unlike RMSE, it is dimensionless.
- Interpretation: RRMSE scores below 10% indicate excellent accuracy, while scores above 30% indicate poor accuracy.Scores from 10% to 20% are classified as good, and 20% to 30% as fair.
- Equation: The RRMSE equation uses N data samples, true values y_i, and predicted values ŷ_i.The supplied passages identify these symbols but do not fully preserve the displayed equation.
- Summary: The paper directs readers to Table XI for the advantages and disadvantages of RRMSE loss.
M. LogCosh Loss
LogCosh computes the logarithm of the hyperbolic cosine of prediction error and combines smooth quadratic-like behavior with reduced sensitivity to large errors. The paper also situates it among the summarized loss functions and contrasts its role with quantile loss.
- Definition: LogCosh computes the logarithm of the hyperbolic cosine of prediction error.It is described as smoother than quadratic loss and less affected by significant prediction errors.
- Properties: The paper reports that LogCosh combines linear and quadratic scoring techniques and is extremely close to Huber loss.
- Properties: LogCosh is twice differentiable, whereas the paper states that Huber loss can only be differentiated once.The text notes that some optimization approaches favor twice-differentiable functions.
- Survey scope: The paper identifies LogCosh as one of 14 evaluated regression loss functions and refers readers to Table XIII for its advantages and disadvantages.
- Contrast with Quantile Loss: Quantile loss forecasts quantiles without assuming a parametric response distribution and can provide prediction intervals for non-constant variance residuals.The quantile parameter controls the balance between positive and negative errors.
IV. EXPERIMENTS
The experiments evaluate forecasting loss functions across four datasets representing electricity consumption, traffic occupancy, grocery sales, and financial volatility. These datasets cover distinct temporal prediction settings and data types.
- Datasets: The electricity dataset contains electricity consumption data from 370 points or clients.
- Datasets: PEMS-SF contains 15 months of daily records describing occupancy rates across San Francisco Bay Area freeway lanes.The dataset contains 440 daily records and occupancy values range from 0 to 1.
- Datasets: The Corporación Favorita dataset contains dates, store and item information, promotion status, and unit sales.
- Datasets: The Volatility dataset contains daily non-parametric measures of historical volatility for financial assets or indexes.
B. Performance Metrics
The experiments evaluate 14 loss functions on four datasets using P10, P50, and P90 metrics, with quantile forecasts providing upper and lower prediction bounds. Performance varies by dataset, although Quantile Loss appears among the better-performing losses in all four reported observations.
- Evaluation setup: The experiment evaluates all loss functions using P10, P50, and P90 metrics.These correspond to the 10th percentile, median, and 90th percentile forecasts.
- Quantile evaluation: Quantile forecasts at P10 and P90 define an 80% interval expected to contain the observed value 80% of the time.The interval is represented by the shaded region between P10 and P90.
- Evaluation setup: The study uses Temporal Fusion Transformers with 1000 training samples, 100 validation samples, and 14 loss functions.The paper states that other losses were resolved into selected functions or were unsuitable for the listed datasets.
- Dataset results: On Electricity, Quantile Loss, MSE, and RRMSE performed well.
- Dataset results: On the traffic dataset, Log Cosh, Quantile, and MAE performed better.
- Dataset results: On Favorita, RAE, LogCosh, and Quantile outperformed other objective functions, while on Volatility, Quantile, MAE, and Huber performed better.
V. CONCLUSION
The paper frames loss-function selection as objective-dependent rather than universal, and summarizes practical choices for different data conditions. It presents 14 regression loss functions and proposes using the study as a baseline for time series forecasting applications.
- Loss-function choice depends on outliers, data skewness, model requirements, computational constraints, and performance requirements.
- The study organizes cases in which particular loss functions may be useful, including datasets containing outliers.
- Mean Squared Error is recommended with many outliers, whereas Mean Absolute Error is preferred when outliers are fewer.
- LogCosh is suggested for maintaining balance, while Quantile loss is suggested when the objective is based on percentiles.
- The paper summarizes 14 well-known loss functions and develops a tractable form intended to support improved and more accurate optimization.