PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning Advances for Time Series Forecasting

Ricardo P. Masini, Marcelo C. Medeiros, Eduardo F. Mendes

arXiv:2012.12802v3econ.EMcs.LGstat.APstat.ML

TL;DR

Reliable forecasting with large economic and financial datasets motivates a survey of supervised machine-learning and high-dimensional methods. The paper reviews linear, nonlinear, ensemble, hybrid, and predictive-ability approaches, finding empirical usefulness for nonlinear models combined with large datasets while identifying substantial theoretical gaps for dependent and high-dimensional settings.

  • Problem

    Reliable and robust economic and financial forecasts are important with vast datasets, motivating synthesis of supervised machine-learning methods for time-series forecasting.

  • Method

    The paper reviews linear and nonlinear forecasting methods, including penalized regressions, neural networks, tree-based models, ensembles, hybrids, and predictive-ability tests.

  • Results

    Recent empirical evidence shows that nonlinear machine-learning models combined with large datasets can be extremely useful for economic forecasting.

  • Takeaways & Limitations

    The survey supports using nonlinear machine-learning models with large datasets for economic forecasting, within the evidence reviewed.

  • Takeaways & Limitations

    Theoretical results for random forests and boosted trees largely concern independent, low-dimensional data, while high-dimensional neural-network behavior remains under study.

Abstract

from arXiv · show

In this paper we survey the most recent advances in supervised machine learning and high-dimensional models for time series forecasting. We consider both linear and nonlinear alternatives. Among the linear methods we pay special attention to penalized regressions and ensemble of models. The nonlinear methods considered in the paper include shallow and deep neural networks, in their feed-forward and recurrent versions, and tree-based methods, such as random forests and boosted trees. We also consider ensemble and hybrid models by combining ingredients from different alternatives. Tests for superior predictive ability are briefly reviewed. Finally, we discuss application of machine learning in economics and finance and provide an illustration with high-frequency financial data.

1 Introduction

The paper surveys supervised machine-learning methods for economic and financial time-series forecasting, covering linear, nonlinear, ensemble, hybrid, and predictive-ability approaches. It frames forecasting as learning functions from input-output data and evaluating their predictive accuracy.

  • Supervised learning maps explanatory variables to dependent variables using input-output pairs, distinguishing it from unsupervised and reinforcement learning.
  • Linear methods emphasize regularized estimators, including Ridge, LASSO, penalty extensions, and inference for dependent data.
  • Nonlinear coverage includes sieve approximations, shallow and deep feed-forward networks, recurrent neural networks, random forests, and boosted trees.
  • Ensemble methods such as Bagging and Complete Subset Regression are surveyed alongside hybrid methods combining linear and nonlinear ideas.
  • The survey also reviews superior predictive-ability tests before presenting an empirical application and discussing economic and financial forecasting.
  • The forecasting framework targets an unknown function fh across horizons, estimates it from data, and evaluates accuracy using losses such as MAPE, MSPE, and RMSPE.

2 Penalized Linear Models

This section reviews penalized linear regression for high-dimensional forecasting, where unrestricted OLS can be nonunique or overfit. It compares penalties designed for shrinkage, variable selection, stability, and improved prediction.

  • When n > T, OLS is not unique because the sample covariance is rank deficient and models can completely overfit.
  • Penalized regression restricts the OLS solution, accepting bias to reduce mean squared error in large or highly correlated designs.
  • Ridge uses an ℓ2 penalty to stabilize correlated regressors, shrinking coefficients toward zero without selecting predictors exactly.
  • LASSO uses an ℓ1 penalty to produce sparse solutions and perform variable selection when regressors greatly outnumber observations.
  • Adaptive LASSO adds data-dependent weights, retaining sparse solutions while targeting the oracle property and handling more variables than observations.
  • Elastic net combines LASSO selection with ridge stabilization, particularly improving prediction when predictors are highly correlated.

2.1 Theoretical properties

This section surveys theoretical guarantees for penalized estimators in finite- and high-dimensional time-series settings. The main properties are selection consistency, oracle behavior, and prediction bounds under model-specific assumptions.

  • Theoretical work studies model-selection consistency, oracle properties, and oracle bounds when the number of predictors is fixed or grows with the sample.
  • Model-selection consistency requires the estimated nonzero coefficient indices to converge in probability to the true active set.
  • Lq-consistency measures whether estimation error converges toward zero under a specified norm.
  • Model-selection consistency and Lq-consistency are distinct properties, each generally requiring specific assumptions.
  • Oracle risk bounds require strong curvature conditions and tuning-rate restrictions involving the number of nonzero parameters.
  • Time-series results cover dependent, non-Gaussian, heavy-tailed, mixed-frequency, integrated, and potentially cointegrated settings.

2.2 Inference

This section reviews inference after regularized model selection, emphasizing methods that recover asymptotic normality or valid confidence intervals under selection and dependence. It also highlights assumptions behind these guarantees.

  • Penalized-estimator asymptotics often assume sufficiently large nonzero coefficients, while uniform post-selection limits can differ substantially from Gaussian behavior.
  • The desparsified LASSO adds an approximate inverse-covariance correction to the LASSO estimate, removing exact zeros and enabling inference.
  • Under regularity conditions, the desparsified estimator satisfies asymptotic Gaussianity, with validity depending critically on approximating the inverse covariance matrix.
  • Nodewise regressions estimate the inverse-covariance approximation by regressing each predictor on all remaining predictors.
  • Double selection combines active regressors from outcome and treatment-control auxiliary regressions before estimating the target parameters in a lower-dimensional regression.
  • Time-series extensions address dependent, serially correlated, heteroskedastic, and non-Gaussian processes, including cases where regressors grow faster than the sample.

3 Nonlinear Models

This section introduces nonlinear sieve methods for forecasting when linearity is too restrictive. It approximates infinite-dimensional function spaces with finite-dimensional spaces and focuses on neural networks and regression trees.

  • When the forecasting function is unknown and linearity is too restrictive, the estimation problem requires more flexible nonlinear forms.
  • Searching directly over an infinite-dimensional function space is infeasible, motivating restrictions or finite-dimensional approximations.
  • Sieve methods replace the original space with finite-dimensional spaces that converge toward it in an appropriate norm.
  • The approximating function is represented through basis functions whose number and space dimension can depend on the sample size.
  • Polynomial spaces illustrate sieves because finite-degree polynomials are simpler while converging toward a dense function class.
  • Known basis functions yield linear parameter problems solvable by ordinary least squares or penalized estimation, whereas parameter-indexed bases require nonlinear least squares.

3.1 Neural Networks

Neural networks provide nonlinear sieve models for forecasting, ranging from shallow feed-forward architectures to deep and recurrent networks. The section covers their approximation structure, regularization, dropout, and LSTM-based sequence modeling.

  • Shallow Neural Networks: Feed-forward neural networks approximate forecasting functions through activation functions applied to hidden-neuron linear combinations of covariates.A single hidden-layer network maps inputs through hidden units to an output layer; activation functions include squashing functions and radial-basis functions.
  • Shallow Neural Networks: Universal-approximation results show that sufficiently wide neural networks can approximate broad classes of functions to arbitrary accuracy.The cited results cover continuous and Borel measurable functions, and later establish the same approximation property for ReLU activations.
  • Regularization: Because neural-network parameter vectors can become very large, estimation uses regularization, including penalty methods and dropout.Dropout randomly removes neurons and connections during estimation, samples thinned networks, and rescales the final unthinned network's weights.
  • Regularization: Dropout is reported to significantly reduce overfitting and improve performance over other regularization methods.The paper describes the improvement as applying to the dropout procedure used for neural-network estimation.
  • Deep and Recurrent Neural Networks: Deep neural networks generalize shallow networks by composing multiple hidden layers, while recurrent networks use hidden-state memory to process input sequences.Deep networks may approximate compositional functions with fewer training parameters and lower sample complexity, whereas RNNs can suffer vanishing or exploding gradients.
  • Deep and Recurrent Neural Networks: LSTM networks address recurrent-network gradient difficulties through cells and gates that filter, attenuate, and combine prior state with new information.The forget, input, and output gates regulate information flow, and predictions are formed as linear combinations of hidden states.

3.2 Regression Trees

Regression trees approximate nonlinear forecasting functions by recursively partitioning covariate space and assigning local predictions to terminal regions. The section then develops random forests and boosted trees as ensemble alternatives.

  • Regression Trees: Regression trees use recursive covariate-space partitioning to produce local predictions in terminal nodes.In the basketball example, terminal-node predictions equal the average score of observations in each group.
  • Regression Trees: Each tree split selects an optimal variable and observation, such as height at 1.85m in the illustrative example.The example first separates players taller than 1.85m from shorter players, then splits groups by weight.
  • Tree Representation: A regression-tree approximation can be written as a linear regression on terminal-node indicators formed from products of split-path functions.The indicator representation assigns observations to regions defined by the tree's recursive splits.
  • Random Forests: Random forests average regression trees fitted on bootstrap samples and randomly selected subsets of regressors.For time series, the paper uses block bootstrap samples and chooses tree size to retain a minimum number of observations in each region.
  • Boosting: Boosted trees sequentially fit small trees to the gradient of the loss, which under quadratic loss means repeatedly refitting previous residuals.A shrinkage parameter controls the learning rate; values near one improve in-sample fit but increase overfitting risk and can worsen out-of-sample results.

3.3 Inference

Inference for nonlinear machine-learning models is difficult, so the paper reviews parametric interpretations and semiparametric sieve approaches. It also describes how inference results depend on covariate dimension and data dependence.

  • Inference Strategies: Nonlinear-machine-learning inference can be approached by interpreting specific nonlinear specifications as parametric models, though this restricts applications to particular settings.The paper gives smooth-transition regressions as an example of this strategy.
  • Inference Strategies: Sieve methods provide a broader route for inference in models such as splines and feed-forward neural networks.Under regularity conditions, cited work establishes consistency and asymptotic normality for semiparametric sieve approximations.
  • Inference Strategies: The reviewed semiparametric framework models a nonlinear component nonparametrically, but the classical results assume fixed covariate dimension.The cited setup estimates both linear and nonlinear components under that fixed-dimension condition.
  • High-Dimensional Inference: When the number of covariates diverges with sample size, earlier asymptotic results are invalid and double-machine-learning methods provide a generalization.The section also notes semiparametric inference results for deep neural networks.
  • Inference for Trees: Random-forest inference results have largely been developed for IID data, with newer work extending concentration and consistency results to nonlinear autoregressive processes.The cited time-series results cover regression trees and a broad class of random forests.

4 Other Methods

The paper surveys additional forecasting methods based on bootstrap aggregation, complete-subset combinations, and hybrid models. These approaches address predictor dimensionality, model instability, computational burden, and combinations of linear and nonlinear structure.

  • Bagging: Bagging, or bootstrap aggregating, reduces the variance of unstable predictors by averaging forecasts across bootstrap samples.In time-series applications, bootstrap construction must preserve temporal dependence through block resampling.
  • Bagging: Time-series bagging arranges forecast tuples, draws blocks of rows with replacement, estimates bootstrap forecasts, and averages them.The described algorithm uses block bootstrap samples to account for serial dependence.
  • Bagging: The standard bagging procedure is infeasible when the number of predictors exceeds the sample size, requiring algorithmic modification.The modified approach first screens regressors using univariate significance tests before applying the remaining steps.
  • Complete Subset Regression: Complete Subset Regression combines all linear models with a fixed number of predictors because exhaustive optimal-subset selection is computationally demanding.With n predictors, the number of k-variable models is n! / ((n-k)!k!), and n = 30 with k = 8 yields 5,852,925 regressions.
  • Hybrid Methods: Hybrid methods combine linear and nonlinear components through LASSO-neural-network, linear-plus-neural-network, and random-forest regression specifications.The reviewed motivations include variable selection, interpretability of nonlinear dependence, and separating variable-selection from model-fitting roles.
  • Forecast Combinations: Forecast combinations can shrink selected weights toward equality, producing forecasts close to simple averages of surviving forecasts.The paper notes that such regularized combinations and subset-average combinations are closely connected.

5 Forecast Comparison

As machine-learning forecasting models proliferate, predictive-ability tests help compare them, including settings with multiple or nested competitors and changing relative performance.

  • Forecasting researchers need statistical tools to compare the growing number of machine-learning models and methods.
  • The Diebold–Mariano test compares two methods under equal unconditional expected loss, with a simple t-test and small-sample adjustment available.
  • Nested competing models can make the original Diebold–Mariano statistic diverge under the null, motivating alternative valid procedures.
  • The unconditional superior predictive ability test accommodates more than two competitors, but can be conservative when inferior methods are included.
  • Conditional predictive-ability tests assess not only whether a model is superior, but also when it outperforms alternatives.

6 Applications of Machine Learning Methods to Economic and Financial Forecasting

Applications span penalized regressions, nonlinear models, ensembles, and high-frequency financial forecasting. Reported evidence includes gains over conventional benchmarks, while the empirical illustration finds adaLASSO-enhanced HAR strongest and gains reaching 10% during the Covid-19 pandemic.

  • 6.1 Linear Methods: Penalized regressions are an established option for economic and financial forecasting, with applications across macroeconomic and financial prediction.
  • 6.1 Linear Methods: High-dimensional penalized models often improve macroeconomic forecasts and can outperform autoregressive and factor-model benchmarks for inflation.
  • 6.1 Linear Methods: Penalized VARs outperform benchmarks for 30 Dow Jones stocks and improve mean-variance portfolio construction, while LASSO forecasts use predictors linked to unexpected news.
  • 6.2 Nonlinear Methods: Earlier nonlinear forecasting studies found only marginal performance differences and no strong evidence that nonlinear models were superior.
  • 6.2 Nonlinear Methods: Recent large-dataset studies report greater accuracy or economic gains from nonlinear models, with tree-based methods and neural networks among the strongest performers.
  • 6.3 Empirical Illustration: The illustration forecasts BOVESPA daily realized variance using five-minute returns, other major indexes, an HAR benchmark, adaLASSO, random forests, and neural networks.
  • 6.3 Empirical Illustration: The empirical exercise uses 4,200 observations, rolling windows of 1,500 observations, and daily model re-estimation, including the 2007–2008 crisis and 2020 pandemic.
  • 6.3 Empirical Illustration: 10%: Forecasting gains reach 10% during the Covid-19 pandemic, with adaLASSO-enhanced HAR winning, random forests underperforming, and neural networks beating HAR.

7 Conclusions and the Road Ahead

The paper reviews linear and nonlinear machine-learning alternatives, ensemble and hybrid methods, and tests for superior predictive ability in time-series forecasting. It concludes that nonlinear models with large datasets can be highly useful, while identifying theoretical gaps and directions for future research.

  • The survey covers linear and nonlinear machine-learning alternatives, ensemble and hybrid models, and tests for superior predictive ability.
  • Theoretical results for Bagging and complete subset regression remain limited, especially relative to advances for penalized linear regression with dependent data.
  • Theoretical results for random forests and boosted trees mainly concern independent, identically distributed, low-dimensional data, while high-dimensional neural-network behavior remains under study.
  • Nonlinear machine-learning models combined with large datasets can be extremely useful for economic forecasting.
  • Further developments: Future work includes developing dependent-data theory for Bagging and Boosting, including consistency and asymptotic normality for random forecast estimators.
  • Further developments: Other priorities are understanding nonlinear variable selection, developing inferential methods for variable importance, incorporating unstructured data, and evaluating nowcasting and structurally unstable environments.
  • Scope: The survey is non-exhaustive and leaves out methods including Support Vector Regressions, autoencoders, and nonlinear factor models.
Loading 2012.12802v3…