PaperlayerView exact revision on arXiv

Source-linked AI summary

How is Machine Learning Useful for Macroeconomic Forecasting?

Philippe Goulet Coulombe, Maxime Leroux, Dalibor Stevanovic, Stéphane Surprenant

arXiv:2008.12477v1econ.EMstat.APstat.ML

TL;DR

The paper addresses the difficult challenge of understanding machine-learning procedures for macroeconomic prediction. It finds that nonlinearities, particularly those created by kernel methods, drive forecasting gains, while noting limits from model instability and extreme events.

  • Problem

    Understanding machine-learning procedures when predicting macroeconomic outcomes remains a difficult challenge.

  • Method

    The analysis considers hyperparameters, including penalty and approximator settings, and the kernel trick that creates nonlinearities.

  • Results

    Nonlinearities are the true game changer in data-rich environments, and most SVR benefits come from kernel-created nonlinearities rather than alternative loss functions.

  • Takeaways & Limitations

    Machine learning is useful for macroeconomic forecasting by mostly capturing important nonlinearities.

  • Takeaways & Limitations

    The evaluation does not address concerns about model instability or methods designed to account for relevant breaks.

Abstract

from arXiv · show

We move beyond "Is Machine Learning Useful for Macroeconomic Forecasting?" by adding the "how". The current forecasting literature has focused on matching specific variables and horizons with a particularly successful algorithm. In contrast, we study the usefulness of the underlying features driving ML gains over standard macroeconometric methods. We distinguish four so-called features (nonlinearities, regularization, cross-validation and alternative loss function) and study their behavior in both the data-rich and data-poor environments. To do so, we design experiments that allow to identify the "treatment" effects of interest. We conclude that (i) nonlinearity is the true game changer for macroeconomic prediction, (ii) the standard factor model remains the best regularization, (iii) K-fold cross-validation is the best practice and (iv) the $L_2$ is preferred to the $\bar ε$-insensitive in-sample loss. The forecasting gains of nonlinear techniques are associated with high macroeconomic uncertainty, financial stress and housing bubble bursts. This suggests that Machine Learning is useful for macroeconomic forecasting by mostly capturing important nonlinearities that arise in the context of uncertainty and financial frictions.

1 Introduction

The paper shifts macroeconomic ML forecasting from identifying a winning algorithm to identifying which ML features generate forecasting gains. It evaluates nonlinearities, regularization, hyperparameter selection, and loss functions across data-rich and data-poor settings.

  • The literature contains many ML tools, but evaluations often consider few models and forecasting targets, leaving their underlying drivers insufficiently understood.
  • Nonlinearities improve forecasting accuracy for all variables, especially at long horizons, and are identified as the most salient ML feature.
  • Alternative regularization methods do not improve over the factor model, indicating that factor representations provide effective dimensionality reduction.
  • K-fold cross-validation and, when available, standard BIC perform best on average, while replacing quadratic loss with ε-insensitive loss is generally unhelpful.
  • Nonlinear ML gains are largest under high uncertainty, tight financial conditions, and housing-bubble bursts, particularly for data-rich models.The results suggest that combining nonlinearities with factors captures complex macroeconomic relationships.

2 Making Predictions with Machine Learning and Big Data

The paper formulates ML forecasting as a prediction problem balancing approximation, estimation, and intrinsic errors, then varies four ML features in data-poor and data-rich forecasting models. It evaluates direct forecasts using macroeconomic targets and predictor panels through a pseudo-out-of-sample treatment design.

  • Forecast error consists of intrinsic, estimation, and approximation components, with more data reducing estimation error and model flexibility controlling approximation error.
  • The four ML features are function flexibility, regularization, hyperparameters, and loss function within a general penalized prediction setup.
  • Data-Poor versus Data-Rich Environments: Data-poor models use lagged target values, whereas data-rich panels additionally include many exogenous predictors.
  • The study compares autoregressive and factor-augmented autoregressive models, with factors typically estimated by principal component analysis.
  • Nonlinear models apply flexible transformations to lagged targets and principal components, allowing latent variables to affect targets flexibly.
  • A pseudo-out-of-sample forecasting horse race generates errors from models differing in the four features and estimates their treatment effects.

3 Four Features of ML

This section isolates how machine-learning forecasting differs from standard macroeconometric methods, emphasizing nonlinearities and alternative regularization designs. It implements nonlinear predictors with kernels and random forests, while varying shrinkage schemes to identify their marginal contributions.

  • Nonlinearities: Nonlinear methods address approximation error from imposing a linear forecasting function when the data-generating process is complex.The paper applies the Kernel trick and random forests to baseline models to test whether generated nonlinearities improve forecasts.
  • Nonlinearities: The Kernel trick implicitly represents numerous basis functions through a reproducing kernel, avoiding explicit specification of those functions.The paper uses the standard radial basis function kernel, with its tuning parameter selected by cross-validation.
  • Nonlinearities: Kernel Ridge Regression makes data-rich and data-poor baseline models nonlinear while retaining a closed-form solution based on matrix inversion.The dual representation expresses predictions through kernel evaluations and a dual Ridge solution.
  • Nonlinearities: Random forests reduce the high variance and overfitting associated with individual regression trees by averaging trees trained on randomized subsamples and regressors.Each tree partitions feature space into regions, and the ensemble averages the resulting tree forecasts.
  • Regularization: The paper varies Elastic Net shrinkage through α and transformation operator B(), generating Lasso, Ridge, Elastic Net, factor-based, and ARDI specifications.The standard ARDI approach extracts principal components and uses them in an ARDL model; alternative schemes create identifying variation for treatment effects.
  • Contribution: The section evaluates which model-selection method is appropriate for macroeconomic forecasting and how much ML gains reflect its underlying features.The contributions explicitly include assessing model selection and isolating gains or losses attributable to widespread ML ingredients.

4 Empirical setup

The empirical setup uses historical US macroeconomic and financial data to compare forecasting models in a pseudo-out-of-sample experiment. Models are recursively estimated over multiple horizons, tuned with alternative criteria, and evaluated against an ARDI-BIC reference using forecast-accuracy tests and treatment-effect regressions.

  • Data: The study uses FRED-MD, containing 134 monthly US macroeconomic and financial indicators observed from 1960M01 to 2017M12.Transformations follow McCracken and Ng (2016) to achieve stationarity, and FRED-MD is retained partly because it supports a long test set.
  • Data: The forecasts target five US indicators: industrial production, unemployment, inflation, the Treasury–federal funds spread, and housing starts.Targets use growth rates, changes, or levels according to the series’ integration properties.
  • Experiment design: The pseudo-out-of-sample period is 1980M01–2017M12, with horizons of 1, 3, 9, 12, and 24 months, yielding 456 evaluation periods per horizon.All models are estimated recursively with an expanding window.
  • Experiment design: Hyperparameters are selected using AIC, BIC, pseudo-out-of-sample cross-validation, and K-fold cross-validation with k = 5.Hyperparameters are re-optimized every two years, while pseudo-out-of-sample validation uses the last 25% of the in-sample period.
  • Limitations: The experiment uses an expanding window rather than a rolling window, while acknowledging that rolling windows may be more robust to model instability.Evaluating instability and structural breaks lies beyond the paper’s scope.
  • Evaluation: Forecast quality is measured by root MSPE, with Diebold–Mariano tests comparing models against the ARDI-BIC reference and Model Confidence Sets identifying best-model subsets.Regression analysis estimates the treatment effect of each ML ingredient.

5 Results

Across five macroeconomic variables, nonlinearities and additional information generally improve forecasts, especially at longer horizons, while standard factor regularization remains difficult to improve upon. Cross-validation and alternative loss effects are mixed or often unfavorable, with some advantages concentrated during recessions and other economically distinct periods.

  • Overall Predictive Performance: Nonlinear and data-rich models generally perform best, and their marginal benefits tend to increase with the forecast horizon.This pattern holds across variables, although spread and housing starts include exceptions.
  • Overall Predictive Performance: Nonlinear models substantially improve forecasting accuracy in almost all situations, with treatment effects reaching up to 23% in pseudo-R2.The gains are significant for all horizons in industrial production and spread, and for most cases in unemployment, inflation, and housing starts.
  • Hyperparameter Optimization: The standard factor model remains the strongest regularization benchmark, while alternative shrinkage can reduce forecast accuracy by up to 20% of pseudo-R2.On average, competing shrinkage schemes perform similarly or worse than ARDI-based factor reduction.
  • Disentangling ML Treatment Effects: Replacing squared loss with the SVR loss function is generally damaging, while nonlinear effects are present during both recessions and expansions and are stronger during expansions.Forecast improvements also show cyclical peaks around the 1981 and 2008 recessions and a decline during the Great Moderation.

6 When are the ML Nonlinearities Important?

The section examines when nonlinear ML treatment effects are strongest by relating them to uncertainty, financial conditions, housing prices, sentiment, and standard macroeconomic controls. Nonlinearities help most during high uncertainty, tight financial conditions, and housing-bubble bursts, especially in data-rich models.

  • Heterogeneity design: The analysis tests whether financial conditions, capital buildups, uncertainty, sentiment, and macroeconomic controls explain heterogeneity in nonlinear ML treatment effects.The regressions interact lagged conditioning variables with the nonlinear ML feature across selected horizons, average horizons, data-rich models, and the last 20 years.
  • Macroeconomic uncertainty: Macroeconomic uncertainty is a true game changer for the nonlinear ML feature.The section identifies uncertainty as important for both macroeconomic prediction and modeling.
  • Financial conditions and housing: Tighter financial conditions and falling house prices are positively correlated with stronger nonlinear treatment effects.These findings support the conclusion that nonlinearities help during tighter financial conditions and housing-bubble bursts.
  • Interpretation: Nonlinear gains are often larger in data-rich models because nonlinear factor relationships can capture complex relationships during uncertainty and financial-friction episodes.Consumer optimism also positively affects nonlinearities, especially in data-rich models.
  • Macroeconomic controls: The importance of nonlinearities is cyclical, with lower interest rates improving the nonlinear treatment effect by as much as 17% in data-rich models.Higher inflation also strengthens nonlinear gains, mainly at shorter horizons and for data-poor models.

7 Conclusion

The conclusion evaluates four ML features using out-of-sample experiments across variables, horizons, and data environments. Nonlinearity provides the main forecasting gains, while PCA factors, K-fold cross-validation, and L2 loss are preferred choices for the other features.

  • Study design: The study classifies ML methods by nonlinearities, regularization, cross-validation, and alternative loss functions in data-rich and data-poor forecasting environments.Experiments over 38 years cover five key variables and five forecasting horizons and identify marginal treatment effects.
  • Main findings: Nonlinearities are the true game changer in data-rich macroeconomic forecasting and substantially improve accuracy across macroeconomic variables and especially long horizons.The conclusion reports that nonlinear models perform particularly well during high uncertainty, financial stress, and housing-bubble bursts.
  • Practical recommendation: The recommended specification is a partially nonlinear factor model that obtains factors by PCA and applies a nonlinear function to an ARDI model’s generated data.This combines standard factor regularization with nonlinear forecasting while retaining the ARDI structure.
  • Other ML features: The standard factor model remains the best regularization, alternative regularization schemes are usually harmful, K-fold cross-validation is preferred, and standard L2 loss beats epsilon-insensitive loss.These conclusions summarize the regularization, model-selection, and loss-function comparisons.
  • Other ML features: Most benefits from support vector regression come from kernel-induced nonlinearities rather than its alternative loss function.The conclusion separates the contribution of the kernel trick from the epsilon-insensitive loss used by SVR.

B Robustness of Treatment Effects Graphs

The robustness graphs report out-of-sample R2 improvements over a basis model across variables and horizons in data-poor and data-rich environments. They also separate recession and expansion subsamples.

  • Data environments: The data-poor and data-rich graphs compare treatment effects across variables and horizons using improvements in OOS R2 over the basis model.Variables are industrial production, unemployment, term spread, inflation, and housing starts; horizons increase from 1 to 24 months within each color block.
  • Business-cycle subsamples: Separate graphs examine the same treatment-effect measures during recessions and expansions.The recession and expansion subsamples retain the same five variables and horizons and use HAC standard errors with 95% confidence bands.

C Additional Results

Additional results extend the treatment-effect graphs to the last 20 years and show rolling, cumulative, and fluctuation-test measures at a 12-month horizon. They also examine loss-function effects in linear and nonlinear models.

  • Recent-period robustness: The last-20-years graphs report treatment-effect improvements in OOS R2 across five variables and horizons from 1 to 24 months.The graphs use HAC standard errors and 95% confidence bands.
  • Heterogeneity variables: Figure 16 plots time series of variables used to explain heterogeneity in nonlinear treatment effects.These variables include measures of financial conditions, uncertainty, housing prices, sentiment, and standard macroeconomic controls.
  • Forecast stability: Figure 17 compares 3-year rolling-window root MSPE, cumulative root MSPE, and Giacomini–Rossi fluctuation tests for linear and nonlinear models at a 12-month horizon.The comparison covers both data-poor and data-rich environments.
  • Loss-function effects: Figures 18 and 19 show marginal improvements or deteriorations from the SVR in-sample loss function for linear and nonlinear models.The x-axis measures improvements in OOS R2 over the basis model, with HAC standard errors and 95% confidence bands, separately for data-poor and data-rich environments.

D Nonlinearites Matter – A Robustness Check

The robustness check replaces Random Forests and kernel ridge regression with boosted trees and neural networks to test whether nonlinearities remain important. It reports similar conclusions, supporting the claim that nonlinearities matter regardless of how they are obtained.

  • Alternative nonlinear methods: Boosted trees and neural networks provide alternative nonlinear forecasting algorithms for the robustness check.The appendix trades Random Forests for Boosted Trees and kernel ridge regression for Neural Networks.
  • Robustness conclusion: The alternative algorithms reach conclusions very similar to section 5.2.1, reinforcing that nonlinearities matter whichever method produces them.The robustness results support the paper’s nonlinear-feature interpretation rather than dependence on one specific nonlinear algorithm.
  • Boosted-tree specification: The boosted-tree models recursively fit trees to pseudo-residuals, using maximum depth 10 and Bayesian optimization for the number of steps and η.The models impose 12 lags and consider all features at each split.
  • Neural-network specification: The neural-network models use one or two hidden layers with ReLU activation functions and produce h-period-ahead forecasts.The two fixed architectures contain either 32 neurons in one hidden layer or 32 and 16 neurons in two hidden layers.
  • Model construction: The neural-network ensemble averages five networks with the same architecture to mitigate overfitting and random initialization effects.The data-poor and data-rich versions use different input sets, while the data-rich specification uses (p_y, p_f, K) = (12, 10, 12).

D.3 Results

The robustness results show that nonlinear treatment effects are strongest at longer horizons and remain broadly homogeneous in data-rich settings. The targets benefiting most differ somewhat across environments, but the appendix largely reproduces the main-text pattern.

  • Horizon patterns: Nonlinear treatment effects are magnified at horizons 9, 12 and 24.The result is reported for both alternative nonlinear algorithms.
  • Data environments: Both algorithms produce very homogeneous improvements in the data-rich environment.By contrast, results in the data-poor environment are more scattered.
  • Target heterogeneity: INF and HOUST benefit most from nonlinear methods in the data-rich environment.This matches the earlier findings reported in the main text.
  • Target heterogeneity: Real-activity targets benefit more from nonlinear methods in the main-text configuration than in this robustness configuration.This is identified as the sole noticeable difference from the reported earlier results.

A Results with Absolute Loss

Using absolute loss yields results broadly consistent with squared loss, including the importance and heterogeneity patterns of the machine-learning features. Nonlinearities and more data become more useful at longer horizons, whereas alternative shrinkage and loss function choices are generally detrimental.

  • Overall robustness: Results using absolute loss are very consistent with those obtained using squared loss.The consistency applies to the main feature-importance conclusions and their behavior across variable-horizon pairs.
  • Heterogeneity: Most heterogeneity is variable-specific, while averaging across variables reveals clear horizon patterns.The figure-based analysis evaluates feature changes while holding the remaining model components fixed.
  • Horizon patterns: More data and nonlinearities become increasingly useful as the forecasting horizon h rises.The passage describes this increase as approximately linear in h when comparing the relevant figures.
  • Feature effects: Cross-validation effects remain near zero, while alternative shrinkage and loss-function choices are negative and follow a boomerang shape across horizons.The negative effects are smaller at short and very long horizons but worse at intermediate horizons.
  • Robustness across losses: The relevance of nonlinearities and the poor performance of alternative shrinkage broadly match the main paper, but data-poor nonlinear advantages are not robust to MSPE versus MAPE.Other results are described as largely alike across the out-of-sample loss choices.
  • Cross-validation: K-fold cross-validation can be significantly worse than BIC in some cases under absolute loss, while its negative effect is stronger for POOS cross-validation.K-fold cross-validation still outperforms other model-selection criteria during recessions.

B Results with Quarterly Data

Quarterly FRED-QD results for GDP components broadly agree with the monthly application, while changing which machine-learning features appear most important. Standard ARDI,BIC models perform best for short-run GDP and consumption growth, whereas random forests dominate at longer horizons.

  • Quarterly experiment: Quarterly results using FRED-QD are consistent with the monthly findings for GDP components.The quarterly dataset contains 248 US macroeconomic and financial aggregates from 1960Q1 to 2018Q4.
  • Forecasting performance: Standard Stock and Watson ARDI,BIC models best predict GDP and consumption growth at short horizons, while random forests dominate at longer horizons.The quarterly forecasting horizons are 1, 2, 3, 4 and 8 quarters.
  • Targets and horizons: The quarterly exercise forecasts real GDP, consumption, investment, disposable income and the PCE deflator using average growth rates over multiple horizons.These are the five target variables used in the quarterly experiment.
  • Feature importance: In quarterly data, shrinkage is the most important ML feature, followed by loss function and nonlinearity, while cross-validation is least relevant.The data-rich component remains very important in this application.

C Results with Canadian data

Canadian forecasts use a monthly panel of 139 macroeconomic and financial variables across five targets and five horizons. Results are broadly similar to the paper’s main findings, but the nonlinear treatment effect is smaller.

  • Data and evaluation: The Canadian dataset contains 139 monthly macroeconomic and financial variables from 1981M01–2017M12, with pseudo-out-of-sample evaluation beginning in 2000M01.The targets are industrial growth, unemployment-rate change, term spread, CPI inflation, and housing-starts growth, forecast at 1, 3, 9, 12, and 24 months.
  • Main results: Canadian results are overall similar to those in the paper, but the nonlinear treatment effect is smaller.The evaluation period lacks a dramatic nonfinancial cycle and housing-bubble burst, potentially explaining the difference.
  • Main results: The pseudo-out-of-sample evaluation covers 2000–2017, when Canada experienced neither a dramatic nonfinancial cycle nor a housing-bubble burst.The passage links this setting to the smaller nonlinear treatment effect as a potential explanation.
  • Cross-validation: The cross-validation comparison uses a last-25% in-sample comparison window and repeats estimation five times as the training set expands.The models are estimated every 12 months while preserving the forecasting horizon.
  • Cross-validation: Five-fold cross-validation randomly splits the in-sample observations into five disjoint subsets, using four subsets for estimation and one for testing in each fold.Average test-subset mean squared error is the performance metric for hyperparameter selection in both cross-validation methods.

E.1 Data-poor (H− t ) models

The data-poor models use lagged dependent-variable values as predictors and vary functional form, regularization, cross-validation, and loss function. The specifications include autoregressive, Ridge, kernel Ridge, support-vector, and random-forest approaches.

  • Model setup: Data-poor forecasting models use only lagged values of the dependent variable, providing a small predictor set.The data-poor setup is denoted H−_t in the supplied passages.
  • Linear models: The autoregressive direct model selects lag order from {1, 3, 6, 12} using BIC, AIC, pseudo-out-of-sample cross-validation, or K-fold cross-validation.This model represents a linear predictive function without regularization.
  • Regularized models: Ridge Regression AR adds penalization and selects its Ridge hyperparameter through pseudo-out-of-sample or K-fold cross-validation.Its feature combination is linearity, Ridge regularization, cross-validation, and quadratic loss.
  • Nonlinear models: Random Forest AR introduces nonlinearities by partitioning predictor space into tree-based dummy variables and interactions, with random forests used to limit overfitting.The method adopts a one-third default share of randomly selected split candidates.
  • Nonlinear models: Kernel Ridge Regression AR introduces a nonlinear approximation of g through the kernel trick while retaining Ridge regularization and quadratic loss.Its hyperparameters are selected using the two cross-validation procedures.
  • Alternative loss: Support Vector Regression AR varies the loss-function dimension using linear and RBF kernels with both cross-validation procedures.The data-poor specifications therefore combine alternative kernels and cross-validation choices.
Loading 2008.12477v1…