PaperlayerView exact revision on arXiv

Source-linked AI summary

Achieving Shrinkage in a Time-Varying Parameter Model Framework

Angela Bitto, Sylvia Frühwirth-Schnatter

arXiv:1611.01310v2stat.MEstat.APstat.CO

TL;DR

TVP models risk overfitting when many coefficients are actually constant. This paper places a double gamma shrinkage prior on process variances and develops an ASIS-based MCMC scheme; the authors report that the priors successfully avoid overfitting when coefficients are static or insignificant.

  • Problem

    TVP models risk overfitting when many coefficients may be constant, motivating methods to identify which coefficients change over time.

    Source: §1 · p. 2

  • Method

    The paper applies a double gamma shrinkage prior to process variances and develops an MCMC sampler that interweaves centered and non-centered parameterizations using ASIS.

    Source: §1 · p. 3, §3 · p. 11

  • Results

    The authors report that double gamma priors successfully avoid overfitting when coefficients are static or insignificant.

    Source: §8 · p. 31

  • Takeaways & Limitations

    The authors suggest the framework may be useful for introducing sparsity in other TVP and state-space models.

    Source: §8 · p. 31

    Inference from the Cholesky-based model is not invariant to data ordering, and functionals of Σ_t^-1 showed ordering sensitivity.

    Source: §7.2 · p. 25

Abstract

from arXiv · show

Shrinkage for time-varying parameter (TVP) models is investigated within a Bayesian framework, with the aim to automatically reduce time-varying parameters to static ones, if the model is overfitting. This is achieved through placing the double gamma shrinkage prior on the process variances. An efficient Markov chain Monte Carlo scheme is developed, exploiting boosting based on the ancillarity-sufficiency interweaving strategy. The method is applicable both to TVP models for univariate as well as multivariate time series. Applications include a TVP generalized Phillips curve for EU area inflation modelling and a multivariate TVP Cholesky stochastic volatility model for joint modelling of the returns from the DAX-30 index.

1 Introduction

The paper develops Bayesian shrinkage methods that can turn overfitting TVP models into sparse models by identifying parameters that are static or insignificant. It introduces a double gamma prior and an efficient MCMC scheme for univariate and multivariate TVP models.

1 Introduction

  • TVP models capture gradual change, but overfitting arises when many coefficients are actually constant, making variance selection a key challenge.The paper illustrates this with a DAX-30 TVP Cholesky stochastic volatility model containing 406 potentially time-varying coefficients, only a small fraction of which change over time.Source: §1 · p. 2
  • The proposed double gamma prior produces a pronounced posterior spike at zero for static coefficients while avoiding overshrinkage of time-varying coefficients.It extends the normal-gamma prior in the non-centered parametrization; the Bayesian Lasso is a special case, while the double gamma is more flexible.Source: §1 · p. 3
  • A second shrinkage prior distinguishes time-varying coefficients, significant static coefficients, and coefficients insignificant over the full observation period.The paper compares prior settings using log predictive density scores and defines sparse TVP models for univariate and multivariate series with either homoscedastic or stochastic-volatility error variances.Source: §1 · p. 3
  • The paper develops a Gibbs-sampling MCMC scheme using the normal-gamma scale-mixture representation and ancillarity-sufficiency interweaving to improve sampling performance.Applications include EU-area inflation modelling with a generalized Phillips curve and multivariate DAX-30 returns using a TVP Cholesky stochastic volatility model.Source: §1 · p. 3

2 Sparse time-varying parameter models

The section formulates TVP regression coefficients as random walks and introduces double-gamma shrinkage on their process variances to allow coefficients to become effectively static. A non-centered parameterization supports this shrinkage, while the hierarchical prior permits global and local adaptation.

2.1 Bayesian inference for time-varying parameter models

  • In the univariate TVP model, regression coefficients follow random walks, and shrinking process variance θj toward zero pulls coefficient paths toward fixed effect βj.The model also considers multivariate time series elsewhere; here it focuses on a univariate series.Source: §2.1 · pp. 4–5
  • The observation equation allows either homoscedastic errors or time-varying variance captured by a stochastic-volatility specification.The stochastic-volatility specification models log volatility as an AR(1) process.Source: §2.1 · pp. 4–5
  • The non-centered parameterization makes state equations parameter-independent and places fixed effects and process variances in the observation equation.It represents the state using random walks with standard-normal independent increments.Source: §2.1 · pp. 5–6

2.2 Shrinking process variances through the double gamma prior

  • The double-gamma prior places mass near zero while allowing global shrinkage through κ2 and local adaptation through aξ.Larger κ2 strengthens global shrinkage; smaller aξ allows more local adaptation.Source: §2.2 · pp. 7–8
  • The hierarchical double-gamma prior induces dependence among process variances, useful when only a few are expected to differ from zero, and a second normal-gamma prior shrinks fixed effects.When θj is near zero, coefficient paths are pulled toward βj; shrinking βj as well can make a practically constant coefficient insignificant.Source: §2.2 · pp. 9–10

3 MCMC Estimation

The MCMC scheme addresses slow mixing when process variances are small by interweaving centered and non-centered TVP parameterizations. Its algorithm combines this strategy with latent-state sampling and updates designed for shrinkage priors.

3 MCMC Estimation

  • For multivariate TVP models, centered sampling suits coefficients that vary over time, while non-centered sampling suits nearly constant coefficients, and both types may occur.When process variances are small, centered Gibbs sampling can mix poorly; choosing one parameterization in advance is therefore not possible for practical analysis.Source: §3 · p. 10
  • Interweaving a non-centered baseline with the centered parameterization through ASIS increases posterior sampling efficiency compared with conventional Gibbs sampling in either parameterization.ASIS resamples parameters conditional on latent variables in the alternative parameterization.Source: §3 · p. 11
  • The algorithm samples non-centered states using an all-without-a-loop multi-move method, then jointly updates coefficients and square roots of process variances to avoid boundary problems for small variances.Bayesian inference uses latent-process data augmentation; log volatilities are also introduced as latent variables for the stochastic-volatility model.Source: §3 · pp. 10, 12
  • During interweaving, the sampler temporarily switches to the centered parameterization to resample coefficients and process variances, matching priors across parameterizations so the posterior remains identical.The centered-parameterization process-variance conditional is generalized inverse Gaussian rather than the usual inverted-gamma conditional.Source: §3 · pp. 12–13
  • For the stochastic-volatility model, Step (f) is modified to sample log volatilities and volatility parameters using an interweaving strategy developed for boosting stochastic-volatility MCMC.The sampled quantities include h0 through hT, µ, φ, and ση^2.Source: §3 · p. 14

4 Comparing shrinkage priors through log predictive density scores

The section uses log predictive density scores (LPDS) to compare shrinkage priors, evaluating one-step-ahead predictions on observations after a training sample. It approximates predictive densities from posterior MCMC draws using Gaussian sums, including a conditionally optimal Kalman mixture for the conditionally Gaussian TVP model.

4 Comparing shrinkage priors through log predictive density scores

  • LPDS compares shrinkage priors by evaluating one-step-ahead predictive densities for observations after the first t0 training observations.The score is based on the remaining observations y_(t0+1), …, y_T.Source: §4 · p. 14
  • Individual LPDS scores assess performance observation by observation, while aggregated LPDS summarizes the time series and can be interpreted as a log marginal likelihood.This interpretation uses the training-sample prior p(ϑ|ytr) for the homoscedastic state-space model, supporting prior comparison.Source: §4 · pp. 14–15
  • Gaussian sum approximations from posterior MCMC draws estimate one-step-ahead predictive densities, with a conditionally optimal Kalman mixture exploiting the TVP model’s conditional Gaussian state-space structure.The draws come from the posterior given information up to the preceding observation.Source: §4 · p. 15

5 Extension to multivariate time series

The paper extends sparse TVP shrinkage to multivariate time series by shrinking both coefficient process variances and fixed coefficients. Its TVP Cholesky stochastic-volatility model uses a Cholesky representation to express the system as independent TVP equations.

5.1 Sparse TVP models for multivariate time series

  • Shrinkage on process variances and fixed coefficients lets each multivariate coefficient be classified as time-varying, constant, or zero.A zero process variance represents a constant coefficient, while a constant coefficient can itself be zero; the hierarchical double gamma prior is applied to both quantities.Source: §5.1 · p. 16
  • With diagonal error covariance, the rows are independent and can be estimated separately or in parallel; for full covariance, a Cholesky decomposition yields such a representation (Lopes et al. (2016), Eisenstat et al. (2014), Zhao et al. (2016)).The row-wise prior independence is advantageous computationally.Source: §5.1 · p. 17

5.2 The sparse TVP Cholesky SV model

  • The TVP Cholesky SV model represents the time-varying covariance through a lower-triangular factor and diagonal matrix, yielding r independent TVP equations with βij,t = −Φij,t.The regressors are the earlier components of yt, and shrinkage priors are used for βij and θij.Source: §5.2 · p. 17
  • Each equation’s log volatility follows an individual stochastic-volatility model, and Bayesian inference applies the MCMC scheme independently to each row.The model has no intercept; the first row has no regressors, while equation i has i − 1 time-varying regression coefficients.Source: §5.2 · p. 18

6 Illustrative Application to Simulated Data

In simulated data, the shrinkage priors distinguish time-varying, static-but-significant, and insignificant coefficients. The double gamma prior improves identification of non-time-varying coefficients and reduces average mean squared error relative to the Bayesian Lasso.

6 Illustrative Application to Simulated Data

  • The simulation uses 100 length-200 series with β1 time-varying, β2 constant but significant, and β3 insignificant, comparing hierarchical double gamma and Bayesian Lasso shrinkage priors.The true coefficients are (β1, β2, β3) = (1.5, −0.3, 0), with process variances (θ1, θ2, θ3) = (0.02, 0, 0).Source: §6 · p. 19
  • Posterior shape helps classify coefficients: a bimodal posterior for θj suggests nonzero variance, whereas a unimodal posterior suggests the variance is zero.The posterior for θj is symmetric around zero by definition.Source: §6 · p. 19
  • The shrinkage priors identify the time-varying β1t, static-but-significant β2t, and insignificant β3t; the latter is also shrunk toward zero.For coefficients with zero process variances, θ2 and θ3, the posterior distributions are shrunken toward zero; Figure 4 shows posterior paths for one series.Source: §6 · p. 19, Figure 4 · p. 20
  • Table 1 reports avMSE, avVAR, and avBIAS2 over 100 simulated time series, comparing hierarchical double gamma priors with aτ ∼ E(10), aξ ∼ E(10) against Bayesian Lasso priors with aτ = aξ = 1.Source: Table 1 · p. 20

7 Applications in Economics and Finance

The applications show how shrinkage priors can distinguish static from time-varying coefficients in EU-area inflation and DAX returns. In both settings, the hierarchical double gamma prior supports sparse TVP models, while predictive comparisons favor shrinkage over the inverted gamma prior for DAX returns.

7.1 Modelling EU area inflation

  • Most of the 37 covariates have posterior densities peaking at zero; among 34 regression coefficients, including inflation lags and monthly dummies, none is significant by its 95% confidence region.M3 and unemployment vary over time. The 1-month interest rate is significantly different from zero but has a constant coefficient, while the 1-year rate is insignificant.Source: §7.1 · pp. 22–23
  • Adding interweaving substantially improves MCMC mixing for the inflation model, with considerably reduced inefficiency factors.The paper illustrates the mixing improvement with sample paths and selected-parameter inefficiency factors.Source: §7.1 · p. 23, Table 3 · p. 24
  • In the EU inflation application, the hierarchical double gamma prior is preferable to the Bayesian Lasso, and the choice of its hyperparameters has little effect on the results.The posterior distributions of the shrinkage parameters favor the double gamma prior over the Lasso setting.Source: §7.1 · pp. 21, 24

7.2 Sparse TVP Cholesky SV modelling of DAX returns

  • DAX inference is not invariant to index ordering: estimates of the covariance matrix are fairly robust, but functionals of its inverse, such as global minimum-variance portfolio weights, are sensitive.The DAX application orders indices alphabetically.Source: §7.2 · p. 25
  • Under the double gamma prior, coefficients j = 1, 2, and 7 are time-varying while the remaining coefficients are static; static coefficients show a more pronounced zero spike than under Bayesian Lasso.Across all 406 posterior paths, a large fraction of coefficients is not significant. In the heat map at t = 1150, the majority of estimated coefficients differs from zero under the inverted gamma prior, versus only a small part significantly different from zero under the double gamma prior.Source: §7.2 · pp. 26–28
  • For DAX returns, predictive scores strongly favor shrinkage priors over the inverted gamma prior, while the Bayesian Lasso and double gamma scores are similar.The inverted gamma results are highly sensitive to its hyperparameters; with financial time series’ small signal-to-noise ratio, predictive scores provide little basis for distinguishing shrinkage priors that behave sensibly at zero.Source: §7.2 · p. 28

8 Conclusion

The paper develops Bayesian shrinkage for TVP models to reduce overfitting by shrinking process variances, and finds that double gamma priors can avoid overfitting when coefficients are static or insignificant. Its general framework uses an efficient MCMC scheme and is applied to inflation and multivariate returns.

Conclusion

  • The framework extends regression shrinkage priors to process-variance selection in TVP models, using efficient MCMC that exploits ASIS.The approach uses a non-centered state-space parametrization to extend shrinkage priors for variable selection to variance selection.Source: §8 · pp. 28, 30, 31
  • Applications cover EU area inflation through a TVP generalized Phillips curve and DAX-30 returns through a sparse TVP Cholesky stochastic-volatility model.The multivariate application estimates a time-varying covariance matrix.Source: §8 · pp. 30–31
  • Double gamma priors successfully avoid overfitting when coefficients are static or insignificant.The paper reports this overall conclusion after investigating prior settings with log predictive density scores.Source: §8 · p. 31
  • The framework may support sparsity in other TVP and state-space models; sparse TVP VARs follow by straightforward extension.The paper also identifies alternative shrinkage priors for variance selection as a worthwhile direction for investigation.Source: §8 · p. 31
  • Extending the normal-gamma-gamma prior to variance selection is a promising direction, with its triple-gamma form including a horseshoe special case when aξ = cξ = 1/2.The horseshoe extension applies to variance selection for state-space models for univariate time series.Source: §8 · p. 31

A.1 Computational Details · A.1.1 Details on the MCMC scheme in Algorithm 1 · A.1.1.1 Step (a): Sampling the latent states

Algorithm 1 samples latent states conditional on known parameters using either FFBS or the faster AWOL alternative. The AWOL implementation exploits a banded precision matrix to draw from the state posterior without forming a matrix inverse.

A.1.1.1 Step (a): Sampling the latent states

  • Step (a) samples the latent states conditional on known parameters using either FFBS or the faster AWOL alternative.The AWOL algorithm is implemented for a slight modification of the non-centered TVP model.Source: §A.1.1.1 · p. 36
  • The AWOL implementation is specified for a slight modification of the non-centered TVP model.Source: §A.1.1.1 · p. 36
  • The conditional joint density of the state process is multivariate normal and can be represented using a tri-diagonal precision matrix and a covector.Source: §A.1.1.1 · p. 36
  • Each precision-matrix submatrix is d × d, with adjacent off-diagonal blocks equal to −Id.Id denotes the d × d identity matrix.Source: §A.1.1.1 · p. 37
  • The precision matrix’s band structure enables AWOL to sample the full state process without a loop from its (T + 1)d-variate normal posterior.Source: §A.1.1.1 · p. 37
  • The band structure makes Cholesky decomposition computationally inexpensive, and the draw is obtained using back-band substitution rather than calculating L−1.After drawing ϵ from a standard normal, the algorithm solves La = c and L′β̃ = a + ϵ.Source: §A.1.1.1 · p. 37

A.1.1.3 Step (c): Sampling the prior variances

The sampler draws prior variances from conditional posteriors implied by the normal-gamma priors. The hyperprior parameter d_2 controls how strongly κ^2 shrinks θ_j toward zero when ξ^2 is small.

Step (c): Sampling the prior variances

  • The conditional posterior for τ^2 is generalized inverse Gaussian under the normal-gamma prior, with an inverted-gamma likelihood kernel.Source: §A.1.1.3 · p. 38
  • With λ^2 ~ G(e_1, e_2) and κ^2 ~ G(d_1, d_2) hyperpriors, the sampler draws both parameters from their conditional posteriors.These posteriors use τ^2 and ξ^2, the averages of the variances in the shrinkage priors.Source: §A.1.1.3 · p. 39
  • When ξ^2 is small, the posterior expectation of κ^2 is proportional to 1/d_2, so smaller d_2 encourages stronger prior shrinkage of θ_j toward zero.Source: §A.1.1.3 · p. 39

A.1.2 Approximating the one-step ahead predictive density … A.3.2 Comparing approximations to the one-step ahead predictive density

The appendix derives one-step-ahead predictive densities with the Kalman filter and compares approximations, finding substantial differences for EU inflation but nearly identical estimates for DAX returns. It also documents the datasets and detailed ECB posterior summaries.

A.1.2.1 Using the Kalman filter for prediction for known parameters

  • For known parameters, the Kalman filter derives the predictive density by iterating prediction and correction steps in the non-centered TVP model.The appendix notes prediction could also be performed in the centered parameterization.Source: §A.1.2.1 · pp. 39–40

A.1.2.2 Approximations for the one-step ahead predictive density in a TVP model

  • The naive mixture can be very imprecise at high signal-to-noise, while Belmonte et al. (2014)’s simulation-and-kernel approach is expected to be inaccurate for multivariate scores.For the naive mixture, this occurs when the error variance is considerably smaller than the forecasting variance; individual multivariate scores are approximated and aggregated, allowing errors to accumulate.Source: §A.1.2.2 · pp. 40–41

A.2.1 ECB data

  • The ECB inflation data are monthly from February 1994 to November 2010, and the series were transformed and standardized for analysis.Core inflation was transformed to have variance one.Source: §A.2.1 · pp. 41–42

A.2.2 DAX data

  • The DAX dataset contains roughly 2,500 daily stock returns from September 4, 2001, through August 31, 2011; one company was excluded because it was not in the index throughout.The index comprises 30 major companies trading on the Frankfurt Stock Exchange.Source: §A.2.2 · p. 42

A.3.1 Detailed estimation results for the ECB data

  • Table A.3 reports posterior summary statistics for β_j and θ_j for 37 ECB inflation covariates under the hierarchical double gamma prior.The prior uses aτ ∼ E(10) and aξ ∼ E(10); the table includes monthly 0/1 dummy variables.Source: §A.3.1 · p. 42, Table A.3 · p. 45

A.3.2 Comparing approximations to the one-step ahead predictive density

  • For EU inflation, the naive and conditionally optimal Kalman mixture approximations differed substantially in log predictive density scores despite using 100,000 mixture components.The comparison covers the hierarchical double gamma prior with aτ ∼ E(10) and aξ ∼ E(10).Source: §A.3.2 · p. 47, Figure A.2 · p. 47
  • The naive mixture’s narrow component densities can yield many zero density evaluations, biasing its log predictive density estimate toward zero and making scores over- or underrated.Figure A.3 illustrates component densities at t = 130 for the hierarchical double gamma and hierarchical Bayesian Lasso priors.Source: §A.3.2 · p. 47, Figure A.3 · p. 48
  • For DAX returns, the naive Gaussian mixture approximation produced identical estimated log predictive density scores at nearly all time points because observation variance dominated under low signal-to-noise.The overall score comparison is shown for aτ = aξ = 0.1.Source: §A.3.2 · pp. 47–48
Loading 1611.01310v2…