PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparsity information and regularization in the horseshoe and other shrinkage priors

Juho Piironen, Aki Vehtari

arXiv:1707.01694v1stat.ME

TL;DR

The paper addresses the lack of a systematic sparsity-informed prior for the horseshoe’s global parameter and the inability to regularize its largest coefficients separately. It introduces effective nonzero-parameter counts and the regularized horseshoe, and experiments show benefits in estimation, prediction, computation, and sampling robustness, while correlated predictors still cause multimodality.

  • Problem

    The horseshoe lacks a systematic way to encode prior sparsity in its global hyperparameter and cannot separately regularize large coefficients, which is problematic for weakly identified parameters.

  • Method

    The paper defines an effective number of nonzero parameters for choosing the global hyperprior and introduces the regularized horseshoe with controllable shrinkage of large coefficients.

  • Results

    Experiments on synthetic and real-world data demonstrate benefits from both advances, including more reasonable estimates, improved prediction accuracy, faster computation, and greater sampling robustness.

  • Takeaways & Limitations

    A sparsity guess can inform the global hyperprior, while finite regularization makes horseshoe estimation more applicable to weakly identified coefficients.

  • Takeaways & Limitations

    With correlated predictors, the posterior can remain multimodal, causing difficult sampling and slow MCMC convergence.

Abstract

from arXiv · show

The horseshoe prior has proven to be a noteworthy alternative for sparse Bayesian estimation, but has previously suffered from two problems. First, there has been no systematic way of specifying a prior for the global shrinkage hyperparameter based on the prior information about the degree of sparsity in the parameter vector. Second, the horseshoe prior has the undesired property that there is no possibility of specifying separately information about sparsity and the amount of regularization for the largest coefficients, which can be problematic with weakly identified parameters, such as the logistic regression coefficients in the case of data separation. This paper proposes solutions to both of these problems. We introduce a concept of effective number of nonzero parameters, show an intuitive way of formulating the prior for the global hyperparameter based on the sparsity assumptions, and argue that the previous default choices are dubious based on their tendency to favor solutions with more unshrunk parameters than we typically expect a priori. Moreover, we introduce a generalization to the horseshoe prior, called the regularized horseshoe, that allows us to specify a minimum level of regularization to the largest values. We show that the new prior can be considered as the continuous counterpart of the spike-and-slab prior with a finite slab width, whereas the original horseshoe resembles the spike-and-slab with an infinitely wide slab. Numerical experiments on synthetic and real world data illustrate the benefit of both of these theoretical advances.

1. Introduction

The paper addresses two shortcomings of the horseshoe prior in sparse Bayesian estimation: choosing global sparsity information and regularizing weakly identified large coefficients. It proposes an effective sparsity measure, a corresponding hyperprior framework, and the regularized horseshoe.

  • Sparse Bayesian estimation targets models with many parameters but only a few coefficients distinguishable from zero.
  • Spike-and-slab priors are appealing but sensitive to slab width and inclusion probability, while large model spaces make inference computationally demanding.
  • Continuous shrinkage priors are easier to implement with generic sampling tools and can perform as well as or better than alternatives.
  • The horseshoe combines global shrinkage toward zero with heavy-tailed local scales that let some coefficients escape.
  • The horseshoe lacks a systematic sparsity-based prior for its global hyperparameter and does not regularize coefficients far from zero.
  • The paper introduces effective nonzero-parameter counts, a sparsity-informed prior for the global parameter, and the regularized horseshoe for controlling shrinkage of large coefficients.

2. Horseshoe prior and its extension

The horseshoe uses global-local shrinkage, while the regularized horseshoe adds finite-slab-like regularization for large or weakly identified coefficients. Their shrinkage profiles connect the continuous priors to spike-and-slab behavior.

  • 2.1. Horseshoe prior for linear regression: The horseshoe prior globally shrinks coefficients toward zero while half-Cauchy local scales allow some coefficients to escape.
  • 2.1. Horseshoe prior for linear regression: The shrinkage factor κ_j measures movement from the maximum-likelihood coefficient toward zero, with κ_j = 1 meaning complete shrinkage and κ_j = 0 no shrinkage.
  • 2.1. Horseshoe prior for linear regression: Changing τ changes the prior sparsity, but fixed-τ sparsity depends on the input dimension D.
  • 2.2. Spike-and-slab: The spike-and-slab prior uses a spike and slab controlled by slab width c and inclusion probability π, which encodes sparsity information.
  • 2.3. Regularized horseshoe: With finite slab width, the regularized horseshoe shrinks small signals like the horseshoe while regularizing even the largest coefficients as a Gaussian slab with variance c^2.
  • 2.3. Regularized horseshoe: The regularized horseshoe resembles a finite-width spike-and-slab and improves robustness when coefficients are weakly identified.
  • 2.4. Hierarchical shrinkage: The original horseshoe can create extreme funnel-shaped posteriors that challenge MCMC sampling.
  • 2.4. Hierarchical shrinkage: Increasing local degrees of freedom is less sparsifying and is no longer recommended because the regularized horseshoe performs better in the demonstrated setting.

3. The global shrinkage parameter

The paper develops a sparsity-informed prior for the horseshoe global parameter by linking τ to the effective number of nonzero coefficients, and extends the horseshoe with separate regularization for large coefficients.

  • 3.3. Effective number of nonzero coefficients: The effective number of nonzero coefficients meff measures the model's effective size through the collection of shrinkage factors.When shrinkage factors are near 0 or 1, meff approximately counts active or unshrunk variables.
  • 3.3. Effective number of nonzero coefficients: τ must scale as σ/√n to keep prior information about meff consistent across noise levels and sample sizes.Priors that do not scale this way can favor models of varying size.
  • 3.3. Effective number of nonzero coefficients: For D = 1000 and n = 200, the prior guess p0 = 5 gives τ0 = 3.6 · 10^-4σ.This reference value is obtained by solving E(meff | τ, σ) = p0.
  • 3.3. Effective number of nonzero coefficients: A half-Cauchy prior τ ∼ C+(0, 1) places substantial mass on large meff, favoring solutions with most coefficients unshrunk when D is large.The paper therefore characterizes this default as dubious when only a small number of variables are expected to be relevant.
  • 3.3. Effective number of nonzero coefficients: The prior for τ should be chosen from prior information about the expected number p0 of relevant variables, rather than using one globally optimal default.The paper proposes studying the induced prior on meff to make this choice.
  • 3.4. Regularized horseshoe and other shrinkage priors: The regularized horseshoe makes effective complexity lower than the original horseshoe for a given τ because large coefficients remain affected by the slab.The same τ0 construction can be used when p0 denotes the prior guess for coefficients far from zero, while retaining slab regularization.

4. Experiments

Experiments show that sparsity-informed global-scale choices can improve estimation, while the regularized horseshoe controls extreme coefficients without sacrificing comparable predictive accuracy. These benefits are accompanied by better sampling robustness, although correlated predictors can produce multimodal posteriors and convergence problems.

  • Toy example: At A = 4, the informative τ choice overshrinks nonzero signals, but at A = 6 it yields substantially better coefficient estimates.Signals near the universal threshold are difficult to distinguish from zero; irrelevant components can consume the limited effective sparsity budget.
  • Toy example: Scaling τ with the noise level σ is important because an unscaled τ increases the prior expectation for meff when observations are rescaled.When observations are multiplied by 0.1, the unscaled choice favors solutions with many more coefficients far from zero.
  • Classification with separable data: In separable classification, the regularized horseshoe cuts down tails for relevant coefficients while shrinking irrelevant coefficients as effectively as the horseshoe.The pure horseshoe produced almost 200 divergent transitions after warmup, whereas tail-thinning alternatives reduced but did not eliminate the problem.
  • Real world data – microarray cancer classification: A sparsity-informed τ prior substantially changes the posterior toward smaller meff and coefficient magnitudes when τ is weakly identified.With τ ∼ C+(0, 1), prior and posterior draws for τ and meff were very similar, producing severe under-regularization under the pure horseshoe.
  • Real world data – microarray cancer classification: Correlated predictors can leave sparsity weakly identified and induce multimodal coefficient posteriors that impair MCMC convergence.Reported ˆR-values of 1.09, 1.20, and 1.05 indicated convergence problems for three coefficients.
  • Real world data – microarray cancer classification: Across real datasets, the regularized horseshoe has predictive accuracy comparable to the original horseshoe and generally improves sampling robustness and computation time.The regularized horseshoe produced few divergent transitions versus 1–30% for the original horseshoe, while Lasso was much faster but generally less accurate.

5. Recommendations

The authors recommend weakly informative priors for both the regularized horseshoe’s slab scale and the global shrinkage parameter, using sparsity information to set the latter.

  • The regularized horseshoe is recommended over the original horseshoe with a weakly informative prior on the slab scale c.The scale and degrees of freedom should be appropriately chosen, and model assessment can check whether the slab width is too restrictive.
  • A very large slab scale may improve predictive fit but is not recommended when it produces many divergent transitions.Such divergences imply inference may have bias of unknown magnitude; tuning Stan’s adaptation can sometimes remove them.
  • For linear regression, the authors recommend replacing τ ∼ C+(0, 1) with a weakly informative prior centered using a prior guess p0 for relevant variables.For generalized linear models, an approximately equivalent choice uses an appropriate plug-in value in place of σ.

6. Discussion

The discussion identifies two advances: sparsity-informed global shrinkage and separate regularization of large coefficients. Experiments support both, while correlated predictors remain a source of posterior multimodality.

  • 6. Discussion: The paper introduces the regularized horseshoe and effective number of nonzero parameters as its two methodological advances.The former controls regularization of large coefficients; the latter guides the global-shrinkage hyperprior.
  • 6. Discussion: Small regularization of the largest coefficients is especially useful under weak identification, producing finite posterior means, more reasonable estimates, and faster posterior exploration in separated logistic regression.
  • 6. Discussion: Sparsity-informed hyperpriors can improve parameter estimates, predictive accuracy, and computation, while the regularized horseshoe generally outperforms Lasso in predictive accuracy.Unlike Lasso, the horseshoe does not produce exact zeros; projective variable selection can recover a sparse solution after fitting.
  • 6. Discussion: With correlated predictors, posterior multimodality can still slow MCMC convergence because the sparsifying prior favors one nonzero coefficient within predictor groups.The regularized horseshoe improves sampling robustness but does not remove this multimodality.
  • 6. Discussion: The effects of applying sparsifying priors to transformed features, such as principal components, on predictive accuracy remain unexplored.

Appendix A: Derivation of the regularized horseshoe

The appendix derives the regularized horseshoe by shifting the horseshoe shrinkage profile so its lower endpoint is b_j rather than zero, with b_j determined by the slab scale.

  • The regularized horseshoe is constructed by designing a prior whose shrinkage factor favors b_j and 1 instead of 0 and 1.
  • The derivation chooses b_j = 1 / (1 + nσ^-2s_j^2c^-2), corresponding to Gaussian shrinkage with variance c^2.
  • The resulting prior has the horseshoe shrinkage profile shifted from (0, 1) to (b_j, 1).The lower bound b_j depends on n, σ, s_j, and c through the stated expression.

Appendix B: Pseudo variance for Non-Gaussian observations

For non-Gaussian observations in the exponential family, the appendix derives the pseudo variance from the likelihood’s natural-parameter representation and its derivatives.

  • The exponential-family log likelihood is written using functions A(·), B(·), and C(·), with natural parameter θ and dispersion φ.
  • Defining μ = B′(θ) and V(μ) = B′′(θ) yields the relation ∂μ / ∂θ = V(μ).
  • The appendix uses the first and second derivatives of the likelihood to derive the pseudo variance for non-Gaussian observations.

C.1. Simple parametrization

The simple parametrization implements the regularized horseshoe for Gaussian and logistic regression, with user-controlled shrinkage and slab settings. It also provides a direct route for adapting the global scale between observation models.

  • The straightforward code generally works, but an alternative parametrization is suggested when sampling produces divergences.The paper also supplies alternative code in Appendix C.2 for such sampling issues.
  • Half-t priors govern τ and λj, while slab scale and slab degrees of freedom regularize the largest coefficients.The prior specification uses an inverse-gamma auxiliary variable for c^2 and allows user-defined degrees of freedom.
  • Setting nu_local = 1 recovers the horseshoe, and nu_global = 1 gives τ a half-Cauchy prior.The global half-t scale is scale_global*sigma in the Gaussian model.
  • The Stan model defines global and local shrinkage parameters, truncated local shrinkage, a slab scale, regression coefficients, and latent function values.The transformed parameters construct sigma, c, lambda_tilde, beta, and f from these quantities.
  • For logistic regression, sigma-related lines are removed, the observation model changes, and scale_global is set using a plug-in value for σ.The recommended scale is τ0 = p0/(D−p0) · σ/√n, with a suitable plug-in σ.

C.2. Alternative parametrization

The alternative parametrization expresses the regularized horseshoe through auxiliary variables and is used for the paper’s experiments. The appendix also shows rstanarm implementations for Gaussian and logistic regression, while noting that rstanarm did not yet implement the regularized horseshoe.

  • C.2. Alternative parametrization: This alternative code was used to generate the reported experiments and is recommended when the straightforward parametrization has sampling issues.The authors report no observed problems with the simple code in practice but provide the alternative as a fallback.
  • C.2. Alternative parametrization: The alternative parametrization represents local and global shrinkage through normal and inverse-gamma auxiliary variables.These auxiliaries construct λ and τ before forming the truncated local shrinkage and regression coefficients.
  • C.2. Alternative parametrization: The transformed parameters compute τ, c, λ_tilde, β, and f from the auxiliary variables and model scales.The resulting latent function is f = beta0 + x*beta for the Gaussian observation model.
  • Appendix D: rstanarm code: At the time of writing, rstanarm implemented the horseshoe prior but not the regularized horseshoe.The authors hoped regularized-horseshoe support would be added later.
  • Appendix D: rstanarm code: The rstanarm example sets p0 = 5 and uses tau0 = p0/(D-p0)/√n for a Gaussian horseshoe prior.The package automatically scales this prior by σ in the Gaussian model.
  • Appendix D: rstanarm code: For binary logistic regression, the rstanarm example uses pseudo σ = 1/√(mean(y)(1 − mean(y))) and tau0 = p0/(D-p0) · σ/√n.The model is fitted with a binomial observation family.
Loading 1707.01694v1…