On the Hyperprior Choice for the Global Shrinkage Parameter in the Horseshoe Prior

1 INTRODUCTION

• Sparse coefficient models are common in regression and classification, where only some parameters may be far from zero; this paper uses full Bayesian inference.The passage contrasts this approach with frequentist LASSO and elastic-net methods.Source: §1 · p. 1
• Spike-and-slab priors can be computationally demanding with many variables, while continuous shrinkage priors are easy to implement and can perform as well or better.Spike-and-slab results can also be sensitive to slab width and prior inclusion probability.Source: §1 · p. 1
• The horseshoe uses a global τ to shrink all parameters and heavy-tailed local half-Cauchy priors that let some coefficients escape shrinkage.Its performance has been comparable to spike-and-slab in examples where sparsifying priors are desirable.Source: §1 · p. 2
• When data weakly identify τ, its hyperprior becomes crucial because τ determines overall sparsity and substantially affects results.The paper focuses on full Bayesian inference for τ.Source: §1 · p. 2
• The paper derives the relationship between τ and the effective number of nonzero coefficients, then formulates p(τ) from prior beliefs about sparsity.The approach is presented for regression and classification and is also applicable to other generalized linear models and shrinkage priors.Source: §1 · p. 2

2 THE HORSESHOE PRIOR FOR LINEAR REGRESSION

• The global parameter τ pulls all weights toward zero, while heavy-tailed half-Cauchy local scales λ_j allow some weights to escape; as τ approaches zero, all weights shrink toward zero.Large τ gives all variables diffuse priors with little shrinkage.Source: §2 · p. 2
• With uncorrelated, zero-mean, unit-variance predictors, X^TX is approximated by nI, which simplifies the coefficient-posterior expression.The approximation is conditional on the stated predictor assumptions.Source: §2 · p. 2
• The shrinkage factor κ_j measures how much coefficient β_j is shrunk from its maximum-likelihood solution, with κ_j = 1 meaning complete shrinkage and κ_j = 0 meaning none..Source: §2 · p. 2
• For fixed τ and σ, the horseshoe shrinkage-factor prior places mass near both κ = 0 and κ = 1; reducing nσ^-2τ^2 to 0.1 favors complete shrinkage.The figure contrasts nσ^-2τ^2 = 1 (solid) with 0.1 (dashed).Source: §2 · p. 3, Figure 1 · p. 3

3.1 Full Bayes versus Point Estimation

• Cross-validation is computationally less efficient and does not account for posterior uncertainty, while maximum marginal likelihood can collapse to zero and is difficult for non-Gaussian likelihoods.The authors therefore recommend full Bayesian inference for τ and focus on specifying its prior.Source: §3.1 · p. 3

3.2 Earlier Approaches

• Earlier default scales can place too much prior mass on large τ, favoring implausibly many unshrunk coefficients when data do not strongly identify τ.The proposed construction uses prior beliefs about the number of nonzeros p* and recovers the oracle result when p* is known.Source: §3.2 · pp. 3–4

3.3 Effective Number of Nonzero Coefficients

• The expected effective number of nonzeros links τ to model size and implies that τ should scale as σ/√n to keep prior beliefs about model size consistent.Given a prior guess p0, the authors choose τ so that the prior mean effective number of nonzeros is p0.Source: §3.3 · p. 4
• With p0 = 5, the half-Cauchy prior on τ gives substantial prior mass to effective model sizes much larger than p0 when D is large; τ ∼ C+(0,1) also changes its implied prior on model size with n or σ.A half-normal scale τ0 favors meff < p0 while allowing larger values; fixing τ = τ0 yields a distribution nearly symmetric around p0.Source: §3.3 · p. 4, Figure 2 · p. 5
• When the prior guess equals the true nonzero count under the stated asymptotic assumptions and σ = 1, τ0 approaches the oracle value p*/D in the simplified model.This supports centering the prior for meff at the true number of nonzeros when that number is known.Source: §3.3 · pp. 4–5

3.4 Non-Gaussian Likelihood

• For non-Gaussian likelihoods, a Gaussian approximation and a common effective variance provide an approximate route to applying the calibration; using σ = 2 tends to favor slightly fewer than p0 nonzeros.The authors report that this approach, despite crude approximations, gives reasonably accurate results.Source: §3.4 · pp. 5–6

3.5 Other Shrinkage Priors

• The framework can extend to other shrinkage priors expressible as scale mixtures of Gaussians, although the closed-form equations in Section 3.3 apply only to the horseshoe.For other priors, sampling τ and the local scales makes it possible to draw from the induced prior on meff.Source: §3.5 · p. 6

• For Ovarian, weak data identification under τ ∼ C+(0, 1) led to severe under-regularization and coefficients with magnitudes in the hundreds; τ0 or half-normal improved estimates.The revised priors concentrated posterior mass for meff on more reasonable values and yielded more sensible coefficient magnitudes.Source: §4 · pp. 6–7
• In Ovarian, predictor correlations may help explain why τ and meff were not strongly identified by the data.One predictor had absolute correlation at least 0.5 with 314 others, including 65 correlations above 0.7.Source: §4 · p. 7
• Prediction comparisons averaged 50 random train/test splits; LASSO’s regularization parameter was tuned by 10-fold cross-validation.Source: §4 · p. 7
• Across classification datasets, a broad range of p0 values produced smaller posterior mean meff, better predictive accuracy, and lower computation time than τ0=1 under either prior.The reduced computation time is attributed to tighter posterior concentration, which aids sampling.Source: §4 · pp. 7–8, Figure 4 · p. 7
• When p0 was well chosen, the tighter half-normal prior produced simpler models with equally good or better predictive accuracy on classification datasets.Source: Figure 4 · p. 7, §4 · p. 8
• For reasonably selected priors, the horseshoe consistently outperformed LASSO in pairwise comparisons, but τ0=1 performed worse on Ovarian and Colon; LASSO took less than a second.The paper also reports better horseshoe regression performance by mean squared error, though those results are not shown.Source: §4 · p. 8, Figure 5 · p. 8

5 CONCLUSION AND RECOMMENDATIONS

• Prior choices can improve parameter estimates, prediction accuracy, and computation time when even a crude guess about relevant variables informs the prior for τ.The results also show that no prior choice is globally optimal across problems.Source: §5 · p. 8
• Previous default choices are often dubious because they tend to favor solutions with too many parameters unshrunk.The paper discusses this issue for Bayesian regression and classification.Source: §5 · p. 8
• For regression, the authors recommend a τ prior scaled using τ0 computed from the prior guess p0 for the number of relevant variables.The recommendation seems to perform well unless p0 is very far from optimal.Source: §5 · p. 8
• For logistic regression, an approximately equivalent recommendation uses σ = 2; if more regularization appears necessary, the authors suggest checking and adjusting the prior on meff to match prior beliefs.The recommendation is to change p(τ) so that its induced prior for meff corresponds as closely as possible to those beliefs.Source: §5 · p. 8

Non-Gaussian Likelihood

• A second-order Taylor expansion of the log-likelihood terms is used to approximate the posterior for non-Gaussian likelihoods.In logistic regression, points on the classification boundary are most informative, and the text gives a pseudo-variance reference value of 4.Source: Non-Gaussian Likelihood · p. 10

Additional experiments

• The synthetic study averages MSE over 100 realizations with n = 400, varying the number of nonzero coefficients and their shared magnitude.It compares τ ~ C+(0, 1) with τ = τ0 computed using the oracle guess p0 = p*; τ remains distributed because it depends on unknown σ.Source: Additional experiments · p. 10, Non-Gaussian Likelihood · p. 10
• For the synthetic experiment, oracle τ performs worse at A = 4 because it overshrinks signals, but yields clearly superior results for larger A.Figure 6 reports MSE; the overshrinkage arises when zero-signal observations resemble actual signals and compete for the limited meff budget.Source: Figure 6 · p. 10, Additional experiments · p. 11

Stan codes

• The Stan code assigns half-t priors to τ and λj; ν_local = 1 gives the horseshoe, while ν_global = 1 gives a half-Cauchy prior for τ and a large value such as 100 approximates a half-normal.For logistic regression, the code follows the recommendation by plugging in σ = 2.Source: Stan codes · pp. 12–13