Source-linked AI summary
A simple sampler for the horseshoe estimator
Enes Makalic, Daniel F. Schmidt
TL;DR
The paper tackles difficult posterior sampling for the horseshoe hierarchy in Bayesian linear regression. It introduces an auxiliary-variable sampler with conjugate conditional posteriors, extends the framework to logistic regression and alternative hierarchies, and reports higher effective sample sizes than monomvn in the evaluated comparison. The hierarchy also supports readily applicable marginal-likelihood computation with Chib’s algorithm.
Problem
Efficient posterior sampling is difficult because standard Gibbs conditionals for the horseshoe hyperparameters have non-standard forms.
Method
The paper introduces auxiliary variables that produce conjugate conditional posterior distributions for the horseshoe hierarchy and discusses extensions to logistic regression and alternative hierarchies.
Results
The bhs sampler achieved an effective sample size of 80% for all ten predictors with thinning level 11, compared with 16 for monomvn.
Takeaways & Limitations
Conjugacy makes Gibbs sampling relatively straightforward and allows Chib’s algorithm to compute the model’s marginal likelihood from Gibbs output.
Takeaways & Limitations
The linear-regression setup assumes standardized predictors and centred data, while logistic regression requires explicit modelling of an unpenalized intercept.
Abstract
from arXiv · showhide
In this note we derive a simple Bayesian sampler for linear regression with the horseshoe hierarchy. A new interpretation of the horseshoe model is presented, and extensions to logistic regression and alternative hierarchies, such as horseshoe$+$, are discussed. Due to the conjugacy of the proposed hierarchy, Chib's algorithm may be used to easily compute the marginal likelihood of the model.
1 Introduction
The horseshoe hierarchy provides coefficient-specific shrinkage that strongly suppresses noise while preserving large signals. This paper addresses difficult posterior sampling by introducing auxiliary variables that yield conjugate conditional posteriors for all parameters.
- The regression setup uses predictor matrix X and a Gaussian distribution within the Bayesian linear regression model.
- The horseshoe is a global-local shrinkage procedure, with λj controlling local shrinkage and τ controlling overall shrinkage.
- Half-Cauchy priors produce aggressive shrinkage of small coefficients and virtually no shrinkage of sufficiently large coefficients.
- Unlike Bayesian lasso and ridge hierarchies, the horseshoe does not apply a uniform shrinkage effect across coefficients.
- Standard Gibbs sampling is difficult because the conditional posteriors for λ1, . . . , λp and τ have non-standard forms.
- The proposed auxiliary-variable scheme makes all model parameters have conjugate conditional posterior distributions, enabling relatively straightforward Gibbs sampling.
2 Bayesian horseshoe with auxiliary variables
Auxiliary-variable representations convert the horseshoe hierarchy into a Gibbs sampler with mostly conjugate conditional updates. The same strategy extends to logistic and negative-binomial regression and to horseshoe+, while runtime comparison is made against monomvn.
- Linear regression: The half-Cauchy scale-mixture representation yields a revised horseshoe hierarchy with auxiliary inverse-gamma variables.The auxiliary variables ν1, …, νp and ξ have IG(1/2, 1) distributions.
- Linear regression: The revised hierarchy makes Gibbs sampling straightforward by providing conditional updates for the regression coefficients, variance, hypervariances, and auxiliary variables.The conditional distributions for the hypervariances are inverse-gamma type.
- Linear regression: All conditional posteriors except those for the regression coefficients are inverse-gamma, allowing efficient samplers for those parameters.This conjugate structure is the central computational feature of the proposed hierarchy.
- Horseshoe logistic regression: For logistic regression, scale-mixture or Pólya-gamma augmentation produces Gaussian coefficient updates and Pólya-gamma updates for the auxiliary variables.The intercept must be explicitly modeled and protected from shrinkage, for example with a uniform prior.
- Horseshoe negative-binomial regression: The Pólya-gamma strategy extends the horseshoe estimator to negative-binomial regression, where the coefficient update is equivalent to the horseshoe logistic-regression model.The intercept remains explicitly modeled and should not be shrunk.
- Alternative prior distributions: Horseshoe+ follows by applying the same auxiliary-variable decomposition, with an additional half-Cauchy mixing variable ηj.The paper also compares bhs and monomvn runtimes using 1,000 posterior samples, with timings reported in seconds.
3 Evaluation
The evaluation compares auxiliary-variable Gibbs sampling in bhs with monomvn’s slice-sampling implementation using runtime and effective sample size. bhs achieved higher effective sample sizes across predictors and thinning levels in the diabetes analysis.
- Runtime comparison: The simulation compares runtimes for bhs and monomvn across sample sizes and numbers of predictors.The comparison is potentially biased because bhs is pure MATLAB while monomvn uses compiled C through an R interface.
- Effective sample size: The diabetes dataset contains 442 observations and 10 predictors, including highly correlated predictors.Both samplers were evaluated on this dataset.
- Effective sample size: bhs produced higher effective sample sizes than monomvn for every regression coefficient and thinning level examined.Figure 1 illustrates the comparison for predictors S2 and S3, which had the smallest observed effective sample sizes.
- Effective sample size: 80% effective sample size for all ten predictors required thinning level 16 with monomvn versus 11 with bhs.The bhs advantage was greatest at thinning levels four to eight.
4 Discussion
The discussion gives the horseshoe hierarchy a Cauchy-prior interpretation and highlights consequences of conjugate conditional posteriors. It also describes implementation choices for efficient coefficient sampling across predictor dimensions.
- Interpretation: Integrating out the hypervariances yields a Cauchy prior for each regression coefficient βj.The scale of each coefficient’s prior is inversely proportional to νj.
- Interpretation: The horseshoe hierarchy can therefore be interpreted as assigning coefficient-specific Cauchy priors with scales inversely proportional to νj.This interpretation follows from the auxiliary-variable decomposition.
- Conjugacy and model comparison: Conjugacy makes Chib’s algorithm readily applicable because the normalizing constants of all conditional posteriors are known.This provides a way to compute the model’s marginal likelihood from Gibbs-sampler output.
- Implementation: For n > p, Rue’s algorithm samples the multivariate Gaussian coefficient conditional using Cholesky factorization with cubic complexity in p.For large numbers of predictors, the implementation instead uses an algorithm with linear complexity in p.