PaperlayerView exact revision on arXiv

Source-linked AI summary

The Bayesian Bridge

Nicholas G. Polson, James G. Scott, Jesse Windle

arXiv:1109.2279v2stat.MEstat.COstat.ML

TL;DR

The paper addresses how to perform Bayesian estimation and computation for bridge regression while handling multimodality, tuning uncertainty, and difficult posterior mixing. It develops complementary mixture-based MCMC methods and finds that the Bayesian bridge improves estimation and prediction, with especially favorable mixing for the global scale. Its main computational limitation is that the novel method mixes poorly under strong collinearity, where the normal-mixture method is preferred.

  • Problem

    Bridge estimation requires posterior computation that can handle multimodality, uncertainty in tuning parameters, and poor MCMC mixing, while classical point estimates can be suboptimal for estimation and prediction.

  • Method

    The paper develops a Bayesian bridge model with stable-normal and Bartlett–Fejer mixture representations, yielding complementary MCMC algorithms for different design structures.

  • Results

    The posterior mean outperforms least squares and the classical bridge in all three test-data comparisons, while the Bayesian approach generally improves estimation and prediction and mixes well for the global scale.

  • Takeaways & Limitations

    Use the normal-scale mixture for strongly collinear designs and the Bartlett–Fejer representation for orthogonal designs, where its effective sampling rate is roughly two to three times higher.

  • Takeaways & Limitations

    The Bartlett–Fejer method suffers from poor mixing with strongly collinear design matrices because truncated multivariate normal sampling remains inefficient.

Abstract

from arXiv · show

We propose the Bayesian bridge estimator for regularized regression and classification. Two key mixture representations for the Bayesian bridge model are developed: (1) a scale mixture of normals with respect to an alpha-stable random variable; and (2) a mixture of Bartlett--Fejer kernels (or triangle densities) with respect to a two-component mixture of gamma random variables. Both lead to MCMC methods for posterior simulation, and these methods turn out to have complementary domains of maximum efficiency. The first representation is a well known result due to West (1987), and is the better choice for collinear design matrices. The second representation is new, and is more efficient for orthogonal problems, largely because it avoids the need to deal with exponentially tilted stable random variables. It also provides insight into the multimodality of the joint posterior distribution, a feature of the bridge model that is notably absent under ridge or lasso-type priors. We prove a theorem that extends this representation to a wider class of densities representable as scale mixtures of betas, and provide an explicit inversion formula for the mixing distribution. The connections with slice sampling and scale mixtures of normals are explored. On the practical side, we find that the Bayesian bridge model outperforms its classical cousin in estimation and prediction across a variety of data sets, both simulated and real. We also show that the MCMC for fitting the bridge model exhibits excellent mixing properties, particularly for the global scale parameter. This makes for a favorable contrast with analogous MCMC algorithms for other sparse Bayesian models. All methods described in this paper are implemented in the R package BayesBridge. An extensive set of simulation results are provided in two supplemental files.

1 Introduction

The paper develops a Bayesian version of bridge estimation that replaces a single penalized estimate with posterior inference, uncertainty quantification, and MCMC methods tailored to design structure. It argues that this approach improves estimation, prediction, and computational mixing while exposing multimodality and differing conclusions about predictor importance.

  • The Bayesian bridge treats the classical bridge objective as a posterior whose global mode is the classical estimator, then samples the joint posterior instead of minimizing the objective.
  • Bridge priors have heavier-than-exponential tails than ridge or lasso-type priors, helping avoid overshrinkage of large coefficients in sparse settings.
  • Full Bayes accounts for uncertainty in the global scale and can extend this treatment to the concavity parameter α, unlike classical tuning based on a selected penalty value.
  • The posterior mean is optimal under squared-error loss, and the paper reports lower risk than classical bridge solutions, often dramatically so, for estimation and prediction.
  • Bayesian and classical analyses can yield different conclusions about which predictors matter, so the authors recommend retaining both perspectives rather than treating one as universally correct.
  • The paper develops two complementary MCMC approaches: a normal-scale mixture suited to collinearity and a novel representation that is more efficient for orthogonal designs.
  • The novel representation addresses multimodal posterior exploration and supports broader representations of non-convex penalty densities, but posterior inference remains more challenging than in many related Bayesian models.

2 Data augmentation for the bridge model

The paper develops two data-augmentation strategies for Bayesian bridge inference: a Gaussian mixture with an alpha-stable mixing variable and a Bartlett–Fejer mixture with a gamma-based mixing distribution. The second representation avoids exponentially tilted stable variables and extends to a broad class of n-monotone densities through an explicit inversion formula.

  • Scale mixture of normals: The first strategy represents the exponential-power prior as a Gaussian mixture whose mixing variable is a positive alpha-stable random variable.This representation is well known since West (1987), but its conditional mixing distribution is an exponentially tilted stable variable.
  • Scale mixture of normals: The Gaussian-mixture approach is difficult because the conditional stable mixing distribution lacks a closed-form density and is expressible only through an infinite series.The resulting latent-variable MCMC is therefore less straightforward than analogous Gaussian-mixture methods for other priors.
  • Scale mixtures of betas: The alternative strategy represents n-monotone densities as scale mixtures of beta kernels and recovers the mixing distribution from derivatives of the target density.The theorem applies to any integer k between 1 and n and provides a finite-term mixing density in the k-monotone case.
  • Scale mixtures of betas: For the bridge density, the Bartlett–Fejer representation yields a two-component gamma mixture, with the Bayesian lasso as the special case where the second component disappears.The bridge construction uses a transformed exponential-power density and normalized Bartlett–Fejer kernels.

3 The connection with other latent-variable methods

The paper connects its beta-mixture augmentation to slice sampling and Gaussian mixtures through a shared latent-variable view of posterior sampling. The representation can simplify slice-region inversion and is especially attractive for the bridge, although higher-order mixtures introduce additional sampling complexity.

  • Connection with slice sampling: The beta-mixture representation replaces slice-region inversion with sampling a latent scale whose support directly defines the set |θ| < ω.This trades inversion of the slice region for sampling ω from a uniquely identified non-uniform mixing density.
  • Latent-variable efficiency: The efficiency of latent-variable schemes depends inversely on the information about θ carried by the latent variable, favoring kernels that closely mimic the target density.The theorem gives algorithm designers latitude to choose among monotonicity-based representations.
  • Application to the bridge: For the Bayesian bridge, the exponential-power density resembles a triangle and its bimodal marginal posteriors align with the two components of g(ωj).This matching is presented as a reason the representation is especially attractive for the bridge model.
  • Trade-offs: Moving beyond the once-monotone case complicates the mixing distribution and requires sampling from a tilted density, although the twice-monotone bridge representation is straightforward.The difficulty of this step depends on the application context.
  • Connection with Gaussian mixtures: As the beta-mixture order increases, the representation approaches a Gaussian scale mixture, linking the Schoenberg–Williamson and Bernstein constructions.Evaluating the limiting representation at s = t^2/2 produces a scale mixture of normals, with a parallel inversion formula.

4 MCMC sampling for the Bayesian bridge

The Bayesian bridge supports complementary MCMC representations: normal mixtures favor collinear designs, while Bartlett–Fejer mixtures are more efficient for orthogonal designs and expose posterior multimodality. The sampler updates latent variables, coefficients, and hyperparameters using exact conditional distributions and marginalization strategies that improve mixing.

  • Computational strategy: A hybrid sampler uses normal mixtures for strongly collinear designs and Bartlett–Fejer mixtures for orthogonal designs.The Bartlett–Fejer approach usually achieves an effective sampling rate roughly two to three times that of the Gaussian method for orthogonal designs.
  • Sampling β and latent variables: The Bartlett–Fejer representation yields an exact Gibbs sampler alternating latent-scale updates with truncated multivariate-normal updates for β.Each latent ωj is sampled from a mixture of truncated gammas, followed by a component-wise update of β.
  • Sampling β and latent variables: The mixture-of-gammas representation naturally captures bimodal marginal posteriors, allowing the sampler to explore multiple joint-posterior modes.In some cases, each mixture component for ωj corresponds to a distinct mode of the marginal posterior for βj.
  • Sampling hyperparameters: Marginalizing latent local scales when updating the global scale parameter produces a gamma conditional and favorable mixing.The transformation ν = τ^-α permits direct sampling of τ after drawing ν from its conditional gamma distribution.
  • Sampling hyperparameters: The concavity parameter α can be assigned a beta-family prior and updated with a random-walk Metropolis sampler.The diabetes example shows substantial uncertainty about α because the problem has only 10 predictors.
  • Extensions: Related latent-variable constructions extend Bayesian bridge MCMC to logistic and quantile regression.Additional likelihood-specific latent variables represent these likelihoods as Gaussian mean–variance mixtures.

5 Examples

Examples show that Bayesian bridge summaries can differ substantially from classical bridge solutions, especially under correlated predictors, while posterior means improve estimation and prediction. Across real and simulated data, the Bayesian posterior mean outperforms least squares and classical bridge estimators.

  • 5.1 Diabetes data: The diabetes posterior is markedly multimodal, with distinct marginal modes for coefficients associated with TCH and Glucose.Additional multimodality appears for HDL and Female, making a single summary potentially inadequate.
  • 5.1 Diabetes data: Uncertainty in τ and σ makes the fully Bayesian joint mode differ from the classical bridge solution.Marginalizing these hyperparameters changes the objective function and therefore the joint posterior mode.
  • 5.1 Diabetes data: For highly correlated predictors, the Bayesian solution can shrink both effects while the classical solution selects large coefficients with opposite signs.For TC and LDL, the predictors have sample correlation −0.897, and the two approaches imply sharply different conclusions.
  • 5.2 Out-of-sample prediction results: Across all three benchmark prediction exercises, the Bayesian posterior mean outperforms least squares and the classical bridge estimator.A fixed α = 0.5 nearly matches the model that marginalizes over a uniform prior for α.
  • 5.3 Simulated data with correlated design: Across all three simulated α settings, the Bayesian posterior mean outperforms both least squares and the classical bridge estimator.At α = 0.5, the Bayesian estimator outperforms the classical estimator by more than a factor of 16.

6 Discussion

The paper develops tools for full Bayesian inference under the bridge model and finds that posterior summaries and computation can differ substantially across settings. Its two MCMC approaches have complementary strengths, with one handling orthogonal designs better and the other strongly collinear designs.

  • Posterior means can differ substantially from classical bridge solutions and posterior modes when summarizing predictor importance.
  • Posterior means substantially improve estimation of β and prediction under squared-error loss compared with posterior modes.
  • The novel two-component mixing-measure sampler can jump easily between modes in the joint posterior and performs best in orthogonal designs.
  • The normal-mixture sampler performs well for strongly collinear design matrices, where the novel sampler mixes poorly because truncated multivariate normal sampling is inefficient.
  • Both MCMC methods incorporate uncertainty about the concavity parameter naturally and are implemented in the paper’s software.

A.1 Theorem 2.1

The theorem uses monotonicity properties of symmetric densities to establish a mixture representation and derive the associated mixing distribution. The proof rewrites the kernel as a scaled beta density and obtains an explicit density expression by differentiating the CDF.

  • The proof reduces the proposition to the case k = n because n-times monotonicity implies monotonicity of every lower order.
  • The Schoenberg–Williamson theorem characterizes membership in Mn through a representation involving a non-decreasing, bounded-below function H(u).
  • When H(u) = 0, the representation is unique at the continuity points of H(u).
  • Rewriting the kernel as a scaled beta density and differentiating the CDF yields the mixing density.
Loading 1109.2279v2…