Source-linked AI summary
Bayesian Adaptive Lasso
Chenlei Leng, Minh Ngoc Tran, David Nott
TL;DR
The paper addresses limitations of Lasso-based variable selection, including nonadaptive shrinkage and incomplete exploration of model uncertainty. It proposes BaLasso with coefficient-specific Bayesian shrinkage, posterior-mode selection, and sparse-model averaging. The paper reports attractive variable-selection performance in simulations and data analyses, while noting that its posterior model probabilities are not formal Bayesian model probabilities.
Problem
Lasso-based procedures can fail under violated design conditions, while Bayesian Lasso posterior exploration does not produce sparse models and does not fully explore model-space uncertainty.
Method
BaLasso estimates coefficient-specific shrinkage parameters, uses posterior conditional modes for sparse model selection, and averages predictions across models induced by posterior shrinkage samples.
Results
The paper reports attractive empirical performance, with BaLasso often producing sparser models and improving variable selection over Lasso and adaptive Lasso.
Takeaways & Limitations
BaLasso provides a unified framework for adaptive variable selection, model-uncertainty exploration, prediction, and structured penalties.
Takeaways & Limitations
The reported posterior model probability represents uncertainty in conditional posterior-mode sparsity induced by shrinkage uncertainty, not formal Bayesian model comparison.
Abstract
from arXiv · showhide
We propose the Bayesian adaptive Lasso (BaLasso) for variable selection and coefficient estimation in linear regression. The BaLasso is adaptive to the signal level by adopting different shrinkage for different coefficients. Furthermore, we provide a model selection machinery for the BaLasso by assessing the posterior conditional mode estimates, motivated by the hierarchical Bayesian interpretation of the Lasso. Our formulation also permits prediction using a model averaging strategy. We discuss other variants of this new approach and provide a unified framework for variable selection using flexible penalties. Empirical evidence of the attractiveness of the method is demonstrated via extensive simulation studies and data analysis.
1 Introduction
The paper motivates BaLasso by limitations of standard Bayesian and frequentist Lasso procedures, then introduces adaptive shrinkage, posterior-mode model selection, model averaging, and extensions to flexible penalties.
- Lasso model-selection consistency depends on the irrepresentable condition, which collinearity can violate.
- Bayesian Lasso posterior exploration does not produce exact zeros, while thresholding or credible intervals can inadequately represent model uncertainty.
- BaLasso uses coefficient-specific penalties so unimportant covariates receive stronger shrinkage than important covariates.
- Posterior draws of penalty parameters generate sparse models for model-uncertainty exploration and Bayesian-like prediction through model averaging.
- The framework extends variable selection to generalized linear, Cox, grouped, and hierarchically structured models through flexible penalties.
2 Bayesian Adaptive Lasso
BaLasso replaces common Lasso shrinkage with coefficient-specific Bayesian penalties estimated by empirical or hierarchical Bayes methods. Simulations show stronger signals receive lighter penalties, while zero coefficients receive heavier shrinkage.
- BaLasso assigns a separate shrinkage parameter λj to each coefficient, allowing adaptive rather than common penalization.
- An efficient Gibbs-sampling hierarchy supports posterior inference for coefficients, variances, and coefficient-specific shrinkage parameters.
- The empirical Bayes approach estimates λj by marginal maximum likelihood, whereas hierarchical Bayes places hyperpriors on the shrinkage parameters.
- 2.1 Choosing the Bayesian Adaptive Lasso Parameters: The framework uses common hyperpriors to reduce prior-specification degrees of freedom despite estimating p penalty parameters.
- 2.2 Adaptive shrinkage: Both EB and posterior estimates of λ2 decrease as β2 increases, indicating lighter penalties for stronger signals.
3 Inference
BaLasso performs selection by applying estimated or sampled shrinkage parameters to conditional posterior modes, producing sparse models and frequency-based selections. It also averages predictions across sparse models induced by posterior shrinkage uncertainty.
- 3.1 Estimation and Model Selection: BaLasso combines coefficient estimation and selection by plugging estimated shrinkage parameters into the conditional posterior-mode problem.
- 3.1 Estimation and Model Selection: BaLasso-Freq selects variables appearing in at least 0.5 of models obtained by applying sampled shrinkage vectors to the Lasso problem.
- 3.1 Estimation and Model Selection: The authors report improved variable selection over the original Lasso and adaptive Lasso for all proposed strategies.
- 3.2 A Model Averaging Strategy: BaLasso-BMA averages predictions across sparse conditional-mode models generated from posterior samples of smoothing parameters.
- 3.2 A Model Averaging Strategy: Under logarithmic scoring, the posterior predictive distribution is superior to prediction based on any fixed smoothing-parameter vector.
4 Examples
The examples evaluate BaLasso against Lasso, adaptive Lasso, and Bayesian Lasso for variable selection and prediction. Across simulations and data analyses, BaLasso methods generally improve model selection, while BaLasso-BMA provides strong predictive performance and accounts for model uncertainty.
- Simulation design: The simulations compare BaLasso with Lasso, adaptive Lasso, and Bayesian Lasso using variable-selection and prediction criteria.The studies use correctly fitted model frequencies and prediction squared errors across multiple examples.
- Variable selection: BaLasso methods outperform Lasso and generally improve on adaptive Lasso in model-selection simulations, including difficult and large-p settings.In the difficult example, correct-selection frequencies for the proposed methods approach 1 as sample size increases and noise decreases; in the large-p example, BaLasso-Mean outperforms adaptive Lasso.
- Prediction: BaLasso-BMA performs slightly better than Bayesian Lasso and BaLasso-Mean, and much better than Lasso and adaptive Lasso in the small-p prediction experiment.The comparison uses prediction squared errors averaged over 100 replications.
- Prediction: BaLasso-BMA outperforms the other methods in the large-p prediction experiment, where Bayesian Lasso performs poorly under common shrinkage.The experiment introduces small nonzero coefficients to create model uncertainty and contrasts adaptive with common shrinkage.
- Real-data analyses: In the body-fat data, BaLasso selects the simplest model with the smallest BIC, while BaLasso-BMA achieves the best reported out-of-sample prediction.The selected BaLasso model has BIC 708.92; BaLasso-BMA has PSE 18.13 versus 18.28 for BaLasso-Mean and 18.69 for BLasso.
- Real-data analyses: The body-fat analysis shows high model uncertainty: the highest-probability model accounts for 2.23% of total posterior model probability, and the ten highest account for 16.8%.The authors therefore caution that prediction using a single model may be risky.
5 A Unified Framework
The unified BaLasso framework extends adaptive Bayesian variable selection beyond linear regression to generalized models and structured penalties. Simulations report improved parsimony, model-selection consistency, and ordering of main and interaction effects relative to competing methods.
- Structured penalties: The framework supports flexible penalties, including group selection and composite penalties encoding hierarchical ordering among groups.
- General models: BaLasso uses a least squares approximation to extend the linear-regression framework to generalized linear and other parametric models.
- Simulation design: The simulations evaluate correct-fitting frequencies and average numbers of unselected factors or effects over 100 replications.
- Simulation evidence: BaLasso produces more parsimonious models than group Lasso and adaptive group Lasso when sample sizes are small, with better overall model-selection consistency.
- Simulation evidence: BaLasso always selects main effects before corresponding interaction effects, whereas competing methods sometimes select interactions without their required main effects.
6 Conclusion
The Bayesian adaptive Lasso combines signal-adaptive shrinkage with posterior-mode model selection and model averaging. The paper reports sparse models, consistent-model behavior, uncertainty exploration, predictive inference, and extensions to structured penalties and broader model classes.
- BaLasso applies different shrinkage levels across coefficients and provides empirical Bayes and hierarchical Bayes approaches for tuning-parameter inference.
- Posterior conditional modes generate an ensemble of sparse models that can support model averaging for prediction and exploration of model uncertainty.
- The approach retains sparse-model behavior associated with Lasso and consistent-model behavior associated with adaptive Lasso.
- The unified framework extends variable selection to groups, constrained penalties, and more general models.