Source-linked AI summary
Generalized double Pareto shrinkage
Artin Armagan, David Dunson, Jaeyong Lee
TL;DR
The paper addresses the need for shrinkage priors that support sparse estimates without over-shrinking coefficients away from zero while also enabling uncertainty quantification. It proposes a generalized double Pareto prior with mixture-based computation and studies its Bayesian, regularization, and estimation properties. The paper reports simple and fast computation and excellent performance in small-sample simulations, while noting scope limitations across dense and sparse settings.
Problem
The paper addresses the need for shrinkage priors that produce sparse estimates, avoid over-shrinking coefficients not close to zero, support computation in large p cases, and quantify uncertainty.
Method
The paper proposes a generalized double Pareto hierarchical prior based on a scale-mixture representation and applies it to Bayesian inference, penalized likelihood, and maximum a posteriori estimation.
Results
The paper reports simple and fast computation and excellent performance in small sample simulation studies, with the prior also argued to mimic Bayesian model averaging through mixing over higher-dimensional models.
Takeaways & Limitations
The generalized double Pareto is presented as useful for Bayesian shrinkage estimation and regularization, bridging those two paradigms through a hierarchical prior.
Takeaways & Limitations
In the cases considered, performance was comparable to other methods when the true model was dense or very sparse.
Abstract
from arXiv · showhide
We propose a generalized double Pareto prior for Bayesian shrinkage estimation and inferences in linear models. The prior can be obtained via a scale mixture of Laplace or normal distributions, forming a bridge between the Laplace and Normal-Jeffreys' priors. While it has a spike at zero like the Laplace density, it also has a Student's $t$-like tail behavior. Bayesian computation is straightforward via a simple Gibbs sampling algorithm. We investigate the properties of the maximum a posteriori estimator, as sparse estimation plays an important role in many problems, reveal connections with some well-established regularization procedures, and show some asymptotic results. The performance of the prior is tested through simulations and an application.
1. Introduction
The paper motivates alternative shrinkage priors that combine sparse estimation, limited shrinkage of large coefficients, computational tractability, and uncertainty quantification. It proposes the generalized double Pareto prior and develops Bayesian and regularization-based analyses around its analytic and mixture representations.
- Normal-Jeffreys’ priors can strongly shrink small coefficients while minimally shrinking large ones, but create an inferential problem because the posterior is improper.
- Existing Laplace-based procedures can over-shrink large coefficients because the Laplace prior has relatively light tails.
- The paper identifies a need for priors that produce sparse point estimates, avoid over-shrinking nonzero coefficients, compute straightforwardly in large p settings, and quantify uncertainty.
- The generalized double Pareto prior has a simple analytic form, a proper posterior, a spike at zero, Student’s t-like tails, and a scale-mixture representation enabling Gibbs sampling.
- The prior is studied through fully Bayesian and frequentist penalized-likelihood approaches, including posterior computation and maximum a posteriori estimation.
- Its induced penalty yields a consistent continuous thresholding rule in the orthogonal case and connects the Laplace-mixture procedure to the adaptive LASSO.
2. Generalized Double Pareto Prior
The generalized double Pareto prior combines a sharp concentration near zero with heavy tails, while its parameters control dispersion, tail behavior, and shrinkage. It admits hierarchical representations that connect it to Laplace and Normal-Jeffreys’ priors.
- Prior behavior: Near zero, the standard double Pareto resembles Laplace, while its Cauchy-like tails help avoid over-shrinkage of large coefficients.These properties motivate its use for sparse estimation and maximum a posteriori estimation.
- Hierarchical representation: A normal-exponential-gamma hierarchy yields the GDP prior marginally, with ξ = η/α.Specifically, θ ∼ N(0, τ), τ ∼ Exp(λ^2/2), and λ ∼ Ga(α, η).
- Limiting cases: The prior forms a bridge between Laplace and Normal-Jeffreys’ priors through limiting choices of α and η.As α →∞ with α/η = λ′, the density approaches a Laplace density with rate λ′.
- Hyperparameter effects: Increasing α sharpens the density and strengthens shrinkage near zero but lightens tails, whereas increasing η flattens the density and weakens shrinkage.Increasing both at the same rate preserves variance while producing lighter tails and convergence toward Laplace behavior.
- Shrinkage implications: Under maximum a posteriori estimation, the prior induces a thresholding rule when η < 2√α + 1, creating a trade-off between shrinkage magnitude and tail robustness.The hyperparameters should therefore be selected carefully, using prior information when available.
3. Bayesian Inference in Linear Models
The paper embeds the generalized double Pareto prior in a normal linear regression model using hierarchical scale mixtures. Conditional posterior distributions support straightforward Gibbs sampling, with optional hyperpriors for α and η.
- Model: The regression model uses y = Xβ + ϵ with normally distributed errors and independently assigned generalized double Pareto priors for the coefficients.The design matrix X has dimensions n × p.
- Hierarchical formulation: The prior is represented hierarchically as β_j|σ ∼ N(0, σ^2τ_j), τ_j ∼ Exp(λ_j^2/2), and λ_j ∼ Ga(α, η).A Jeffreys’ prior is placed on the error scale parameter σ.
- Computation: The scale-mixture representation produces a simple data augmentation scheme with gamma and inverse Gaussian conditional distributions.These conditionals enable direct Gibbs updates for λ_j and τ_j.
- Computation: The authors report that the Gibbs sampler has fast convergence and mixing in their experience.
4. Sparse Maximum a Posteriori Estimation
The generalized double Pareto prior induces a sparsity-favoring penalty for regularized least squares. Its penalty can reduce shrinkage bias for large coefficients while producing thresholding under a condition on η and α.
- Penalty formulation: The generalized double Pareto prior induces a sparsity-favoring penalty in regularized least squares, analyzed initially for orthonormal design matrices.The resulting penalty is p(|β_j|) = (α + 1) log(ση + |β_j|).
- Near-unbiasedness: The derivative of the induced penalty decreases with |β_j|, so shrinkage bias diminishes as coefficient magnitude increases.This contrasts with LASSO, whose bias term remains constant regardless of |β_j|.
- Near-unbiasedness: The ratio governing the penalty derivative’s decay converges to α + 1, while larger α produces lighter tails and changes that convergence rate.
- Sparsity: The induced estimator is a thresholding rule when η < 2√α + 1, automatically setting sufficiently small estimated coefficients to zero.When the observed coefficient magnitude is below the threshold region, the penalized least-squares estimator is zero.
4.3. Continuity
The continuity analysis identifies a hyperparameter choice that makes generalized double Pareto thresholding continuous in the data. The section also describes EM computation and compares the GDP estimator with established regularization procedures.
- Continuity: Continuity in the data is linked to avoiding prediction instabilities caused by abrupt changes in regularized estimates.Hard thresholding is described as unstable, whereas ridge and LASSO estimates are considered stable.
- Continuity: For the generalized double Pareto penalty, the minimum determining continuity occurs at |β_j| = σ(√α + 1 − η).
- Continuity: Choosing η = √α + 1 yields an estimator with the continuity property.This choice creates a trade-off between sparsity and tail robustness, with larger values penalized less than under LASSO.
- Non-orthogonal regression: For non-orthogonal linear regression, the authors use a normal likelihood and estimate the model with an expectation-maximization algorithm.The resulting estimator is called GDP(MAP).
- Regularization connections: The one-step GDP estimator connects adaptive LASSO and LASSO solution paths, but the authors do not use it in their experiments.As η† → ∞ with α†/η† → λ† < ∞, GDP(OS) gives the LASSO solution with penalty parameter λ†.
4.5. Oracle Properties
The GDP(MAP) and GDP(OS) estimators possess oracle properties under stated regularity conditions, including i.i.d. mean-zero errors and a positive-definite design limit.
- GDP(MAP) and GDP(OS) estimators possess oracle properties.The result follows work by Zou (2006) and Zou and Li (2008).
- The error terms are assumed independent and identically distributed with mean 0 and variance σ2.
- The normalized design cross-product n^-1X′X converges to a positive definite matrix C.
- The notation βA retains coefficients indexed by A, while CA retains the corresponding rows and columns of C.
Theorem 1. Let
The theorem defines the GDP(OS) estimator and an associated active-set sequence under asymptotic conditions on the estimator and tuning parameters.
- The GDP(OS) estimator is denoted by α′n in equation (4.5), with An defined through nonzero components of β(1).
- The theorem assumes α†n tends to infinity, with additional asymptotic conditions stated for the estimator sequence.
- The proofs of the theorem are deferred to Section 8.
5. Experiments
The experiments compare GDP estimators with Bayesian and regularization competitors across sparse and dense simulated models, examining model error and hyper-parameter adaptation. GDP(PM) performs well across sparsity settings, while GDP(MAP) is especially competitive as sparsity, signal, or sample size increases.
- Simulation design: The simulations use four models varying the number of nonzero coefficients between 5 and 10 and signal values between 1 and 3.
- Simulation design: 100 data sets were generated for each simulation case, and median estimator performance was reported.
- Simulation comparisons: GDP(PM) performs similarly to the horseshoe prior in sparse setups and adapts flexibly to dense models with small signals.
- Simulation comparisons: GDP(MAP) performs similarly to SCAD and much better than LASSO, particularly with increasing sparsity, signal, or sample size.
- Computation: GDP(PM) and GDP(MAP) computations are straightforward and computationally inexpensive through normal and Laplace scale-mixture representations.
- Computation: A simple Gibbs sampler is available, especially when α = η = 1, making the procedure attractive for average users.
- Limitations and defaults: Setting α = η = 1 may be restrictive for very dense or very sparse models, although it performed comparably to other methods in the considered cases.
- Limitations and defaults: For p ≫ n cases, the paper does not provide results and states that much larger α values would be needed for adjustment.
6. Data Example
The ozone-data example evaluates GDP estimators and competitors on repeated training-test splits using prediction performance and model sparsity. GDP(MAP) selects substantially smaller models, while GDP(PM)1 achieves the second-best predictive results after BMA.
- Data and evaluation: The dataset has p = 12 predictors and n = 203 complete observations, split into 180 training and 23 test observations.
- Data and evaluation: Models include main effects, quadratic terms, and two-way interactions, producing 290 possible subsets.
- Data and evaluation: Prediction summaries use median R2test values and ±2 standard-error intervals from 100 random training-test splits, with bootstrap standard errors.
- Results: The median number of retained predictors is 4 for all three GDP(MAP) estimators, compared with 14 for LASSO and 9 for SCAD.
- Results: GDP(PM)1 yields the second-best prediction results after BMA, while GDP(PM)2, GDP(PM), and horseshoe perform somewhat worse.
- Results: GDP(PM)1 and SCAD perform within LASSO’s standard-error range while retaining fewer variables in the model.
- Interpretation: The posterior mean under the GDP prior is presented as appealing for mimicking BMA while producing relatively small predictor sets.
7. Discussion
The discussion presents the generalized double Pareto prior as a hierarchical shrinkage framework connecting Bayesian inference with regularization. It supports sparse estimation, posterior uncertainty and model-averaging-like behavior, while retaining computational simplicity and broad applicability.
- The proposed hierarchical prior is obtained through a particular scale mixture and supports both Bayesian inference and regularization-oriented analysis.
- Its marginal prior combines a spike near zero with heavy, Cauchy-like tails, supporting shrinkage of small coefficients while accommodating larger values.
- The framework yields a sparse maximum a posteriori estimator, a posterior mean estimator that mimics model averaging, and uncertainty measures around estimates.
- Simple block-updated Gibbs sampling provides computational advantages over Bayesian model averaging, which requires sampling from a model space with 2p models.
- Small-sample simulation studies report simple, fast computation and excellent performance, while proposed methods are suggested for generalized linear, nonparametric regression, factor-analysis and nonparametric Bayes settings.
8. Technical Details
The technical details establish asymptotic properties of the estimators using convergence arguments, convexity, epiconvergence and KKT optimality conditions. The proofs address asymptotic normality, consistency and exclusion of irrelevant variables.
- The proofs use Slutsky’s theorem and convergence in distribution to establish asymptotic normality.
- The analysis assumes X′X/n converges to C and ϵ′X/√n converges in distribution to W distributed as N(0, σ2C).
- Epiconvergence and convexity are used to identify the limiting objective and its unique minimum.
- KKT optimality conditions are used to show that irrelevant variables are excluded with probability tending to one.
- The technical results include consistency of the estimator.