PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive mixture variational inference for spike-and-slab regression

Hanqing Li, Yaroslav Golub, Xuewen Lu

arXiv:2609.38656v1stat.MEstat.ML

TL;DR

Correlated predictors can support competing sparse explanations, making joint inclusion uncertainty difficult for mean-field approximations to capture. The paper develops adaptive mixtures with direct joint refinement and finds lower posterior-approximation errors than mean field on exact-reference datasets, while complete stagewise fitting can be more accurate near collinearity.

  • Problem

    Correlated predictors can support competing sparse explanations with similar predictions, while mean-field approximations can distort their joint inclusion uncertainty.

    Source: §1 · p. 1

  • Method

    The paper adaptively grows mixtures of product distributions and directly optimizes inclusion indicators and active coefficients while jointly refining component parameters and weights.

    Source: §1 · p. 2

  • Results

    On all 250 simulated datasets with exact posterior references, mixtures reduce inclusion-probability, grouped-support and coefficient-covariance errors relative to multistart mean field.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Posterior comparisons reveal differences in inclusion patterns and coefficient uncertainty that predictive performance alone may miss.

    Source: §6 · p. 9

    Direct joint refinement wins in tested common-initialization comparisons, but complete stagewise fitting is more accurate near collinearity; the discussion does not identify what limits fitted mixtures.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Correlated predictors can support competing sparse explanations with similar predictions, making joint uncertainty about variable inclusion difficult to capture with mean-field approximations. We develop an adaptive fitting procedure for mixtures of product distributions in Gaussian regression with a point-mass spike-and-slab prior. It minimizes reverse Kullback-Leibler divergence directly on inclusion indicators and active coefficients, jointly refining component parameters and weights as the mixture grows. This avoids an additional divergence penalty on unused latent coefficients under independent augmentation. Our analysis relates approximation accuracy to mixture size, support coverage and dependence within supports, and establishes contraction, selection consistency and a Bernstein-von Mises approximation under explicit conditions on the prior, posterior concentration and variational error. On all 250 simulated datasets with exact posterior references, mixtures reduce errors in inclusion probabilities, grouped support probabilities and coefficient covariance relative to multistart mean field. Comparisons at fixed mixture size and common initialization favor direct joint refinement over augmented or restricted refinement in posterior divergence. Complete stagewise fitting can nevertheless be more accurate near collinearity. The results support direct joint refinement for posterior approximation while showing that local gains do not ensure superiority of the full adaptive search.

1 Introduction

Correlated predictors can support competing sparse explanations, so accurate prediction does not ensure accurate joint uncertainty about inclusion. The paper develops adaptive mixtures with direct joint refinement and studies their approximation and fitting performance.

1 Introduction

  • Under predictor correlation, mean-field approximations can preserve marginal inclusion probabilities while distorting the joint probability of selecting competing predictors.A useful approximation must represent uncertainty about model choice as well as coefficients.Source: §1 · p. 1
  • The augmented boosting baseline induces the same exact-zero mixture family when its weights and component parameters vary freely, motivating comparisons of objectives and updates.The paper focuses on how these choices affect approximation within that family.Source: §1 · p. 1
  • Directly optimizing inclusion indicators and active coefficients avoids an extra divergence cost on latent coefficients for excluded predictors.All component parameters and weights are jointly refined as the mixture grows.Source: §1 · p. 2
  • The analysis relates approximation accuracy to mixture size, support coverage and within-support dependence, and gives conditions for contraction, selection consistency and a Gaussian limit.The results require explicit assumptions on the model, posterior concentration and variational error.Source: §1 · p. 2
  • The adaptive procedure is evaluated separately from its proposed improvements, distinguishing local refinement gains from the performance of the full search.The introduction reports that direct joint refinement can win in controlled comparisons while stagewise fitting can perform better near collinearity.Source: §1 · p. 2

2 Model and mixtures

The paper formulates Gaussian regression with a point-mass spike-and-slab prior and approximates its posterior using mixtures of product distributions. The direct objective works on inclusion indicators and active coefficients, avoiding the extra cost that augmentation can impose on excluded predictors.

2.1 Spike-and-slab regression

  • The model uses Gaussian regression and an independent point-mass spike-and-slab prior, with each coefficient inactive at zero or active under a Gaussian slab.Each coordinate is represented jointly by its inclusion indicator and coefficient.Source: §2.1 · p. 2

2.2 Mixture family

  • Each mixture component is a product of Bernoulli–Gaussian coordinates, while averaging over components can represent dependence among inclusion indicators.Two components selecting either of two predictors represent competing supports that a product law with the same marginals spreads across all four supports.Source: §2.2 · pp. 2–3
  • The mixture yields posterior inclusion probabilities and joint inclusion probabilities by averaging component probabilities with the mixture weights.The between-component covariance has rank at most K−1.Source: §2.2 · p. 3

2.3 Direct and augmented objectives

  • The augmented boosting baseline also induces this mixture family when its weights and component parameters vary freely, but its augmented objective is compared with the direct objective.The baseline differs from Spence’s principal product method, which already avoids auxiliary independence.Source: §2.3 · p. 3
  • On the induced Bernoulli–Gaussian mixture family, direct optimization removes the extra divergence cost from auxiliary independence while retaining the same variational family.That cost arises when inactive predictors’ latent coefficients are penalized for departing from their prior.Source: §2.3 · p. 3

3 Computation

The fitting procedure expands a mean-field approximation into a mixture, jointly refining its components and weights. It evaluates objectives using an analytic component contribution and numerical integration for component overlap, with separate integration draws for validation.

3 Computation

  • Starting from mean field, the adaptive procedure adds components and jointly refines their parameters and weights, using separate integration draws to assess proposed improvements.The fit records whether search stops at the component cap, budget or failed proposals.Source: §3 · p. 4, §3.2 · p. 5

3.1 Objective evaluation

  • The mixture objective combines component objectives with a mutual-information term, using the mixture entropy identity.Each component objective is analytic; only the overlap term requires numerical integration for K greater than one.Source: §3.1 · p. 4

3.2 Joint refinement

  • Joint refinement holds the component count fixed while updating all component weights, inclusion probabilities, active means and variances.Bounded L-BFGS-B updates these parameters, and differentiation includes trial density ratios and weights.Source: §3.2 · pp. 4–5
  • For local refinement, fixed reference-mixture points approximate expectations under trial parameters, with density ratios accounting for changes in the trial distribution.The reference and its points remain fixed during each optimization step.Source: §3.2 · p. 4
  • Updates pass overlap and importance-weight checks, then are accepted or rejected using validation batches separate from optimization.Adaptive expansion tries split and residual-guided proposals, refining each candidate before comparing it with the incumbent using fresh draws.Source: §3.2 · pp. 5–6
  • Frozen refinement updates weights and the new component, whereas stagewise fitting adds a product component while holding the incumbent mixture fixed.Joint refinement also updates the existing component parameters.Source: §3.2 · p. 6

4 Experiments

The experiments compare adaptive mixtures with mean field and compare fitting objectives and strategies under controlled settings. Mixtures improve posterior-approximation errors in the reported settings, but controlled refinement gains do not imply that joint refinement always wins as a complete procedure.

4 Experiments

  • The study simulates 550 datasets across predictor counts and correlations, using exact posterior references at p=10 and eligible MCMC references for larger-dimensional PIP and support errors.Paired intervals are descriptive, unadjusted 95% t intervals across datasets.Source: §4 · p. 6

4.1 Posterior approximation

  • On every exact-reference dataset, mixtures reduce PIP, grouped-support and covariance errors versus MFVI; for one-group data at ρ=0.7 and 0.9, mean reverse KL falls from 0.377 to 0.090 and from 0.692 to 0.338.Figure 1 shows support and covariance errors for the one-group datasets.Source: §4.1 · p. 6, Figure 1 · p. 7
  • At p=20,30, mean PIP errors fall by 24.7–41.0% and grouped-support TV by 22.8–51.0%; at p=100, both error intervals include zero at both correlations.For two-group data, KL and support-TV reductions relative to MFVI shrink as correlation increases, with support TV still 0.646 at ρ=0.99.Source: §4.1 · p. 6

4.2 Objective and refinement comparisons

  • From common initializations, direct joint refinement has lower direct posterior KL than augmented joint on all 60 datasets at both refresh caps.Increasing the cap enlarges every mean advantage; at ρ=0.99 grouped-support TV favors augmentation at cap 16 but direct joint at cap 128.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Complete fitting procedures

  • Joint refinement has lower mean KL than frozen refinement at every correlation, but stagewise fitting performs better near collinearity across the reported posterior metrics.At ρ=0.99, stagewise fitting has lower KL, TV, PIP and covariance errors; prediction intervals between mixture strategies include zero.Source: §4.3 · p. 7

5 Theoretical analysis

The analysis relates mixture approximation error to component count, retained-support coverage, and dependence within supports. Under stated posterior, model, and variational-error conditions, it establishes contraction, support-selection consistency, and a Gaussian approximation.

5.1 Mixture capacity

  • For K components, the best reverse-KL improvement over mean field is at most log K, so vanishing error for a fixed-support Gaussian target may require log K to grow with support dimension.The product-Gaussian minimum error grows proportionally to support size when the stated log-determinant quantity does.Source: §5.1 · p. 8

5.2 Support approximation

  • A mixture covering selected supports has an error bound that separates omitted posterior mass from retained-support product-Gaussian error.Conditional dependence can distort support weights as well as coefficient uncertainty; two competing singleton supports with total mass 1 − δ give a2 ≤ −log(1 − δ).Source: §5.2 · p. 8

5.3 Asymptotic results

  • The asymptotic claims rely on explicit sparsity, design, prior, posterior-concentration, and variational-error conditions.For example, the contraction result assumes p ≥ n, s0 log p = o(n), bounded column norms, and bounded known noise and slab variances.Source: §5.3 · p. 9
  • Under Assumption 1 and an optimization-error bound, the variational distribution contracts in prediction and controls expected model size; a sparse eigenvalue condition also yields coefficient contraction.The prediction rate is of order s0 log p, and the expected model size is of order s0.Source: §5.3 · p. 9
  • If the posterior concentrates sufficiently on the true support and the stated condition-number and error bounds hold, thresholding variational PIPs at 1/2 consistently recovers that support.The variational probability assigned to the true support tends to one under these conditions.Source: §5.3 · p. 9
  • When the true-support posterior mass tends to one and the approximation error vanishes under the stated conditions, the variational law has a Bernstein–von Mises Gaussian approximation.The approximation is centered at the least-squares estimator on the true support and is asymptotically accurate uniformly over measurable events.Source: §5.3 · p. 9

6 Discussion

Posterior approximation captures uncertainty over competing supports and coefficient dependence that prediction alone may not reveal. Local refinement gains do not ensure that a complete fitting procedure is superior, and the analysis does not identify what limits fitted mixtures.

6 Discussion

  • Improved support probabilities and coefficient covariance matter when multiple sparse explanations are plausible, because similar predictions can arise from different inclusion patterns.Posterior comparisons can reveal differences that predictive performance alone may miss.Source: §6 · p. 9
  • Direct joint refinement lowers KL from tested common initializations, but complete stagewise fitting is more accurate near collinearity.This distinction motivates examining proposals and search paths as well as refinement.Source: §6 · p. 9
  • The analysis distinguishes approximation from optimization error and highlights support coverage and within-support dependence, but does not identify which limits the fitted mixtures.Source: §6 · p. 10
  • Crossing proposal strategies with joint and stagewise refinement, and independently varying mixture size and refinement effort, are proposed as ways to clarify their effects.Source: §6 · p. 10

AI use statement

The appendices detail the mixture objective and its numerical treatment, including the distinction between direct and augmented objectives and conditions needed to certify descent.

AI use statement

  • AI tools assisted with writing, proofs, simulation code, execution, and result interpretation; the authors take responsibility for the methods, results, and conclusions.Source: AI use statement · p. 10

A.1 Objective and moments

  • The mixture objective separates expected loss from a nonnegative dependence term bounded by the entropy of component weights, which is at most log K.Zero-weight components contribute nothing.Source: §A.1 · p. 11
  • Conditioning on the component label yields the covariance decomposition and joint-inclusion formula, while each component's coefficient moments are its mean and diagonal covariance.Source: §A.1 · p. 11

A.2 Augmented objectives

  • Independent augmentation can add divergence cost for inactive latent coefficients; completing the augmented law with prior draws on inactive coordinates can instead give zero conditional cost.The extra cost is induced by independence between the indicator and latent Gaussian variable.Source: §A.2 · p. 12

A.3 Mixture derivatives

  • The mixture derivatives include an inclusion score at the inactive point, and the finite-objective formulas differentiate the numerical objective used for optimization.The formulas are checked by finite differences, with separate draws assessing integration variability.Source: §A.3 · p. 12

A.5 Acceptance bounds

  • A proposed numerical update certifies exact descent only when its estimated objective decrease exceeds the stated absolute-error bounds.Empirical scramble standard errors alone do not supply those bounds, particularly for adaptive comparisons.Source: §A.5 · p. 13

B Theoretical results

The theoretical results connect mixture approximation and optimization error to posterior approximation, contraction, and selection. Their proofs use support-level bounds, comparator distributions, and transfer from posterior concentration.

B Theoretical results

  • Uniform objective error and proximity to the global infimum bound within-family optimization error; restricted-domain bounds also require accounting for the gap to the full family infimum.Local convergence and finitely many validation checks do not certify these global bounds.Source: B Theoretical results · p. 13

B.1 Capacity bound

  • For mixtures with one product Gaussian per retained support, the capacity bound separates omitted support mass from conditional product-Gaussian error.The conditional error is nonnegative and reflects dependence among active coefficients.Source: §B.1 · pp. 13–14

B.2 Proof of contraction

  • Under the stated sparse-regression assumptions and an optimization error of order s0 log p, the proof establishes prediction contraction and an expected model size of order s0.A sparse eigenvalue condition also yields coefficient contraction without a beta-min condition.Source: §B.2 · pp. 14–15

B.3 Proof of baseline transfer

  • A controlled comparator objective can transfer the prediction and model-size conclusions, provided the comparison error is controlled at the stated order.The truth-centered product distribution in the theorem proof supplies one such comparator bound.Source: §B.3 · p. 15

B.4 Posterior transfer

  • Posterior transfer bounds variational error using true-support posterior mass, conditional Gaussian approximation error, and within-family error.Under the stated assumptions, concentration on the true support yields consistent thresholded-PIP recovery; vanishing approximation error also gives the Gaussian limit.Source: §B.4 · pp. 15–16

C.1 Design and metrics

The simulations use correlated groups with one active predictor per group, alongside independent active predictors. They assess posterior-approximation errors against exact or eligible MCMC references, with descriptive paired intervals.

C.1 Design and metrics

  • 550 datasets vary predictor count, within-group correlation, and one- versus two-group designs, with 50 datasets per (p, ρ) cell.Each group has three correlated predictors and one randomly chosen signal; the one- and two-group designs add four and three independent signals, respectively.Source: Table 2 · p. 17
  • PIP error averages absolute inclusion-probability error, grouped-support TV compares joint inclusion patterns, and covariance error uses the Frobenius norm.Prediction MSE is evaluated on 1,000 independent test rows.Source: §C.1 · p. 17
  • At p=10, all supports are enumerated for exact references; larger dimensions use MCMC references, and reported paired intervals are descriptive, unadjusted 95% t intervals.Integration variability and reference error are assessed separately and are not fully incorporated into these intervals.Source: §C.1 · p. 17

C.2 Posterior approximation

The adaptive mixtures generally improve posterior inclusion and grouped-support approximation over MFVI on the evaluated datasets, while the gains weaken at p=100. Improvements in posterior summaries do not establish prediction or selection advantages.

C.2 Posterior approximation

  • PIP error improves on 263 of 264 eligible datasets and support TV on 262 across p=10, 20, and 30, while neither paired interval excludes zero at p=100.At p=30, ρ=0.7, prediction MSE increases by 0.00344 (95% interval [0.00159, 0.00529]); the other one-group prediction intervals include zero.Source: §C.2 · p. 17
  • At p=10, mixtures have larger PIP, support, and covariance errors than MCMC estimates, although the empirical MCMC law has infinite reverse KL to the continuous-slab posterior.At ρ=0.9, mean errors are 0.03651, 0.1834, and 0.0790 for mixtures versus 0.00221, 0.0090, and 0.0033 for MCMC, respectively.Source: §C.2 · p. 17
  • At ρ=0.7 and 0.9, MFVI and mixtures share true-positive rates of 0.992 and 0.972, while false-discovery proportions rise from 0.0330 and 0.0393 to 0.0497 and 0.0493.Active-coefficient 95% coverage is 0.956 for both at ρ=0.7 and rises from 0.920 to 0.928 at ρ=0.9.Source: §C.2 · p. 17

C.3 Controlled objective and refinement comparisons

Controlled comparisons favor direct joint refinement in posterior KL under common initialization, but complete fitting comparisons show that stagewise fitting can outperform it on some posterior-summary errors near collinearity. Reference eligibility and finite search budgets qualify these results.

C.3 Controlled objective and refinement comparisons

  • At both refresh caps, direct joint refinement has lower direct posterior KL than augmentation on all 60 datasets and than each direct restriction on 51, with nine numerical ties.The compared fits share prepared candidates and do not reproduce each method’s complete adaptive pipeline.Source: §C.3 · pp. 18–19, Table 5 · p. 19
  • Increasing the refresh cap lowers direct joint’s mean KL by 8.0%, 5.9%, and 3.0% across correlations; eight of nine paired change intervals exclude zero.Augmentation improves its own objective on all datasets, but its mean direct KL increases, with all three increase intervals including zero.Source: §C.3 · p. 19, Table 6 · p. 19
  • The direct-versus-augmented KL gap shifts from mainly coefficient-conditional at cap 16 to mainly support KL at cap 128, and contributions use each fit’s own support law.At ρ=0.99 and cap 128, the total contrast −0.6530 equals support contrast −1.0118 plus conditional contrast 0.3588.Source: §C.3 · p. 19

C.4 Complete fitting procedures

  • In complete fitting, joint refinement improves KL, support TV, PIP, and covariance over MFVI on every dataset, and has lower KL than stagewise in 49, 38, and 11 of 50 cases as ρ increases.At ρ=0.99, PIP and covariance contrasts favor stagewise: 0.0444 [0.0333, 0.0555] and 0.0514 [0.0367, 0.0661]; all prediction intervals between mixture strategies include zero.Source: §C.4 · p. 20
  • At ρ=0.9, joint refinement’s conditional-KL contribution is lower than stagewise’s, while the support-KL contrast interval includes zero; at ρ=0.99, stagewise improves both contributions.At ρ=0.99, support KL accounts for approximately 85% of stagewise’s mean advantage; this decomposition does not attribute residual error to proposals, optimization, or capacity.Source: §C.4 · p. 20

C.5 Reference reliability

  • Reference eligibility varies by dimension and correlation, and a probability-specific MCMC screen leaves 24 eligible references at p=100, ρ=0.9; neither screen resolves mean PIP improvement at p=100.Longer chains on six large-case datasets pass the probability screen, with mean absolute PIP changes from 0.00091 to 0.00125.Source: §C.5 · pp. 20–21

C.6 Implementation and supplementary records

  • All 100 primary p=100 mixture fits reach the time limit; extending the allowance first lowers mean objective, but the further increase yields only 0.00052 and 0.00153 more.PIP intervals mostly include zero, and all prediction intervals include zero.Source: §C.6 · p. 21

D Algorithm details

The adaptive search initializes a multistart mean-field product, then grows a mixture through component-splitting proposals and refinement. Proposals are checked through overlap and validation rules, with fallback to the incumbent or the original mean-field fit when improvement is unresolved.

Initialization

  • The mean-field incumbent is the lowest-objective product among ten CAVI fits using two initializations and independently permuted coordinate orders.Each fit holds variances fixed and stops at a maximum-change tolerance of 10^-8 or 1,500 sweeps.Source: D Algorithm details · pp. 21–22

Adaptive expansion

  • The search selects the largest-weight component, splits it along a data- and variance-based direction, and makes all coordinates available for joint refinement.The split copies inclusion probabilities and variances, divides the weight, and tries scales 0.5 and 1.Source: D Algorithm details · p. 22

Alternative proposal

  • If split proposals fail, the search can use a released-tilt proposal, whose temporary tilt constructs an initializer before ordinary CAVI resumes.The resulting product is appended at weight 0.1, with incumbent weights scaled by 0.9; the mixture objective and posterior target remain unchanged.Source: D Algorithm details · p. 22

Clipping limitation

  • The reported fixed clipping bounds prevent full support-mass concentration, although they do not rule out consistent thresholded-PIP decisions.An asymptotic implementation intended to concentrate on the true support must relax clipping at least so that p_n epsilon_n tends to zero, alongside other conditions.Source: D Algorithm details · pp. 22–23

Acceptance and fallback

  • Expansion and final-return decisions use fresh scrambled evaluations, retaining the incumbent if proposals fail and returning mean field if improvement remains unresolved.The final comparison with mean field and the independent final assessment each use four fresh scrambles at 16,384 points.Source: D Algorithm details · p. 23

Stagewise initialization

  • Stagewise adaptation initializes proposals with importance-weighted EM, while later refinement, trust checks, validation, and fallback use the joint-refinement solver’s rules.The first two proposals use adapted importance/weighted-EM initializers; if both fail and time remains, an ordinary released-tilt proposal is appended.Source: D Algorithm details · pp. 23–24

Local refinement

  • JointRefine optimizes a fixed-size mixture against a reference mixture, accepting proposed parameters only after overlap checks and local validation succeed.It uses scrambled Sobol points and bounded L-BFGS-B; failed checks retain the current parameters.Source: D Algorithm details · pp. 22–24
Loading 2609.38656v1…