Source-linked AI summary
Vector Approximate Message Passing for the Generalized Linear Model
Philip Schniter, Sundeep Rangan, Alyson K. Fletcher
TL;DR
The paper addresses GLM inference when existing GAMP methods can fail for general linear transforms. It extends VAMP to the GLM and reports greater robustness to ill-conditioning than adaptively damped GAMP.
Problem
GAMP is efficient and rigorously characterized for suitable i.i.d. matrices but can misbehave for other A, while damping and sequential updating have limited effects.
Method
The paper extends the VAMP framework to provide a methodology for MAP estimation and approximate inference under the GLM.
Results
GLM-VAMP accurately recovered x across the tested condition-number range, whereas adaptively damped GAMP failed at higher condition numbers.
Takeaways & Limitations
VAMP provides a more robust GLM inference approach for ill-conditioned rotationally invariant matrices than adaptively damped GAMP in the reported experiments.
Abstract
from arXiv · showhide
The generalized linear model (GLM), where a random vector $\boldsymbol{x}$ is observed through a noisy, possibly nonlinear, function of a linear transform output $\boldsymbol{z}=\boldsymbol{Ax}$, arises in a range of applications such as robust regression, binary classification, quantized compressed sensing, phase retrieval, photon-limited imaging, and inference from neural spike trains. When $\boldsymbol{A}$ is large and i.i.d. Gaussian, the generalized approximate message passing (GAMP) algorithm is an efficient means of MAP or marginal inference, and its performance can be rigorously characterized by a scalar state evolution. For general $\boldsymbol{A}$, though, GAMP can misbehave. Damping and sequential-updating help to robustify GAMP, but their effects are limited. Recently, a "vector AMP" (VAMP) algorithm was proposed for additive white Gaussian noise channels. VAMP extends AMP's guarantees from i.i.d. Gaussian $\boldsymbol{A}$ to the larger class of rotationally invariant $\boldsymbol{A}$. In this paper, we show how VAMP can be extended to the GLM. Numerical experiments show that the proposed GLM-VAMP is much more robust to ill-conditioning in $\boldsymbol{A}$ than damped GAMP.
I. INTRODUCTION
The paper studies estimating a random vector x from observations generated by a generalized linear model, where measurements depend probabilistically on the linear transform z = Ax.
- The GLM estimates x from observations y whose likelihood depends on hidden transform outputs z = Ax through a probabilistic measurement channel.The model specifies a prior p_x for x and a conditional density p_y|z for measurements given z.
A. The Generalized Linear Model
The GLM covers applications from regression and imaging to communications, with measurement channels that may be additive, binary, quantized, magnitude-based, or count-valued.
- The GLM applies across statistics, computer science, and engineering, including regression, imaging, communications, photon-limited imaging, and neural spike-train inference.The roles of x, A, and y vary by application, while p_y|z characterizes the measurement relationship.
- Robust regression models measurements as y = z + w with i.i.d. heavy-tailed noise to handle outliers beyond the standard AWGN model.The standard linear model uses additive white Gaussian noise, whereas robust methods use heavy-tailed noise distributions.
- Binary classification uses y_m = sgn(z_m + w_m), with Gaussian or logistic errors producing probit or logistic models.
- Quantized compressive sensing uses y_m = Q(z_m + w_m), where Q is a scalar quantizer.
- Phase retrieval observes magnitudes |z_m + w_m|, while photon-limited imaging models photon counts with Poisson rates z_m.
B. Inference under the Generalized Linear Model
Inference under the GLM can target MAP estimates or approximate marginals, but existing methods face tractability or robustness limits; the paper proposes a VAMP-based methodology to address this gap.
- MAP estimation converts GLM inference into minimizing a likelihood-derived loss plus a prior-derived regularizer, including LASSO for AWGN measurements and a Laplacian prior.
- Approximate marginal inference seeks posterior marginals and MMSE estimates because MAP objectives may be surrogates and point estimates do not quantify uncertainty.Exact marginal inference is generally intractable, motivating approximation.
- MCMC can be expensive and difficult to assess in high dimensions, while variational methods restrict distributions and may require large matrix inversions.
- GAMP is computationally efficient for MAP or marginal inference with few restrictions, but its rigorous scalar-state-evolution guarantees rely on i.i.d. sub-Gaussian A and it can diverge for perturbed or correlated matrices.Damping and sequential updating improve robustness only to a limited extent.
- The paper proposes a VAMP-based methodology for both MAP estimation and approximate inference under the GLM.The approach extends the VAMP framework to generalized linear measurements.
II. VAMP FOR THE STANDARD LINEAR MODEL
VAMP extends AMP-style inference to right-rotationally invariant matrices and supports both marginal and MAP inference through denoising and LMMSE modules.
- VAMP extends AMP's rigorous state-evolution guarantees from i.i.d. sub-Gaussian matrices to right-rotationally invariant matrices.Its fixed points agree with the replica prediction of MMSE, and numerical experiments suggest near-replica performance at moderate dimensions.
- VAMP can perform approximate marginal inference by choosing the denoiser to approximate each marginal posterior p(xn|y).
- VAMP also supports MAP inference through an alternative choice of its denoising function.
- Algorithm 1 alternates a prior-based denoiser with LMMSE estimation under an AWGN pseudo-prior.The denoiser operates on pseudo-measurements r1k = x + N(0, I/γ1k), while the LMMSE module estimates x from AWGN-corrupted measurements.
- After SVD precomputation, VAMP requires two matrix-vector multiplies per iteration, matching AMP's dominant per-iteration cost.
III. VAMP FOR THE GENERALIZED LINEAR MODEL
The paper extends VAMP to GLMs by exploiting an equivalence to a constrained standard linear model and separately tracking the x- and z-subvectors.
- The GLM is represented through an equivalent standard linear model with an auxiliary Gaussian-noise parameter taken toward the noiseless constraint limit.
- GLM-VAMP modifies VAMP by separately tracking precisions for the x and z subvectors, yielding Algorithm 2.The two subvectors can behave differently, motivating separate precision parameters γ and τ.
- Algorithm 2 alternates denoising of x and z with LMMSE estimation under corresponding pseudo-priors.The x and z modules use separate pseudo-measurements and precisions, while the linear coupling is handled by the LMMSE steps.
- The x- and z-denoisers decouple because x and y are conditionally independent given z.
- The LMMSE updates are obtained by solving the coupled x–z optimization and substituting the SVD of A.The resulting Jacobian traces determine the α and β coefficients used by the algorithm.
IV. NUMERICAL EXPERIMENTS
In one-bit compressed sensing with rotationally invariant matrices, GLM-VAMP remains accurate across severe ill-conditioning and converges faster than adaptively damped GAMP.
- The experiment used sparse 512-dimensional signals, 2048 measurements, 40 dB signal-to-noise ratio, and rotationally invariant matrices with geometrically spaced singular values.
- Performance was measured with debiased NMSE because the one-bit measurement channel discards amplitude information.
- VAMP accurately recovered x across the full tested condition-number range up to κ(A)=10^6, whereas AD-GAMP failed above κ(A)=10^3.The comparison used average debiased NMSE over 500 independent realizations.
- VAMP converged in about 10 iterations versus 30–40 for AD-GAMP where both methods recovered accurately.
- At κ(A)=10^6, VAMP converged in fewer than 20 iterations.
APPENDIX
The appendix derives the GLM-VAMP LMMSE updates by differentiating the coupled quadratic cost, applying a block-matrix inversion, and substituting the SVD of A.
- The appendix derives the coupled x- and z-stationarity equations by setting the gradients of the quadratic cost to zero.
- The iteration-level figure tracks debiased NMSE across κ(A)=1, 316.23, and 10^6 over 500 realizations.
- A Schur-complement inversion of the block system produces expressions for the LMMSE estimates.
- Substituting A=USV^T into the resulting expressions yields the SVD-based forms used for the GLM-VAMP updates.