PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-System Analysis of Sparse Bayesian Learning

Fangqing Xiao, Dirk T. M. Slock

arXiv:2609.06560v1eess.SPcs.ITmath.IT

TL;DR

Large-system stationary behavior of SBL is difficult to characterize because its variance hyperparameters are jointly estimated from the data. The paper uses leave-one-out analysis and adaptive-response corrections to derive scalar descriptions of selected stationary branches, including a noise-learning identity and conditional null-law result.

  • Problem

    The large-system stationary behavior of classical SBL is poorly understood when all variance hyperparameters are estimated from the same data.

    Source: §I · p. 1

  • Method

    A one-coordinate leave-one-out construction transfers an exact conditional Gaussian law to a selected full stationary branch without assuming reduced-to-full stationary-vector proximity.

    Source: §I · p. 1

  • Results

    Joint noise-variance estimation yields the exact identity V_n = η_n at interior joint stationarity and a parameter-free asymptotic chi-square null law when P_X({0}) > 0.

    Source: §I · p. 2, §VI · p. 6

  • Takeaways & Limitations

    The stationary system reduces to a scalar Gaussian law, a generally set-valued scalar KKT graph, and three branchwise balance equations.

    Source: §VI · p. 6

    The scalar equations are necessary branchwise consistency conditions, and the paper does not claim that every algebraic scalar root is realizable by SBL.

    Source: §I · p. 2

Abstract

from arXiv · show

Sparse Bayesian learning is widely used for sparse linear inverse problems, yet its large-system stationary behavior remains poorly understood because all variance hyperparameters are estimated from the same data. We study classical sparse Bayesian learning, formulated as evidence maximization (type-II maximum likelihood), for underdetermined linear models with sensing matrices having independent and identically distributed Gaussian entries, Gaussian measurement noise, and an unknown deterministic signal sequence. Hyperparameter reoptimization induces a nonvanishing feedback term: a typical coordinate obeys a reoptimization-corrected scalar Gaussian law whose signal coefficient is governed by the normalized adaptive response rather than the frozen resolvent trace. A one-coordinate leave-one-out construction gives an exact conditional Gaussian law, which is transferred to a selected full stationary branch without assuming asymptotic closeness of the reduced and full stationary vectors. Combining this law with the Karush--Kuhn--Tucker conditions of the evidence objective yields a generally set-valued scalar relation and three branchwise large-system consistency relations. If the model noise variance is jointly estimated by evidence maximization, interior joint stationarity yields an exact finite-dimensional equality between the normalized residual energy and normalized resolvent trace. When the limiting signal law has nonzero mass at zero, this identity further yields a parameter-free asymptotic chi-square null law. Under an additional differentiability condition on the selected scalar branch, the large-system characterization also gives a closed relation for the reconstruction error of the posterior mean. The analysis is stationary-point based and permits multiple stationary branches.

I. INTRODUCTION

The paper studies classical SBL evidence maximization in Gaussian linear inverse problems, where reoptimizing variance hyperparameters couples coordinates. It develops a leave-one-out analysis of selected stationary branches and derives branchwise scalar characterizations.

B. SBL Evidence Objective and KKT Conditions

  • Classical SBL assigns independent zero-mean Gaussian priors with variance hyperparameters estimated by maximizing evidence, without imposing a hyperprior.The model noise variance is fixed or estimated from the same evidence objective.Source: §I · p. 1, B. SBL Evidence Objective and KKT… · p. 2

I. INTRODUCTION

  • The analysis uses a genuine leave-one-out construction to derive a reoptimization-corrected coordinate law without assuming proximity between reduced and full stationary vectors.The correction reflects feedback from reoptimizing the remaining hyperparameters.Source: §I · p. 1
  • The scalar law and finite-dimensional KKT conditions yield a generally set-valued scalar relation and three branchwise large-system consistency relations.The analysis permits multiple stationary points and scalar roots; the equations are necessary conditions, not an update rule, and not every root is claimed realizable.Source: §I · p. 2
  • Joint evidence estimation of noise gives an exact interior-stationarity identity and, when the limiting signal law has mass at zero, a parameter-free asymptotic chi-square null law.An additional differentiability condition on the selected scalar branch yields a closed posterior-mean MSE relation.Source: §I · p. 2

A. Gaussian Linear Model

  • The analysis concerns underdetermined Gaussian sensing with Gaussian measurement noise and an unknown deterministic signal sequence; it imposes no generative prior or sparsity assumption on that sequence.A nonzero limiting mass at zero is required only for the stated null-law result.Source: A. Gaussian Linear Model · p. 2, §I · p. 2

B. SBL Evidence Objective and KKT Conditions

  • The paper characterizes stationary KKT points of the nonconvex evidence objective rather than the trajectory of a particular optimization algorithm.Different algorithms or initializations may select different stationary points.Source: B. SBL Evidence Objective and KKT… · p. 2

C. Regular Stationary-Point Class

The analysis follows selected finite-coordinate continuations of regular SBL stationary branches without requiring uniqueness or proximity between full and reduced points. It distinguishes the adaptive response from the frozen resolvent response, then uses a genuine reduced problem to obtain a coordinate law.

C. Regular Stationary-Point Class

  • Because the evidence objective is nonconvex, algorithms or initializations may select different stationary points, so the analysis does not assume uniqueness or global optimality.The compatible branch family includes selected full and auxiliary stationary points connected by continuous KKT paths under finite-coordinate changes.Source: C. Regular Stationary-Point Class · pp. 2–3
  • The selected branch is required to have bounded hyperparameters, uniformly positive active-face curvature, and locally finite, isolated, simple support transitions.These regularity conditions support implicit continuation and Gaussian integration by parts; they assume neither sparsity nor full-to-reduced proximity.Source: C. Regular Stationary-Point Class · p. 3

D. Adaptive Response at an SBL Stationary Point

  • The adaptive response accounts for reoptimization of active hyperparameters and differs from the frozen-resolvent trace by a KKT-response term.The proposition gives h_n ≤ η_n under the stated local regularity conditions.Source: D. Adaptive Response at an SBL… · p. 3

III. REOPTIMIZATION-CORRECTED SCALAR LAW

  • A full-data leave-one-out resolvent is not independent of the deleted column, so the analysis starts from a genuine reduced SBL problem and transfers its conditional Gaussian law to the selected full branch.The construction does not assume asymptotic closeness of the reduced and full stationary vectors.Source: §III · p. 3

A. Genuine One-Column Cavity

A genuine one-column cavity yields an exact conditional Gaussian law that is transferred to a selected full stationary branch; combining it with KKT conditions gives a branch-selected, generally set-valued scalar relation.

A. Genuine One-Column Cavity

  • Deleting coordinate i and its true signal contribution makes the reduced branch independent of the deleted Gaussian column, so its cavity projection is exactly conditionally Gaussian.Continuations transfer the law back to the selected full stationary point.Source: A. Genuine One-Column Cavity · pp. 3–4
  • The transfer does not require asymptotic closeness between genuine reduced and full stationary vectors, under the stated pathwise regularity and sublinear support-transition condition.The transition-count condition also applies after deleting one additional uniformly sampled coordinate.Source: A. Genuine One-Column Cavity · p. 4

B. Reoptimization-Corrected Scalar Gaussian Law

  • The typical-coordinate scalar Gaussian law retains reoptimization feedback, with adaptive response h rather than frozen response η governing the effective signal gain.The empirical law extends this scalar description to the asymptotic coordinate population.Source: B. Reoptimization-Corrected Scalar… · p. 4

C. Scalar KKT Geometry

  • Combining the scalar law with finite-dimensional KKT conditions yields a generally set-valued relation whose selected root follows the finite-dimensional stationary branch.When h = η, the relation reduces to the fixed-point relation of the preliminary analysis.Source: C. Scalar KKT Geometry · p. 4

IV. BRANCHWISE LARGE-SYSTEM CHARACTERIZATION

The branchwise characterization imposes macroscopic balances on every joint subsequential limit. With fixed noise variance, the first two constrain frozen-resolvent geometry and noise coupling, while the third closes the adaptive response.

A. Resolvent and Noise Balances

  • Every joint subsequential limit satisfies three branchwise large-system balances, with the first two constraining frozen-resolvent geometry and noise coupling and the third closing adaptive response.The first balance follows from an exact trace identity; the noise balance is branchwise, and closing h requires the directional KKT response.Source: §IV · p. 4, A. Resolvent and Noise Balances · p. 5
  • The adaptive noise term cannot be conditionally centered because the residual is adaptive, so its balance is established using Gaussian integration by parts.The adaptive-response balance additionally requires the stated directional KKT response.Source: A. Resolvent and Noise Balances · p. 5

B. Adaptive-Response Balance

The adaptive-response balance closes the response variable h under an additional endpoint differentiability condition, while joint noise estimation forces residual energy and resolvent response to coincide exactly. Under regular jointly profiled paths, the leading scalar and large-system laws persist.

B. Adaptive-Response Balance

  • Assumption 4 is needed only for derivative-level adaptive-response transfer and the closed reconstruction-error relation, not Theorem 1 or the first two balances.Source: B. Adaptive-Response Balance · p. 5
  • Under Assumption 4, the adaptive-response balance closes h, which cannot generally be replaced by the frozen-resolvent response η.The balance also implies h ≠ 0 for every regular limiting state.Source: B. Adaptive-Response Balance · p. 5

A. Noise-Variance Stationarity

  • At every interior joint stationary point, normalized residual energy Vn equals normalized resolvent response ηn exactly, removing one macroscopic degree of freedom.With fixed noise variance, those two states are distinct.Source: A. Noise-Variance Stationarity · p. 5

V. JOINT NOISE VARIANCE ESTIMATION AND RECONSTRUCTION ERROR

B. Normalized Scalar Distribution

Joint noise estimation yields a parameter-free chi-square null law when the limiting signal has mass at zero, and a differentiable selected scalar branch gives a relation between effective noise and reconstruction error. The characterization applies to regular stationary branches; extending it beyond i.i.d. Gaussian sensing remains future work.

B. Normalized Scalar Distribution

  • When PX({0}) > 0, the normalized scalar field has a parameter-free chi-square null law because X = 0 gives U/√η = G.For nonzero coordinates, the noncentrality remains branch dependent through α.Source: B. Normalized Scalar Distribution · p. 6

C. Reconstruction Error

  • Under an additional differentiability condition on the selected scalar branch, effective scalar noise V/h^2 equals physical noise σ0^2 plus reconstruction error Eb/δ.The condition selects a differentiable branch when several positive KKT roots coexist.Source: C. Reconstruction Error · p. 6

VI. CONCLUSION

  • The characterization reduces the stationary system to a scalar Gaussian law, a set-valued scalar KKT graph, and three branchwise balances.Multiple stationary branches are permitted.Source: §VI · p. 6
  • Whether standard SBL algorithms select compatible regular branches, and whether the analysis extends beyond i.i.d. Gaussian sensing, remain open directions.Source: §VI · p. 6

APPENDIX A LOCAL GEOMETRY OF REGULAR SBL STATIONARY

The appendix establishes regularity for the selected residual map and develops a leave-one-out transfer that accounts for hyperparameter reoptimization. It uses this construction to relate adaptive responses along reduced paths to full-state quantities.

APPENDIX A LOCAL GEOMETRY OF REGULAR SBL STATIONARY

  • The selected residual map is Gaussian-Sobolev regular, its weak Jacobian matches (11), and the adaptive trace satisfies h_n ≤ η_n.Source: Appendix A · pp. 6–7

APPENDIX B PROOF OF THE REOPTIMIZATION-CORRECTED SCALAR

  • The proof combines deleted-column Gaussian concentration, insertion and correction transfer, integrated adaptive response, and two-column cavity self-averaging.Source: Appendix B · p. 7

A. Reduced Cavity and Local Transfer

B. Integrated Adaptive Response and Large-System State Replacement

  • Smooth response and insertion paths have controlled derivatives for residual energy and normalized traces, supporting replacement of reduced-state quantities by full-state quantities.The normalized residual-energy and trace derivatives are O_P(m^-1/2).Source: B. Integrated Adaptive Response and… · p. 8

C. Two-Column Cavity and Empirical Scalar-Law Convergence

The two-column cavity argument establishes empirical scalar-law convergence, while the subsequent results connect leave-one-out resolvents and finite-dimensional KKT conditions to large-system relations. The appendix also develops balances and a profiled coordinate-response curvature.

C. Two-Column Cavity and Empirical Scalar-Law Convergence

A. Adaptive Leave-One-Out Diagonal Resolvent Relation

  • The leave-one-out resolvent relation combines with finite-dimensional KKT inequalities to yield limiting inactive and active scalar relations, including active sequences approaching Γ = 0.Source: A. Adaptive Leave-One-Out Diagonal… · p. 9

B. Large-System Balances

  • At a KKT point, γ_i d_i = μ_i p_i; Gaussian column moments and the limiting marked law then provide the large-system balance relations.Source: B. Large-System Balances · p. 9
  • The noise-balance proof applies a weakly differentiable Gaussian Stein identity to the complete adaptive residual map, without assuming independence between the noise and residual.Source: B. Large-System Balances · p. 9

APPENDIX D ADAPTIVE-RESPONSE CLOSURE

  • The appendix identifies the adaptive-response closure through a random-endpoint response-trace argument.Source: Appendix D · p. 9

A. Finite-Dimensional Profiled Coordinate Response

  • The profiled active-coordinate KKT curvature κ_i is defined through a Schur complement and matches the one-dimensional profiled curvature at regular support transitions.Source: A. Finite-Dimensional Profiled… · p. 9

B. Adaptive-Response Trace Transfer

The trace-transfer argument controls the adaptive response at a random coordinate endpoint and carries it into the limiting marked law. It also identifies activity in that law through the positive-variance marks.

B. Adaptive-Response Trace Transfer

C. Random-Endpoint Adaptive-Response Concentration

  • At the random endpoint, activity equals 1{Γ > 0} almost surely, and the limiting directional response satisfies S_dir = hΨ_η,h(Γ).The interpolation regularity needed for this passage follows from the regular branch bounds rather than an additional assumption.Source: C. Random-Endpoint Adaptive-Response… · pp. 10–12

RECONSTRUCTION ERROR

The appendix studies reconstruction-error relations through noise sensitivity and direct transport along a jointly profiled branch. It concludes that the scalar law and adaptive-response limit extend to joint noise estimation.

A. Noise-Variance Sensitivity

  • The posterior-mean sensitivity satisfies ∥∇_y b_τ∥₂ = O_P(m^-1/2), while the normalized joint-versus-profiled Jacobian trace difference is O_P(m^-1).Source: A. Noise-Variance Sensitivity · p. 12

B. Direct Jointly Profiled Cavity Transport

  • A data-dependent profiled noise variance cannot be handled by simply inserting a random τ into the fixed-τ cavity theorem, so the analysis transports the cavity on the jointly profiled branch.Source: B. Direct Jointly Profiled Cavity… · p. 12
  • On the jointly selected branch, the scalar law and empirical scalar-law convergence hold, the scalar KKT relations remain unchanged, and the adaptive-response limit agrees with the fixed-noise branch.Source: B. Direct Jointly Profiled Cavity… · pp. 12, 14, 15

C. Full-to-Reduced τ Transfer

The joint selected residual map inherits local Lipschitz, piecewise-C1 chart structure, and its weak Jacobian is identified almost everywhere. Large-system state replacement follows from Lemma 2.

C. Full-to-Reduced τ Transfer

  • Large-system state replacement follows from Lemma 2.The passage also identifies (184) as the rate used in Section V.Source: C. Full-to-Reduced τ Transfer · p. 15
  • Assumption 5 and positive profiled curvature identify the joint selected residual map’s weak Jacobian with Jjoint almost everywhere.The map inherits the locally Lipschitz, piecewise-C1 chart structure of Assumption 2.Source: C. Full-to-Reduced τ Transfer · p. 15

D. Branchwise Reconstruction Error

The branchwise reconstruction-error analysis establishes an MSE limit using uniform integrability. Differentiating the scalar KKT relation and applying conditional Stein arguments yields a closed MSE relation on realized branch limits.

D. Branchwise Reconstruction Error

  • Bounded xi and Gaussian column moments give uniform integrability, yielding the MSE limit stated in Section V.For deterministic limits, the convergence is in probability.Source: D. Branchwise Reconstruction Error · p. 16

E. Closed MSE Relation

  • Differentiating the scalar KKT relation gives ∂U b X = (2 + ηΓ)/[(2h−η)+ηhΓ] on the active branch and zero otherwise.The passage also gives P = −h( b X −X)+.Source: E. Closed MSE Relation · p. 16
  • For each realized branch limit (M, νb), conditional scalar Stein arguments, Theorem 3, and (33) yield the stated relation.The passage specifies applying Stein conditionally on each realized branch limit.Source: E. Closed MSE Relation · p. 16
  • Because h is nonzero by Theorem 3, the last identity gives (48), and setting V = η gives (49)–(50).Source: E. Closed MSE Relation · p. 16

Related papers

Loading 2609.06560v1…