PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Information Bottleneck for Stable Variables

Dongmin Park, Jihad Fahs, Mohamad Assaad

arXiv:2610.08007v1cs.ITmath.IT

TL;DR

The paper extends the Gaussian information bottleneck to jointly α-stable variables, a heavy-tailed setting beyond the jointly Gaussian case. It characterizes a linear stable information bottleneck with stable additive noise and finds that these encoders are generally suboptimal, though asymptotically optimal at large compression rates.

  • Problem

    Exact information bottleneck treatments are limited to discrete alphabets and jointly Gaussian variables, leaving heavy-tailed α-stable sources beyond the Gaussian setting to be studied.

    Source: §I · p. 1

  • Method

    The authors study linear encoders T = kX + N with stable additive noise independent of X and characterize the linear stable information bottleneck in closed form.

    Source: §I · p. 2, B. Linear Stable Information Bottleneck · p. 3

  • Results

    Except in the Gaussian case, stable stochastic linear encoders are strictly suboptimal for the general information bottleneck, but are asymptotically optimal at large compression rates.

    Source: §IV · p. 5, §V · p. 7

  • Takeaways & Limitations

    The linear solution recovers the Gaussian information bottleneck at α = 2, while at fixed rate heavier tails reduce the fraction of relevant information retained.

    Source: §I · p. 2, §IV · p. 6

    For α < 2, the optimal stable linear encoder for the linear stable information bottleneck is not necessarily optimal for the original information bottleneck.

    Source: B. Linear Stable Information Bottleneck · p. 3

Abstract

from arXiv · show

We consider a generalization of the Gaussian information bottleneck problem to jointly stable variables $X$ and $Y$. Specifically, for $X = Y + A$, where $A$ is stable and independent of $Y$, we find optimal stochastic linear encoders of the form $T = kX + N$, $N$ being stable and independent of $X$. We characterize the linear stable information bottleneck in closed-form and its critical value. Furthermore, we show that, except for the Gaussian case, such stochastic linear encoders are strictly suboptimal, however asymptotically optimal at large compression rate. Our linear solution recovers the Gaussian information bottleneck for jointly Gaussian variables.

I. INTRODUCTION

The information bottleneck seeks representations that compress X while preserving information about Y; exact solutions are known for discrete and jointly Gaussian variables. This work extends the study to heavy-tailed α-stable sources with linear stochastic encoders, characterizing their tradeoff and limits.

I. INTRODUCTION

  • For jointly Gaussian X and Y, the optimal Gaussian information bottleneck representation is linear with Gaussian noise, while exact treatments otherwise are confined to discrete alphabets.Its linear transform is determined by the regression matrix's eigenstructure associated with canonical correlation analysis.Source: §I · p. 1
  • Heavy-tailed α-stable variables extend the Gaussian family but have infinite variance, motivating information bottleneck study beyond the jointly Gaussian setting.The paper notes examples in EEG signals, SAR images, and neural-network weight distributions.Source: §I · p. 1
  • The paper asks for the best stable linear encoder and whether such encoders are optimal for α-stable information bottleneck problems.The proposed structure uses linear encoders with additive noise from the same statistical family as X.Source: §I · p. 1
  • For jointly α-stable variables, the authors characterize the linear information tradeoff and critical value; linear encoders are strictly suboptimal except in the Gaussian case but asymptotically optimal at large compression rate.The linear solution recovers the Gaussian information bottleneck at α = 2, and heavier tails make relevant information harder to capture.Source: §I · p. 2

A. Information Bottleneck

The information bottleneck balances compression of X against preservation of information about Y. Its tradeoff parameter controls that balance, and an equivalent constrained formulation fixes the rate.

A. Information Bottleneck

  • The positive parameter β sets the tradeoff between compression and preserving information about Y, with larger β favoring preservation.Source: A. Information Bottleneck · p. 2
  • For β in [0, 1], the optimal encoder is degenerate and preserves no information about X or Y; as β approaches infinity, an optimal encoder is T = X.The degenerate solution has I(X; T) = I(Y; T) = 0.Source: A. Information Bottleneck · p. 2
  • An equivalent formulation constrains the optimization to a fixed rate R > 0.Source: A. Information Bottleneck · p. 2

B. Symmetric α-Stable Laws

Symmetric α-stable distributions extend the Gaussian family to heavy-tailed variables. Their stability property preserves the stability index under sums of independent variables.

B. Symmetric α-Stable Laws

III. MAIN RESULTS

The paper treats jointly SαS variables as heavy-tailed analogues of jointly Gaussian variables. The stability index α controls departure from the Gaussian regime, with α = 2 recovering the finite-variance Gaussian case.

III. MAIN RESULTS

  • Jointly SαS variables are treated as heavy-tailed analogues of the jointly Gaussian variables in the classical Gaussian information bottleneck setting.Source: §III · p. 2
  • The stability index α controls departure from the Gaussian regime, and α = 2 recovers the finite-variance Gaussian case.Source: §III · p. 2

A. Additive Stable Noise Model

The model treats X as a noisy measurement of Y, with independent additive stable noise A. The encoder aims to preserve information about Y while discarding information introduced by A.

A. Additive Stable Noise Model

B. Linear Stable Information Bottleneck

The LSIB restricts the encoder to a noisy linear form with stable noise and characterizes its optimal information tradeoff. Its solution has a critical β value, coincides with the Gaussian information bottleneck at α = 2, and is not generally optimal for the unrestricted problem when α < 2.

B. Linear Stable Information Bottleneck

  • Stable linear encoders use a noisy linear projection of X, with independent stable additive noise; deterministic continuous encoders generally have infinite mutual information I(X; T).The LSIB optimization is restricted to this encoder class.Source: B. Linear Stable Information Bottleneck · p. 3
  • For α < 2, linear encoders are not necessarily optimal for the original IB problem, unlike the Gaussian case.LSIB studies an optimization restricted to linear encoders.Source: B. Linear Stable Information Bottleneck · p. 3
  • The optimal LSIB encoder has the form T = kX + N, with independent N ∼ SαS(1).The theorem identifies this encoder as optimal within the LSIB.Source: B. Linear Stable Information Bottleneck · p. 3
  • At βc = ρ^-α, the optimal encoder changes from degenerate k = 0 to nontrivial k ≠ 0; at α = 2, it coincides with the GIB solution.The transition corresponds to activation of a representation component.Source: B. Linear Stable Information Bottleneck · p. 3
  • For β below βc, snrα = 0 is the unique global minimizer, while the optimal snrα increases with β when β ≥ βc.The proof establishes the zero-SNR minimizer below the critical value and monotonicity above it.Source: B. Linear Stable Information Bottleneck · p. 4

C. LSIB Information Curve

The LSIB information curve describes the relevant information attainable at each compression level for linear encoders. It starts at zero at the critical tradeoff value and approaches the information shared by X and Y as β grows.

C. LSIB Information Curve

  • The LSIB curve is increasing and concave as β varies, reflecting diminishing returns in relevant information as more information about X is retained.Figure 1(a) plots the mutual-information tradeoff for α = 1.5 and ρ = 0.6, 0.8.Source: C. LSIB Information Curve · p. 5, Figure 1 · p. 5
  • As β crosses βc = ρ^-α, the curve emerges continuously from the origin, and as β → ∞, I(Y; T) approaches I(X; Y).At β = βc, both mutual informations are zero.Source: C. LSIB Information Curve · pp. 4–5
  • The curve is characterized by the stable parameter α and correspondence parameter ρ, and it can also be expressed directly as relevant information versus compression rate R.The rate is R := I(X; T).Source: C. LSIB Information Curve · p. 5

IV. NON-OPTIMALITY OF LINEAR ENCODERS

The paper shows that nonzero stable linear encoders are strictly suboptimal for the general information bottleneck above the critical value, then bounds their relevant-information loss. The bound and reported trends indicate that the linear solution approaches optimality at large compression rates.

IV. NON-OPTIMALITY OF LINEAR ENCODERS

  • Linear conditional-mean estimators for jointly stable variables do not imply that linear encoders are optimal for the information bottleneck.The paper cites, for the conditional-mean result and contrasts it with encoder optimality.Source: §IV · p. 5
  • For β > βc = ρ^-α, every nonzero stable linear encoder is strictly suboptimal for the general information bottleneck.The proof shows that such an encoder fails a necessary condition for optimality among all channels.Source: §IV · pp. 5–6
  • The paper bounds the relevant-information loss ℓ(R) of the best linear encoder at fixed compression rate R.The loss is the difference between the optimal relevant information and the linear encoder's relevant information.Source: §IV · p. 6
  • The lower bound on the fraction of relevant information preserved converges to 1 at large compression rates, with the gap upper-bounded by a quantity that decays exponentially in R.The paper evaluates this bound for different α and ρ values.Source: §IV · p. 6
  • At fixed rate, the fraction of relevant information retained decreases as α falls and increases as ρ decreases and α approaches 2.The paper reports that in the latter regimes, most relevant information can be retained at moderate rates with only a small loss.Source: §IV · p. 6

V. CONCLUSION

The conclusion summarizes the information bottleneck for jointly α-stable variables in an additive model. It emphasizes the linear solution's closed form, its strict suboptimality outside the Gaussian case, and its asymptotic behavior at large rate.

V. CONCLUSION

  • For jointly α-stable variables related through an additive model, stochastic stable linear encoders are not generally optimal.The conclusion states that the linear solution is asymptotically optimal at large rate.Source: §V · p. 7
  • The best encoder of the form T = kX + N, with stable N, has a closed-form linear information bottleneck expression and a critical value βc.The stated noise N is a stable random variable.Source: §V · p. 7
  • The linear information bottleneck expression reduces to the optimal jointly Gaussian solution as a special case.Source: §V · p. 7

APPENDIX A α-STABLE PROBABILITY LAWS

The appendices review properties of α-stable laws and provide technical lemmas used in the analysis. They include tail and score-function results, plus a necessary condition and asymptotic expression for the linear encoder.

APPENDIX A α-STABLE PROBABILITY LAWS

  • Symmetric α-stable variables are heavy-tailed, power-law variables with infinite variance, and stable laws arise as limits of normalized IID sums.The appendix defines the symmetric case by β = 0 and δ = 0.Source: Appendix A · p. 7
  • The appendix gives asymptotic series expansions for the stable density and its derivatives, and states that the density score is bounded and decays at least as C/|u| for |u| ≥ 1.Source: Appendix A · pp. 7, 9

APPENDIX B TECHNICAL LEMMAS

  • A necessary condition for global optimality is that ξ(t) be constant in t for every distinct pair x1, x2.The lemma states this condition for an optimal encoder satisfying the specified stationary condition.Source: Appendix B · p. 10
  • Under the best linear encoder, the appendix derives an asymptotic expression for ξlin(t) that supports testing the necessary optimality condition.The displayed expression includes a term proportional to k(x1 − x2), plus o(1).Source: Appendix B · pp. 11–12
Loading 2610.08007v1…