PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy and mutual information in models of deep neural networks

Marylou Gabrié, Andre Manoel, Clément Luneau, Jean Barbier, Nicolas Macris, Florent Krzakala, Lenka Zdeborová

arXiv:1805.09785v2cs.LGcond-mat.dis-nncs.ITstat.ML

TL;DR

The paper addresses how to compute and interpret entropies and mutual informations in deep neural networks under analytically tractable assumptions. It derives a replica-based method, proves it for two-layer Gaussian networks, and tests it with constrained networks on synthetic data. Compression appears during learning, including with ReLU, but its relationship with generalization remains unclear in the proposed setting.

  • Problem

    Computing entropies and mutual informations for high-dimensional deep-network representations is difficult, while the relationship between information compression and generalization remains unresolved.

  • Method

    The paper derives a heuristic replica formula under independent orthogonally-invariant weights, proves the two-layer Gaussian case using adaptive interpolation, and evaluates it with USV-constrained networks on synthetic datasets.

  • Results

    Compression can occur during learning even with ReLU activations, but no clear link between compression and generalization is observed in the experiments.

  • Takeaways & Limitations

    The framework provides tractable information-theoretic analyses for multilayer models and supports further study of regularizers, unsupervised settings, and learned constrained weights.

  • Takeaways & Limitations

    The replica formula is not proved in full generality, and its validity after learning USV-layer weights still requires confirmation.

Abstract

from arXiv · show

We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physics methods, under the assumption that weight matrices are independent and orthogonally-invariant. (ii) We extend particular cases in which this result is known to be rigorously exact by providing a proof for two-layers networks with Gaussian random weights, using the recently introduced adaptive interpolation method. (iii) We propose an experiment framework with generative models of synthetic datasets, on which we train deep neural networks with a weight constraint designed so that the assumption in (i) is verified during learning. We study the behavior of entropies and mutual informations throughout learning and conclude that, in the proposed setting, the relationship between compression and generalization remains elusive.

1 Multi-layer model and main theoretical results

The paper derives a replica formula for entropies and mutual informations in high-dimensional stochastic multilayer networks with independent orthogonally-invariant weights. It also proves the formula for two-layer networks with Gaussian i.i.d. weights under regularity assumptions.

  • Model: The model is a Markov chain X → T1 → T2 → · · · → TL with componentwise activations and stochastic noise.Hidden variables are generated layer by layer from the preceding representation and the corresponding weight matrix.
  • Replica formula: Under independent orthogonally-invariant weight matrices, the replica method gives normalized entropies of hidden layers through stationary points of a replica potential.The potential depends on layer-wise scalar mutual informations, conditional entropies, and weight spectra.
  • Tractability: The high-dimensional entropy computation reduces to extremizing a function of 4ℓ variables with single- or bidimensional integrals.A fixed-point iteration can perform the extremization, and a Python package supports different priors, activations, and weight spectra.
  • Assumptions: The replica prediction is conjectured under large dimensions, order-one adjacent-layer aspect ratios, independent layers, and orthogonally-invariant random matrices.These assumptions include a decomposition into Haar-distributed singular-vector factors and a diagonal singular-value matrix.
  • Connections: The formula also agrees with known exact Gaussian cases, rigorous single-layer results, broader random-matrix proofs, and AMP/VAMP predictions.These connections provide supporting evidence but do not establish the formula in full multilayer generality.
  • Rigorous result: For two layers with separable bounded-support inputs, bounded C2 activations, and Gaussian i.i.d. weights, adaptive interpolation proves the replica formula in the high-dimensional limit.This theorem closes the corresponding two-layer conjecture.

2 Tractable models for deep learning

The paper constructs two synthetic-data settings that make information quantities tractable while representing either independent inputs or correlated inputs generated from latent codes. It then preserves the required weight structure during learning with constrained USV-layers and adds noise only for information estimation.

  • Motivation: The framework connects information-bottleneck analyses with synthetic learning experiments using replica-computable entropies and mutual informations.The motivating information-bottleneck view seeks label information while reducing input information, but the paper uses the framework to examine this behavior rather than prescribe a new optimizer.
  • Teacher-student: The teacher-student scenario uses separable input variables and labels generated by applying a teacher mapping to each input.The student network is trained on synthetic data whose inputs have a simple factorized structure.
  • Generative-recognition: The generative-recognition scenario samples a separable latent code, generates correlated inputs, and trains a recognition model to recover the code from those inputs.The combined generative and recognition chain restores a separable starting distribution while retaining correlated observed data.
  • Weight constraint: USV-layers keep the singular-vector factors Uℓ and Vℓ fixed while learning only diagonal Sℓ matrices, reducing parameters from nℓ×nℓ−1 to min(nℓ,nℓ−1).The constraint is designed to keep weight matrices close to independent and orthogonally invariant during learning.
  • Weight constraint: USV-layers can learn complex functions despite their restriction, and stacking them increases the number of trainable parameters for experimental comparisons.The paper relates this factorization to structured efficient linear layers that learn diagonal matrices.
  • Information estimation: The experiments train deterministic networks without injected noise, while assuming small additive white Gaussian noise only when computing finite mutual informations.With noise only at the last layer, adjacent-layer mutual information equals the information between that layer and the input.

3 Numerical experiments

The numerical experiments compare replica-based information estimates with non-parametric methods and examine entropy and mutual-information behavior across activations and training. In constrained-weight learning experiments, compression and generalization show no clear relationship.

  • Estimators and activation comparisons: The experiments validate the replica estimator against exact and non-parametric entropy calculations in tractable two-layer networks.The first experiment uses i.i.d. Gaussian weights, where the replica formula is rigorously exact in the large-network limit.
  • Estimators and activation comparisons: Non-parametric estimators systematically over-estimate entropy for linear and hardtanh networks while following the replica estimate's trend as σ varies.The linear case also permits direct closed-form entropy computation for comparison.
  • Estimators and activation comparisons: For ReLU networks, the replica estimate captures the activation cutoff and yields an entropy slope equal to half the linear-network approximation.Non-parametric methods instead predict entropy increasing like that of a linear network with identical weights.
  • Estimators and activation comparisons: Bounded hardtanh entropies decrease after reaching a maximum as weight magnitude increases, unlike the unbounded activations considered, including ReLU.This indicates activation-dependent information behavior even without learning.
  • Learning experiments with linear networks: In linear student training, mutual informations between input and hidden layers increase throughout learning without compromising generalization, while replica estimates approach exact values as width grows.The experiments use USV-constrained layers and compare estimation errors across widths and independent training runs.
  • Learning experiments with linear networks: Other learning experiments show hidden-layer compression, with a longer and more pronounced initial increase for hardtanh layers, but no clear link between compression and generalization.Compression is not more pronounced in deeper layers, and no delayed generalization is observed in the cited hardtanh recognition experiment.

4 Conclusion and perspectives

The paper presents a tractable framework for computing entropies and mutual informations in deep learning models and provides packages supporting its use. Experiments find that compression can occur during learning, but its relationship with generalization remains unclear in the proposed setting.

  • The paper presents a class of deep learning models with a tractable method for computing entropy and mutual information between layers.
  • Python packages facilitate mutual-information computation and training for arbitrary implementations of the model.
  • Compression can occur during learning even with ReLU activations.
  • The experiments do not show a clear link between generalization and compression in this setting.
  • The replica formula may extend to unsupervised contexts, while proving the general multilayer case and validating it after USV-layer learning remain open directions.

1 Replica formula for the entropy

The paper extends replica-based entropy calculations from generalized linear models to multi-layer networks, using orthogonally-invariant weight ensembles and tractable extremization formulas. It also relates the resulting expressions to conditional entropies, mutual informations, and computational procedures.

  • 1.1 Background: The replica method evaluates typical model properties through an extremization over order parameters of a replica-symmetric free energy.The method connects average log-marginal likelihoods with conditional output entropy.
  • 1.1 Background: Orthogonally-invariant matrix ensembles generalize the usual i.i.d. weight assumption and can be analyzed with related replica techniques.Each matrix distribution is invariant under orthogonal transformations, extending prior generalized linear model analyses.
  • 1.2.1 Single-layer: For single-layer generalized linear models, entropy is expressed as an extremization involving the weight-spectrum eigenvalue distribution.The formulation uses α, the input second moment ρ, and a function determined by the spectrum of W^T W.
  • 1.2.1 Single-layer: For Gaussian i.i.d. weights, the eigenvalue spectrum is Marchenko-Pastur, reducing the orthogonally-invariant expression to the known formula.In this case, the result has been proven rigorously.
  • 1.2.2 Multi-layer: The multi-layer analysis computes the entropy of the final-layer representation in the high-dimensional limit with finite layer-size ratios.Posterior inference is generally hard because it requires evaluating a marginal likelihood.
  • 1.3 A simple heuristic derivation of the multi-layer formula: A heuristic multi-layer derivation recursively combines single-layer replica free energies for estimating intermediate representations.Repeating the procedure yields equations for arbitrary depth.
  • 1.4 Formulation in terms of tractable integrals: The formulas can be rewritten using conditional entropies and mutual informations, with Gaussian integrations and auxiliary measures making the expressions explicit.The resulting framework accommodates likelihoods such as linear, probit, and ReLU when the required integrals are computationally feasible.
  • 1.4 Formulation in terms of tractable integrals: The replica potential extends to orthogonally-invariant weights through eigenvalue-dependent functions and recovers the known Gaussian and single-layer formulas.For Gaussian weights, the Marchenko-Pastur spectrum gives FWℓ(AℓVℓ)=αℓAℓVℓ and recovers equation (31).

2 Proof of the replica formula by the adaptive interpolation method

The section proves a replica-symmetric free-entropy formula for a two-layer Gaussian generalized linear model using adaptive interpolation. The proof connects the original high-dimensional estimation problem to analytically tractable scalar channels under boundedness, smoothness, and Gaussian-weight assumptions.

  • 2.1 Two-layer generalized linear estimation: Problem statement: The model uses a signal prior P0, component-wise functions ϕ1 and ϕ2, random auxiliary variables, and two Gaussian measurement matrices.The matrices W1 and W2 have i.i.d. N(0, 1) entries, and the observations include additive Gaussian noise.
  • 2.1 Two-layer generalized linear estimation: Problem statement: The free entropy is defined through the normalization Z(Y, W1, W2), which is the conditional density of Y given the measurement matrices.Its normalized logarithm corresponds to minus the conditional entropy −H(Y|W1, W2)/n0.
  • 2.1 Two-layer generalized linear estimation: Problem statement: In the high-dimensional regime, n0, n1, and n2 diverge while n1/n0 → α1 and n2/n1 → α2, giving n2/n0 → α1 · α2.These ratios define the limiting measurement-rate parameters used in the thermodynamic analysis.
  • 2.2 Important scalar inference channels: The replica calculation decouples the model into scalar inference channels whose free entropies form the replica-symmetric potential.The channels include additive Gaussian denoising, an output-linked channel, and a hidden-layer channel.
  • 2.3 Replica-symmetric formula and mutual information: The proof targets the thermodynamic-limit free entropy of a two-layer generalized linear model and its associated mutual-information formula.The model estimates a signal from measurements generated through two component-wise nonlinear transformations.
  • 2.3 Replica-symmetric formula and mutual information: The replica-symmetric formula holds under a bounded-support prior, bounded C2 activations with bounded first and second derivatives, and i.i.d. Gaussian weights.These are the stated hypotheses H1–H3 for Theorem 1.
  • 2.4 Interpolating estimation problem: Adaptive interpolation connects the original two-layer model at t=0 to two analytically tractable problems at t=1.The endpoint at t=1 combines one scalar output channel with a one-layer GLM whose limiting free entropy is already known.
  • 2.4 Interpolating estimation problem: The perturbation changes the interpolating free entropy by at most order s_n0, while regularity controls the interpolation functions and parameter bounds.The proof also establishes convergence of ρ1(n0) to ρ1 under the theorem’s hypotheses.

3 Numerical experiments

The experiments evaluate mutual information across activations, constrained USV-layers, and synthetic classification and regression settings. Results show that compression behavior can vary with activation, depth, and initialization, while generalization does not track it simply.

  • 3.1 Activations comparison in terms of mutual informations: Mutual information increases with weight scale for linear and ReLU activations, whereas hardtanh saturation causes information loss at large weights.The comparison uses stochastic two-layer networks with additive Gaussian noise inside the non-linearity.
  • 3.2 Learning ability of USV-layers: USV-layers outperform random projections but remain below unconstrained fully connected layers on MNIST classification.USV-layers decompose weights as UℓSℓVℓ, with only diagonal Sℓ learned.
  • 3.2 Learning ability of USV-layers: Adding six USV-layers improves training accuracy but not testing accuracy to the same extent, producing training-set overfitting.The testing losses grow toward the end of learning in the six-layer experiment.
  • 3.3 Additional learning experiments on synthetic data: A 10-hidden-layer USV student reaches best training and testing MSE losses of 0.015 and 0.018, with slightly stronger compression in deeper layers.The regression task uses ReLU activations and a final unconstrained linear layer.
  • 3.3 Additional learning experiments on synthetic data: Changing weight initialization can flip mutual-information evolution while final classification performance remains similar.Mutual informations may decrease throughout learning, increase throughout learning, or follow a hybrid trajectory.

A Proofs of some technical propositions

This appendix develops technical ingredients for the paper’s asymptotic information-theoretic analysis. It proves properties of the replica quantities using the Nishimori identity, Gaussian calculations, and regularity arguments.

  • A.1 The Nishimori identity: The Nishimori identity equates expectations using an independent conditional sample with expectations replacing the final sample by the original jointly distributed variable.The identity follows directly from Bayes’ formula and equality in law.
  • A.2 Technical propositions: Under the stated hypotheses, the first-layer order parameter converges to an expectation involving a Gaussian variable and an independent parameter.The result is immediate when the input variance is zero.
  • A.2 Technical propositions: The function Ψϕ1 is convex, non-decreasing, and Lipschitz under the imposed activation assumptions.Gaussian integration by parts and the Nishimori property establish monotonicity and convexity, while the definition yields Lipschitz control.
  • A.2 Technical propositions: The appendix establishes these properties through differentiability, Gaussian integration by parts, and concentration-related bounds.The arguments rely on the assumptions imposed on the activation functions.

B.1 Computing the derivative: proof of(132)

This section computes the derivative of the interpolating free entropy. The derivation repeatedly applies Gaussian integration by parts and the Nishimori identity to simplify the derivative terms.

  • B.1 Computing the derivative: proof of(132): The derivative of the interpolating free entropy is expressed for 0 < t < 1 using the interpolating Hamiltonian.The partition function and auxiliary variables define the quantities entering the derivative.
  • B.1 Computing the derivative: proof of(132): Gaussian integration by parts is applied successively with respect to W2, V, U, and Gaussian auxiliary variables.These operations transform expectations involving derivatives of the Hamiltonian and output channel.
  • B.1 Computing the derivative: proof of(132): The Nishimori identity removes the term T2, completing the algebra needed to obtain equation (132).The cancellation combines the identities derived in the preceding steps.

B.2 Proof that An,ϵ(t) vanishes uniformly as n0 →+∞

This section proves that the interpolation error term vanishes uniformly in the asymptotic limit. The proof establishes concentration of the normalized free entropy through sequential variance bounds.

  • B.2 Proof that An,ϵ(t) vanishes uniformly as n0 →+∞: The proof aims to show that An,ϵ(t) converges uniformly to zero as the layer dimensions grow with fixed aspect ratios.The argument controls the term uniformly over t and ϵ.
  • B.2 Proof that An,ϵ(t) vanishes uniformly as n0 →+∞: The normalized free entropy concentrates because its variance is bounded by C(ϕ1,ϕ2,α1,α2,S)/n0.This concentration, together with the free-entropy representation, implies the theorem.
  • B.2 Proof that An,ϵ(t) vanishes uniformly as n0 →+∞: Gaussian Poincaré and bounded-difference inequalities provide the variance bounds for the successive concentration steps.The appendix applies these inequalities conditionally to different subsets of random variables.
  • B.2 Proof that An,ϵ(t) vanishes uniformly as n0 →+∞: The proof first controls Gaussian variables and A2, then establishes concentration with respect to W1 and X1.The order of concentration arguments is explicitly used in the proof.

C.1.2 Concentration with respect to X1 and W1

This section establishes concentration bounds for the interpolating quantities with respect to X1 and the weight matrices W1 and W2. The proof uses bounded-difference arguments together with derivative bounds obtained through Gaussian integration by parts.

  • Concentration with respect to X1 and W1: Under (H1), (H2), and (H3), Lemmas 7, 9, and 10 provide positive constants controlling the interpolating quantities uniformly for t ∈ [0, 1].The constants depend on the activation functions, aspect ratios, and singular-value bound S.
  • Concentration with respect to X1 and W1: The proof treats ln b Zn,t,ϵ/n0 conditionally on X1, W1, and X0, then establishes bounded differences when one coordinate of X1 changes.The interpolation between vectors c and c(i) reduces coordinate sensitivity to a derivative bound for ψ(s).
  • Concentration with respect to X1 and W1: Integration by parts with respect to entries of W2 bounds the relevant conditional expectations by constants or terms vanishing as n1 grows.The displayed argument separates diagonal, off-diagonal, and additional terms, with bounds scaling as 1/√n1 or 1/n0.
  • Concentration with respect to X1 and W1: The resulting almost-sure bound, followed by expectation, completes the concentration argument for the quantities considered.The section repeatedly invokes the established lemmas and bounded-difference propositions to conclude the proof.
  • Concentration with respect to X1 and W1: The derivative with respect to each W1 entry is bounded almost surely by C2(ϕ1,ϕ2,α1,α2,S)/n1^3/2.This bound is obtained after controlling the derivative terms and applying a bounded-difference proposition.

C.1.3 Proof of Theorem 2

The proof of Theorem 2 follows directly from the bound established using Lemmas 3 and 7. The resulting statement holds almost surely, and the theorem follows from the cited prior identity.

  • Proof of Theorem 2: Lemmas 3 and 7 directly provide the bound required for the proof.The section states that this bound is the immediate input to the theorem.
  • Proof of Theorem 2: The bound implies Theorem 2 through identity (169), completing the proof.The implication is stated explicitly after the bound.

C.2 Concentration of the overlap

This section proves concentration of the overlap by combining posterior concentration with concentration over realizations of the quenched variables. The argument relies on regularity, convexity, derivative bounds, and concentration of the free entropy.

  • C.2 Concentration of the overlap: The section concludes that Proposition 3 follows from the concentration of L on its expectation.This conclusion is stated after the concentration argument for L.
  • C.2 Concentration of the overlap: Under (H1), (H2), and (H3), regular parameters yield concentration results for L and its Gibbs average when n0 is sufficiently large.The relevant statements are given in Proposition 13 and Lemma 12.
  • C.2 Concentration of the overlap: The proof decomposes overlap concentration into posterior, or “thermal,” fluctuations and fluctuations over quenched-variable realizations.The two parts correspond to concentration of L around its Gibbs average and concentration of that average.
  • C.2 Concentration of the overlap: Regularity makes R a C1-diffeomorphism with Jacobian JR(ϵ) ≥ 1, enabling the integration argument over the parameter set Bn0.The map is defined from (ϵ1, ϵ2) to (R1(t,ϵ), R2(t,ϵ)).
  • C.2 Concentration of the overlap: Adding a second term makes ˜F(R1) convex, while derivative bounds and Jensen’s inequality control the remaining deviations.The proof also uses the Nishimori identity and Gaussian integration by parts for intermediate equalities.
Loading 1805.09785v2…