Emergence of Invariance and Disentanglement in Deep Representations

Alessandro Achille; Stefano Soatto

Source-linked AI summary

Emergence of Invariance and Disentanglement in Deep Representations

Alessandro Achille, Stefano Soatto

arXiv:1706.01350v3Jun 5, 2017cs.LGcs.AIstat.ML

TL;DR

The paper studies why over-parameterized deep networks can generalize and how their learned representations become invariant and disentangled. It establishes information-theoretic connections among representation minimality, weight complexity, regularization, and generalization, and validates predicted random-label transitions experimentally.

Problem
Over-parameterized deep networks can fit random labels yet generalize on real labels, while effective representations should be invariant to nuisance factors and disentangle data-generating factors.
Method
The paper derives information-theoretic bounds linking weights, activations, invariance, and disentanglement, and regularizes cross-entropy through PAC-Bayes or an information bottleneck on the weights.
Results
The analysis shows that sufficient representations are invariant exactly when minimal, weight information bounds representation information and disentanglement, and weight-based complexity predicts random-label underfitting and overfitting transitions.
Takeaways & Limitations
Information in the weights provides a complexity measure connecting regularization, invariant and disentangled representations, loss geometry, and generalization error.
Takeaways & Limitations
The work does not address transfer learning, and its multilayer information bound need not be tight because the bottleneck may transmit different amounts of information across layers.

Abstract

from arXiv · show

Using established principles from Statistics and Information Theory, we show that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation, and that stacking layers and injecting noise during training naturally bias the network towards learning invariant representations. We then decompose the cross-entropy loss used during training and highlight the presence of an inherent overfitting term. We propose regularizing the loss by bounding such a term in two equivalent ways: One with a Kullbach-Leibler term, which relates to a PAC-Bayes perspective; the other using the information in the weights as a measure of complexity of a learned model, yielding a novel Information Bottleneck for the weights. Finally, we show that invariance and independence of the components of the representation learned by the network are bounded above and below by the information in the weights, and therefore are implicitly optimized during training. The theory enables us to quantify and predict sharp phase transitions between underfitting and overfitting of random labels when using our regularized loss, which we verify in experiments, and sheds light on the relation between the geometry of the loss function, invariance properties of the learned representation, and generalization error.

1. Introduction

The paper connects representation learning and optimization through information theory, explaining how minimal representations, weight information, and regularization relate to invariance, disentanglement, and generalization.

Deep networks can generalize despite severe over-parameterization, but the roles of loss geometry and implicit SGD regularization remain unclear.
A sufficient representation is invariant to nuisances if and only if it is minimal, even when its dimension is not small.
Information in activations and their total correlation are tightly bounded by information that the weights contain about the dataset.
Explicit or implicit regularization can control weight information, biasing networks toward invariant and disentangled representations.
Experiments show a sharp, theoretically predicted transition between overfitting and underfitting random labels when complexity is measured by weight information rather than parameter count.
The paper relates representation invariance to optimization-residual geometry and flat minima, while noting that PAC-Bayes offers an alternative derivation and test-error bound.

2. Preliminaries

The preliminaries define the information-theoretic language used to characterize representations, nuisances, sufficiency, minimality, invariance, and disentanglement.

The paper uses entropy, conditional entropy, mutual information, KL divergence, cross-entropy, and total correlation as core quantities.
Total correlation is zero exactly when representation components are independent, which the paper calls disentanglement.
The Data Processing Inequality states that a representation cannot contain more information about a target than the original data.
A representation is sufficient when it preserves all information about the task and minimal when it minimizes information about the input among sufficient representations.
The Information Bottleneck Lagrangian trades off task sufficiency against representation minimality through β.
A nuisance affects the observed input but is independent of the task, while invariance means the representation is independent of that nuisance.

3. Properties of optimal representations

The paper shows that sufficiency and minimality are enough to obtain invariance and disentanglement in deep networks, with noisy bottlenecks and stacking strengthening invariance.

For a sufficient representation, maximal insensitivity to nuisances is equivalent to minimality, up to a residual ε bounded by H(y|x).The residual vanishes when y is a deterministic function of x.
Reducing information about x while retaining task-sufficient information provides a route to learning invariance to complex nuisances.
A sufficient bottlenecked layer is more invariant than its predecessor when the bottleneck reduces mutual information with the input.Dimensionality reduction and noisy channels such as dropout can create such bottlenecks.
When the final layer is sufficient, stacking layers makes it more insensitive to nuisances than all preceding layers.
Minimizing the Information Bottleneck Lagrangian for a deterministic invariant also maximizes Actionable Information.

4. Learning minimal weights

The paper reframes overfitting control as minimizing information stored in network weights, using regularized objectives connected to PAC-Bayes and the Information Bottleneck. This weight complexity measure is linked to generalization, implicit SGD regularization, and invariant, disentangled representations.

Cross-entropy decomposition: Cross-entropy decomposes into intrinsic error, captured information about data-generating parameters, efficiency, and a negative memorization term that can drive overfitting.Without regularization, minimizing the loss can maximize label information unrelated to the underlying distribution.
Regularizing overfitting: The paper proposes neutralizing the memorization term by upper-bounding it through either a PAC-Bayes KL term or an Information Bottleneck objective.The two routes yield equivalent conclusions under different premises and assumptions.
Weight Information Bottleneck: The weight Information Bottleneck is a novel IB Lagrangian applied to network weights rather than activations, and it generalizes variational-learning formulations.For β = 1, the objective reduces to the variational lower-bound of the marginal log-likelihood.
Information as complexity: Information in the weights is advocated as effective network complexity because it recovers a bias-variance trade-off and predicts random-label underfitting and overfitting more precisely than parameter count.The experiments report no clear relationship between parameter number and overfitting, while information complexity predicts the transition behavior.
Representations and optimization: Under the paper’s modeling assumptions, low-information weights realize invariant and disentangled representations, with explicit IB regularization and implicit SGD regularization both favoring these properties.The paper also connects this weight information to flat minima and representation minimality.

5. Duality of the Bottleneck

The paper links information in network weights to representation minimality, invariance, and disentanglement, deriving bounds for one-layer and multilayer networks. Lower weight information improves these properties, although multilayer bounds need not be tight.

The central proposition connects information in the weights with flatness of local minima, representation minimality, and disentanglement.
Under Gaussian assumptions, I(z; x) + TC(z) decreases monotonically as weight variances α_i increase.The variance is defined by ˜α_i = exp(α_i)−1.
For one layer with uncorrelated inputs and bounded kurtosis, I(x; z) + TC(z) is tightly bounded by weight information and increases strictly with it.The result assumes z = Wx and a log-normal multiplicative-noise model for W.
Reducing information in the weights improves representation minimality, invariance to nuisance factors, and disentanglement, including under implicit regularization such as SGD.The multilayer extension uses the Markov property of network layers.
The multilayer bound need not be tight because a weight-information bottleneck does not determine how much information about x passes through each layer.Initial layers may discard more information, while final layers may transmit most of it.
The framework assumes sufficiency rather than identical training and test distributions, and sufficiency may be unattainable when the test distribution is entirely different.

6. Connection with PAC-Bayes bounds

The paper derives its weight Information Bottleneck regularizer from PAC-Bayes bounds, showing that the two formulations coincide up to constants. The practical factorized-prior version is tractable, but the modeled log-uniform prior makes the resulting test-error bound vacuous.

A PAC-Bayes test-error bound yields the same regularized loss as the weight Information Bottleneck without approximation.The bound applies for fixed λ > 1/2, a prior p(w), and weight distribution q(w|D).
Choosing the prior p(w) = q(w) gives the sharpest PAC-Bayes bound and reduces it, up to a multiplicative constant, to the weight Information Bottleneck Lagrangian.
Because the joint weight marginal is intractable, the sharpest tractable factorized prior uses the product of the marginal weight distributions and recovers the practical loss.
The modeled improper log-uniform marginal is non-informative and closely matches real weight marginals, but its additive-constant ambiguity makes the test-error bound vacuous.
Gaussian PAC-Bayes priors and posteriors have been used elsewhere to obtain non-vacuous generalization bounds for deep networks.

7. Empirical validation

Experiments validate weight information as a complexity measure: it predicts transitions between overfitting and underfitting, tracks the bias-variance trade-off, and correlates with nuisance invariance while preserving sufficiency.

7.1 Transition from overfitting to underfitting: For completely random labels, β > 1 theoretically prevents memorization, whereas β < 1 makes memorization favorable in the regularized objective.The transition follows because each random label requires additional information in the weights.
7.1 Transition from overfitting to underfitting: Experiments across dataset sizes show a phase transition between overfitting and underfitting near the critical value β = 1.
7.1 Transition from overfitting to underfitting: Increasing label randomness requires more information in the weights, reaching about 2.30 nats/sample for entirely random 10-class labels.
7.1 Transition from overfitting to underfitting: For real labels, a substantial interval with β > 1 permits fitting without excessive overfitting once βN ∝ I(w; D) exceeds H(θ).
7.1 Transition from overfitting to underfitting: Different architectures exhibit a similar phase transition for random CIFAR-10 labels, with ResNets showing a particularly sharp transition near β = 1.
7.2 Bias-variance trade-off: Controlling weight information recovers the bias-variance trade-off: too little information underfits, while excessive information supports memorization.
7.3 Nuisance invariance: Reducing weight information lowers I(z; n) while retaining sufficiency, with the MNIST nuisance experiment showing increasing representation insensitivity.The experiment estimates I(z; n) using a discriminator distinguishing samples from p(z|n) and p(z).

8. Discussion and conclusion

The discussion connects low information in weights to desirable representations and relates the theory to stochastic representations, PAC-Bayes, and scope limitations. The work focuses on task-specific optimal representations rather than transfer learning.

Reducing information stored in weights about the dataset yields desirable properties in learned internal representations.
The theory uses intrinsically stochastic representations, while deterministic representations can be converted into stochastic ones depending on minimum flatness.
The Information Bottleneck principle is not required, because PAC-Bayes can re-derive the results and provide additional test-error bounds.
The work addresses task-specific optimal representations but does not address transfer learning, which is left for future investigation.

A.1 Random labels

The random-label experiments evaluate the information-regularized loss across architectures and training settings. They use CIFAR-10 with random labels and examine how information changes with corruption, architecture, and regularization.

The experiments train a Small AlexNet on centrally cropped CIFAR-10 images with completely random labels and no additional data augmentation.
Information content is measured across label-corruption levels with β = 0.1, N = 30000, and η = 0.005.
The phase transition is tested on Small AlexNet, AllCNN, and ResNet using N = 10000 random labels and β values from 10^-2 to 10^2.
A separate AllCNN experiment varies filter count for standard training and varies β for the information-bottleneck loss.
The appendix specifies the three architectures and identifies the AllCNN final representation as the last “conv 192” activations.

A.4 Visualizing the representation

The representation-visualization experiment reconstructs images from stochastic representations to reveal which information is retained or discarded. Increasing β produces reconstructions that preserve task-relevant facial attributes while regenerating nuisance details.

The method learns a distribution q(x̂|z) that generates images likely to have representation z.
The reconstructed images preserve hair color, sex, and expression, while background, hairstyle, and other nuisances are generated anew.
The reconstruction objective combines a representation reconstruction term with a divergence-based distance from the image prior, estimated using a GAN discriminator.
Increasing β reduces I(w; D) and produces more random backgrounds and hairstyles while retaining facial features in CelebA reconstructions.
Training uses a CelebA classifier for 40 binary attributes followed by a GAN that reconstructs the input from its representation.

Appendix C. Proofs of theorems

The appendix proves information-theoretic relationships among task nuisances, representations, weights, and loss geometry. It establishes invariance–minimality equivalence and bounds representation information and total correlation using noisy weights under stated assumptions.

Task-nuisance decomposition: A task-nuisance decomposition represents x as a deterministic function of task variable y and nuisance n independent of y.
Invariance and minimality: A sufficient representation is invariant to nuisances if and only if it is minimal under the stated conditions.
Invariance and minimality: For sufficient z, nuisance information satisfies I(z; n) = I(z; x) − I(x; y) − ϵ, with ϵ bounded by H(y|x).
Information decomposition: The cross-entropy decomposition includes data uncertainty, parameter uncertainty, inference mismatch, and a conditional information term involving the weights.
Information in the weights: The information in weights is upper-bounded through a KL-based variational formulation with an arbitrary constant arising from the improper prior.
Flat minima: Flat minima have low information under the optimal noisy posterior centered at the local minimum.
Information bounds: For noisy linear layers, I(z; x) + TC(z) decreases with weight variance and is tightly bounded by weight information under Gaussian and distributional assumptions.
Multilayer case: The multilayer bound follows from the Markov structure of layers, the data processing inequality, and the one-layer information bound.

Appendix D. Q&A

The Q&A clarifies how weight information, flat minima, nuisance definitions, and minimal representations relate to generalization and invariance. It also qualifies the PAC-Bayes construction and explains why weight information need not be computed explicitly.

Optimization and generalization: Flat minima can limit information in the weights, which promotes good generalization without requiring the Hessian’s nuclear norm to be computed.The nuclear norm is treated as a proxy for flatness; the Hessian trace could instead be approximated in linear time.
Nuisance assumptions: The condition I(y; n) = 0 is preferred to I(y; n|x) = 0 because labels determined by x would make the conditional condition hold for every nuisance.Under the weaker condition, every variable could be classified as a nuisance for the task.
PAC-Bayes qualification: The log-uniform-prior PAC-Bayes formulation gives a vacuous bound because the prior is improper, although it matches empirical network behavior and simplifies the analysis.Gaussian priors and posteriors are mentioned as an alternative, while obtaining a good PAC-Bayes bound is outside the paper’s scope.
Representation properties: The paper argues that invariance to nuisance variation follows from minimality of a task representation, while noting that compression and invariance are not generally identical.The authors state that a proof beyond special tasks or small perturbations was not previously known to them.
Optimization and generalization: The weight information need not be computed explicitly: SGD can implicitly approach the solution minimizing the weights’ Information Bottleneck, and stochastic training yields a weight posterior.The discussion includes annealed learning rates, Information Dropout, and Dropout as relevant cases.