PaperlayerView exact revision on arXiv

Source-linked AI summary

Information Dropout: Learning Optimal Representations Through Noisy Computation

Alessandro Achille, Stefano Soatto

arXiv:1611.01353v3stat.MLcs.LGstat.CO

TL;DR

The paper addresses the gap between cross-entropy training and the fuller properties of optimal representations. It adds an Information Bottleneck-inspired regularizer implemented through adaptive multiplicative noise, with Information Dropout as a generalization of dropout. The approach connects representation learning with variational autoencoding and disentanglement, and is reported to improve generalization especially for limited-capacity models.

  • Problem

    Cross-entropy enforces sufficiency but does not explicitly enforce minimality and invariance to nuisance factors in learned representations.

  • Method

    The paper adds an Information Bottleneck-inspired regularizer and minimizes it with Information Dropout, which selectively injects data-adaptive multiplicative noise into activations.

  • Results

    Information Dropout achieves comparable or better generalization than binary dropout, especially on smaller models, while adapting noise to network structure and test samples.

  • Takeaways & Limitations

    The framework links optimal representations, dropout, variational inference, variational autoencoding, and disentanglement through information-theoretic principles.

  • Takeaways & Limitations

    Because the method restricts the family of distributions, the resulting representation is not guaranteed to be optimal; iteration yields incrementally improved approximations.

Abstract

from arXiv · show

The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. We show that our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity. When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of disentangled representations simply by enforcing a factorized prior, a fact that has been observed empirically in recent work. Our experiments validate the theoretical intuitions behind our method, and we find that information dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.

1 INTRODUCTION

The paper defines optimal representations through sufficiency, minimality, and invariance, then proposes regularized noisy computation to approximate them. Information Dropout connects these ideas to dropout, variational autoencoding, and disentanglement.

  • Optimal representations are defined by sufficiency, minimality, and invariance to nuisance factors.
  • Cross-entropy encourages sufficient representations but does not explicitly enforce minimality and invariance.
  • The proposed loss adds a regularizer related to multiplicative activation noise, whose Bernoulli special case recovers standard dropout.
  • Information Dropout adapts noise to data and improves efficiency over regular dropout for limited-capacity networks.
  • For reconstruction, the procedure generalizes the Variational Autoencoder and links information-theoretic and Bayesian representations.
  • A factorized prior promotes disentanglement by minimizing total correlation among representation components.

2 PRELIMINARIES

The preliminaries frame representation learning as retaining task-relevant variation while discarding nuisance factors that can produce spurious correlations. The paper addresses this with an Information Bottleneck-inspired objective and adaptive multiplicative noise.

  • High-dimensional inputs contain nuisance variation that can induce spurious training correlations and poor generalization.
  • Common training may preserve easily ignorable nuisance dependencies even in deep network top layers.
  • Information Dropout minimizes a modified Information Bottleneck cost by selectively injecting multiplicative noise into layer activations.
  • The method encourages increasingly disentangled and nuisance-insensitive representations while controlling information flow.
  • Information Dropout improves generalization and outperforms binary dropout on smaller models by adapting noise to network structure and test samples.
  • The framework unifies several dropout methods and connects Information Dropout to Variational Dropout and variational autoencoders.

3 RELATED WORK

The paper links dropout methods to optimal representations through the Information Bottleneck principle, contrasting task sufficiency with nuisance invariance and disentanglement approaches. It proposes noise injection as an information-theoretic route to improved nuisance insensitivity and disentangling.

  • The paper links dropout methods for preventing overfitting with the study of optimal representations through the Information Bottleneck principle.
  • The related-work discussion positions Information Dropout as information-theoretically derived, unlike Variational Dropout's Bayesian derivation.
  • Unlike approaches seeking equivalence under group nuisances, this work targets representations sufficient for the task.
  • Injecting noise into network layers yields nondeterministic representations that can simplify analysis and lead to disentangling and improved nuisance insensitivity.

4 OPTIMAL REPRESENTATIONS AND THE INFORMATION BOTTLENECK LOSS

The paper formulates optimal representations through the Information Bottleneck, balancing task sufficiency against representation minimality. It adds a KL regularizer to cross-entropy and efficiently approximates the resulting objective with Information Dropout, while allowing stochastic representations.

  • An optimal representation preserves task-relevant information while discarding variability irrelevant to the task.
  • The Information Bottleneck Lagrangian balances sufficiency, measured by task performance, against minimality, measured by representation complexity.
  • Restricting representations to those produced by injected layer noise makes the Information Bottleneck objective efficiently approximable through Information Dropout.
  • The empirical objective combines average cross-entropy with a KL regularizer that penalizes information transfer from x to z.
  • Stochastic representations can achieve a lower Information Bottleneck objective than deterministic representations for some tasks and β values.
  • Minimizing the Information Bottleneck objective produces representations that become increasingly sufficient in the sense of approaching the task-relevant Markov structure.

5 DISENTANGLEMENT

The paper formalizes disentanglement through total correlation and shows that an independent prior can make this objective tractable. Under this factorized assumption, minimizing the standard Information Bottleneck objective is equivalent to enforcing disentanglement.

  • Disentanglement is quantified by total correlation, which is zero exactly when representation components are mutually independent.
  • Adding total correlation as a penalty augments the Information Bottleneck objective with an explicit independence preference.
  • The augmented objective is generally intractable because the representation marginal pθ(z) is difficult to compute.
  • When γ = β, introducing an auxiliary variable simplifies the augmented minimization and makes it tractable.
  • Assuming a factorized prior makes minimizing the standard Information Bottleneck objective equivalent to enforcing disentanglement of hidden factors.
  • The paper notes that factorized models are often adopted for simplicity and that their disentangling effect had previously been observed empirically.

6 INFORMATION DROPOUT

Information Dropout represents activations as deterministic features multiplied component-wise by input-dependent unit-mean noise. Its regularization uses activation-matched priors and penalizes low noise variance, controlling information flow.

  • Noise model: Information Dropout multiplies f(x) component-wise by noise whose distribution and parameters depend on the input x.The noise has unit mean and variance, and is sampled from a parametric distribution.
  • Relation to dropout: If the noise is rescaled Bernoulli noise with mean 1, Information Dropout becomes classic binary dropout.The element-wise product is the mechanism connecting the general layer to binary dropout.
  • Activation-matched priors: For ReLU activations, the proposed prior is a scale-invariant log-uniform distribution with a point mass at zero.The prior is q(z) = q0δ0(z) + c/z, reflecting that ReLU activations are frequently zero.
  • Activation-matched priors: For Softplus activations, the paper proposes a log-normal prior because it fits the expected activation distribution.Figure 1 compares these priors with empirical post-noise activation distributions from All-CNN-32 trained on CIFAR-10.
  • Regularization: The regularized loss combines a sampled cross-entropy term with a penalty for low noise variance, which corresponds to transmitting more information.The loss can be optimized with stochastic gradient descent and reparameterization.

TION DROPOUT

Information Dropout connects information-theoretic representation learning with variational inference. In the reconstruction setting, choosing β = 1 yields the VAE objective exactly, while larger β values are associated with more disentangled representations.

  • Variational autoencoders: A variational autoencoder reconstructs x through a latent variable z generated by an unknown process pθ(x|z).Its training objective is the negative variational lower bound on the data marginal log-likelihood.
  • Connection to VAEs: When reconstruction is the task, the Information Bottleneck loss reduces to the corresponding variational autoencoder loss.This is the special case y = x.
  • Connection to VAEs: Setting β = 1 makes Information Dropout and the VAE use exactly the same loss, with their representations serving the same reconstruction role.The paper identifies the Information Dropout representation with the VAE latent variable in this case.
  • Disentanglement: Higher β values are experimentally associated with increasingly disentangled latent representations in related VAE work.The paper relates this observation to its own factorized-prior analysis and experiments.

8 EXPERIMENTS

Experiments test whether Information Dropout suppresses nuisance information, improves limited-capacity networks, adapts information flow, and promotes disentanglement. Across these settings, the reported results support the proposed information-theoretic behavior, while excessive regularization can damage task-relevant information.

  • Experimental goals: The experiments target reduced nuisance dependence, more disentangled representations, and better use of limited-capacity architectures through adaptive noise.These goals are evaluated across Cluttered MNIST, Occluded CIFAR, MNIST, and CIFAR-10.
  • Nuisance suppression: Higher β makes Cluttered MNIST representations discard distractors once receptive fields can identify them as nuisances.The resulting representation is described as more robust to nuisances and better generalizing.
  • Invariance to occlusion: On Occluded CIFAR, increasing β makes nuisance classification harder while improving the main CIFAR classification task.The same learned representation is evaluated with noisy and deterministic representations for the main task.
  • Limited capacity: Information Dropout is comparable to or better than binary dropout, especially for smaller MNIST and CIFAR-10 networks.The paper attributes this pattern to adapting noise to both the data and network size so relevant information can continue flowing.
  • Disentanglement: Increasing β reduces total correlation and test error until excessive β prevents sufficient information flow and sharply increases error.This trade-off is measured in the final All-CNN-32 layer with 25% of the filters.
  • VAE validation: Information Dropout achieves similar testing variational lower-bound performance to a Gaussian latent-variable implementation of a simple VAE.The comparison varies latent size and encoder/decoder size together through k.
  • Information allocation: The total transmitted information changes little with filter count, while fewer filters require each unit to transmit more information.The upper dropout layer carries more task-relevant information per unit than the lower layer.

9 DISCUSSION

The paper connects Information Dropout to information-theoretic representation learning, variational inference, and disentanglement. It interprets multiplicative noise as both a regularizer and an optimization-related operation, with activation-specific cost analyses.

  • Information Dropout connects the Information Bottleneck principle with dropout, disentanglement, and variational autoencoding.
  • Bernoulli multiplicative noise recovers standard dropout, while adaptive noise improves exploitation of limited capacity and adapts to data.
  • Disentanglement is quantified by total correlation and promoted by enforcing independence among representation components through a factorized prior.
  • Information Dropout can bias representations toward compatibility with a Markov chain generative model, complementing architectural constraints such as convolutions.
  • Multiplicative activation noise can be viewed as a loss minimizer choice, a cost regularizer, or an optimization procedure.
  • The appendix derives Information Dropout costs for ReLU and Softplus activations using specified noise distributions and KL-divergence invariance under reparameterization.

APPENDIX B DISENTANGLEMENT

This appendix shows that optimizing over a factorized prior decomposes the regularizer into mutual information and total correlation terms. The optimal factorized prior is the product of the representation marginals.

  • The appendix studies a minimization problem that is difficult because the joint distribution p(z) is unavailable, and reformulates it as a simpler optimization.
  • For discrete z and a factorized prior q(z)=Q_i q_i(z_i), the regularizer decomposes into mutual information I_p(z; x) and total correlation TC_p(z).
  • The optimal factorized prior sets each q_i(z_i) equal to the corresponding marginal p(z_i), making the prior the product of marginals.

APPENDIX C ADDITIONAL PLOTS

Figure 7 plots total KL divergence across spatial locations for the first three dropout layers on two additional input samples.

  • Figure 7 shows total KL divergence evaluated at each spatial location.
  • The plot covers the first three dropout layers.
  • Results are shown for two additional input samples and linked to the detailed description in Section 8 and Figure 2.
Loading 1611.01353v3…