PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Variational Information Bottleneck

Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, Kevin Murphy

arXiv:1612.00410v7cs.LGcs.IT

TL;DR

The paper addresses how to optimize the information bottleneck for flexible deep-network representations without restrictive discrete or Gaussian assumptions. It introduces VIB, a variational lower-bound method trained with reparameterized stochastic gradients, and reports improved generalization and adversarial robustness, including better adversarial accuracy than deterministic models whose accuracy is 0%.

  • Problem

    The information bottleneck is difficult to compute generally, while discrete and jointly Gaussian formulations severely constrain learnable models.

  • Method

    VIB constructs a variational lower bound on the information-bottleneck objective, parameterizes distributions with neural networks, and uses the reparameterization trick for stochastic-gradient training.

  • Results

    VIB models show improved generalization and adversarial robustness, with significantly better adversarial-example accuracy than deterministic models whose accuracy is 0%.

  • Takeaways & Limitations

    Stochastic information-bottleneck representations can ignore input details while retaining predictive information and provide robustness benefits over deterministic models.

  • Takeaways & Limitations

    The paper leaves richer marginal approximations, real-image testing, and other extensions for future work.

Abstract

from arXiv · show

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

1 INTRODUCTION

The paper formulates representation learning as an information-bottleneck tradeoff: retain information about targets while compressing information about inputs. VIB uses variational inference and neural parameterizations to make this objective trainable on high-dimensional continuous data, with reported benefits for generalization and adversarial robustness.

  • Information bottleneck objective: The information bottleneck seeks representations that are predictive of Y while remaining compressed with respect to X.The tradeoff is controlled by β, which weights compression relative to informativeness about the target.
  • Motivation: Mutual-information optimization is computationally challenging except under discrete or jointly Gaussian assumptions, which constrain learnable models.These restrictions limit applicability to high-dimensional continuous data such as images.
  • VIB contribution: The paper constructs a variational lower bound on the IB objective and calls the resulting method VIB.The bound enables stochastic-gradient optimization with neural-network parameterizations.
  • VIB contribution: The reparameterization trick provides unbiased Monte Carlo gradient estimates, enabling stochastic-gradient training of deep neural networks.This approach supports high-dimensional, continuous inputs while avoiding the earlier discrete or Gaussian restrictions.
  • Reported benefits: VIB-trained stochastic networks are reported to resist overfitting and adversarial inputs better than deterministic models trained with penalized maximum likelihood.The proposed explanation is that stochastic encodings map each image to a distribution rather than a unique latent representation.

2 RELATED WORK

The related work positions VIB among information-theoretic, variational, supervised-regularization, generative, and fairness-oriented approaches. Its distinctions include deep-neural-network parameterization, stochastic-gradient scalability, discriminative modeling, and automatic discovery of irrelevant input content.

  • Information bottleneck: Earlier deep-network information-bottleneck work lacked experiments because Blahut–Arimoto optimization was infeasible for deep neural networks.This motivates a computationally tractable variational approach.
  • Variational approaches: Chalk et al. independently developed the same variational lower bound but applied it to sparse coding with kernel mappings and batch variational EM.The paper instead applies the bound to deep neural networks and large datasets using stochastic gradient descent.
  • Supervised regularization: Confidence penalty regularizes deterministic networks by combining cross-entropy with a penalty for low-entropy predictive distributions.The cited work reports that confidence penalty outperforms label smoothing.
  • Generative models: Variational autoencoders are presented as a special case of an unsupervised VIB with β fixed at 1.0.Other cited work explored VAE objectives with different β values from a different perspective.
  • Fair representation learning: Unlike the variational fair autoencoder, VIB can discover irrelevant input components automatically rather than requiring users to specify sensitive aspects.Both approaches share the idea of ignoring parts of the input.

3 METHOD

The method replaces intractable information terms with variational bounds, using a neural decoder for target prediction and a variational marginal for compression. Reparameterization then makes the resulting stochastic objective amenable to unbiased backpropagation.

  • Model assumptions: The method assumes the Markov chain Y ↔ X ↔ Z, so the stochastic representation Z depends on X rather than directly on labels Y.This assumption also supports an unsupervised representation-learning extension.
  • Variational bounds: A variational decoder q(y|z) lower-bounds the target-information term by replacing the intractable conditional p(y|z).The decoder is parameterized as another neural network.
  • Variational bounds: The target-information bound becomes an expectation of log q(y|z) over the data distribution and stochastic encoder.It requires samples from the joint data distribution and encoder, plus a tractable variational decoder.
  • Variational bounds: A variational marginal r(z) upper-bounds the encoder–input mutual information through a KL-based compression term.This avoids directly computing the generally difficult marginal p(z).
  • Optimization: For Gaussian encoders, reparameterization writes z=f(x,ϵ) with parameter-independent noise, allowing gradients through stochastic codes.The encoder network outputs the latent mean and covariance, and the resulting objective can be optimized by backpropagation.

4 EXPERIMENTAL RESULTS

The experiments compare standard deterministic networks with stochastic neural networks trained by optimizing the VIB objective.

  • Experimental setup: The experiments compare deterministic networks with stochastic neural networks trained using the VIB objective.The comparison examines the behavior of the two model types across the paper’s experiments.

4.1 BEHAVIOR ON MNIST

On permutation-invariant MNIST, VIB uses a stochastic bottleneck whose behavior depends strongly on β: moderate compression improves regularization, while excessive compression destroys class information. The experiments compare this approach with deterministic baselines and visualize how bottleneck uncertainty affects classification.

  • Setup: The MNIST baseline is a fully connected 784-1024-1024-10 MLP with 1.38% error, while the VIB encoder uses a Gaussian K-dimensional bottleneck.The stochastic encoder predicts Gaussian means and scales; its decoder maps the K-dimensional latent code to class logits.
  • Higher-dimensional embedding: VIB improves the baseline to 1.13% error, compared with 1.17% for confidence-penalty regularization, under the reported MNIST setup.The comparison uses the same network architecture for the regularized models.
  • Higher-dimensional embedding: For β approaching zero, the encoder variance tends toward zero, making the VIB model nearly deterministic and reducing its regularization benefit.The experiments also include an explicitly deterministic model obtained by using the predicted mean as the hidden encoding.
  • Higher-dimensional embedding: When β exceeds approximately 10^-2, the error rate rises sharply because the bottleneck carries less than log2(10) bits and cannot represent all 10 classes.At intermediate β values, Monte Carlo averaging helps, whereas the authors have not extensively studied selecting β without a validation set.
  • Higher-dimensional embedding: The best results occur for β between 10^-3 and 10^-2, where the stochastic encoding retains approximately 10 to 100 bits of information about the images.The reported information measure is an upper bound based on the relative entropy to a fixed isotropic unit Gaussian prior.
  • Two dimensional embedding: In a two-dimensional bottleneck, increasing β enlarges embedding covariances until classes overlap and eventually the encoding collapses, although intermediate β can retain good predictive performance.The visualization plots 95% confidence ellipses for posterior encodings of 1,000 test images, colored by true class, over classifier entropy.

4.2 BEHAVIOR ON ADVERSARIAL EXAMPLES

The paper evaluates VIB against several adversarial attacks and finds that stochastic bottleneck models become substantially more robust than deterministic and dropout baselines. Robustness is observed across MNIST attacks and motivates evaluation on ImageNet.

  • Motivation: VIB training makes models significantly more robust to adversarial examples than deterministic models trained by penalized maximum likelihood.The proposed mechanism maps each input to a distribution rather than a unique latent representation.
  • Attack types: Adversarial examples can be untargeted or targeted, with targeted attacks requiring the input to reach a specified decision region.The paper studies both attack types for the L2 optimization method.
  • MNIST attacks: VIB models often resist MNIST L2 attacks or require much larger perturbations when attacks succeed, whereas deterministic models are easily fooled by small perturbations.The experiments use the first 10 zeros from MNIST and compare deterministic, dropout, and VIB models.
  • Interpretation: VIB shows adversarial robustness even as β →0, consistent with the encoder-decoder being approximately quadratic for finite β.The paper connects this behavior to theoretical results about quadratic classifiers.

Architecture

The experiments compare deterministic, dropout, and stochastic VIB classifiers across MNIST and ImageNet adversarial settings. Figures vary the compression parameter β and show how perturbation size and classification accuracy change relative to baseline models.

  • MNIST attack examples: Figure 3 compares original MNIST zeros with adversarial examples targeting deterministic, dropout, and VIB models at different β values.Red, green, and purple backgrounds indicate successful, unsuccessful, and wrong-label outcomes, respectively.
  • Perturbation magnitude: Figure 4 plots relative L0, L2, and L∞ perturbation magnitudes against β for deterministic and dropout baselines.As β increases, L0 decreases while L2 and L∞ increase, indicating larger changes concentrated in fewer pixels.
  • FGS evaluation: Figure 5 shows VIB adversarial accuracy divided by baseline accuracy for FGS examples as a function of β.Panels use deterministic and dropout baselines, with higher normalized accuracy indicating better robustness.
  • L2 evaluation: Figure 6 plots targeted and untargeted L2-adversarial classification accuracy from 0 to 1 as a function of β.The deterministic and dropout baselines remain at 0% accuracy, while untargeted attacks are easier to resist than targeted attacks.
  • Architecture: The ImageNet architecture uses two 1024-unit fully connected layers, a 1024-dimensional spherical-Gaussian stochastic encoding, and a logistic-regression variational classifier.The experiment operates on 1536-dimensional representations extracted from a pretrained Inception ResNet V2 checkpoint.

Classification

VIB achieved strong ImageNet classification with a substantially compressed representation and improved resistance to targeted L2 adversarial attacks. The VIB model required larger perturbations and resisted attacks that consistently fooled deterministic models.

  • Classification: At large β accuracy suffered, while intermediate β values improved performance over both a deterministic baseline and the β = 0 regime.The ImageNet accuracy remained somewhat below the original 80.4%, possibly because of inadequate training time or suboptimal hyperparameters.
  • Classification: 80.12% accuracy at β = 0.01 used only I(X, Z) ∼45 bits, compared with 78.87% at β = 0 using over 10,000 bits.The β = 0.01 result was nearly the same as the state-of-the-art unmodified network.
  • Classification: VIB offered an accuracy benefit with a mere ∼45 bits of information from each image.Across β values and a deterministic baseline, the best classification accuracy occurred at β = 0.01 ∈(0, 1).
  • Adversarial robustness: The VIB network was more robust to targeted L2 attacks in both perturbation magnitude and successful-attack frequency.The comparison included VIB, a deterministic baseline, and the original pretrained Inception ResNet V2 network.
  • Adversarial robustness: 17 out of 30 targeted attacks succeeded against VIB, whereas deterministic models always misclassified into the target label.The VIB model resisted about 43.3% of the attacks.
  • Adversarial robustness: VIB required much larger perturbations to fool the classifier than the compared deterministic models.Figure 8 visualized absolute pixel differences, with quantitative perturbation results reported in Table 2.

5 FUTURE DIRECTIONS

The paper identifies several extensions for VIB, spanning network depth, data realism, marginal approximations, privacy, open-universe recognition, and sequence prediction.

  • Future directions: Future work includes applying the VIB objective at multiple or every network layer and testing it on real images.These directions extend the method's architectural placement and evaluation setting.
  • Future directions: The authors propose richer parametric marginal approximations instead of assuming r(z) = N(0, I).They also suggest exploring connections to differential privacy.
  • Future directions: Additional directions include open universe classification and sequence prediction using X as the past, Y as the future, and Z as the current representation.This sequence-prediction formulation is identified as predictive information.

A HYPERPARAMETERS AND ARCHITECTURE DETAILS FOR EXPERIMENTS

The experiments used TensorFlow networks optimized with Adam, exponential learning-rate decay, and Polyak-style parameter averaging, with architecture-specific Gaussian embedding initialization choices.

  • Training: All networks used TensorFlow, Xavier weight initialization, zero biases, and Adam with initial learning rate 10^-4 and exponential decay.The Adam settings were β1 = 0.5 and β2 = 0.999, with decay by 0.97 every 2 epochs.
  • Training: Test-time parameters were exponential moving averages with decay constant 0.999, and mutual information estimates were measured in bits.No other form of regularization was used for the VIB experiments.
  • Embedding architectures: The 256-dimensional Gaussian embedding used a 512-unit linear layer to produce means and standard deviations, with softplus-transformed standard deviations initially biased by -5.0.This initialization kept the initial standard deviations small.
  • Embedding architectures: The 1024-dimensional ImageNet embedding used sigma bias 0.57 and batch size 200.The sigma bias kept initial standard deviations near 1.
  • Embedding architectures: The 2-dimensional Gaussian embedding predicted means and a Cholesky covariance parameterization using six linear outputs.The covariance center used softplus bias -5.0, off-diagonal components were scaled by 10^-2, and the upper-triangular element was dropped.

B CONNECTION TO VARIATIONAL AUTOENCODERS

The unsupervised information bottleneck admits variational bounds that produce a VAE-like objective, but its interpretation remains information-theoretic rather than generative. This connection motivates alternative marginal approximations and tighter bounds.

  • Unsupervised information bottleneck: The unsupervised information bottleneck maximizes information in Z while restricting information about each data element's identity i.This objective is described as similar to an information-theoretic clustering objective.
  • Variational bounds: A variational decoder q(x|z) bounds the first mutual-information term after replacing the intractable conditional with q(x|z).The derivation drops H(X) because it is outside the model's control and uses KL-divergence nonnegativity.
  • Variational bounds: The intractable marginal p(z) is replaced by a variational marginal r(z) to bound the second term from above.The construction combines the two bounds into the unsupervised variational information bottleneck objective.
  • Connection to VAEs: The resulting objective has the mathematical form of a variational autoencoder, except that the second KL-divergence term has arbitrary weight β.The paper emphasizes that the shared form does not imply the same interpretation.
  • Connection to VAEs: Unlike a VAE's generative interpretation, VIB starts from the stochastic encoder p(z|x), while q(x|z) and r(z) are variational approximations.This different interpretation suggests targeting the exact marginal p(z) and seeking tighter bounds on the first I(Z, X) term.
  • Relation to prior work: The information bottleneck provides a principled motivation for changing the relative weighting of the objective terms and studying β's tradeoffs.The paper connects this setup to prior work on β-weighted VAE regularization and latent representations.

C QUADRATIC BOUNDS FOR STOCHASTIC LOGISTIC REGRESSION DECODER

This section derives an approximate quadratic upper bound for the negative log-likelihood of a stochastic soft-max decoder with Gaussian bottleneck variables. A sharper bound preserves the log-quadratic form and yields a classifier with quadratic capacity relevant to adversarial robustness.

  • Gaussian bottleneck: The bottleneck variable Z is modeled as a multivariate Normal with mean µ_x and positive-definite covariance Σ_x generated by a deep neural network.The covariance matrix is K × K.
  • Decoder setup: The categorical prediction is computed as S(Wz), where W is a C × K weight matrix and S is the soft-max function.The log-sum-exp function defines the normalization in the log-soft-max.
  • Robustness implication: The setup induces a classifier bounded by a quadratic function, which has greater theoretical capacity for adversarial robustness than a linear function.This comparison is attributed to the framework of Fawzi et al. (2016).
  • Approximate bound: Jensen’s inequality upper-bounds the negative log-likelihood, after which a second-order Taylor expansion approximates the expected log-sum-exp around the bottleneck mean.The derivation takes the expectation of the Taylor expansion at the mean parameters µ_x and Σ_x.
  • Quadratic form: The resulting approximation is log-quadratic in the decoder parameters, and a sharp quadratic upper bound changes only the scaling inside the exponential.The sharper bound is obtained via Browne & McNicholas (2015).
Loading 1612.00410v7…