PaperlayerView exact revision on arXiv

Source-linked AI summary

Restricting the Flow: Information Bottlenecks for Attribution

Karl Schulz, Leon Sixt, Federico Tombari, Tim Landgraf

arXiv:2001.00396v4stat.MLcs.CVcs.LG

TL;DR

Attribution methods help explain neural-network decisions, but existing evaluation lacks a standard benchmark and interpretability remains important for sensitive applications. The paper introduces Information Bottleneck Attribution, which injects noise into intermediate feature maps to estimate each region’s information for prediction, and reports competitive performance with guarantees for zero-valued regions.

  • Problem

    Attribution methods assign relevance scores to input variables, but no standard evaluation benchmark exists and performance depends strongly on the model and dataset.

  • Method

    Information Bottleneck Attribution injects noise into intermediate feature maps and learns a bottleneck to estimate an upper bound on the information each image region provides for prediction.

  • Results

    Both bottleneck variants show competitive results on all evaluated metrics and outperform state-of-the-art methods by a significant margin on some tasks.

  • Takeaways & Limitations

    The information-theoretic foundation provides attribution scores in bits and guarantees that regions with zero-valued attribution are not required for correct classification.

  • Takeaways & Limitations

    The mutual information required by the method involves an intractable integral, so the paper uses a variational approximation to upper-bound the estimate.

Abstract

from arXiv · show

Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks. For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image. In this work we adapt the information bottleneck concept for attribution. By adding noise to intermediate feature maps we restrict the flow of information and can quantify (in bits) how much information image regions provide. We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings. The method's information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision. For reviews: https://openreview.net/forum?id=S1xWh1rYwB For code: https://github.com/BioroboticsLab/IBA

1 INTRODUCTION

The paper develops Information Bottleneck Attribution, which estimates how much information image regions provide to a network and evaluates it against existing attribution methods. Its information-theoretic scores provide a bits-based reference and guarantee that zero-information regions are unnecessary for prediction.

  • Motivation: Attribution methods explain model behavior by assigning relevance scores to input variables, often visualized as image heatmaps.These methods are important for interpreting complex neural networks in applications such as medical decision making and autonomous driving.
  • Motivation: Current attribution methods cannot guarantee that low-scored regions are unnecessary for the network’s prediction.A heatmap may highlight tracks rather than a locomotive, but this does not establish whether the ignored region is irrelevant to the model.
  • Method: The proposed method estimates the information an image region provides to the network’s prediction using a variational upper bound.Regions with zero bits of information are guaranteed not to be necessary for the prediction.
  • Method: The information bottleneck restricts feature-map information flow by adding noise, with Per-Sample and Readout Bottleneck variants for learning its parameters.The Per-Sample Bottleneck learns from one sample, whereas the Readout Bottleneck uses the entire dataset.
  • Evaluation: The method is evaluated against ten baselines using Sensitivity-n, object localization, degradation, and layer-wise weight-randomization analyses.The authors report consistent outperformance across the attribution metrics and provide source code for reproducibility.
  • Contribution: IBA provides a theoretical upper bound on used information while demonstrating strong empirical performance.The authors frame this as improving interpretability and increasing trust in attribution results.

2 RELATED WORK

The related work spans gradient, propagation-based, perturbation-based, concept-level, and information-bottleneck attribution methods. It also highlights the lack of a standard evaluation benchmark and distinguishes this work from bottlenecks introduced during network training.

  • Attribution methods: Existing attribution methods include gradient-based, propagation-based, perturbation-based, concept-based, and neuron-pruning approaches.Examples include Integrated Gradients, SmoothGrad, LRP, Deep Taylor Decomposition, Occlusion, TCAV, and irrelevant-neuron pruning.
  • Evaluation: No standard attribution benchmark exists, making state-of-the-art comparisons difficult and results dependent on the model and dataset.Visual inspection and degradation-based evaluations are commonly used, alongside Sensitivity-n.
  • Information bottlenecks: Information bottlenecks reduce information by adding noise and have been used for regularization, latent-capacity control, robustness, and salient-region extraction.Unlike those approaches, this work inserts the bottleneck into an existing network for post-hoc explanations rather than during network training.

3 INFORMATION BOTTLENECK FOR ATTRIBUTION

The method inserts a noise-based information bottleneck into intermediate feature maps, optimizing information flow against classification performance. It estimates regional information in bits while providing an upper-bound guarantee for regions assigned zero information.

  • Information Bottleneck for Attribution: The bottleneck restricts information flow by injecting noise into intermediate feature maps of a pretrained network while keeping the original model fixed.Noise can replace unimportant activations, removing their information for subsequent layers.
  • Information Bottleneck for Attribution: The bottleneck output interpolates between signal and noise: λi = 1 transmits Ri, whereas λi = 0 replaces it with noise.The mask λ has the same dimensions as the intermediate representation and contains values between 0 and 1.
  • Information Bottleneck for Attribution: The method estimates regional information using a variational upper bound on mutual information, computed per feature dimension and aggregated into an image-sized heatmap.The heatmap sums KL-divergence values over channels and is resized by bilinear interpolation.
  • Information Bottleneck for Attribution: If the information loss LI is zero for an area, the method guarantees that information from that area is not necessary for the network’s prediction.The bound may overestimate mutual information because the variational distribution introduces a nonnegative additional KL-divergence term.
  • Information Bottleneck for Attribution: The optimization minimizes mutual information while preserving classification performance through L = LCE + βLI, where β controls the trade-off.Higher β favors transmitting fewer bits, whereas lower β permits more information flow.
  • Bottleneck Variants: The Per-Sample Bottleneck fits λ separately for each image, while the Readout Bottleneck trains a network to predict bottleneck parameters from collected feature maps.The readout procedure uses two forward passes and leaves the analyzed network’s parameters fixed.

4 EVALUATION

The evaluation compares IBA variants with established attribution baselines on VGG-16 and ResNet-50 using localization, degradation, Sensitivity-n, and parameter-randomization tests. Across these evaluations, the Per-Sample Bottleneck generally provides focused heatmaps and strong benchmark performance, while some baselines fail the sanity check.

  • 4.1 EXPERIMENTAL SETUP: VGG-16 and ResNet-50 were evaluated with pretrained weights, alongside random, occlusion, gradient, gradient-accumulation, modified-backpropagation, and Grad-CAM baselines.PatternAttribution and LRP were not reported for ResNet-50 because the implementation did not support skip connections.
  • 4.2 QUALITATIVE RESULTS: The evaluated IBA heatmaps identify classification-relevant image regions, whereas Guided Backpropagation and PatternAttribution tend to highlight edges.Compared with Grad-CAM, both IBA variants assign high scores to fewer pixels; Readout Bottleneck attribution is somewhat more concentrated on object edges.
  • 4.3 SANITY CHECK: RANDOMIZATION OF MODEL PARAMETERS: Randomizing the final dense layer drops IBA mean SSIM by around 0.4, while GuidedBP and LRP remain similar after large portions of the network are randomized.Readout Bottleneck values have limited expressiveness because it was not retrained after randomization; SmoothGrad and Integrated Gradients drop by more than 0.4.
  • 4.4 SENSITIVITY-N: For Sensitivity-n, Per-Sample Bottleneck β = 10/k performs best for both models above n = 2 · 10^3 masked pixels.Pixel-wise masking produces poorly discriminable scores, while 8x8 tiles increase scores to 30% of the scale.
  • 4.5 BOUNDING BOX: 0.152: the Per-Sample Bottleneck outperforms every baseline by this margin on the bounding-box evaluation for both VGG-16 and ResNet-50.The evaluation uses images whose bounding boxes cover less than 33% of the input and includes 11,849 ImageNet validation images.
  • 4.6 IMAGE DEGRADATION: The Per-Sample Bottleneck outperforms all other methods in the degradation benchmark except GuidedGrad-CAM on ResNet-50, where the score difference is 0.004.The benchmark combines MoRF and LeRF through the integral between their curves; Readout Bottleneck remains competitive but generally scores lower.

5 CONCLUSION

The paper introduces two information-bottleneck attribution methods that quantify information provided by input regions while supporting broad network architectures. Both methods perform competitively, but the authors generally recommend the Per-Sample Bottleneck despite practical trade-offs.

  • The Per-Sample and Readout Bottlenecks inject learned noise into feature layers to upper-bound information supplied by each input region.The Per-Sample Bottleneck is optimized for one data point, whereas the Readout Bottleneck is trained across the dataset.
  • Both methods support arbitrary activation functions and network architectures because they do not constrain internal network structure.
  • Both bottlenecks outperform state-of-the-art baselines on all evaluated metrics, with significant margins on some tasks.
  • The authors generally recommend the Per-Sample Bottleneck because it performs better and requires only feature-map mean and variance estimates.The Readout Bottleneck instead produces attribution maps with one forward pass after training.
  • Per-Sample heatmaps may discard redundant class evidence and vary slightly because optimization is random.These caveats are especially relevant for images containing multiple object instances or when interpreting repeated runs.
  • Information-theoretic scores in bits provide an absolute comparison frame and guarantee that zero-valued regions are unnecessary for correct classification.

A VISUAL COMPARISON OF ATTRIBUTION METHODS

The figure uses blue for negative relevance and red for positive relevance, with samples reportedly selected randomly rather than cherry-picked.

  • Blue indicates negative relevance, while red indicates positive relevance.
  • The authors state that the displayed samples were selected truly randomly without cherry-picking.

B GRID ARTIFACTS WHEN NOT USING SMOOTHING

For ResNet-50 conv1 3, smoothing changes the Per-Sample Bottleneck information-flow visualization by removing grid artifacts and reducing overfitting to small areas.

  • The figure shows DKL(Q(Z|X)||Q(Z)) for the Per-Sample Bottleneck at ResNet-50 layer conv1 3.
  • Red marks areas with maximal information flow, while semi-transparent green marks zero information flow.
  • Without smoothing, the top-row mask exhibits a grid structure; smoothing with σs = 2 removes these artifacts and reduces overfitting to small areas.

C EFFECTS OF DIFFERENT β AND LAYER DEPTH FOR THE VGG-16

The figure examines how varying layer depth and β affects the Per-Sample Bottleneck for VGG-16, while reporting the resulting correct-class output probability.

  • The figure varies layer depth and β to examine their effects on the VGG-16 Per-Sample Bottleneck.
  • The resulting output probability for the correct class is reported as p.

D DERIVATION OF THE UPPER-BOUND OF MUTUAL INFORMATION

The bottleneck forms Z by interpolating feature-map activations with Gaussian noise, while input-dependent λ controls how much signal is transmitted. Because λ(X) and R both depend on X, the resulting Z does not generally preserve R’s variance.

  • Z combines the feature map R with Gaussian noise ϵ through the input-dependent damping tensor λ(X).λ controls signal transmission and noise addition at each feature-map location.
  • The expected value of Z is approximately the expected value of R when λ(X) and R do not correlate.The derivation uses independence between λ and ϵ and the noise mean µ_R.
  • Var[Z] ≠ Var[R] in general because multiplying λ(X) and R changes the variance.

F HYPERPARAMETERS

The appendix specifies hyperparameters for Per-Sample and Readout Bottlenecks across ResNet-50 and VGG-16. It also reports how β affects information loss and accuracy for the two bottleneck types.

  • Table 2 lists Per-Sample Bottleneck hyperparameters for ResNet-50 and VGG-16 layer configurations.ResNet-50 uses block-and-layer indices, while VGG-16 uses convolutional-layer numbers.
  • Table 3 provides the hyperparameters for the Readout Bottleneck.
  • For the Per-Sample Bottleneck, accuracy stays close to 1 for β ≤10/k, but falls to 0.43 at β = 100/k.At β = 100/k, positive evidence is also discarded.
  • For the Readout Bottleneck, small β ≤0.1/k suppresses negative evidence and slightly improves final accuracy.

G EVALUATION METRICS

The evaluation uses a cascading parameter-randomization sanity check with SSIM-based heatmap comparisons. MoRF and LeRF paths are evaluated for ResNet-50 and VGG-16 using 14x14 tiles.

  • The cascading parameter-randomization sanity check uses SSIM with a window size of 5.
  • For LRP, randomized weights can flip saliency-map values, so the evaluation compares SSIM against both the normalized and sign-inverted randomized heatmaps.
  • Figure 11 presents MoRF and LeRF results for ResNet-50 using 14x14 tiles.
  • Figure 12 presents MoRF and LeRF paths for VGG-16 using 14x14 tiles.
Loading 2001.00396v4…