PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Deep Networks via Extremal Perturbations and Smooth Masks

Ruth Fong, Mandela Patrick, Andrea Vedaldi

arXiv:1910.08485v1cs.CVcs.LGstat.ML

TL;DR

Attribution methods often lack a principled definition of input importance. This paper introduces extremal perturbations and smooth, area-constrained masks, showing sensitivity to network spatial properties and enabling salient-channel analysis in intermediate layers.

  • Problem

    Most attribution methods lack a principled definition of which input regions are important to a neural network.

  • Method

    The paper optimizes maximal output-changing perturbations at fixed area using smooth masks and extends the framework from inputs to intermediate activations.

  • Results

    Sweeping perturbation area reveals sensitivity to network spatial properties, while intermediate-layer perturbations identify salient classification channels visualized through feature inversion.

  • Takeaways & Limitations

    Extremal perturbation analysis provides an interpretable way to study spatial input importance and channel-level representations in deep networks.

  • Takeaways & Limitations

    Prior perturbation formulations produce masks whose meaning depends on incomparable trade-offs and tunable weights, making results difficult to compare.

Abstract

from arXiv · show

The problem of attribution is concerned with identifying the parts of an input that are responsible for a model's output. An important family of attribution methods is based on measuring the effect of perturbations applied to the input. In this paper, we discuss some of the shortcomings of existing approaches to perturbation analysis and address them by introducing the concept of extremal perturbations, which are theoretically grounded and interpretable. We also introduce a number of technical innovations to compute extremal perturbations, including a new area constraint and a parametric family of smooth perturbations, which allow us to remove all tunable hyper-parameters from the optimization problem. We analyze the effect of perturbations as a function of their area, demonstrating excellent sensitivity to the spatial properties of the deep neural network under stimulation. We also extend perturbation analysis to the intermediate layers of a network. This application allows us to identify the salient channels necessary for classification, which, when visualized using feature inversion, can be used to elucidate model behavior. Lastly, we introduce TorchRay, an interpretability library built on PyTorch.

1. Introduction

The paper addresses attribution’s lack of a principled definition of input importance by introducing extremal perturbations, which maximize output effects at fixed area with guaranteed smoothness. It also develops stable optimization tools and extends perturbation analysis to intermediate-layer channels, supported by feature inversion and TorchRay.

  • Motivation: Most attribution methods lack a definition of what makes an input region important and are commonly validated a-posteriori.
  • Implementation: The paper introduces TorchRay, an interpretability library built on PyTorch.
  • Extremal perturbations: Extremal perturbations maximize a network’s output effect among perturbations with a fixed area, while restricting optimization to a family with minimum guaranteed smoothness.
  • Optimization: The method introduces a ranking-based area loss for enforcing target perturbation sizes stably and efficiently, plus a smooth perturbation family built with smooth-max convolution and a perturbation pyramid.
  • Intermediate-layer attribution: Intermediate-layer perturbation analysis identifies salient channels beyond spatial input attribution, and feature inversion helps visualize their role in model behavior.

2. Related work

Related work includes efficient backpropagation-based attribution, perturbation-based analysis, local model approximations, and intermediate-activation visualization. The paper highlights limitations in output- and channel-specific interpretability and combines channel attribution with visualization techniques.

  • Backpropagation-based methods: Backpropagation-based methods trace information from outputs to inputs or intermediate layers and require only one forward and backward pass.Variants reduce gradient noise, combine gradients with weights or activations, or use probabilistic and local-approximation interpretations.
  • Backpropagation-based methods: Some backpropagation-based methods produce identical saliency maps across output neurons or network parameters, limiting characterization of individual outputs, activations, or models.The cited studies report that this behavior occurs for some, but not all, such methods.
  • Perturbation-based methods: Perturbation-based methods alter inputs and measure output changes, using regular or random occlusions or optimized spatial masks.Occlusion and RISE weight output changes by their occlusion patterns, while meaningful perturbations optimize a mask that maximally affects the output.
  • Approximation-based methods: Approximation-based methods replace black-box models locally with simpler interpretable models, especially linear models such as LIME.Decision-tree approximations are also possible but are less applicable to visual inputs.
  • Visualizations of intermediate activations: Intermediate activations can be visualized through maximally activating examples, activation maximization, or feature inversion, but important channels are not naturally interpretable alone.The paper addresses this difficulty by combining channel attribution with these visualization techniques.

3. Method

The method replaces ambiguous multi-term perturbation optimization with fixed-area, smooth-mask extremal perturbations. Extremality identifies the smallest mask reaching a target response, while max-convolution constructs masks that remain nearly binary and smoothly varying.

  • The original objective has an unclear trade-off because model response, mask area, and mask regularity are not commensurate, making λ and β produce incomparable masks.
  • The proposed formulation fixes the mask area to a fraction a|Ω| and restricts masks to a fixed smooth function set M.
  • An extremal perturbation is the smallest mask whose preserved input achieves at least a chosen output threshold Φ0.The threshold can be set as Φ0 = τΦ(x), and the smallest mask is found by sweeping the area parameter a.
  • A single extremal mask characterizes an entire family of perturbations because every smaller mask produces an output below Φ0.
  • The area constraint is enforced by sorting mask values and penalizing their distance from (1−a)|Ω| zeros followed by a|Ω| ones.The regularizer is Ra(m) = ∥vecsort(m) − ra∥2; λ is set as large as numerics allow so the constraint is nearly exact.
  • Max-convolution produces smooth masks that preserve unit values and inherit the smoothing kernel’s Lipschitz bound.If k(0) = 1, then ¯m(u) ≤ m(u) ≤ 1; if k is Lipschitz with constant K, m has a constant at most K.

4. Experiments

Experiments show that area-controlled extremal perturbations produce localized, stable masks that reveal how networks prioritize and monotonically integrate visual evidence. The method is sensitive to model weights and competitive in the pointing game across PASCAL VOC and COCO.

  • Implementation details: Experiments use VGG16 on ImageNet with preservation masks at areas 0.05, 0.1, 0.2, 0.4, 0.6, and 0.8.The classification score Φ(x) is the pre-softmax ground-truth score, and Φ0 is set to the unperturbed score Φ(x).
  • Qualitative analysis: The criterion Φ0 = Φ(x) yields localized, sharp masks that tightly cover objects and identify regions of interest.As area increases, masks expose discriminative and complete regions, including foreground objects and parts such as a dog’s nose.
  • Stability and sanity checks: Area control provides visibly greater stability than Fong and Vedaldi’s masks, while weight randomization produces significantly different visualizations.The weight-randomization sanity check uses the hybrid formulation and progressively randomizes VGG16 weights from fc8 to conv1_1.
  • Pointing game: On the pointing game, the method is competitive with VGG16 and ResNet50 on PASCAL VOC and COCO, whereas Fong and Vedaldi’s method was not competitive.The benchmark uses PASCAL VOC 2007 test images and approximately 50k COCO 2014 validation images.
  • Evidence integration: Evidence integration is monotonic in 98.45% of ImageNet validation images for areas below the optimal perturbation area.This supports the interpretation that networks usually integrate evidence consistently as the perturbation area grows.

5. Attribution at intermediate layers

The paper extends extremal perturbations to intermediate-layer channels, using area-constrained masks to identify channels salient for classification. Feature inversion makes these channel attributions interpretable and also enables class-specific channel discovery from per-instance masks.

  • 5. Attribution at intermediate layers: Extremal perturbations are extended to intermediate layers to identify channels salient for classification.The method studies channel-wise perturbations of intermediate representations using the area loss.
  • 5. Attribution at intermediate layers: Class scores monotonically increase as the channel area increases, with the optimal mask selecting a specified number of channels.Figure 8 illustrates an optimal mask with a∗ = 25 channels and plots class score against the number of channels.
  • 5. Attribution at intermediate layers: Experiments use GoogLeNet’s inception4d layer with Hl = 14, Wl = 14, and Kl = 528, optimizing channel masks for 300 iterations.The learning rate is 10^-2, while λ increases from 0 to 1500 during the first 150 iterations and then remains constant.
  • 5. Attribution at intermediate layers: Feature inversion compares unperturbed and channel-masked activations to reveal the discriminative input regions encoded by salient channels.Because intermediate channel identities are difficult to visualize directly, feature inversion provides an image-based interpretation of the attribution mask.
  • 5. Attribution at intermediate layers: Per-class masks are obtained by averaging per-instance optimal masks across ImageNet validation images, yielding highly class-specific channel feature inversions.This identifies class-specific channels without explicitly optimizing for class specificity at a global level.

6. Conclusion

The paper introduces extremal perturbation analysis to address shortcomings of prior perturbation-based approaches and contributes techniques for computing it, including a rank-order area constraint and intermediate-activation perturbations.

  • The paper introduces extremal perturbation analysis as a framework that avoids some issues in prior perturbation-based neural-network analysis.
  • Its technical contributions include methods for computing extremal perturbations and a rank-order area constraint with potential applications beyond extremal perturbation computation.
  • The framework is extended to perturb intermediate activations.

A. Implementation details · A.1. Generating smooth masks

The smooth-mask generator maps input mask samples to output locations using a finite-radius kernel, then computes masks efficiently through unpooling, nearest-neighbour upsampling, pooling, and cropping. Parameters are selected from the kernel width, step, margin, and desired output width.

  • A.1. Generating smooth masks: Input samples i = 0, . . . , N −1 map to output samples through ui = ai + b, while kernel radius σ limits relevant samples to |u −ui| ≤σ.The kernel uses only samples within radius σ of each mapped output location.
  • A.1. Generating smooth masks: Nearest-neighbour upsampling brings the unpooled signal to output resolution, with factor s = W ′′/W ′ determined by the input and output sizes.In PyTorch, the output size W ′′ must be chosen explicitly from the desired scaling factor.
  • A.1. Generating smooth masks: The generator chooses kernel width σ, step s, and margin b ≥0, sets a = s, and uses these settings to determine the samples required for ˆm(u).The margin b is used to avoid border effects.
  • A.1. Generating smooth masks: The pooled mask is formed by unpooling input parameters mi into m′ and upsampling into m′′ before pooling with precomputed weights gk,u.This sequence defines the practical computation of the smooth mask.
  • A.1. Generating smooth masks: The generator takes s, σ, and desired width W, constructs an intermediate mask of width W ′′ = sW ′, and sets W ′ = N −K+2P +1.The construction uses the earlier formulas to determine R, K, P, W ′, and W ′′.
  • A.1. Generating smooth masks: As a rule of thumb, N = ⌈W/s⌉ spreads samples regularly across W, after which a W-sized crop shifted by b pixels produces the final mask.The crop converts the intermediate width W ′′ into the desired output width W.
Loading 1910.08485v1…