PaperlayerView exact revision on arXiv

Source-linked AI summary

A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations

Weili Nie, Yang Zhang, Ankit Patel

arXiv:1805.07039v4cs.CVcs.AI

TL;DR

The paper addresses the lack of theory explaining why GBP and DeconvNet produce visually compelling but less class-sensitive CNN visualizations. It develops an analysis of their backward propagation and supports it with experiments, concluding that these methods perform partial image recovery driven by backward ReLUs and local CNN connections rather than network decisions.

  • Problem

    The relationship between the clean visualizations produced by GBP and DeconvNet and neural-network decision-making lacks a theoretical explanation.

  • Method

    The paper analyzes backpropagation-based visualizations in random, pooled, deep, and trained CNN settings, focusing on backward ReLUs, local connections, and max-pooling.

  • Results

    GBP and DeconvNet are essentially performing partial image recovery and are class-insensitive, while saliency maps are class-sensitive; experiments support this analysis.

  • Takeaways & Limitations

    Compelling visual quality in GBP and DeconvNet does not by itself indicate that the visualization reflects the network’s decision-making.

  • Takeaways & Limitations

    DeconvNet with max-pooling may never recover the input perfectly, and the random-CNN analysis does not directly apply to trained networks because their weights may not be i.i.d. Gaussian.

Abstract

from arXiv · show

Backpropagation-based visualizations have been proposed to interpret convolutional neural networks (CNNs), however a theory is missing to justify their behaviors: Guided backpropagation (GBP) and deconvolutional network (DeconvNet) generate more human-interpretable but less class-sensitive visualizations than saliency map. Motivated by this, we develop a theoretical explanation revealing that GBP and DeconvNet are essentially doing (partial) image recovery which is unrelated to the network decisions. Specifically, our analysis shows that the backward ReLU introduced by GBP and DeconvNet, and the local connections in CNNs are the two main causes of compelling visualizations. Extensive experiments are provided that support the theoretical analysis.

1. Introduction

Backpropagation-based visualizations aim to make neural-network decisions interpretable, but the relationship between visually compelling outputs and decision-making remains unclear. The paper explains why GBP and DeconvNet look cleaner than saliency maps while being less class-sensitive.

  • Safety-critical applications have increased demand for human-interpretable explanations of neural-network decisions.
  • Backpropagation-based visualizations highlight input pixels by propagating network outputs back into image space.
  • Saliency maps use the true gradient but are often noisy and difficult to interpret.
  • GBP and DeconvNet produce cleaner visualizations than saliency maps but are less sensitive to the selected class label.
  • The paper argues that GBP and DeconvNet perform partial image recovery rather than highlighting class-relevant pixels or learned weights.

2. Backpropagation-based Visualizations

The paper formalizes how saliency maps, DeconvNet, and GBP propagate output scores backward through ReLU activations. Empirically, saliency maps are noisy but class-sensitive, whereas DeconvNet and GBP are cleaner and nearly invariant across class logits.

  • Backpropagation-based methods differ primarily in how they propagate output scores through nonlinear activations.
  • The formalism distinguishes the forward ReLU from modified backward propagation through each activation.
  • Visualizations are computed by differentiating a selected class logit with respect to the input image.
  • Saliency maps are noisy, while DeconvNet and GBP produce more human-interpretable visualizations; DeconvNet shows texture-like patterns and GBP filters some background information.
  • Saliency maps change substantially across class logits, whereas DeconvNet and GBP remain almost invariant to the chosen logit.

3. Theoretical Explanations

The analysis explains how backward ReLU, local connections, max-pooling, network depth, and trained weights shape backpropagation-based visualizations. GBP can recover image structure independently of class labels, while DeconvNet and saliency map differ depending on pooling and weight assumptions.

  • Analysis setup: The theory analyzes saliency map, GBP, and DeconvNet first in random three-layer CNNs, then extends the analysis to max-pooling, deeper networks, and trained weights.The random-network setting assumes i.i.d. Gaussian weights and normalized visualization outputs.
  • Random three-layer CNN: With sufficiently many filters, GBP approximately recovers the input image regardless of the selected class logit.This result follows from the backward ReLU rather than from class-specific learned weights.
  • Random three-layer CNN: For filter size 3 × 3 × 3, at most O(10^3) filters can achieve estimation error below 0.1, linking local connections to crisp GBP visualizations.The required filter count depends heavily on filter size, which is usually small in CNNs.
  • Random three-layer CNN: In the random three-layer setting, saliency map and DeconvNet become Gaussian random noise conveying little information about the input image or class logits.Saliency map depends strongly on random weights, while DeconvNet becomes more similar to GBP once max-pooling is introduced.
  • CNNs with max-pooling: Max-pooling makes DeconvNet behave like GBP, because its visualization becomes the same form and therefore performs image recovery rather than class-specific highlighting.The analysis also attributes DeconvNet’s image-specific information to max-pooling.
  • CNNs with max-pooling: DeconvNet with max-pooling cannot theoretically recover the input perfectly for every pooling window, which may explain its texture-like artifacts.Contradictory activation and selection cases prevent the same arbitrarily accurate recovery available to GBP.
  • Deep CNNs: For deep CNNs, the relevant activation terms are approximately unchanged from the shallow analysis, so visualization behavior should barely change with increased depth.The shallow three-layer analysis therefore extends to deeper networks.
  • CNNs with trained weights: In trained CNNs, GBP and DeconvNet perform partial image recovery: trained weights select active image patches, but the filtering process is not class-sensitive.Similar weight distributions across fc1 neurons make the resulting visualization approximately independent of the selected class logit.

4. Experiments

Experiments support the analysis that local connections and backward ReLUs drive visually interpretable GBP and DeconvNet outputs, while saliency maps remain more class-sensitive. Across random and trained networks, GBP and DeconvNet change little across classes or adversarial relabelings, consistent with partial image recovery.

  • Random CNN and FCN: In a random CNN, only GBP produces a human-interpretable visualization; DeconvNet and saliency map produce random noise.In a random FCN, all methods produce random noise because local connections are absent and the hidden layer is too small to recover the image.
  • Impact of Local Connections: Increasing CNN filters or FCN hidden neurons improves GBP visual quality, but even Nh = 70000 in the FCN cannot match the CNN with N = 64.This supports the contribution of local connections to GBP’s visual quality.
  • Max-Pooling and Network Depth: Adding max-pooling makes DeconvNet human-interpretable in the random CNN, while GBP and saliency map remain largely unaffected.In a random VGG-16, GBP and DeconvNet remain interpretable while saliency map produces random noise.
  • Class Sensitivity: Across 10K ImageNet images, saliency maps have much larger average l2 distances across class logits than GBP or DeconvNet in random and trained VGG-16 networks.The comparison indicates greater class sensitivity for saliency maps and relative class insensitivity for GBP and DeconvNet.
  • Adversarial Attack: Under an FGSM change from “panda” to adversarial “busby,” saliency maps change significantly while GBP and DeconvNet remain almost unchanged.This further supports the distinction between class-sensitive visualization and partial image recovery.
  • Trained Weights: In trained VGG-16, trained convolutional layers filter out progressively more image patches, whereas trained dense layers have little additional effect on GBP.The layerwise experiment also supports GBP’s class-insensitive behavior.

5. Conclusions

The paper explains why GBP and DeconvNet look interpretable despite being class-insensitive. Its analysis attributes this behavior to partial image recovery driven by backward ReLUs, CNN local connections, and, for DeconvNet, max-pooling.

  • Conclusions: GBP and DeconvNet perform partial image recovery rather than highlighting class-relevant pixels or learned weights.The paper therefore characterizes them as unrelated in principle to neural-network decision-making.
  • Conclusions: Backward ReLUs and local CNN connections are identified as the main causes of human-interpretable GBP and DeconvNet visualizations.The paper also identifies max-pooling as important for DeconvNet input recovery.
  • Conclusions: The theoretical analysis begins with a random three-layer CNN and is generalized to more realistic cases, with support from extensive experiments.

A. Proof of Lemma 1

The proof unifies saliency map, DeconvNet, and GBP through their different combinations of forward and backward ReLUs and a common normalization of visualization outputs.

  • Proof of Lemma 1: Saliency map uses forward ReLUs only, DeconvNet uses backward ReLUs only, and GBP uses both types.
  • Proof of Lemma 1: All visualization results are normalized so that their norms lie in the range [0, 1].The normalization coefficient Zk ensures the stated range for the visualization associated with the k-th logit.
  • Proof of Lemma 1: The unified expression follows by differentiating fk(x) with the appropriate modified derivative and applying the definitions for each ReLU configuration.

B. Proof of Theorem 1

The proof analyzes GBP in a random CNN by modeling filter and output weights as independent Gaussian variables, then uses expectation and normalization arguments to approximate its visualization.

  • Random-Network Assumptions: The random-network proof assumes entries of V and W are independently Gaussian with zero mean and variance c2.It also assumes sufficiently many filters for the approximation.
  • Distributional Analysis: The derivation represents GBP filter contributions with rectified Gaussian variables induced by the backward ReLU.Their distributions are analyzed through projected Gaussian densities on a manifold defined by y(j)T w = 0.
  • Approximation: The proof obtains the GBP approximation by replacing sample means with expectations and using independence between Vqij,k and w(i).
  • Image-Recovery Approximation: Under constant patch norms and ignoring boundary points, the patch-overlap sum is approximated by pId×d.The padding choice can reduce boundary inconsistency.

C. Proof of Theorem 2

The proof shows that saliency map and DeconvNet outputs become approximately Gaussian under sufficiently many filters, while the identity term dominates image-information terms in high-dimensional inputs. Consequently, both methods preserve no input information.

  • Gaussian approximation: The central limit theorem approximates the intermediate quantity Θ_j as Gaussian when the number of filters N is sufficiently large.The proof treats Θ_j as a sum of independent identically distributed random variables.
  • Moment analysis: Saliency map and DeconvNet responses are modeled through means and variances of intermediate random variables.The proof separately evaluates saliency-map and DeconvNet moments, using independence assumptions and Gaussian approximations.
  • High-dimensional input: For a typical 3 × 3 or 7 × 7 patch, image dimension d greatly exceeds J/(2p), so the identity term dominates the image-information term.The trace comparison gives Trace(I − 1/(2p)Λ) ≈ d.
  • Conclusion: Both saliency map and DeconvNet are therefore standard Gaussian variables that preserve no input information.This is the proof’s concluding claim for Theorem 2.

D. Proof of Proposition 1

The proof establishes an independence approximation for GBP and extends the same reasoning to saliency map and DeconvNet. High-dimensional Gaussian intermediate variables support the required uncorrelatedness arguments.

  • GBP case: The GBP proof begins by analyzing the propagated quantity defined through the network’s backward computation.The argument starts from the GBP expression and tracks the relevant intermediate variables across layers.
  • Independence approximation: When the relevant layer dimension d2 is sufficiently high, the contribution of Γ^(2) to the dependence structure can be ignored.This approximation makes b_t independent of q_ij under the stated assumptions.
  • Gaussian approximation: The central limit theorem makes the propagated quantity approximately Gaussian with zero mean when d3 is sufficiently large.The proof then checks independence by showing that the Gaussian variables are uncorrelated.
  • Other visualization methods: The same independence approximation applies to saliency map, while DeconvNet follows because its case is a special case of GBP.The proof explicitly states that the relevant parameters are unchanged for saliency map and that DeconvNet inherits the GBP analysis.

E. More Experiments on Random/Trained VGG-16 Net

Experiments on randomly initialized and trained VGG-16 networks reproduce the predicted contrast: GBP and DeconvNet look more compelling but are less class-sensitive than saliency map. GBP also resembles an edge detector while retaining subtle differences.

  • Overall comparison: Across random and trained VGG-16 networks, GBP and DeconvNet are more visually compelling but less class-sensitive than saliency map.The experiments use randomly selected ImageNet images labeled dog, panda, forest, and mastiff.
  • Edge-detector comparison: GBP visualizations in trained VGG-16 resemble a linear vertical edge detector but differ in which intensity changes they highlight.The comparison connects GBP’s behavior to edge information while noting that the two visualizations are not identical.

G. More Experiments on Partly Trained VGG-16 Net

Experiments with partly trained VGG-16 networks show that convolutional-layer weights, especially from earlier layers, determine GBP’s filtering of image patches more strongly than dense-layer weights.

  • Layer contributions: Trained convolutional-layer weights, rather than dense-layer weights, account for filtering out image patches in GBP visualizations.This conclusion follows from loading trained weights either up to a layer or everywhere except a selected layer.
  • Layer contributions: Earlier convolutional layers have a greater impact on GBP visualizations than later convolutional layers.The result is reported consistently across the partly trained VGG-16 experiments.

H. More Experiments on ResNet

Experiments on ResNet-50 largely reproduce the VGG-16 visualization patterns for both random and trained networks, while revealing additional grid-like textures. The authors conjecture that skip connections cause this quality deterioration and leave rigorous analysis for future work.

  • ResNet-50 comparisons: ResNet-50 produces visualization results generally similar to those observed with VGG-16 for randomly initialized and trained networks.The experiments use saliency maps, DeconvNet, and GBP on random and trained ResNet-50 models.
  • ResNet-50 comparisons: Additional grid-like textures appear in the ResNet-50 visualizations, reducing their visual quality.This deterioration is observed beyond the patterns reported for VGG-16.
  • Interpretation: The authors conjecture that skip connections cause the additional grid-like textures because network structure significantly affects visualization quality.They explicitly leave a rigorous analysis of this phenomenon for future work.
Loading 1805.07039v4…