PaperlayerView exact revision on arXiv

Source-linked AI summary

When Explanations Lie: Why Many Modified BP Attributions Fail

Leon Sixt, Maximilian Granz, Tim Landgraf

arXiv:1912.09818v7cs.LGstat.ML

TL;DR

Modified BP attribution methods are intended to explain neural-network predictions, but their faithfulness is uncertain. The paper combines theoretical matrix analysis with sanity checks and CSC evaluation, finding that all tested methods except DeepLIFT can ignore later-layer parameters and become class-insensitive. It identifies negative relevance and revised theoretical analyses as directions for addressing this convergence problem.

  • Problem

    The paper investigates whether modified BP attribution methods faithfully explain deep-network predictions rather than producing explanations insensitive to later-layer parameters.

  • Method

    The authors analyze modified BP rules theoretically and empirically, using matrix-convergence arguments, randomized-parameter sanity checks, and the cosine similarity convergence metric.

  • Results

    All tested modified BP methods except DeepLIFT can produce explanations independent of later-layer parameters and class-insensitive maps.

  • Takeaways & Limitations

    The findings support assessing attribution faithfulness through theoretical and empirical convergence tests and highlight negative relevance as a possible way to improve class sensitivity.

  • Takeaways & Limitations

    The evaluation omits some modified BP methods, and the PatternAttribution convergence argument relies on observations from a single set of pattern matrices.

Abstract

from arXiv · show

Attribution methods aim to explain a neural network's prediction by highlighting the most relevant image areas. A popular approach is to backpropagate (BP) a custom relevance score using modified rules, rather than the gradient. We analyze an extensive set of modified BP methods: Deep Taylor Decomposition, Layer-wise Relevance Propagation (LRP), Excitation BP, PatternAttribution, DeepLIFT, Deconv, RectGrad, and Guided BP. We find empirically that the explanations of all mentioned methods, except for DeepLIFT, are independent of the parameters of later layers. We provide theoretical insights for this surprising behavior and also analyze why DeepLIFT does not suffer from this limitation. Empirically, we measure how information of later layers is ignored by using our new metric, cosine similarity convergence (CSC). The paper provides a framework to assess the faithfulness of new and existing modified BP methods theoretically and empirically. For code see: https://github.com/berleon/when-explanations-lie

1. Introduction

The paper asks whether modified backpropagation explanations faithfully reflect deep-network predictions. It finds that most tested methods can ignore later-layer parameters and become class-insensitive, whereas DeepLIFT passes the tested sanity check.

  • 1. Introduction: GuidedBP explanations remain identical when the final layer is randomized, despite the network output changing.Ignoring the final layer prevents the explanation from faithfully representing the network prediction.
  • 1. Introduction: Seven tested modified BP methods fail the sanity check, while DeepLIFT is the only tested method that passes.The failures include LRP, Deep Taylor Decomposition, PatternAttribution, Excitation BP, Deconv, GuidedBP, and RectGrad.
  • 1. Introduction: Modified BP methods backpropagate custom relevance scores instead of gradients to identify image regions relevant to predictions.For example, Deep Taylor Decomposition backpropagates only positive relevance scores.
  • 1. Introduction: The theoretical explanation is that the z+-rule creates a chain of non-negative matrices that converges toward rank one, reducing dependence on later layers.After convergence, multiplying by different vectors changes mainly the scale or sign of the saliency map, not its direction.
  • 1. Introduction: The cosine similarity convergence metric tracks layer-by-layer loss of information, revealing rank-1 convergence for all analyzed methods except DeepLIFT on VGG-16 and ResNet-50.For sufficiently large α and β, LRPαβ instead produces noisy saliency maps without converging.
  • 1. Introduction: Many modified BP methods produce class-insensitive maps that emphasize low-level features, while negative relevance may help prevent rank-1 convergence.The paper presents negative relevance as a possible direction for improving faithfulness.

2. Theoretical Analysis

The theoretical analysis models modified backpropagation as products of layer-dependent matrices. Non-negative chains contract toward a single direction, while DeepLIFT’s treatment of positive and negative contributions avoids this convergence.

  • 2.1. z+-Rule: The z+-rule used by DTD, LRPα1β0, and Excitation BP forms a product of non-negative matrices across layers.The resulting relevance map is determined by this matrix chain rather than by the original gradient masks.
  • 2.1. z+-Rule: Under the theorem’s norm and overlap conditions, products of non-negative matrices converge to a rank-1 matrix.The geometric proof interprets multiplication as repeatedly shrinking a cone of non-negative column directions toward one vector.
  • 2.1. z+-Rule: After convergence, later layers can alter saliency scaling but cannot substantially change its direction, even though the final layer determines the network prediction.Thus contributions from deeper layers shrink with depth and the explanation can ignore prediction-relevant information.
  • 2.2. Modified BP algorithms: DTD necessarily converges because its z+-, w2-, and bounded-input rules produce non-negative outputs to which the theorem applies.This conclusion covers the specific DTD rules described for nonnegative, unrestricted, and bounded inputs.
  • 2.2. Modified BP algorithms: PatternNet empirically exhibits strong single-direction alignment, with σ1(A)/σ2(A) > 6 for almost all VGG-16 patterns.The authors interpret this as evidence that its matrix chain strongly converges toward one direction.
  • 2.2. Modified BP algorithms: PatternAttribution’s Hadamard product complicates theory, although empirical singular-value ratios indicate increased matrix alignment in some layers.The authors could not prove necessary rank-1 convergence for PatternAttribution.
  • 2.2. Modified BP algorithms: DeepLIFT avoids rank-1 convergence by backpropagating finite differences from a reference point while separating positive and negative contributions.Removing positive-negative intermixing in an ablation decouples the chains and restores convergence, supporting the proposed mechanism.

3. Evaluation

The evaluation tests attribution faithfulness through randomized logits, randomized network parameters, and the CSC metric across several networks. These procedures assess whether explanations retain sensitivity to predictions and later-layer information.

  • 3. Evaluation: The evaluation covers a CIFAR-10 convolutional network, VGG-16, and ResNet-50 using 200 validation images per network.The models vary in depth, residual connections, dense layers, and batch normalization.
  • 3. Evaluation: Random-logit tests compare saliency maps for the ground-truth and a random logit because explanations should change with the explained output.The comparison uses SSIM, following the cited sanity-check procedure.
  • 3. Evaluation: Parameter-randomization tests proceed from the last layer to the first, using SSIM to measure change from the original saliency map.For DTD and LRPα1β0, inverted maps are also compared because randomizing the last layer can flip their sign.
  • 3. Evaluation: CSC randomizes relevance vectors at an intermediate layer and measures how closely two resulting backpropagated vectors align with cosine similarity.Rank-1 convergence corresponds to cosine similarity approaching one.
  • 3. Evaluation: The direct singular-value-ratio alternative is computationally costly because the full derivation matrix can be large.CSC provides a more practical way to trace convergence layer by layer.
  • 3. Evaluation: For convolutional layers, CSC is computed separately at each feature-map location, and the evaluation reports median similarities plus first-layer histograms.The input-layer cosine-similarity jump reflects the input’s low dimensionality of three channels.

4. Results

The results show that converging modified BP methods produce saliency maps that are largely insensitive to output logits and parameter randomization. CSC analysis confirms strong layerwise alignment for nearly all methods except LRPz and DeepLIFT.

  • SSIM values very close to 1 show that converging methods produce almost identical saliency maps independently of the output logit.Methods with SSIM between 0.4 and 0.8 produce maps that differ from the ground-truth logit’s map.
  • One method group retains similar saliency maps even when convolutional parameters are randomized, while the remainder is sensitive to randomization.The same clustering appears for ResNet-50.
  • Except for LRPz and DeepLIFT, all methods reach at least 0.99 cosine similarity in CSC, although convergence rates differ across methods.LRPα1β0, PatternNet, and the DeepLIFT ablation converge fastest, while PatternAttribution converges more slowly but still exponentially.
  • LRPα5β4 converges less strongly for VGG-16, and numerical instabilities reduce CSC to 0 in early ResNet-50 layers for the DeepLIFT ablation.Even on a small 6-layer network, LRPα1β0 reaches median CSC greater than 1-1e-6.

5. Discussion

The discussion examines why convergence can go unnoticed, how proposed remedies affect class sensitivity, and where the analysis remains bounded. It emphasizes that modified BP convergence is distinct from methods based directly on gradients or black-box access.

  • Modified BP methods can appear acceptable because datasets often contain one object class per image, while some applications need only predictive low-level features.The paper notes that this can obscure class insensitivity and later-layer dependence.
  • LRPCMP: LRPCMP changes saliency maps after VGG-16 parameter randomization but cannot alter the direction of earlier-layer relevance vectors, and it does not resolve global convergence in ResNet-50.The dense layers change coarse local scaling, while convergence begins when LRPαβ is applied in convolutional layers.
  • Contrastive LRP: Contrastive LRP subtracts normalized saliency maps for the explained and opposite logits, yet its underlying convergence problem remains unresolved.The resulting maps are similar to the paper’s earlier converged example.
  • Contrastive Excitation BP: Contrastive Excitation BP adds negative-weight contributions to increase class sensitivity, but additional fully connected layers would restore global class insensitivity.The modification therefore does not resolve the underlying convergence problem.
  • Limitations: The study focuses mainly on convolutional vision models, and the theoretical PatternAttribution argument relies on observations from one set of pattern matrices.Some modified BP methods were omitted from evaluation.
  • Scope: Gradient-based and black-box attribution methods do not exhibit this convergence problem, although they may have other limitations.The paper restricts its convergence claim to modified BP methods.

6. Related Work

Related work has identified attribution failures through sanity checks, proxy evaluations, and human-centered assessment. The paper positions CSC as an efficient measure specifically for tracing convergence in modified BP methods.

  • Prior work showed that attribution maps can be altered by small architectural changes, conceal classifier bias, reconstruct inputs, or produce incorrect explanations for noisy linear models.Other studies also reported class insensitivity in several modified BP methods.
  • Bounding-box relevance, MoRF/LeRF, and ROAR are proxy evaluations because attribution ground truth is unavailable, but ROAR does not separate converging from non-converging methods.ROAR retrains models to reduce concerns about out-of-distribution masked samples, at substantial computational cost.
  • CSC resembles effective-rank analyses of gradient convergence but is designed for modified BP methods and efficiently traces their degree of convergence.It measures convergence layer by layer.
  • Human-centered evaluation provides another way to assess attribution methods by measuring how helpful explanations are to people.

7. Conclusion

The conclusion argues that most analyzed modified BP methods have properties contrary to their explanatory goal. It highlights the need to revisit theoretical foundations and the importance of negative relevance.

  • Most analyzed modified BP attribution methods have theoretical properties contrary to their goal of explaining deep neural network predictions.The conclusion identifies PatternAttribution and LRP’s reliance on Deep Taylor Decomposition as a reason to revisit its theoretical derivation.
  • The theoretical analysis stresses the importance of negative relevance for avoiding the convergence behavior identified in modified BP methods.

A. Proof of Theorem 1

Theorem 1 establishes that products of suitably constrained non-negative matrices converge, after normalization, to a rank-1 matrix. The proof uses cosine similarities between product-matrix columns to show monotonic convergence toward a common direction.

  • Theorem conditions: Under non-negativity, bounded dimensions and column norms, nonzero columns, and infinitely many sufficiently non-orthogonal matrices, the normalized product converges to rank 1.The theorem requires every column norm to exceed ϵ0 and infinitely many matrices to contain column pairs with dot product at least ϵ⟨·,·⟩.
  • Proof extensions: The proof permits finitely many matrices outside the main class and extends from square to compatible non-square matrix sequences by grouping subsequent matrices into square products.Matrices that do not meet the regularity class finitely often do not change the convergence result.
  • Cosine-similarity proof: The proof defines s_n as the cosine similarity corresponding to the maximal angle between columns of the matrix product M_n.The product is formed from A_1 through A_n, and the column angles are well defined because the columns are nonzero.
  • Conclusion: Because s_n → 1, the maximal column angle tends to zero, making all normalized product columns identical up to scalar multiples and yielding a rank-1 limit.The unnormalized matrix entries may diverge, so the proof explicitly normalizes the product before taking the limit.
  • Cosine-similarity proof: The sequence s_n is monotonic and bounded above by 1, so it converges; the argument then shows its limit must equal 1.Assuming a positive limiting gap leads to a contradiction, forcing ε_n = 1 − s_n toward zero.

B. Convergence Speed

The analysis shows that non-negative matrix chains converge exponentially toward rank-1 structure, explaining rapid loss of directional information in modified backpropagation. Simulations confirm this behavior within a few iterations, while sufficiently negative contributions can reduce convergence but introduce noise.

  • Theory: The convergence bound implies that cosine similarity approaches 1 exponentially fast, with typical matrices reaching rank-1 structure within a few steps.The rate parameter is bounded using the minimum dot product and maximum column L1-norm.
  • Experimental setup: The experiments simulate a VGG-16-like matrix chain by starting from the last layer and varying matrix distributions, ReLU processing, sparsity, and positivity.Convolutional kernels are represented as 1x1 matrix transformations, and median cosine similarity tracks alignment among columns.
  • Simulation: After 7 iterations, positive, stochastic, and non-negative matrix chains converged to a single vector up to floating-point imprecision.The observed linear path on a logarithmic scale indicates exponential convergence; 50% zeros only slightly reduced the convergence slope.
  • Negative contributions: Introducing negative matrix contributions preserves convergence for sufficiently small β < 4, but the resulting LRPα5β4 maps show less convergence and more noise.This simulation motivated evaluating LRPα5β4 on VGG-16.
  • Attribution connection: The convergence analysis also motivates the connection between matrix-chain behavior and modified attribution rules derived from linear-layer explanations and covariance-based pattern computation.The supplied passages describe the PatternAttribution derivation and its relation to a power-iteration step.

D. CIFAR-10 Network Architecture

The CIFAR-10 experiment uses a small convolutional network with four convolutional layers, two dense layers, and a softmax output.

  • Architecture: The CIFAR-10 model contains four convolutional layers, two dense layers, and a final softmax classifier.The architecture includes ReLU activations, max pooling, flattening, and dropout between the convolutional and dense blocks.

E. Results on ResNet-50

The supplied passage identifies Figure 8 as evaluating how randomizing logits or parameters affects a ResNet-50.

  • ResNet-50 evaluation: Figure 8 evaluates the effect of randomizing logits and model parameters on a ResNet-50.The supplied passage provides the figure scope but no outcome values or stated winner.

F. Additional Cosine Similarity Figures

Additional figures examine CSC convergence and saliency-map normalization during parameter-randomization sanity checks. These materials define the evaluation views but do not supply further numerical outcomes.

  • CSC evaluation: Figure 9 presents convergence measured with CSC for different starting layers.The caption identifies the metric and the varying starting-layer condition, but not the plotted outcomes.
  • Visualization: For visualization, saliency maps with only positive relevance are normalized to [0, 1], while maps containing negative relevance use [−1, 1].Sanity-check maps are scaled to [0, 1] regardless of whether the attribution method estimates negative relevance.
  • Sanity checks: Figure 10 shows saliency maps for sanity checks while parameters are randomized from the last layer toward the first.The passage identifies the randomization order but does not state the comparative result.
Loading 1912.09818v7…