PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Better Understanding Attribution Methods

Sukrut Rao, Moritz Böhle, Bernt Schiele

arXiv:2205.10435v1cs.CVcs.LG

TL;DR

Deep networks are difficult to interpret, and the lack of ground-truth importance values makes attribution methods hard to evaluate systematically. The paper proposes DiFull, ML-Att, and AggAtt for faithfulness, fair comparison, and dataset-wide visual inspection, and reports that fair comparisons narrow some method gaps while smoothing improves localization for some methods.

  • Problem

    Deep neural networks are hard to interpret, and without ground-truth importance values attribution methods are difficult to compare and evaluate holistically.

    Source: §1 · p. 1

  • Method

    The paper proposes DiFull to distinguish possible from impossible attributions, ML-Att for same-layer comparisons, AggAtt for dataset-wide qualitative inspection, and a smoothing step for some attribution maps.

    Source: §1 · pp. 1–2

  • Results

    Fair same-layer comparisons make some apparent performance differences vanish, while smoothing significantly improves localization for some methods and results differ across architectures.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Architectural design choices can influence the applicability of an attribution method.

    Source: §1 · p. 2

    In GridPG, images in a grid can influence one another's outputs, leaving it unclear whether some attributions are model-faithful.

    Source: §3.1.1 · p. 3

Abstract

from arXiv · show

Deep neural networks are very successful on many vision tasks, but hard to interpret due to their black box nature. To overcome this, various post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. Evaluating such methods is challenging since no ground truth attributions exist. We thus propose three novel evaluation schemes to more reliably measure the faithfulness of those methods, to make comparisons between them more fair, and to make visual inspection more systematic. To address faithfulness, we propose a novel evaluation setting (DiFull) in which we carefully control which parts of the input can influence the output in order to distinguish possible from impossible attributions. To address fairness, we note that different methods are applied at different layers, which skews any comparison, and so evaluate all methods on the same layers (ML-Att) and discuss how this impacts their performance on quantitative metrics. For more systematic visualizations, we propose a scheme (AggAtt) to qualitatively evaluate the methods on complete datasets. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods. Finally, we propose a post-processing smoothing step that significantly improves the performance of some attribution methods, and discuss its applicability.

1. Introduction

Because deep neural networks are hard to interpret and ground-truth feature importance is unavailable, attribution methods are difficult to evaluate holistically and systematically. The paper introduces three evaluation schemes and reports that fair, same-layer comparisons narrow some apparent performance differences.

1. Introduction

  • The paper targets holistic, systematic evaluation of attribution methods because deep networks are difficult to interpret and ground-truth importance values are lacking.The stated evaluation aims include measuring model-faithfulness, ensuring fair comparisons, and enabling systematic visual inspection.Source: §1 · p. 1
  • DiFull distinguishes possible from impossible attributions by controlling which input regions can influence the model output.This provides ground-truth annotations for whether an input feature could have influenced the output and can reveal distinct failure modes.Source: §1 · p. 1
  • Same-layer evaluation makes some apparent performance differences between attribution methods essentially vanish.ML-Att compares methods at the same network layers to even the playing field.Source: §1 · pp. 1–2
  • AggAtt aggregates attribution maps across many samples to show performance trends across complete datasets, complementing individual examples.The paper notes that comparisons based on individual examples can be skewed and may not represent methods fully.Source: §1 · p. 2
  • Post-processing smoothing significantly improves localization for some attribution methods, with differences across architectures highlighting the influence of architectural design on applicability.The paper proposes smoothing existing IntGrad and IxG attributions.Source: §1 · p. 2

2. Related Work

Prior work uses several mechanisms to produce attributions and evaluates them with different kinds of metrics. The paper situates its evaluation among these approaches and identifies limits of grid-based localization.

2. Related Work

  • Attribution methods use backpropagation, activation weighting, or input perturbation to assign feature importance.Backpropagation methods use input or intermediate-layer gradients; activation methods weight maps; perturbation methods observe output changes after occlusion or mask optimization.Source: §2 · p. 2
  • Evaluation metrics include sanity checks, localization measures, and perturbation-based tests; this work combines localization with sanity-check aspects to assess model-faithfulness.Localization measures compare attributions with bounding boxes or grid cells, while perturbation metrics measure model behavior after input changes.Source: §2 · p. 2
  • Grid-based localization evaluates attribution within class-specific image cells, but does not ensure that only the corresponding cell influences the model output.The paper proposes controlling information flow so grid cells are classified independently.Source: §2 · p. 2

3. Evaluating Attribution Methods

The section introduces evaluation settings that test attribution faithfulness, support systematic qualitative inspection, and compare methods at matched network layers. Together, these settings address limits in existing quantitative and example-based comparisons.

3.1. Quantitative Evaluation: Disconnecting Inputs

  • The quantitative settings complement GridPG’s localization evaluation by testing whether attributions reflect features that could influence the model output.DiFull provides ground truth for regions that can or cannot have influenced that output.Source: §3.1 · pp. 2–3

3.1.1 Grid Data and GridPG

  • GridPG measures the fraction of positive attribution assigned to a class’s grid cell, but shared low-level features can let other cells influence its score.In that case, whether an attribution outside the class cell is unfaithful is unclear.Source: §3.1.1 · p. 3

3.1.2 Proposed Metric: DiFull

  • DiFull disconnects each subimage from other classes’ outputs, guaranteeing that outside features cannot influence its class logit.This provides ground truth for whether a region could have influenced the output and can reveal impossible attributions.Source: §3.1.2 · p. 3

3.1.3 Natural Extension: DiPart

  • DiPart partially disconnects subimages by using local classification heads while retaining influence from other subimages within a head’s receptive field.For models with small receptive fields, its results are very similar to DiFull.Source: §3.1.3 · pp. 3–4

3.2. Qualitative Evaluation: AggAtt

  • AggAtt sorts attribution maps by localization score and averages them within percentile bins, providing a systematic view across samples.Unequal bin sizes emphasize the best and worst cases, where the authors observed significant deviations.Source: §3.2 · p. 4

3.3. Attributions Across Network Layers: ML-Att

  • When attribution methods are compared at the same network layer, apparent performance differences between some methods vanish.ML-Att evaluates methods at the input, an intermediate layer, and the final spatial layer across multiple architectures.Source: §3.3 · p. 4

4. Experimental Setup

Experiments evaluate a diverse set of attribution methods on ImageNet-trained VGG11 and ResNet18, using confidently classified images and multiple evaluation settings and layers.

  • Experiments use VGG11 and ResNet18 trained on ImageNet, selecting validation images classified with confidence of at least 0.99.The authors report similar results on CIFAR10 in the supplement.Source: §4 · p. 4
  • For each of GridPG, DiFull, and DiPart, the study generates 2,000 attributions per method on randomly constructed 2×2 grids.DiFull and DiPart repeat the top-left class in the bottom-right cell to test whether methods highlight class-related features unused by the model.Source: §4 · p. 4
  • Methods are evaluated at the input, middle, and final spatial layers, with lower-dimensional attribution maps upsampled to 448 × 448.The middle and final layers correspond to Conv5/Conv3 x and Conv8/Conv5 x for VGG11/ResNet18, respectively.Source: §4 · p. 4
  • The evaluated methods span backpropagation, activation, and perturbation approaches, including Guided Backpropagation, Grad-CAM, Occlusion, and RISE.Occlusion uses different kernel sizes and strides at the input versus intermediate layers; RISE uses 1,000 random masks per layer.Source: §4 · p. 5

5. Experimental Results and Discussion

Results across GridPG, DiFull, and DiPart vary with layer and method family. Gaussian smoothing improves some input-layer attributions, while AggAtt reveals dataset-level patterns and failure cases.

5.1. Evaluation on GridPG, DiFull, and DiPart

  • On GridPG, input-layer gradient attributions are noisy, but IntGrad and IxG reach localization comparable to Grad-CAM at the final layer; DiFull localization is near-perfect across layers.On DiPart, localization decreases moving backward from the classifier, where receptive fields can overlap adjacent subimages.Source: §5.1 · p. 5
  • Activation-based methods generally improve from input to final layer, except Layer-CAM; global pooling also leads most methods to attribute both repeated-class instances on DiFull.The authors suggest improved localization of later-layer activations may explain the layer trend; Layer-CAM avoids the DiFull issue because it does not pool.Source: §5.1 · p. 6
  • Occlusion performs well across DiFull layers but drops slightly at later layers, while on GridPG its performance improves with depth; RISE performs poorly across settings and layers.The authors link Occlusion’s DiFull drop to overlap between kernels and adjacent subimages, and RISE’s inconsistency to random masks attributing shared pixels equally.Source: §5.1 · p. 6

5.2. Smoothing Attributions

  • On VGG11, smoothing makes S-IntGrad and S-IxG localize significantly better, with performance increasing as kernel size grows; input-layer S-IntGrad with K=257 outperforms final-layer Grad-CAM.Both smoothed methods localize well across settings and layers, though smoothing slightly leaks attributions across grid boundaries on DiFull.Source: §5.2 · p. 7
  • S-IxG’s improvement with larger kernels does not generalize across architectures: it lacks systematic improvement on Resnet18 and VGG11 with BatchNorm, unlike S-IntGrad.The authors propose that BatchNorm randomizes contribution signs, with valence encoded in its biases.Source: §5.2 · p. 7

5.3. Qualitative Evaluation using AggAtt

  • AggAtt visualizations track quantitative trends: IxG and Grad-CAM improve from input to final layer, while S-IxG localizes well at all three layers.Some GridPG cases assign importance to other subimages with class-consistent features, and the authors note that absent ground truth makes it difficult to distinguish model behavior from attribution-method deficiencies.Source: §5.3 · p. 8
  • On DiFull, AggAtt shows near-perfect localization for backpropagation methods, Grad-CAM attribution to both same-class instances, and noisy RISE attributions for roughly half the images.The authors attribute Grad-CAM’s behavior to global gradient pooling and RISE’s noise to random masks.Source: §5.3 · p. 8

6. Conclusion

The conclusion presents DiFull, ML-Att, and AggAtt as complementary ways to evaluate attribution faithfulness, fairness, and qualitative performance. It also states that DiFull cannot determine whether attributions within target grid cells are correct.

6. Conclusion

  • DiFull constrains which input regions can affect classification, identifying regions where any attribution is necessarily unfaithful.It distinguishes possibly correct from impossibly correct attributions, but cannot evaluate correctness within target grid cells because their ground truth is unknown.Source: §6 · p. 8
  • ML-Att compares methods at the same layers, narrowing performance gaps between some methods..Source: §6 · p. 8
  • AggAtt aggregates attribution maps to visualize performance variation systematically and succinctly.The conclusion frames fair comparisons, holistic evaluation, and disentangling model behavior from explanations as useful for better understanding method performance.Source: §6 · p. 8

Supplementary Material

The supplement adds quantitative and qualitative results, method analyses, computational-cost comparisons, and further implementation details. It also reports similar trends on CIFAR10 to those on ImageNet.

Supplementary Material

  • The supplement provides additional ImageNet quantitative results, including complete DiPart results, and complete AggAtt qualitative results with examples.It also reports results on CIFAR10, finding trends similar to ImageNet across the evaluation settings.Source: Supplementary Material · p. 11
  • The supplement analyzes localization-score correlations, smoothing effects, and Grad-CAM similarity to S-IntGrad and S-IxG across AggAtt bins.It also compares the proposed smoothing technique with SmoothGrad.Source: Supplementary Material · p. 11
  • The supplement reports results at all spatial layers for both models, analyzes evaluation-setting computational costs, and provides implementation details..Source: Supplementary Material · p. 11

A. Quantitative Results Including DiPart

The quantitative and AggAtt evaluations compare attribution methods across GridPG, DiFull, and DiPart. DiFull and DiPart show very similar performance, with a small visual difference for backpropagation-based methods and Layer-CAM where DiPart permits partial overlap between grid cells.

A. Quantitative Results Including DiPart

  • DiFull and DiPart performance is very similar across evaluation settings and layers, except for backpropagation-based methods and Layer-CAM, whose DiFull scores are near-perfect.DiPart receptive fields can overlap adjacent cells, producing nonzero gradients there and lower localization scores when moving backward from the classifier.Source: A. Quantitative Results Including DiPart · p. 12, Figure 9 · p. 12
  • Resnet18’s final-layer Gradient and Guided Backprop localization scores are constant because global average pooling gives all activations an equal share of gradients.The final layer is immediately followed by global average pooling.Source: A. Quantitative Results Including DiPart · p. 12, Figure 9 · p. 12

B.1. GridPG

  • On GridPG, input-layer backpropagation maps are noisy, most activation methods localize poorly, and perturbation methods vary across the dataset, as AggAtt reveals.Perturbation methods localize well for about half the dataset but show noisy, poorly localized maps in the remaining bins.Source: §B.1 · p. 13, Figure 10 · p. 13
  • At GridPG’s final layer, IntGrad, IxG, Grad-CAM, and Occlusion localize better, while Gradient and Guided Backprop remain noisy and some activation methods and RISE remain poorly localized.The AggAtt bins across methods and layers reflect the trends in the individual examples.Source: §B.1 · pp. 13–14, Figure 11 · p. 14, Figure 12 · p. 14

B.2. DiFull

  • On DiFull, input-layer backpropagation methods, Layer-CAM, and Occlusion localize nearly perfectly, whereas activation methods localize poorly and RISE remains noisy.RISE can attribute regions sharing random masks with the influential cell; Occlusion is near-perfect because non-overlapping kernel placements cannot affect the output.Source: §B.2 · p. 15, Figure 13 · p. 15, Figure 15 · p. 16
  • On DiFull’s final layer, activation methods apart from Layer-CAM often attribute both same-class cells, while backpropagation methods and Layer-CAM retain perfect localization.Occlusion maps have lower resolution at the final layer because its kernel is relatively larger than the activation map.Source: §B.2 · p. 15, Figure 14 · p. 16, Figure 15 · p. 16

B.3. DiPart

  • DiPart’s visual performance across layers resembles DiFull’s, with only a small boundary region around the target cell receiving extra attributions from partial connectivity.The authors describe DiPart as a natural extension of DiFull that mostly retains its requisite property without being entirely constructed.Source: §B.3 · p. 17, Figure 16 · p. 17, Figure 17 · p. 18

C. Correlation between Attributions

The correlation analysis finds that agreement among attribution methods generally increases at deeper layers. Smoothing input-layer IntGrad and IxG improves their similarity to final-layer Grad-CAM, while its effect varies by model and method.

C. Correlation between Attributions

  • At the final layer, all methods show moderate-to-strong correlations, and diverse methods can perform similarly when compared fairly.Activation-based methods correlate poorly with others at the input layer; their correlations generally strengthen at the middle layer.Source: C. Correlation between Attributions · p. 19, Figure 19 · p. 20
  • 0.80 is the correlation between input-layer S-IntGrad and final-layer Grad-CAM, versus 0.34 for unsmoothed IntGrad.S-IntGrad and S-IxG also correlate well with the best-performing final-layer methods, unlike their unsmoothed input-layer versions.Source: C. Correlation between Attributions · p. 19, Figure 19 · p. 20

D.1. Effect of Smoothing

  • Smoothing IntGrad and IxG can make input-layer explanations individually similar to final-layer Grad-CAM, but the correlation gain is small for Resnet18 S-IxG.The authors suggest Gaussian smoothing averages local attribution values; for piece-wise linear models, IxG sums to the model output apart from biases, and local averaging appears to reduce noise and improve localization.Source: C. Correlation between Attributions · p. 19, Table 2 · p. 19, §D.1 · p. 21

D.2. AggAtt Evaluation after Smoothing

AggAtt visualizations show that smoothing improves localization for IntGrad and IxG as kernel size increases, with the effect weaker for Resnet18 IxG. Smoothed input-layer maps are visually similar to final-layer Grad-CAM across bins, though they can be coarser.

D.2. AggAtt Evaluation after Smoothing

  • Larger smoothing kernels strengthen target-cell attributions for S-IntGrad and S-IxG, while smoothing IxG is relatively ineffective for Resnet18.The AggAtt bins on both VGG11 and Resnet18 reflect the example-level trends.Source: §D.2 · p. 21, Figure 20 · p. 21, Figure 21 · p. 21

D.3. Comparing Grad-CAM with S-IntGrad and S-IxG

  • With K = 129, S-IntGrad and S-IxG visually resemble Grad-CAM across bins, including poorly localized examples, but Resnet18 S-IxG performs relatively worse.Smoothed maps are somewhat coarser than Grad-CAM, particularly in early bins.Source: §D.3 · p. 22, Figure 23 · p. 22

E. Quantitative Evaluation on All Layers

The authors evaluate localization across all convolutional layers and find that the trends match those from the selected input, middle, and final layers. They also compare computational cost and smoothing against SmoothGrad, and report dataset and implementation details.

E. Quantitative Evaluation on All Layers

F. Computational Cost

  • DiFull and DiPart’s CNN-module cost matches GridPG’s, and their classifier-module and total pipeline cost is at most GridPG’s.DiFull processes each grid cell separately, while DiFull and DiPart use one classification head per cell.Source: F. Computational Cost · p. 25

G. Comparison with SmoothGrad

  • Gaussian smoothing improves GridPG localization over SmoothGrad in most tested cases, but not Resnet18 IxG, and can leak attributions to neighboring DiFull cells.SmoothGrad costs more because it averages attributions from multiple noisy samples and is sensitive to noise percentage and sample count.Source: G. Comparison with SmoothGrad · p. 26, Figure 26 · p. 26, Figure 27 · p. 26

H.1. Dataset

  • The evaluation uses 2,000 attributions per method on each dataset, with four subimages arranged in 2 × 2 grids.GridPG uses 500 grid images; DiFull and DiPart use 1,000 grid images and evaluate the top-left and bottom-right locations, which contain same-class images.Source: §H.1 · p. 27

H.2. Models and Attribution Methods

  • Experiments use pretrained VGG11 and Resnet18, with attribution implementations from Captum or the cited method sources and layer-specific Occlusion kernel settings.RISE uses 1,000 random masks; Gradient and Guided Backprop use absolute attributions, and all methods are summed across channels.Source: §H.2 · p. 27

H.3. Localization Metric

The localization score measures positive attribution assigned to a target subimage, with a zero-denominator convention for undefined cases. AggAtt uses that score to order attribution maps, while the CIFAR10 evaluation repeats the study with a modified VGG11 setup.

H.3. Localization Metric

  • When the localization-score denominator is zero, the evaluation sets the score to 0; this can occur when all input attributions are negative.The score uses positive attribution values to assess localization for a subimage.Source: §H.3 · p. 27

H.4. AggAtt Visualizations

  • AggAtt sorts maps by localization score, breaks ties by target-cell attribution sum, and normalizes aggregate maps with a shared factor per method.The tie-break is used when localization scores are equal, including the perfect-localization case observed for backpropagation methods on DiFull.Source: §H.4 · p. 27

I. Evaluation on CIFAR10

  • The CIFAR10 evaluation reports performance trends similar to ImageNet across its GridPG, DiFull, and DiPart results.The section presents quantitative results and AggAtt qualitative results.Source: §I · p. 28

I.1. Experimental Setup

  • CIFAR10 experiments use a modified VGG11 with its last two convolutional layers removed and evaluate input, middle, and final layers on 2 × 2 and 3 × 3 grids.The removed layers leave 4 × 4 activations before pooling at the new final layer; evaluation uses correctly classified images with confidence of at least 0.99.Source: §I.1 · p. 28

I.2. Quantitative Evaluation on GridPG, DiFull, and DiPart

On CIFAR10, quantitative attribution trends resemble those on ImageNet, though localization is generally poorer on 3 × 3 grids. AggAtt visualizations and median examples show the qualitative evaluation across methods and layers.

I.2. Quantitative Evaluation on GridPG, DiFull, and DiPart

  • 3 × 3 grids generally yield poorer localization than 2 × 2 grids, while CIFAR10 methods show quantitative trends similar to ImageNet.The results cover GridPG, DiFull, and DiPart.Source: §I.2 · p. 28, Figure 28 · p. 28

I.3. Qualitative Results using AggAtt

  • AggAtt evaluates 3 × 3 GridPG maps across methods and layers; median examples are shown for input- and final-layer attributions.The illustrated methods include IxG, Grad-CAM, and Occlusion, with trends reported as similar to ImageNet.Source: §I.3 · p. 29, Figure 29 · p. 29, Figure 30 · p. 29

Related papers

Loading 2205.10435v1…