Benchmarking Attribution Methods with Relative Feature Importance

Mengjiao Yang; Been Kim

Source-linked AI summary

Benchmarking Attribution Methods with Relative Feature Importance

Mengjiao Yang, Been Kim

arXiv:1907.09701v2Jul 23, 2019cs.LGstat.ML

TL;DR

Quantitative evaluation of feature attribution methods is difficult because ground-truth feature importance is generally unknown and human judgments can be subjective. BAM addresses this with controlled relative feature importance, complementary metrics, and released resources; evaluations show that method rankings depend on the metric and that some methods produce false positive explanations.

Problem
Quantitative evaluation is difficult because absolute feature importance is unknown, while human assessment can be biased and subjective.
Method
BAM combines a semi-natural dataset and models with known relative feature importance and three metrics comparing attributions across models and inputs.
Results
Attribution-method rankings differ across metrics, while GC and VG are least likely to produce false positive explanations in the reported evaluations.
Takeaways & Limitations
BAM provides an open framework for quantitatively evaluating attribution methods and choosing methods or metrics for an application.
Takeaways & Limitations
Performance on BAM is not guaranteed to generalize to real images, although failure on this simpler test may indicate likely failure on harder tests.

Abstract

from arXiv · show

Interpretability is an important area of research for safe deployment of machine learning systems. One particular type of interpretability method attributes model decisions to input features. Despite active development, quantitative evaluation of feature attribution methods remains difficult due to the lack of ground truth: we do not know which input features are in fact important to a model. In this work, we propose a framework for Benchmarking Attribution Methods (BAM) with a priori knowledge of relative feature importance. BAM includes 1) a carefully crafted dataset and models trained with known relative feature importance and 2) three complementary metrics to quantitatively evaluate attribution methods by comparing feature attributions between pairs of models and pairs of inputs. Our evaluation on several widely-used attribution methods suggests that certain methods are more likely to produce false positive explanations---features that are incorrectly attributed as more important to model prediction. We open source our dataset, models, and metrics.

1 Introduction

BAM addresses the difficulty of quantitatively evaluating feature attributions when true feature importance is unknown. It uses controlled relative importance to expose false positive explanations and shows that attribution quality depends on the evaluation metric.

Qualitative explanation assessment can be biased because visually plausible features may have no role in prediction.The paper motivates metrics that detect mismatches between explanations and model rationales.
BAM controls relative feature importance across models and inputs, then quantitatively compares their feature attributions with three metrics.The framework includes a semi-natural dataset and models with known relative importance.
Certain attribution methods produce false positives by assigning higher importance to less important features.The evaluation covers six feature-based and one concept-based attribution methods.
Attribution-method rankings differ across metrics, so the preferred method depends on the application and metric of interest.

2 Related work

Prior attribution evaluations study perturbation sensitivity, feature-removal effects, controlled importance, or human understanding. BAM builds on controlled comparisons while emphasizing quantitative correctness checks that can precede human evaluation.

Prior work evaluates attribution sensitivity, accuracy drops after feature removal, controlled feature importance, and human understanding.
Evaluating sensitivity of explanations: BAM trains models with different labels to create relative importance differences rather than randomizing parameters, and uses semantically meaningful input perturbations.
Evaluating correctness of explanations: Feature-removal evaluations can confound attribution quality with out-of-distribution inputs or changes caused by retraining.BAM enables evaluation on the original model using in-distribution inputs.
Evaluating with knowledge of feature importance: Compared with controlled human-evaluation work, BAM adds quantitative metrics in a finer-grained semi-natural image setting.
Evaluating with humans in the loop: BAM complements rather than replaces human-in-the-loop evaluation by providing an efficient correctness pre-check.

3 BAM dataset and models

BAM constructs semi-natural images and models with known relative feature importance, enabling attribution tests across models and inputs. Its controlled object, scene, and dog-feature settings verify that the intended importance relationships are reflected in model behavior.

BAM combines a semi-natural dataset, models with known relative feature importance, and metrics for quantitative attribution evaluation.
3.1 BAM dataset construction: The dataset pastes MSCOCO objects into MiniPlaces scenes, covering 10 object classes, 10 scene classes, and 100,000 images.Each object-scene class pair has 1,000 images, and every object class appears in every scene class.
3.3 Relative importance of BAM models: Removing object pixels makes object-class accuracy fall to random guessing but leaves scene-class accuracy unchanged, establishing objects as more important to the object model.Conversely, removing scenes harms the scene model while leaving the object model unaffected.
BAM varies dog commonality across 10 scene-class subsets to control the dog feature’s relative importance to scene classifiers.Lower commonality makes the dog feature more important, while higher commonality makes it less important.
Training scene classifiers with fine-grained CFs: Dog-removal accuracy drops generally decrease as commonality increases, confirming that the dog feature becomes less important to prediction.
3.4 Relative importance of BAM inputs: Removing object pixels can increase scene-model accuracy, and 86.7% of correctly classified inputs also gain confidence after removal.This supports testing whether explanations assign lower attribution to the removed object than to the scene pixels it covered.

4 Metrics with/without BAM dataset

BAM evaluates attribution methods through three complementary metrics that compare attributions across models and inputs with known relative feature importance. MCS measures model contrast, IDR tests attribution changes when less-important counterfactual regions are present, and IIR tests attribution stability across functionally similar inputs.

BAM defines three complementary metrics—MCS, IDR, and IIR—to evaluate false positive explanations using relative feature importance.MCS compares two models on the same input, IDR compares two inputs for one model, and IIR compares functionally similar inputs; MCS and IDR require BAM models, whereas IIR applies to any model.
The metrics are intended as inexpensive pre-checks before human evaluations, but they focus only on false positive explanations and are not exhaustive.The threshold used by IIR depends on human subjects and the application.
4.1 Setup: Attribution regions are summarized by average saliency over human-friendly concepts on correctly classified inputs, preventing classifier errors from propagating into attribution evaluation.Feature-based explanations use normalized saliency maps and binary region masks; concept-based methods can supply their own region attribution values.
4.2 Model contrast score (MCS): MCS compares concept attributions between models whose relative importance for that concept differs, with the largest score expected when the concept is important to one model and less important to the other.Training models across different feature-frequency settings produces a spectrum of contrast scores for graded relative importance.
4.3 Input dependence rate (IDR): IDR is the percentage of correctly classified inputs where a counterfactual region receives less attribution than the original regions it replaces.A high IDR indicates more correct attribution of the counterfactual, while 1 − IDR represents the false positive rate for misleadingly emphasizing less-important features.
4.4 Input independence rate (IIR): IIR measures whether attribution to a region changes by less than a human-visibility threshold when a functionally similar input is created by adding a semantically meaningful perturbation.The perturbation δ is optimized to preserve model output, regularized to look natural, and constrained near an initialization such as dog pixels.

5 Evaluation with/without BAM

BAM evaluates attribution methods across model changes, input changes, and functionally similar perturbations using complementary quantitative metrics. Results show substantial differences: GC and VG minimize false positives in some settings, while TCAV and GC achieve the strongest MCS.

5 Evaluation with/without BAM: The metrics are complementary: VG has high IDR and IIR but low MCS, so method choice depends on the evaluation criterion.Applications may prioritize contrast, low false positive rates, or robustness to functionless input changes.
5.1 Attributions between models with MCS: TCAV and GC have the best MCS rankings across evaluations using models with different labels and different CF commonality.The rankings are similar across both MCS evaluation settings.
5 Evaluation with/without BAM: Qualitative saliency-map inspection alone is insufficient to rank methods, motivating quantitative comparisons with MCS, IDR, and IIR.MCS also includes robustness measurements and random-feature baselines.
5.1 Attributions between models with MCS: TCAV and GC most closely follow accuracy-drop trends as dog-CF commonality k increases, whereas other methods change at a much smaller scale.GB changes minimally, with dog edges remaining visible across k values.
5.2 Attributions between inputs with IDR: GC and VG produce the most correctly attributed CFs under IDR, indicating the fewest false positive explanations among applicable methods.VG is the original gradient method and offers nearly the best performance at the lowest computational cost.
5.3 Attributions between functionally similar inputs with IIR: Most methods except GC and VG assign much higher attributions to prediction-irrelevant δ for over 80% of examples under IIR.This effect is especially visible for IG, IG-SG, GB, and GxI-related explanations, whose attributions depend directly on the input.

6 Discussion

BAM differs from perturbation tests by evaluating relative feature importance without assuming a unique set of important features. Its results may not generalize to real datasets, although failure on BAM may indicate difficulty on harder tests.

Assumptions about feature importance: BAM does not assume that individual features have a unique importance set, unlike perturbation-based evaluations that remove highly ranked features and measure accuracy drops.Perturbation tests can be confounded because accuracy changes may reflect out-of-distribution inputs or retraining a different model.
Qualitative comparisons: Figure 7 compares saliency maps with and without a common feature on fs, where the dog common feature is less important than the replaced scene region.
Qualitative comparisons: Figure 9 presents saliency maps for functionally similar inputs x and x + δ, whose attributions can be compared for unwanted sensitivity.
Computational efficiency: BAM trains a fixed number of models once and avoids the small perturbation steps that make many perturbation tests computationally expensive.
Generalization to real datasets: BAM evaluation results are not guaranteed to generalize to natural images, but failure on this simpler test is likely to indicate failure on harder tests.
Quantitative comparisons: Figure 10 reports IIR at t = 10%, with higher IIR better and standard deviations computed over 10 trials of 100 image pairs.

7 Conclusions

The paper concludes that quantitative attribution evaluation can use relative feature importance between models and inputs. It releases the BAM dataset, models, and metrics, while explaining how δ-based testing differs from common-feature training.

7 Conclusions: The work takes a step toward quantitative attribution evaluation using relative feature importance between different models and inputs, supported by released datasets, models, and metrics.
7 Conclusions: Additional regularization penalizes invalid pixel values and limits updates outside the δ region.
7 Conclusions: The update rule uses step size λ, while δ_0 is initialized from a dog image to produce semantically meaningful solutions.
7 Conclusions: The δ optimization and common-feature training differ because δ moves an input with a fixed model gradient, whereas training makes the gradient small with respect to the common feature.This distinction predicts minimal attribution change for δ in input-independence testing but small attribution to dog common features in input-dependence testing.

C Other measures for input independence

The paper also evaluates input independence through average attribution differences after adding δ. Lower attribution difference is better, and this ranking roughly matches IIR.

C Other measures for input independence: Average attribution difference measures how much attributions change when δ is added to the input.
C Other measures for input independence: Lower average attribution difference is better, and Figure 11 shows a ranking roughly matching the IIR metric across saliency methods.The averages are computed over 100 images for each method.

D DNN Architecture and training

BAM models are fine-tuned ResNet50 models trained on the BAM dataset from an ImageNet-pretrained checkpoint. Training uses a 90%/10% train-test split and a single Tesla V100 GPU.

D DNN Architecture and training: All BAM models are ResNet50 architectures initialized from an ImageNet-pretrained checkpoint, with all layers fine-tuned on the BAM dataset.
D DNN Architecture and training: The dataset is split into 90% training data and 10% testing data, and models are implemented in TensorFlow and trained on one Nvidia Tesla V100 GPU.

E Details of interpretability methods compared

The paper compares eight saliency methods, including gradient-, perturbation-, concept-, and activation-based approaches, using their stated attribution procedures and visualization normalizations. It also evaluates TCAV through concept-vector construction, statistical testing, and model-score differences.

GradCAM computes class-logit gradients at the last convolutional feature map, while Guided GradCAM multiplies GradCAM with Guided Backpropagation.
Vanilla Gradient measures each input pixel’s effect through the target-class logit gradient, whereas Gradient x Input multiplies that gradient element-wise by the original image.
SmoothGrad averages explanations over noisy neighboring images, and Integrated Gradients sums gradients along a path from a baseline image to the input; smoothing can also be applied to IG.
Guided Backpropagation attributes input importance by backpropagating neuron activations from the logit layer to the input layer.
For visualizations, selected saliency maps are channel-averaged, capped at the 99th percentile, and normalized to [0, 1].
TCAV learns dog concept activation vectors from 100 positive dog images and 100 negative images, tests scores at ResNet50 block and logit layers, and averages statistically accepted layers.
Model contrast compares TCAV scores between two models for dog and random concept activation vectors, alongside a random-mask baseline for model contrast scores.

H Full size figures

The full-size figures provide visual examples and metric plots for models trained with different contrast factors, relative model contrast testing, and input independence testing.

Figure 12 shows saliency-map examples as k increases from left to right, where greater contrast within each row is better.
Figure 13 plots MCS across contrast factors against the accuracy drop from removing the contrast feature, with TCAV as a comparison and correlations shown below.
Figures 14 and 15 provide additional saliency-map examples from relative model contrast testing, while Figure 16 provides examples from input independence testing.