PaperlayerView exact revision on arXiv

Source-linked AI summary

The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

Giang Nguyen, Daeyoung Kim, Anh Nguyen

arXiv:2105.14944v5cs.CVcs.AIcs.HC

TL;DR

Most attribution methods were evaluated with proxy metrics even though humans are their intended users, leaving their effect on human-AI team performance uncertain. This paper studies that effect on ImageNet and fine-grained Stanford Dogs, finding maps no better than nearest-neighbor examples overall, harmful on the dog task, and poorly aligned with automatic evaluation scores.

  • Problem

    Most attribution methods were assessed with proxy automatic metrics rather than tested with human end-users, and it was unknown whether those scores predict human-AI team performance.

    Source: §1 · p. 1

  • Method

    The authors conduct a user study measuring attribution-map effectiveness on ImageNet and Stanford Dogs classification with natural and adversarial images.

    Source: §1 · p. 2

  • Results

    Attribution maps were no more effective than 3-NN examples on both datasets, hurt human-AI team performance on fine-grained dogs versus AI alone, and automatic metrics correlated poorly with team performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support evaluating explanations with human users and rethinking automatic metrics, while suggesting 3-NN as an alternative worth exploring.

    Source: §5 · p. 10

    The study could not control participants' prior knowledge, leaving explanation utility for people with strong dog expertise unknown.

    Source: §5 · p. 10

Abstract

from arXiv · show

Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications. Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work. However, despite humans being the target end-users, most attribution methods were only evaluated on proxy automatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et al. 2018). In this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine-grained classification, and when an image is natural or adversarial (i.e., contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-AI teams compared to AI alone. Importantly, we found automatic attribution-map evaluation measures to correlate poorly with the actual human-AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-loop applications and to rethink the existing evaluation metrics.

1 Introduction

Attribution maps can expose image-classifier evidence, but whether they help people judge AI decisions—and whether automatic metrics predict that help—remains uncertain. The study finds that maps do not outperform nearest-neighbor examples overall, can hurt teams on fine-grained dog classification, and correlate poorly with team performance by automatic measures.

Motivation

  • Attribution maps highlight pixels that support or oppose a predicted label and have been used for applications such as tumor localization and bias reflection.The paper frames their human usefulness as important because explanations of high-stakes predictions can affect people in social, scientific, and legal contexts.Source: §1 · p. 1

Study scope

  • The study tests attribution maps with human users on ImageNet and fine-grained Stanford Dogs classification, including natural and adversarial images.Participants judged whether the model's top-1 prediction was correct after seeing the image, prediction, and explanation.Source: §1 · p. 2

Main findings

  • Attribution maps did not outperform 3-NN examples in improving human-AI team performance on ImageNet or fine-grained dog classification.On fine-grained dogs, maps instead hurt team performance relative to AI alone.Source: §1 · p. 2
  • On adversarial Stanford Dog images that are very hard for humans to label, confidence scores alone were more effective than the tested explanations.The alternatives included attribution maps and 3-NN examples.Source: §1 · p. 2
  • Automatic measures including Pointing Game, IoU, and WSL correlated poorly with actual human-AI team performance.The authors argue that this weak correlation calls the predictive value of these proxy measures into question.Source: §1 · p. 2

2 Methods

The methods compare human-AI teams across two image-classification tasks, several explanation conditions, and natural or adversarial images. Participants judge whether a model prediction is correct, while the study also measures how automatic attribution-map scores relate to human-team performance.

2.1 Image classification tasks

  • ImageNet and 120-class Stanford Dogs provide classification tasks of differing difficulty, with the dog task chosen to test generalization to fine-grained categories.Dog breeds are similar and vary substantially within classes; lay users are expected to know less about them than everyday ImageNet objects.Source: §2.1 · p. 3

2.2 Image classifiers

  • A pretrained ResNet-34 serves as the classifier for both tasks because its ImageNet pretraining includes the 120 Stanford Dogs classes.The classifier has 73.31% top-1 accuracy on ImageNet.Source: §2.2 · p. 3

2.3 Visual explanation methods

  • The explanation conditions include confidence scores alone, GradCAM, Extremal Perturbation, salient-object detection, and three nearest training examples from the predicted class.The 3-NN images are retrieved by L2 distance in the ResNet-34 layer4 feature space; the study presents this as a user test of post-hoc nearest-neighbor explanations.Source: §2.3 · pp. 3–4

2.4.1 Participants

  • The study recruited lay participants through Prolific and collected more than 466 complete submissions across pilot and main studies.In the main study, 161 ImageNet and 159 Dogs submissions qualified after validation filtering.Source: §2.4.1 · p. 4

2.4.2 Tasks

  • Each participant was assigned one explanation method and answered 40 yes/no trials after training, with validation trials used to filter responses.Only the 30 non-validation trials from qualified submissions entered the analysis.Source: §2.4.2 · p. 4

2.4.3 Images

  • For each dataset, the study sampled 300 natural images across controlled difficulty and correctness bins plus 150 adversarial images.Adversarial examples were generated with PGD at an L∞ bound of 16/255 for 40 steps, then checked to remain misclassified after JPEG saving.Source: §2.4.3 · p. 5

2.5 Automatic evaluation metrics for attribution maps

  • Pointing Game, IoU, and weakly-supervised localization assess overlap between an attribution map and a human-annotated bounding box.The metrics assume an explanation for a predicted label should highlight its annotated region; IoU uses a thresholded heatmap and WSL counts overlap above 0.5.Source: §2.5 · pp. 5–6

3 Results

Across natural and adversarial image classification, nearest-neighbor examples often helped human-AI teams more than attribution maps, while results differed by task and difficulty. Automatic attribution-map scores correlated poorly with team accuracy, and the expert study also favored 3-NN.

3.1 3-NN is more effective than AMs on ImageNet but both hurt user accuracy on Dogs

  • On ImageNet, 3-NN users outperformed all other users by about 3% (76.08% mean per-user accuracy), while on Stanford Dogs, 3-NN and EP users performed worse than users with no explanations.The ImageNet difference was significant versus SOD and GradCAM; on Dogs, visual explanations, including 3-NN and attribution maps, hurt user classification accuracy.Source: §3.1 · p. 6

3.2 On natural images, how effective are attribution maps in human-AI team image classification?

  • On natural ImageNet, teams using heatmaps and confidence reached 88.77% accuracy versus 80.79% for AI-only; on Dogs, only 3-NN teams beat AI-only.On Dogs, confidence-only or confidence-plus-heatmap teams scored 76.45% versus 81.14% for AI-only, while 3-NN teams reached 82.88%; on ImageNet, confidence alone reached 84.79%.Source: §3.2 · p. 6

3.3 On adversarial images, how effective are attribution maps in human-AI team image classification?

  • On adversarial ImageNet, 3-NN remained best at 75.07%, while on adversarial Dogs, confidence scores alone helped users detect AI mistakes more than explanations did.On adversarial Dogs, heatmaps and 3-NN examples tended to make users agree with the model's incorrect decisions.Source: §3.3 · p. 7, Figure 2 · p. 7

3.4 On ImageNet, why is 3-NN more effective than attribution maps?

  • 3-NN was most effective on Easy and Hard ImageNet images, where it could provide contrasting evidence for low-confidence errors or examples that helped confirm correct decisions.It was not best on Medium images, and about 63% of misclassified images in 3-NN teams had debatable ground-truth labels.Source: §3.4 · pp. 7–8

3.5 How do automatic evaluation scores correlate with human-AI team performance?

  • Attribution-map scores correlated poorly with team accuracy: only GradCAM on ImageNet showed small positive correlations, while other tested settings had negligible correlations.For GradCAM on ImageNet, correlations were r = 0.22 for IoU, r = 0.15 for WSL, and r = 0.21 for Pointing Game; other cases ranged from −0.12 to 0.07.Source: §3.5 · p. 8

3.6 Machine learning experts found Nearest-Neighbors more effective than GradCAM

  • Users working with 3-NN achieved mean accuracy of 76.67%, compared with 68.00% for GradCAM users, and 3-NN was consistently more effective on natural and adversarial image subsets.Standard deviations were 2.98% for 3-NN and 8.69% for GradCAM; Table 1 reports mean and standard deviation of per-user accuracy over all images.Source: §3.6 · p. 9, Table 1 · p. 9

4 Related Work

Related work reports that attribution maps can help users in several non-image tasks, while image-domain findings are mixed. The paper distinguishes its evaluation by measuring human-AI team accuracy on standard image classification, including adversarial examples.

Attribution maps improved human accuracy in non-image domains

  • Prior studies found attribution maps improved human performance in text classification, sentiment analysis, hypoxemia-risk prediction, and fake-review detection, but had marginal or negative utility in house-price prediction.The cited tasks include book categories [66], movie-review sentiment [12] [66], hypoxemia-risk, fake reviews, and house prices [21].Source: §4 · p. 9

Evaluating confidence scores on humans

  • Earlier work found confidence scores could improve trust and prediction accuracy, but this study measured team performance rather than user trust.The paper also describes its human evaluation of confidence scores for image classification as, to its knowledge, the first of its kind.Source: §4 · p. 9

Evaluating explanations on humans

  • Prior image studies examined user understanding or error identification, whereas this work measured human-AI team accuracy on ImageNet and compared attribution methods separately.The paper distinguishes its study from work on facial-age prediction [19], user understanding [27], and AI debugging [6].Source: §4 · p. 10

Adversarial attacks

  • The paper reports evaluating users on well-known adversarial examples, unlike a related study of natural adversarial images, and distinguishes its scale, image type, and setup.Earlier work found gradient-based attribution maps could change radically after minute input changes, but had not studied the impact on heatmap end-users.Source: §4 · p. 10

5 Discussion and Conclusion

The authors recommend evaluating explanations with human users and reconsidering automatic metrics after finding weak correspondence with team performance. They also note limits in controlling participants' prior knowledge and study conditions.

Limitations

  • The study's uncontrolled participant prior knowledge limits what it establishes about explanation utility for people with stronger domain expertise.The authors note that dog experts might perform better on fine-grained dog classification and that explanation utility in that case is unknown.Source: §5 · p. 10
  • The online study could not control physical factors such as participants doing other activities, and the authors did not consider state-of-the-art explanation-training methods.They note that each explanation method may require a different optimal training procedure.Source: §5 · p. 10

Discussion and Conclusion

  • The authors call for human-in-the-loop evaluation and reconsideration of automatic metrics, and suggest exploring combinations of 3-NN with attribution maps.They also suggest prototypical and visually grounded explanations may be more effective than heatmaps.Source: §5 · p. 10

A1 Break-down human-AI team accuracy on image subsets

The appendix estimates human accuracy on the original natural-image distributions from controlled-bin results and real bin ratios. Adversarial images are excluded because they have no original distribution.

A1 Break-down human-AI team accuracy on image subsets

A2 How effective attribution maps in improving human-AI team accuracy on the controlled distribution?

On the controlled distribution, 3-NN leads on ImageNet, while explanation methods offer little utility for fine-grained dog classification.

A2 How effective attribution maps in improving human-AI team accuracy on the controlled distribution?

A3 Are attribution maps more effective than 3-NN in improving human accuracy on Easy, Medium, and Hard images of AI on the controlled distribution?

Results differ by task and image difficulty: 3-NN leads on easy and hard ImageNet images, while dog-image explanations show little utility.

A3 Are attribution maps more effective than 3-NN in improving human accuracy on Easy, Medium, and Hard images of AI on the controlled distribution?

A4 Are attribution maps more effective than 3-NN in improving human accuracy on correct/wrong images of the controlled distribution?

The strongest method varies with whether the model’s prediction is correct: 3-NN leads for ImageNet error identification, while dog-image errors remain difficult.

A4 Are attribution maps more effective than 3-NN in improving human accuracy on correct/wrong images of the controlled distribution?

A5 How does prior knowledge affect human accuracy?

The authors examined whether prior knowledge affected human accuracy across known and unknown images. SOD showed the largest accuracy gap, which the authors associate with its lower classifier relevance.

A5 How does prior knowledge affect human accuracy?

A6 AI-only thresholds

The appendix describes study thresholds, sampling and procedures, then details how explanation methods shaped users’ decisions across ImageNet and Dogs tasks. It reports that explanations generally increased acceptance of AI predictions except SOD, while 3-NN helped on several ImageNet cases and no explanation method benefited participants on adversarial Dogs.

A6 AI-only thresholds; A7 Original distributions of ImageNet and Stanford Dogs

A8 Participants’ statistics; A9 Participants’ background and payment rate

A10 Participants’ acceptance-rejection rate

A11 Sample experiment screens; A11.1 Instructions; A11.2 Training; A11.3 Validation and Test; A12 Images mislabeled into non-dog categories (MIND samples); A13 Example explanations displayed to users

A14.1 Hard, real ImageNet images that were correctly labeled by 3-NN users but not GradCAM, EP, or SOD users

  • On hard real ImageNet images, 3-NN helped users gain confidence in AI predictions when heatmaps missed discriminative features or multiple labels seemed plausible.The cases included low-quality images, look-alike objects, partial objects, multiple concepts, and unusual appearances.Source: A14.1 Hard, real ImageNet images that… · p. 27

A14.4 Adversarial ImageNet images that were correctly rejected by 3-NN users but not GradCAM, EP, or SOD users

  • On adversarial ImageNet images, heatmaps could make wrong predictions compelling, while 3-NN contrastive images helped users differentiate the predicted class from the ground truth.The adversarial mislabels were often visually close to the ground-truth classes.Source: A14.4 Adversarial ImageNet images that… · p. 39

A14.5 AI-misclassified Dogs images that are correctly rejected by SOD users but not GradCAM, EP, or 3-NN users; A14.6 Adversarial Stanford Dogs images that were correctly rejected by Confidence-only users but not GradCAM, EP, and 3-NN users

Loading 2105.14944v5…