Source-linked AI summary
How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels
Hua Shen, Ting-Hao Kenneth Huang
TL;DR
It is unclear whether machine-generated interpretations help users understand why image classifiers make errors. The paper tests this with crowd workers guessing incorrect labels with or without visual interpretations, and finds that interpretations decreased average accuracy by roughly 10%.
Problem
It remains unclear whether machine-generated interpretations help users figure out why they are getting an error, beyond assessing model fidelity or user trust.
Source: Introduction · p. 2Method
The study showed crowd workers images and correct labels, then asked them to choose the classifier’s incorrect label from five options, with interpretations shown in one condition.
Source: Introduction · p. 2, Method · p. 3Results
Roughly 10%: across two control experiments with 3,800 submissions from 150 online crowd workers, interpretations decreased average guessing accuracy.
Source: Conclusion · p. 5Takeaways & Limitations
For image classification, interpretations are not necessarily useful for understanding model failures and may harm accuracy in similar-appearance or background-correlation errors, so designers should use caution.
Source: Discussion · p. 4The authors hypothesize that limited interpreter representational power, saliency-map format, or poor performance on classifier-failure cases may constrain usefulness.
Source: Discussion · pp. 4–5
Abstract
from arXiv · showhide
Explaining to users why automated systems make certain mistakes is important and challenging. Researchers have proposed ways to automatically produce interpretations for deep neural network models. However, it is unclear how useful these interpretations are in helping users figure out why they are getting an error. If an interpretation effectively explains to users how the underlying deep neural network model works, people who were presented with the interpretation should be better at predicting the model's outputs than those who were not. This paper presents an investigation on whether or not showing machine-generated visual interpretations helps users understand the incorrectly predicted labels produced by image classifiers. We showed the images and the correct labels to 150 online crowd workers and asked them to select the incorrectly predicted labels with or without showing them the machine-generated visual interpretations. The results demonstrated that displaying the visual interpretations did not increase, but rather decreased, the average guessing accuracy by roughly 10%.
Introduction
The paper asks whether machine-generated interpretations help end users make sense of image-classifier errors, a question not answered by fidelity or trust evaluations alone. It tests this by asking workers to guess the model’s incorrect label with or without visual interpretations.
Introduction
- The paper focuses on end users, who may not understand neural-network mechanisms but are strongly affected by system outputs.It examines whether machine-generated interpretations help them make sense of algorithmic errors.Source: Introduction · p. 1
- The study tests post-hoc visual interpretations in image classification, including saliency maps that highlight image regions informative to a predicted label.Post-hoc interpretations are generated outside the model’s workflow; the paper focuses on these rather than interpretable proxy models.Source: Introduction · p. 1
- It remains unclear whether interpretations help users figure out why they are getting an error, beyond reflecting model behavior or boosting trust.Prior evaluations measured fidelity and trust, but those measures do not directly answer the error-understanding question.Source: Introduction · p. 2
- Workers see an image and its correct label, may see interpretations, then choose the model’s incorrect label from five options.The evaluation assumes effective interpretations should help users predict the model’s outputs.Source: Introduction · p. 2, Figure 1 · p. 2
- The paper asks whether interpretations improve identification of predicted labels and when they do or do not help.Source: Introduction · p. 2
Related Work
Prior work developed multiple post-hoc saliency-map approaches and evaluated interpretations for model fidelity and user outcomes. The authors identify a remaining need to study how general users perceive and predict neural-network failure cases.
Related Work
- Post-hoc interpretation methods include gradient-based pixel scoring, intermediate-feature maps, meta-model saliency prediction, and perturbation-based maps.Examples include SmoothGrad, GradCAM, RTS, and ExtremalPerturb, respectively.Source: Related Work · p. 2
- Human evaluations have measured trust, satisfaction, mental-model understanding, and performance in human-computer cooperative tasks.The cited work includes Smith-Renner et al. (2020), Bucinca et al. (2020), Chu, Roy, and Andreas (2020), and Feng and Boyd-Graber (2019).Source: Related Work · p. 2
- The authors identify a need to study how general users perceive and predict neural-network failure cases.Interpretation is framed as human-centric, with related work examining perception of interpretation representations in visual interfaces.Source: Related Work · p. 2
Method
The study applies several post-hoc interpreters to misclassified ImageNet images and asks crowd workers to guess the classifier’s incorrect label. It also categorizes failures into five types for fine-grained analysis.
Method
- The classifier was trained on ImageNet with 78.67% TOP-1 accuracy, and the researchers randomly selected images it misclassified.Source: Method · p. 3
- Three interpreters produced five visualizations per image: three ExtremalPerturb masks, one GradCAM map, and one SmoothGrad map.The ExtremalPerturb maps constrained preserved pixels to 20%, 40%, or 50%; GradCAM used last-convolutional-layer gradient information, while SmoothGrad averaged heatmaps from noisy image samples.Source: Method · p. 3
- Workers saw the image and correct label, were told the algorithm misidentified it, and guessed its incorrect label from five choices.The four distractors were randomly drawn from ImageNet’s 1,000 labels, excluding the correct gold-standard label.Source: Method · p. 3
- The interpretation condition showed maps and explanations of how the algorithm sees the image; the comparison condition did not show interpretations.The study treated crowd workers as general users who need not understand neural networks or be trained to reason about their errors.Source: Method · p. 3
- The authors inspected 1,000 misclassified images and grouped errors into five types for fine-grained analysis.Source: Method · p. 3, Figure 2 · p. 3
- Local-character inference and multiple-object selection describe errors from partial-object cues or choosing another object in the image.Source: Method · pp. 3–4
- Similar-appearance inference and correlation learning describe confusion between visually similar classes or reliance on background context.The correlation example predicts “shower curtain” from bathroom context despite no curtain being present.Source: Method · p. 4
- Incorrect gold-standard labels form the fifth category of error cases.Source: Method · p. 4
Experimental Results
In both experiments, showing interpretations lowered average accuracy when workers guessed the classifier’s incorrect label. The category analyses found lower accuracy for C3 and C4 errors, while C1 and C2 accuracy increased nonsignificantly in Experiment 2.
Experimental Results
- 0.73 vs 0.81 accuracy ([Int] vs [No-Int]) in Experiment 1 and 0.63 vs 0.73 in Experiment 2; interpretations reduced average guessing accuracy in both.The Experiment 1 difference was significant (unpaired t-test, p<0.05); Experiment 2’s difference was also significant (paired t-test, p<0.01).Source: Experimental Results · p. 4, Table 2 · p. 4
- Experiment 2 used another 200 images, showed each image in both conditions, and excluded workers from participating in both conditions.Five workers per image were assigned to each condition.Source: Experimental Results · p. 4
- Both experiments found significantly lower accuracy with interpretations for C3 similar-appearance and C4 background-correlation errors.Source: Experimental Results · p. 4
- In Experiment 2, accuracy increased for C1 and C2 errors with interpretations, but the differences were not statistically significant.Source: Experimental Results · p. 4
Discussion
In image classification, visual interpretations may fail to help users understand model errors and may be harmful for some error types. The authors offer possible explanations and identify limits on the study’s scope.
Discussion
- Visual interpretations may be harmful when errors involve similar-looking items (C3) or mistaken reliance on image backgrounds or scenes (C4).The authors advise system designers to use caution when displaying interpretations to users.Source: Discussion · p. 4
- Possible reasons include insufficient interpreter representational power, saliency maps that do not convey changes clearly, and poor interpreter performance when classifiers fail.The authors present these as hypotheses and say more research is needed to determine why interpretations were ineffective.Source: Discussion · pp. 4–5
- The study used a relatively small sample, three interpretation types shown together, and qualified MTurk workers whose misunderstanding of images or options could add noise.The authors note that 200 images are a small portion of the more than 10,000 images misclassified in the ImageNet validation set.Source: Discussion · p. 5
- The evaluation covered only visual interpretations for image classifiers; whether similar effects generalize to other tasks remains unstudied.The authors call for more research on generalization to other tasks.Source: Discussion · p. 5
Conclusion
The study evaluated machine-generated visual interpretations for general users reasoning about model errors. Across two control experiments, showing interpretations decreased average guessing accuracy by roughly 10%.
Conclusion
- Roughly 10%: across two control experiments, showing interpretations decreased average human guessing accuracy.The experiments included 3,800 submissions from 150 online crowd workers.Source: Conclusion · p. 5