Source-linked AI summary
Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study
Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, Nadia Berthouze
TL;DR
CNN explanations have been widely proposed, but their user evaluation remains limited, especially for complex image-classification models. This paper reports an online between-group study of LRP saliency maps, finding that they highlighted features participants noticed while providing limited predictive help for new images. The authors therefore call for explainable-AI research beyond instance-level explanations.
Problem
User evaluation of post-hoc explanations for complex CNNs, including saliency maps, has been limited.
Method
An online between-group user study tested LRP saliency maps with participants forecasting a trained CNN’s outputs from examples.
Results
Saliency maps helped participants learn some image features the system was sensitive to, but the CNN remained largely unpredictable, with 60.7% prediction accuracy.
Takeaways & Limitations
The findings motivate design and further research on explanation techniques beyond instance-level explanations.
Takeaways & Limitations
The study used only a small number of image classes, one VGG16 architecture, and one LRP saliency-map technique.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) offer great machine learning performance over a range of applications, but their operation is hard to interpret, even for experts. Various explanation algorithms have been proposed to address this issue, yet limited research effort has been reported concerning their user evaluation. In this paper, we report on an online between-group user study designed to evaluate the performance of "saliency maps" - a popular explanation algorithm for image classification applications of CNNs. Our results indicate that saliency maps produced by the LRP algorithm helped participants to learn about some specific image features the system is sensitive to. However, the maps seem to provide very limited help for participants to anticipate the network's output for new images. Drawing on our findings, we highlight implications for design and further research on explainable AI. In particular, we argue the HCI and AI communities should look beyond instance-level explanations.
1 INTRODUCTION
The paper addresses limited user evaluation of saliency maps for CNN image classification through an online study. LRP maps helped participants notice certain image features, but offered limited help in anticipating CNN outputs.
- CNNs are increasingly used in consequential applications, yet many ML algorithms remain difficult to interpret.
- Saliency maps highlight pixels considered important for CNN image classification and are claimed to support interpretation and trust calibration.
- The study evaluated LRP-generated saliency maps in an online user study with 64 participants predicting CNN recognition outcomes from examples.
- The full-factorial 2x2 design varied whether participants saw saliency maps and detailed CNN classification scores.
- Saliency maps influenced participants to notice highlighted features, while it remained unclear whether they diverted attention from other attributes.
- The authors argue that explainable-AI research should examine techniques beyond instance-level explanations.
2 RELATED WORK
Prior work spans feature-attribution methods, mental-model research, and user studies of explanations, but user evaluation of saliency maps for complex CNNs remains limited. The paper positions its study as a bridge between AI/ML and HCI explainability research.
- Feature-attribution methods assign relevance scores to input features; LRP propagates relevance from a neural-network output backward through the network.
- Mental models can support efficient interaction when accurate, but flawed models may produce confusion, misconceptions, and erroneous interactions.
- The paper seeks to bridge relatively isolated AI/ML and HCI research streams by evaluating explanations for highly complex models.
- Existing explainable-ML studies have examined transparency, presentation, explanation detail, over-reliance, and trust across different models and user groups.
- Prior example-based studies found that normative explanations could improve understanding, whereas comparative explanations did not consistently improve perceptions.
- User evaluations of post-hoc explanations for CNNs were identified as rare, with saliency maps not previously evaluated on CNNs or comparably complex models.
3 METHOD
The study used an online between-group design to test whether saliency maps and classification scores help users understand and forecast a complex CNN’s multi-label predictions.
- The study evaluated saliency maps as explanations for a highly complex CNN performing multi-label image classification.
- The full-factorial design independently varied the presence of saliency maps and classification scores between participant groups.
- The design balanced participant numbers, study duration, and variation in experimental factors.
3.1 Materials
Materials combined PASCAL VOC images with a fine-tuned VGG16 classifier and LRP saliency maps. Class-specific thresholds converted CNN scores into detected or missed outcomes.
- The study used the 19,714-image PASCAL Visual Object Classes dataset, selected for popularity and its 20 classes.
- The classifier started from ImageNet-trained VGG16, was fine-tuned on PASCAL VOC 2012, and achieved AP 0.74 on the PASCAL VOC 2007 hold-out test set.
- LRP was selected over LIME because pilot participants preferred its contour-based maps to super-pixel patches; the study used α-β propagation with α = 2 and β = 1.
- Saliency-map examples illustrated correct train recognition through rail-related regions and a false positive associated with rail-like lower-image edges.
- For each object class, a training-set F1-maximizing threshold converted scores from 0 to 1 into detected or missed outcomes.
- The study interface was a Django web application requiring a browser viewport at least 1000px wide and 600px high.
3.2 Tasks
Participants predicted CNN outcomes from similar examples and explained their reasoning by identifying features the classifier might use or ignore. The study used 14 cat and horse images, each paired with 12 training examples, while controlling task difficulty and participant burden.
- Task: Participants predicted whether the CNN would recognise each of 14 hold-out images and listed features they believed the system was sensitive to or ignored.The task included seven cat images and seven horse images.
- Task: Each task image was accompanied by 12 visually similar PASCAL training examples selected using penultimate-layer embedding distance.All participants saw the same task and example images.
- Task: The examples contained six true positives, three false negatives, and three false positives to expose participants to varied classifier outcomes.The outcome labels were correctly assigned, missed, or incorrectly assigned labels, respectively.
- Task: The researchers limited the study to two classes and a 40-minute session because switching classes confused participants and increased task difficulty.This design reduced the number and variety of task images.
- Task: Task images were sampled with mid-range classification scores because low-score images were nearly unpredictable and high-score images easily predictable in pilot studies.The sampling aimed to produce the greatest performance variation.
3.3 Conditions
The experiment crossed whether saliency maps and classification scores were shown, producing four between-group conditions. Saliency maps appeared only for example images, while scores were presented as top-10 score bar charts when enabled.
- Experimental factors: The study manipulated two independent variables: saliency maps shown versus omitted, and classification scores shown versus omitted.Both factors had two levels.
- Saliency maps: Saliency maps were displayed next to each example image for the relevant class, not next to the task image.This distinction defined how participants received visual explanation information.
- Classification scores: When enabled, classification scores appeared as a bar chart of the top 10 CNN scores beside each example image.The study tested whether numerical information complemented or interacted with saliency maps.
- Conditions: The full-factorial design produced four conditions, including a baseline with neither saliency maps nor scores shown.The other conditions independently added scores, saliency maps, or both.
4 RESULTS
Saliency maps improved outcome-prediction accuracy and shifted participants toward mentioning highlighted object features, whereas classification scores had no significant effects. Prediction success remained difficult, and accuracy varied substantially by classifier outcome type.
- 4.1 Outcome prediction accuracy: Prediction success rates were relatively low across conditions, indicating that the forecasting tasks were challenging.The figure caption characterizes performance as difficult despite the saliency-map benefit.
- 4.1 Outcome prediction accuracy: 60.7% versus 55.1%: saliency maps significantly increased participants’ prediction accuracy compared with no maps.The main effect was F(1,60) = 4.191, p = 0.045, η2 = 0.063.
- 4.1 Outcome prediction accuracy: Classification scores produced no significant main effect on prediction accuracy, and scores did not interact significantly with saliency maps.The score effect was F(1,60) = 1.938, p = 0.169; the interaction was F(1,60) = 0.060, p = 0.807.
- 4.2 Confidence: 79.4% for true positives, 46.9% for false positives, and 36.7% for false negatives: accuracy was highest for true-positive examples.Participants found false-negative and false-positive outcomes harder than true-positive outcomes.
- 4.3.2 Mixed-Method Analysis of Answers.: The feature analysis excluded two participants and omitted responses about ignored features because many participants misunderstood that question.The analysis therefore focused on features participants thought the classifier was sensitive to.
- 4.3.2 Mixed-Method Analysis of Answers.: Saliency maps shifted attention toward specific object parts such as legs, outline, tail, and belly, while maps-absent responses more often mentioned background and colour.The qualitative analysis grouped localized object features as Saliency-Features and generic image properties as General-Attributes.
- 4.3.2 Mixed-Method Analysis of Answers.: 83.9% versus 54.6%: saliency maps significantly increased the share of mentioned Saliency-Features relative to General-Attributes.The effect was F(1,58) = 23.427, p < 0.01, η2 = 0.295; scores had no significant main or interaction effect.
5 DISCUSSION
Saliency maps helped participants notice highlighted image features and modestly improved classifier-outcome predictions, but they offered limited support for understanding model behavior globally. The discussion therefore emphasizes complementing instance-level explanations with global feature information and better example-selection strategies.
- 5.1 The utility of saliency maps exists, but it is limited: Saliency maps improved prediction accuracy, but success remained low at 60.7%, leaving new-image prediction challenging.Self-reported confidence was unaffected by saliency maps.
- 5.1 The utility of saliency maps exists, but it is limited: Participants struggled especially with classifier errors, achieving 46.9% on false positives and 36.7% on false negatives.They predicted the system would be correct in 67.3% of cases, although it was correct in only 42.9%.
- 5.1.1 Reasoning on Examples: False-negative saliency maps often highlighted little evidence, limiting their usefulness for explaining why an image was not recognized.The utility of saliency maps therefore varied with the classification outcome.
- 5.1.1 Reasoning on Examples: Choosing representative saliency-map examples remains under-explored, and future research should examine how sampling strategies affect users’ interpretations.Users can inspect only a limited number of maps, making example selection consequential.
- 5.2 Saliency maps can help participants notice features: Saliency maps increased attention to highlighted features such as horse legs, outlines, tails, and bellies, relative to general attributes.General attributes included background and colour.
- 5.2.1 Facilitating global model understanding by explaining local features: Noticing local features did not ensure global model understanding because saliency maps visualize pixel importance for only one image.Transferring feature knowledge across orientations, scales, and forms remained difficult.
- 5.2.1 Facilitating global model understanding by explaining local features: The authors recommend complementing saliency maps with global measures of feature sensitivity, such as how strongly a nose relates to predicting the cat class.Such measures could help users assess feature importance beyond individual instances.
- 5.2.1 Facilitating global model understanding by explaining local features: General attributes such as colour and contrast can influence classification but are usually not localized by saliency maps, potentially narrowing users’ attention.The authors suggest pairing saliency information with global image descriptors such as contrast or brightness.
6 LIMITATIONS
The study’s design introduced limitations concerning the number of image classes, the model and saliency-map technique, outcome types, and participant expertise.
- The study considered a small number of image classes, limiting how deeply participants could learn class-specific behaviour.The authors propose evaluations lasting several days or weeks with larger, multi-class datasets.
- Results may differ with another combination of CNN architecture and saliency-map technique because the study used VGG16 and LRP.
- The analysis could not assess performance across true-positive, false-negative, false-positive, and true-negative outcomes.True negatives were not included because the tasks were fully counterbalanced.
- Participants had technical backgrounds, but their level of machine-learning expertise was not controlled.The authors suggest repeating the study with machine-learning experts and lay users.
7 CONCLUSION AND FUTURE WORK
The study found that LRP saliency maps helped users learn some image features relevant to CNN decisions, but offered limited support for predicting outputs on new images. The authors therefore call for explanation methods beyond instance-level explanations and continued research on explainable AI.
- Saliency maps helped users learn about some image features to which the CNN was sensitive, but the model remained largely unpredictable.Overall prediction accuracy was 60.7%.
- Prediction accuracy remained below chance for misclassified images: 43.8% for False Negatives and 49.2% for False Positives.
- Instance-level explanations alone cannot provide a solid understanding of how the CNN classifies images.Saliency maps highlight features localisable to pixel regions but do not quantify their importance for future classifications.
- Making AI explainable remains an open technical challenge requiring further studies and collaboration between HCI and AI researchers.