Source-linked AI summary
HIVE: Evaluating the Human Interpretability of Visual Explanations
Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, Olga Russakovsky
TL;DR
Interpretability methods have grown, but systematic evaluation of their usefulness to people remains limited. HIVE is a human evaluation framework tested across visual explanation methods in AI-assisted decision settings. The studies find that explanations can foster trust in incorrect predictions, yet users struggle to distinguish correct from incorrect predictions.
Problem
Existing computer-vision evaluation relies on proxy metrics that do not capture end-user usefulness and do not cover prototype-based explanations.
Source: §1 · p. 2Method
HIVE evaluates diverse visual interpretability methods on a common task designed to reduce confirmation bias and prior-knowledge effects.
Source: §1 · pp. 2–3Results
Participants found 60% of explanations for incorrect predictions convincing and achieved only 40% accuracy distinguishing correct from incorrect predictions in a four-option task.
Source: §1 · p. 3Takeaways & Limitations
The findings underscore the need for evaluation methods that rigorously assess the usefulness and effects of explanations.
Source: §6 · p. 14Each study used 50 participants, and the evaluation setup remains far from real-world use; an ideal study would involve domain experts or real-world end-users in context.
Source: §6 · p. 14
Abstract
from arXiv · showhide
As AI technology is increasingly applied to high-impact, high-risk domains, there have been a number of new methods aimed at making AI models more human interpretable. Despite the recent growth of interpretability work, there is a lack of systematic evaluation of proposed techniques. In this work, we introduce HIVE (Human Interpretability of Visual Explanations), a novel human evaluation framework that assesses the utility of explanations to human users in AI-assisted decision making scenarios, and enables falsifiable hypothesis testing, cross-method comparison, and human-centered evaluation of visual interpretability methods. To the best of our knowledge, this is the first work of its kind. Using HIVE, we conduct IRB-approved human studies with nearly 1000 participants and evaluate four methods that represent the diversity of computer vision interpretability works: GradCAM, BagNet, ProtoPNet, and ProtoTree. Our results suggest that explanations engender human trust, even for incorrect predictions, yet are not distinct enough for users to distinguish between correct and incorrect predictions. We open-source HIVE to enable future studies and encourage more human-centered approaches to interpretability research.
1 Introduction
HIVE addresses the lack of human-centered, standardized evaluation for visual explanations by testing their usefulness in AI-assisted decisions across diverse methods. Human studies reveal that explanations can encourage trust in incorrect predictions, while users struggle to distinguish correct from incorrect predictions.
1 Introduction
- Existing computer-vision evaluation often relies on proxy metrics for heatmaps that do not capture end-user usefulness and do not apply to prototype-based explanations.The proxies include deleting image regions or comparing highlighted regions with ground-truth objects.Source: §1 · p. 2
- HIVE evaluates diverse visual interpretability methods on a common task, assessing explanation utility in AI-assisted decisions while reducing confirmation bias and prior-knowledge effects.It also examines how interpretable-by-design models’ reasoning aligns with humans and how users trade off interpretability and accuracy.Source: §1 · pp. 2–3
- The framework was tested in IRB-approved studies with nearly 1,000 participants across GradCAM, BagNet, ProtoPNet, and ProtoTree.The methods span post-hoc and interpretable-by-design approaches, as well as heatmap and prototype explanations.Source: Figure 1 · p. 2, §1 · p. 3
- Participants found 60% of explanations for incorrect predictions convincing, and achieved only 40% accuracy when distinguishing correct from incorrect predictions in a four-option task.These findings point to trust even in incorrect predictions and difficulty using explanations to assess correctness.Source: §1 · p. 3
- Participants preferred models with explanations, but switching to models without explanations required 6.2% to 10.9% higher accuracy.The comparison is framed around participants’ preference for using models with explanations over a baseline without explanations.Source: §1 · p. 3
2 Related work
Related work spans post-hoc and interpretable-by-design methods, local and global explanations, and multiple explanation forms. The paper situates HIVE as a human evaluation approach that covers heatmaps and prototypes and compares methods beyond heatmaps.
Interpretability landscape in computer vision
- The paper studies four methods spanning post-hoc heatmaps, interpretable-by-design heatmaps, and interpretable-by-design prototypes: GradCAM, BagNet, ProtoPNet, and ProtoTree.Post-hoc explanations explain predictions from already-trained models; interpretable-by-design models are intentionally designed with interpretable decision processes.Source: §2 · p. 4
Evaluating heatmaps
- Automatic heatmap metrics lack consensus, while HIVE evaluates how useful heatmaps are to users in AI-assisted decision-making scenarios.Prior work also reported failed sanity checks for several methods.Source: §2 · p. 4
Evaluating interpretable-by-design models
- Interpretable-by-design evaluations often emphasize competitive performance and qualitative examples; this work instead conducts human studies of three such models.It quantifies prior anecdotal observations of misalignment between prototype models and human similarity judgments.Source: §2 · p. 4
Evaluating interpretability with human studies
- Unlike related human studies, HIVE selects the correct prediction from multiple options, omits class labels, and evaluates methods beyond heatmaps.The omitted labels are intended to prevent reliance on prior knowledge; the approach also differs from by showing explanations during output identification.Source: §2 · p. 5
3 HIVE design principles
HIVE evaluates visual explanations in image-classification decisions through falsifiable behavioral tasks, common cross-method comparisons, and human-centered assessment. Its design addresses participant knowledge and method complexity while supporting varied explanations and study settings.
3 HIVE design principles
- HIVE is designed to test explanation utility for identifying model errors, compare different approaches, and assess their practical effectiveness for human users.The target setting is image-classification decision making, where users see a prediction with an explanation and decide whether it is correct or whether to use the model.Source: §3 · p. 5
3.1 Falsifiable hypothesis testing
- The agreement task measures confidence in one prediction-explanation pair, while the distinction task asks users to choose the correct prediction among four and is the main evaluation task.The distinction task is designed to assess whether explanations distinguish correct from incorrect predictions and to mitigate confirmation bias by presenting multiple explanations at once.Source: §3.1 · pp. 5–6
- Fine-grained bird classification and omitted prediction labels reduce reliance on prior knowledge, while omitted ground-truth labels simulate decisions without access to the answer.Fine-grained bird classification is challenging for non-experts; omitting semantic class labels also matters in easier settings where labels could make the task too easy.Source: §3.1 · pp. 6–7
3.2 Cross-method comparison
- HIVE enables comparisons across explanation forms by evaluating methods on a common downstream task, with method-specific interfaces designed around their characteristics.The setup accounts for differences in scenario and digestibility, and aims to present each method favorably.Source: §3.2 · p. 7
- The evaluation adapts to distinct reasoning displays: GradCAM heatmaps, BagNet’s regional evidence maps, and prototype-based ProtoPNet and ProtoTree explanations.For ProtoPNet, participants focus on prototype-image-region matches; for ProtoTree, the 511-node model is simplified to the final two decisions after an introductory walkthrough and warm-ups.Source: §3.2 · pp. 7–8
3.3 Human-centered evaluation
- HIVE combines behavioral assessment of users’ ability to distinguish correct from incorrect predictions with alignment tests of human judgments and prototype-model reasoning.The framework also examines the tradeoff users make between interpretability and accuracy.Source: §3.3 · p. 8
3.4 Generalizability & Scalability
- HIVE is demonstrated across varied interpretability methods and ImageNet and CUB settings; a later study used it for example-based and concept-based explanation research.A GradCAM distinction study cost $70 for 50 participants paid $12 per hour; the authors identify study design and UI development effort and expertise as the main obstacles, mitigated by open-sourcing HIVE.Source: §3.4 · p. 9
4 HIVE study design
The study introduces participants to the method and task, then combines behavioral evaluation with self-ratings and questions about interpretability–accuracy tradeoffs. The tradeoff questions span low-, medium-, and high-risk scenarios.
4 HIVE study design
- Participants consented, could provide optional demographics and machine-learning experience, received a jargon-light method explanation, and saw task previews with correct and incorrect prediction examples.No personally identifiable information was collected, and participants could revisit the method description during the task.Source: §4 · p. 9
- Behavioral tasks assess agreement or distinction, while participants self-rate understanding before and after the task and again after learning their performance.The objective tasks are the study’s core; subjective questions supplement them.Source: §4 · p. 9
- Participants state the baseline model’s minimum accuracy that would persuade them to use it without explanations across low-, medium-, and high-risk scenarios.They also describe their reasons for each choice.Source: §4 · pp. 9–10
5 Experiments
Across six experiments, HIVE evaluates how people use visual explanations in AI-assisted decisions. Participants often trusted incorrect predictions, struggled to identify correct predictions from explanations, and showed a risk-sensitive preference for model accuracy over explanations.
5.1 Experimental details
- The study used 19 IRB-approved studies with 50 participants each, evaluating four methods on CUB and GradCAM and BagNet on ImageNet.The researchers compared study accuracy with chance using one-sample t-tests and used two-sample t-tests to compare groups.Source: §5.1 · p. 10
5.2 The issue of confirmation bias
- 72.4% of correct CUB predictions and 67.2% of incorrect GradCAM predictions were judged correct, illustrating participants’ tendency to trust predictions shown with explanations.Across methods and datasets, participants leaned toward believing predictions were correct, including when they were wrong.Source: Table 1 · p. 11, §5.2 · p. 11
5.3 Objective assessment of interpretability
- 71.2% on correctly predicted CUB samples for GradCAM fell to 26.4% on incorrect samples; overall, explanations were not reliably useful for decision making.Participants performed above the 25% chance baseline on correctly predicted CUB samples, but accuracy remained far from 100%.Source: §5.3 · p. 11, Table 2 · p. 12
- Automatic localization metrics for GradCAM showed near-zero correlation with confidence ratings and no meaningful relationship with distinction choices.For BagNet, participants also struggled to distinguish top-class heatmaps, which looked similar to human observers.Source: §5.3 · pp. 12–13
5.4 A closer examination of prototype-based models
- ProtoTree similarity scores had little relationship with human ratings (Pearson r = 0.06), while ProtoPNet correlations were not significantly negative.Participants scored 86.5% on a basic decision-tree comprehension task, despite not outperforming chance on ProtoTree explanation judgments.Source: §5.4 · p. 13
5.5 Subjective evaluation of interpretability
- Self-rated method understanding averaged between 3.5 and 3.8 on the study’s rating scale, and declined after participants saw their performance (p < 0.05).Several participants said their performance was lower than expected.Source: §5.5 · p. 13
5.6 Interpretability-accuracy tradeoff
- +6.2%, +8.2%, and +10.9% were the average accuracy advantages participants required from a baseline over an explained model in low-, medium-, and high-risk settings, respectively.Participants required a larger accuracy advantage in higher-risk settings.Source: §5.6 · p. 14
6 Conclusion
The paper introduces and open-sources HIVE, a human evaluation framework, and uses it to assess four visual interpretability methods. Its findings indicate that explanations can engender trust even for incorrect predictions, while the study’s sample size and setup limit its scope.
6 Conclusion
- HIVE is introduced and open-sourced as a human evaluation framework for diverse visual interpretability methods, and is used to evaluate GradCAM, BagNet, ProtoPNet, and ProtoTree.The authors say their evaluation reveals that explanations can engender trust even when predictions are incorrect, motivating rigorous assessment of explanation usefulness.Source: §6 · p. 14
- Each study used 50 participants, and the evaluation setup remained far from real-world use; the authors identify contextually situated studies with domain experts or end-users as an ideal.The limited per-study sample size reflected the decision to evaluate four methods, some under multiple conditions.Source: §6 · p. 14