Source-linked AI summary
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
Shane T. Mueller, Robert R. Hoffman, William Clancey, Abigail Emrey, Gary Klein
TL;DR
The review addresses the need for explanations in modern AI systems and examines what makes explanations effective for their recipients. It synthesizes prior work on explanatory systems, psychological theories, and XAI-relevant studies, highlighting evidence that scaffolding can support sound mental models and that rationales influence expert-system use.
Problem
Modern deep-net and machine-learning systems create a need for explanations, including explanations that their recipients can understand.
Method
The paper integratively reviews computer-science work, psychological theories, and selected studies relevant to explainable AI.
Results
With scaffolding, participants quickly built sound mental models of a music recommender, whereas participants without scaffolding did not do so over five days.
Takeaways & Limitations
An explanation has explanatory value for particular individuals, and rationales can affect whether physicians follow expert-system advice.
Takeaways & Limitations
The review is bounded by the vastness of the relevant literature, and it notes that layperson understanding may not be possible in some cases.
Abstract
from arXiv · showhide
This is an integrative review that address the question, "What makes for a good explanation?" with reference to AI systems. Pertinent literatures are vast. Thus, this review is necessarily selective. That said, most of the key concepts and issues are expressed in this Report. The Report encapsulates the history of computer science efforts to create systems that explain and instruct (intelligent tutoring systems and expert systems). The Report expresses the explainability issues and challenges in modern AI, and presents capsule views of the leading psychological theories of explanation. Certain articles stand out by virtue of their particular relevance to XAI, and their methods, results, and key points are highlighted. It is recommended that AI/XAI researchers be encouraged to include in their research reports fuller details on their empirical or experimental methods, in the fashion of experimental psychology research reports: details on Participants, Instructions, Procedures, Tasks, Dependent Variables (operational definitions of the measures and metrics), Independent Variables (conditions), and Control Conditions.
Executive Summary
The Report highlights key ideas and research relevant to XAI, including explanation-as-mental-models.
- The Report highlights research and key points that are particularly relevant to XAI.It frames explanation-as-mental-models as a promising perspective for XAI evaluation.
- Tasks involving prediction of an AI's determinations, combined with post-experimental interviews, hold promise for evaluating XAI.
Disclaimer
The Report addresses explainability challenges as AI systems become more complex, pervasive, and autonomous, while synthesizing diverse literatures around what makes explanations useful. It emphasizes that explanation quality depends on the recipient's understanding and goals, and that the review is selective rather than exhaustive.
- Challenges and motivation: Contemporary AI systems are more complex, less interpretable, more pervasive, and increasingly autonomous, making justification of their decisions more crucial.
- Limitations and risks: A central limitation is that some algorithms are confidential or legally secret, making their biases difficult for outsiders to identify.
- Scope: The Report focuses on AI systems that make determinations or reach conclusions requiring explanation to users.
- Review approach: The review draws on computer science, philosophy, psychology, and human factors, along with research on causal reasoning, abduction, and concept formation.
- What makes an explanation useful: Learners' goals determine what counts as a good explanation, and explanatory value depends on its effect for particular individuals.
- Review approach: The Report synthesizes concepts across publications and disciplines rather than exhaustively summarizing every individual publication.
- What makes an explanation useful: Explanation matters because recipients may need to understand the system, concepts, or knowledge conveyed by an explanation.
- Implications: Explanations should facilitate transfer of concepts and principles, while explanation systems can help users make decisions and take actions.
4. Key Papers and Their Contributions that are Specifically Pertinent to XAI
The review identifies key XAI contributions spanning explanation taxonomies, human evaluation, expert-system history, user mental models, and adaptive, user-centered explanation design.
- The review organizes XAI-relevant work across historical explanation systems, psychological theories, empirical evaluations, and contemporary explainability challenges.
- Human-subjects studies evaluate users’ mental models by testing predictions about machine classifications and scoring participants’ reasons.These studies also examine how explanations change users’ mental models.
- Several contributions argue that explanations should be taxonomized by properties, grounded in causal reasoning and abduction, and designed for users rather than developers.
- Empirical findings show that explanations can affect adoption, reasoning, and reliance, while users may accept explanations containing flaws or gaps.Effects can depend on user skill, and explanations may support adoption without ensuring satisfaction with decisions.
- Explanation is treated as a problem-solving and discourse process rather than merely replaying internal reasoning chains.Moore and Swartout emphasized text planning, natural-language discourse, and relations between data and interpretations.
- The review highlights adaptive explanation strategies that model the human-AI system, track user knowledge, support exploration, and tailor explanations to users and dialogue settings.It also recommends clarifying explanation goals and testing each goal in an appropriate context.
7. Synopsis of Key XAI Concepts
Good explanations are judged by their explanatory value for particular users, shaped by context, knowledge, beliefs, and goals. In XAI, they should support effective use, prediction, error understanding, trust calibration, and appropriate reliance.
- Explanation and Justification: An explanation is an interaction among offered material, learner knowledge and beliefs, context, and learner goals.Its explanatory value is therefore not a fixed property of text, diagrams, or other media.
- Explanation Goodness and Explanation Satisfaction: The goodness of an explanation depends on the beneficiary’s use context and goals, while simple clear explanations can be more satisfying than formal accounts of system operations.Formal model accuracy is not equivalent to user-perceived goodness or explainability.
- Human Learning and Interaction: Explanation is a co-adaptive tuning process in which explainer and learner take each other’s perspectives, and self-explanation can improve learning.User experience with AI can also help people discount initial misconceptions, including among users without technical knowledge.
- Explanation Goodness: Successful explanations help users use AI effectively, select among models, predict system behavior, and explain errors.Prediction should include both correct and incorrect cases, including failures and anomalies.
- Explanation Goodness: Successful explanations also help users assess correctness, recalibrate trust, explain AI behavior to others, and recognize competence boundaries.These functions connect explanation with informed reliance rather than simple acceptance.
Performance Evaluation Using Human Participants
The review frames XAI evaluation around human mental models, performance, trust, and reliance, while emphasizing empirical evaluation of the joint human-AI system. It highlights interactive tasks and fuller reporting of experimental methods as priorities.
- Evaluation framework: The DARPA framework places mental models, performance, and trust or reliance among the central classes of XAI evaluation measures.It also presents explanations as supporting improved mental models and performance, which contribute to appropriate trust.
- Mental models: Mental models can be assessed through knowledge tests, system predictions, generative exercises, diagrams, or explanations of how the system works.These measures target users’ knowledge about the system rather than only subjective workload or satisfaction.
- Performance: Performance evaluation may target the system, the human user, or the effectiveness of their joint work, and can involve multiple dimensions.An explainable system may operate more slowly while increasing the likelihood of success.
- Method reporting: The review recommends that AI/XAI reports provide fuller details about participants, instructions, procedures, tasks, dependent variables, independent conditions, and controls.This recommendation follows the reporting style of experimental psychology research.
- Interactive evaluation: Tasks involving human-AI interactivity, co-adaptation, prediction of AI determinations, and post-experimental interviews are presented as promising for studying explanation and mental models.Bug or oddity detection is identified as one example of an interactive evaluation task.
APPENDIX Evaluations of XAI System Performance Using Human Participants
Human-participant evaluations examine how explanations affect task performance, trust, reliance, learning, mental models, and debugging outcomes across varied AI systems.
- Participants who could ask “why” questions about computer recommendations performed better on a river pollutant search task.
- Reliable decision aids were rated as more trustworthy, but explanations of why errors occurred increased reliance after participants observed errors.
- Explanation research also examined collaborative dialogue, user experience and knowledge, learning transfer, and prediction correction across human-participant studies.
- Explanation widgets helped users find assistant mistakes more effectively, with the Confidence widget revealing significantly more incorrect classifications.
- Scaffolding produced significantly better comprehension-test performance, while initial instruction was sufficient to support good mental models over five days.
- Explanatory debugging improved users’ understanding of classifier operation by about 50% and improved classifier performance by 10% overall.