Source-linked AI summary
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Sina Mohseni, Niloofar Zarei, Eric D. Ragan
TL;DR
XAI researchers across disciplines pursue different goals and use different evaluation measures, while accountable AI is increasingly important. The authors review work in machine learning, visualization, and HCI, then present a categorization and a framework pairing design steps with evaluation methods. Their synthesis maps goals for different user groups to evaluation approaches, while the framework remains light on step-specific detail.
Problem
XAI disciplines pursue different goals and use different evaluation measures, complicating the choice and consolidation of design and evaluation approaches as accountable AI becomes more important.
Source: §1 · p. 2Method
Results
The synthesis maps design goals for different XAI user groups to evaluation methods and presents step-by-step design guidance paired with evaluation.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that designing and evaluating XAI systems requires interdisciplinary effort and point to social-science resources for examining social and cognitive aspects of explanations.
Source: §10 · p. 36The framework is light on specific details at each step, leaving room to extend it with more tailored domain guidelines.
Source: §9.7 · p. 36
Abstract
from arXiv · showhide
The need for interpretable and accountable intelligent systems grows along with the prevalence of artificial intelligence applications used in everyday life. Explainable intelligent systems are designed to self-explain the reasoning behind system decisions and predictions, and researchers from different disciplines work together to define, design, and evaluate interpretable systems. However, scholars from different disciplines focus on different objectives and fairly independent topics of interpretable machine learning research, which poses challenges for identifying appropriate design and evaluation methodology and consolidating knowledge across efforts. To this end, this paper presents a survey and framework intended to share knowledge and experiences of XAI design and evaluation methods across multiple disciplines. Aiming to support diverse design goals and evaluation methods in XAI research, after a thorough review of XAI related papers in the fields of machine learning, visualization, and human-computer interaction, we present a categorization of interpretable machine learning design goals and evaluation methods to show a mapping between design goals for different XAI user groups and their evaluation methods. From our findings, we develop a framework with step-by-step design guidelines paired with evaluation methods to close the iterative design and evaluation cycles in multidisciplinary XAI teams. Further, we provide summarized ready-to-use tables of evaluation methods and recommendations for different goals in XAI research.
1 INTRODUCTION
As AI is entrusted with higher-impact tasks, accountability and transparency become more important, while disciplines bring different priorities to XAI. The paper surveys those efforts and maps design goals to evaluation methods in a multidisciplinary framework.
1 INTRODUCTION
- The demand for accountable AI grows as AI takes on more sensitive, socially impactful tasks, making algorithm transparency essential to accountability.The introduction presents XAI as a possible means of explaining decisions and predictions, enabling oversight of adverse effects such as bias or discrimination.Source: §1 · p. 2
- Machine learning, visualization, HCI, and psychology approach XAI with distinct goals, priorities, and areas of specialization.The paper contrasts interpretable-model and ad-hoc-explainer work with visualization tools and HCI research on end-user trust and understanding.Source: §1 · p. 2
- Disciplines use different evaluation measures, from computational analyses in machine learning to human-subject evaluations in HCI and visualization.The authors identify a mismatch between design objectives and evaluation approaches and argue that convergence in goals benefits XAI.Source: §1 · p. 2
- The paper reviews machine learning, visualization, and HCI research to map XAI design goals for different user groups to evaluation methods.It presents step-by-step guidelines paired with evaluation methods and ready-to-use evaluation summaries for multidisciplinary teams.Source: §1 · p. 2
2 BACKGROUND
The background links transparency and explanation to accountability, then surveys auditing, XAI explanations, and existing work across disciplines. It positions this paper as a multidisciplinary account of design goals and evaluation methods paired in a framework.
2 BACKGROUND
- Algorithms affect consequential decisions and can discriminate, while transparency can inform users and support awareness, bias detection, and accountability.The background notes that auditing can investigate algorithm behavior by testing inputs and outputs, but is time-intensive and difficult to scale.Source: §2 · p. 3
2.1 Auditing Inexplicable AI
- Algorithm auditing tests inputs and reviews outputs to detect bias or other unwanted behavior without requiring knowledge of the system's design.The authors characterize this as valuable but time-intensive and not easily scalable to many algorithms.Source: §2.1 · p. 3
2.2 Explainable AI
- XAI systems generate explanations alongside predictions, aiming to help users understand data processing and notice possible bias or malfunction.The figure describes users querying an interface and receiving model predictions and explanations; cited studies examine user awareness, comprehension, trust, and collaboration.Source: §2.2 · pp. 3–4, Figure 1 · p. 4
2.3 Related Surveys and Guidelines
- The authors identify a gap: existing surveys do not provide a comprehensive survey and framework for evaluating explainable machine-learning systems.Their review is limited to computer-science literature while also summarizing relevant peer-reviewed surveys across active disciplines.Source: §2.3 · p. 4
2.3.1 Social Science Surveys.
- Social-science work examines how people formulate and understand explanations, including cognitive biases, social expectations, trust, reliance, and human-XAI performance.The reviewed work also considers explanation quality, satisfaction, understanding, and curiosity.Source: §2.3.1 · pp. 4–5, §2.3.2 · p. 5
2.3.2 Human Computer Interactions Surveys.
- Visualization surveys cover visual-analytic tools for understanding and interacting with machine-learning systems, including an iterative workflow of understanding, diagnosis, and refinement.The cited explAIner system instantiates the three-stage pipeline.Source: §2.3.3 · p. 5
2.3.3 Visual Analytics Surveys.
- Machine-learning surveys classify interpretability techniques for general models and deep neural networks, including convolutional-network visualization and evaluation approaches.One reviewed survey categorizes evaluation approaches to bridge machine learning and HCI.Source: §2.3.4 · p. 5
2.3.4 Machine Learning Surveys.
- The proposed framework extends prior approaches by including interpretability-algorithm design and pairing evaluation methods with each design step.It organizes a multidisciplinary literature review into a unified framework for teamwork; the review process also added literature in three passes.Source: §2.3.4 · p. 6, Figure 2 · p. 6
3 SURVEY METHOD
The authors used an iterative literature survey and coding process to organize XAI design and evaluation research. They reviewed 226 papers and compiled focused digests of papers jointly designing and evaluating XAI systems and of evaluation methods.
3 SURVEY METHOD
- The survey began with 40 XAI papers from HCI, visualization, and machine-learning venues, including work on interpretable models, explanations, interfaces, debugging, transparency, and fairness.The selection also included relevant preprints and workshop discussions.Source: §3 · p. 6
- Selective coding identified 10 attributes, including discipline, paper type, domain, model, modality, explanation type, design goal, evaluation type, targeted user, and evaluation measure.Examples include user reliance as a design goal and task performance as an evaluation measure.Source: §3 · p. 6
- The researchers added 80 papers through upward and downward literature searches, then another 80 from conferences and journals to balance category coverage.The second round used XAI-related keywords and axial coding to organize literature and discuss the categorization.Source: §3 · p. 7
- The review covered 226 papers, with a digest of 42 papers that both designed and evaluated XAI systems and an in-depth analysis of 69 papers on evaluation methods.The reference table was made available online.Source: §3 · p. 7
4 XAI TERMINOLOGY
The survey distinguishes XAI terminology and explanation scope, and relates explanation goals and formats to different users. It also highlights trade-offs in explaining complex models and risks from misleading presentations.
4 XAI TERMINOLOGY
- Transparent AI is treated as the AI-based class of Intelligible Systems, achievable through interpretable or explainable approaches, though terminology varies across disciplines.The survey distinguishes system transparency from the interpretability of internal machine-learning models.Source: §4 · pp. 7–8
4.1 Global and Local Explanations
- Global explanations describe how an overall model works, while local explanations address individual input-output pairs and can support edge-case investigation or debugging.Global examples include model visualizations and decision rules; local methods include saliency and local approximation.Source: §4.1 · p. 9
4.2 Interpretable Models vs. Ad-hoc Explainers
- Interpretability trades off with model size and complexity, while ad-hoc explainers can explain why a prediction occurred but may have uncertain fidelity and omit how it was made.The survey contrasts more interpretable linear models and trees with higher-performing but less understandable complex models.Source: §4.2 · p. 9
4.3 What to Explain
- The survey distinguishes explanations of how, why, why-not, what-if, how-to, and what-else, with example-based explanations potentially misleading when training data lack uniform distribution.What-if explanations explore changes to inputs or models; how-to explanations describe hypothetical adjustments that would yield a different output.Source: §4.3 · pp. 9–10
4.4 How to Explain
- XAI explanations may be visual, verbal, analytic, or multimodal, and similar goals across user groups can require different research objectives, design methods, and implementation paths.The survey also notes that deceptive phrasing or interface design can contribute to inattention, incorrect mental models, or algorithmic anxiety.Source: §4.4 · pp. 10–11, Figure 3 · p. 11
5 CATEGORIZATION OF XAI DESIGN GOALS AND EVALUATION METHODS
The categorization organizes XAI research around design goals and evaluation measures, with user groups serving as an auxiliary dimension. It emphasizes that similar goals across groups may require different research and implementation approaches.
5 CATEGORIZATION OF XAI DESIGN GOALS AND EVALUATION METHODS
- The categorization uses design goals and evaluation measures as its two main attributes, while giving particular attention to evaluation methods as a relatively less explored area.Measures include mental models, trust and reliance, usefulness and satisfaction, human-machine task performance, and computational measures.Source: §5 · p. 11
- The framework pairs XAI design goals with evaluation measures and uses user groups to emphasize the role of end users in system goals.Figure 3 presents this pairing across user groups.Source: §5 · p. 11
- Overlapping goals across user groups do not imply the same research objectives, design methods, or implementation paths.The paper reviews eight design goals by user group and six main evaluation measures and their methods.Source: §5 · p. 12
6 XAI DESIGN GOALS
The paper organizes XAI design goals by intended user and evaluation subject, while noting that group boundaries are organizational rather than exclusive. The goals range from end-user understanding and fairness to model analysis and improvement.
6 XAI DESIGN GOALS
- The survey groups design goals around AI novices, data experts, and AI experts, while noting that goals overlap and priorities depend on the project.It recommends choosing explanation type, scope, and detail in light of the application, goal, and user expertise.Source: §6 · p. 12
6.1 AI Novices
- For AI novices, the goals include transparency, trust and reliance, bias mitigation, and privacy awareness, alongside a trade-off between comprehensible and accurate explanations.The paper describes novices as everyday AI-product users with little or no machine-learning expertise.Source: §6.1 · pp. 12, 14
6.2 Data Experts
- For data experts, XAI supports model inspection, including uncertainty, bias, and failure cases, as well as interactive model tuning and selection.These users apply machine learning to analysis, decision-making, or research without necessarily knowing algorithmic technical details.Source: §6.2 · p. 15
6.3 AI Experts
- For AI experts, interpretability supports insight into learned patterns, while debugging uses explanations and visualization to identify or address model and training issues.The surveyed examples include revising training hyperparameters and feature engineering after reviewing instance explanations.Source: §6.3 · pp. 15–16
- The review also notes uses of explanations beyond the main goals, including dataset-bias detection, adversarial-attack detection, failure prediction, object localization, and knowledge transfer.These additional uses are described as other purposes of machine-learning explanations.Source: §6.3 · p. 16
7 XAI EVALUATION MEASURES
XAI evaluations should match the intended explanation goal, with methods ranging from human studies to computational measures. The section organizes five evaluation measures and summarizes methods used to assess them.
7.1 M1: Mental Model
- Mental-model evaluations probe users’ understanding through interviews, think-alouds, self-explanations, prediction accuracy, and model-failure prediction; completeness can matter more than soundness.Explanations are also studied for how they shape users’ mental models of how AI works.Source: §7.1 · pp. 16–17
7.2 M2: Explanation Usefulness and Satisfaction
- Satisfaction studies assess explanation usefulness and user response; more detail can help in critical scenarios, while greater complexity or viewing costs can reduce satisfaction or benefits.For domain and AI experts, case studies and participatory design can include users directly in design-cycle evaluation.Source: §7.2 · pp. 17–18, Table 4 · p. 18
7.3 M3: User Trust and Reliance
- Trust evaluations use interviews, questionnaires, and context-specific scales, and repeated measures can capture how trust changes as users gain experience.The review did not find direct trust measurement commonly prioritized for data and machine-learning experts, though reliance and continued use are considered.Source: §7.3 · pp. 18–20
7.4 M4: Human-AI Task Performance
- Human-AI task performance can be measured through user success and completion time, classifier outcomes during debugging, and expert case studies of visual analytics tools.The reviewed work also considers model diagnosis, tuning, and training-process understanding as expert tasks.Source: §7.4 · pp. 20–21
7.5 M5: Computational Measures
- Computational evaluation tests explanation fidelity, consistency, correctness, and completeness, and can assess model trustworthiness goals such as fairness, reliability, and safety.Methods include interpretable-model gold standards and human-annotated benchmarks; the former is limited to gold standards generated by interpretable models.Source: §7.5 · pp. 21–23
8 XAI DESIGN AND EVALUATION FRAMEWORK
The framework links XAI design goals to evaluation methods in a nested, iterative process spanning system outcomes, user interfaces, and interpretable algorithms. Its guidelines are illustrated through a multidisciplinary fake-news detection case study.
8 XAI DESIGN AND EVALUATION FRAMEWORK
- The nested framework pairs design choices with evaluation methods across system goals, explainable interfaces, and interpretable algorithms.Design proceeds from system goals inward to user needs and then underlying algorithms, with iterative design and evaluation cycles.Source: §8 · pp. 23–24, Figure 4 · p. 24
- The case study applies the framework to a fake-news detection assistant for non-expert newsreaders, developed by an eight-researcher team spanning HCI, visualization, and AI.The team described an ongoing collaborative design effort over one year.Source: §8 · pp. 24–25
8.1 XAI System Goals Layer
- Establish system goals early around the application and target users, then revisit requirements against evaluation results during design cycles.Prioritize multiple goals and choose an appropriate baseline, since subjective measures can otherwise make progress difficult to track.Source: §8.1 · p. 25
- Choose what to explain by considering user needs alongside algorithmic constraints, using interviews, surveys, observations, and preliminary experiments to narrow options.Global explanations from deep neural networks may be infeasible or incomprehensible, and instance explanations may lack completeness or omit salient features [3].Source: §8.1 · p. 26
8.2 User Interface Design Layer
- Match explanation format and interaction to the audience; evaluate interface usefulness through user understanding, satisfaction, and mental models.Novices may prefer simplified representations, while detailed interactive representations may target experts; evaluation can compare explanation types or explanations with a non-explainable baseline.Source: §8.2 · pp. 28–29
8.3 Interpretable Algorithm Design Layer
- Select and evaluate interpretable algorithms before human-subject interface studies, accounting for model performance, explanation limits, and explainer fidelity.Shallow models can struggle with complex, high-dimensional data, while ad-hoc explainers for black-box models require fidelity validation; unreliable inner-layer explanations may propagate into later evaluations.Source: §8.3 · pp. 30–31
- Dataset cleaning after qualitative explanation review improved model performance, while user mental models later exposed imbalance behind greater confidence in fake-news predictions.The team also lowered the attention-map visualization threshold after users reported negative effects from low-attention keyword explanations.Source: §8.3 · p. 31
9 DISCUSSION
The discussion connects XAI design goals with suitable evaluation methods, emphasizing that these goals overlap across disciplines and require different priorities. It also considers evaluation over time, ground truth, interaction design, framework extension, and limitations.
9 DISCUSSION
- The framework organizes XAI design goals and evaluation methods for different user groups, linking them in an end-to-end design process.The review also identifies the need for interdisciplinary effort in XAI design and evaluation.Source: §9 · p. 31
9.1 Pairing Design Goals with Evaluation Methods
- Evaluation measures should match specific design goals; iterative work can begin with small qualitative studies and progress to larger quantitative studies.Separate measures can capture distinct attributes such as trust, understandability, usefulness, and detail sufficiency; interviews offer depth while remote measures scale.Source: §9.1 · pp. 31–32
9.2 Overlap Among Design Goals
- XAI goals can overlap across disciplines, but differences in user groups and objectives lead to distinct requirements and implementation paths.The framework treats HCI novices, visual-analytics data experts, and machine-learning AI experts as priority groups, not as absolute divisions.Source: §9.2 · p. 32
9.3 System Evaluation Over Time
- Repeated evaluation can track how interaction, changing system behavior, and users’ prior information relate to trust, mental models, and explanation usability over time.The discussion distinguishes time-based evaluation from static measurement and notes that real-world performance variation can shape trust.Source: §9.3 · p. 33
9.4 Evaluation Ground Truth
- Evaluation findings depend on ground-truth choices, including human-study baselines and computational annotations of relevant input features.The review covers no-explanation, no-AI, and placebic baselines, plus segmentation maps, feature annotations, and bounding boxes for computational evaluation.Source: §9.4 · pp. 33–34
9.5 Role of User Interactions in XAI
- Interactive explanations let novices explore model behavior through trial and error, while expert tools support inspection of models, training, and biases.Examples include What-If exploration, parameter control, and interactive selection or filtering of neurons and feature representations.Source: §9.5 · pp. 34–35
9.6 Generalization and Extension of the Framework
- The framework can incorporate existing AI interaction guidelines and encourages review across layers because errors may propagate from algorithms to system outcomes.The discussion pairs expert review of outcomes with user-centered interface evaluation and computational algorithm evaluation.Source: §9.6 · p. 35
9.7 Limitations of the Framework
- A year-long, ongoing case study with eight university researchers informed the guidelines, but further case studies are needed to establish framework usefulness.The framework is intentionally light on step-specific detail, allowing extension with more tailored guidance for particular domains.Source: §9.7 · pp. 35–36
10 CONCLUSION
The conclusion summarizes a review of XAI design goals and evaluation methods, and presents a framework linking them. It offers high-level guidance for multidisciplinary XAI design while leaving detailed interface and interpretability methods outside its scope.
10 CONCLUSION
- The review organizes XAI design goals and evaluation methods, including ready-to-use tables of evaluation methods and recommendations.The categorization arranges the literature by design goals, evaluation methods, and targeted users.Source: §10 · p. 36
- The framework connects design goals with evaluation methods through a model and guidelines for end-to-end XAI system design.The authors describe it as a high-level guideline for multidisciplinary work.Source: §10 · p. 36
- The framework is not intended to cover every aspect of interface design or interpretable machine-learning technique development.The conclusion also points to social-science resources concerning social and cognitive aspects of explanations.Source: §10 · p. 36