Source-linked AI summary
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Q. Vera Liao, Daniel Gruen, Sarah Miller
TL;DR
Existing XAI work offers many algorithmic explanation techniques, but how to address real-world user needs remains underexplored. The paper interviews 20 UX and design practitioners using an algorithm-informed question bank, finding varied motivations and practical tensions around explainability. It contributes guidance for connecting user questions with technical capabilities and presents an extended question bank for user-centered XAI.
Problem
XAI research has produced many algorithmic techniques, but how to design explanations that address real-world users’ needs remains insufficiently understood.
Method
The authors interviewed 20 UX and design practitioners and used an algorithm-informed question bank representing explainability needs as prototypical user questions.
Results
The study found varied motivations and question types for explanations, alongside tensions between user needs, available XAI techniques, and other product goals.
Takeaways & Limitations
The extended XAI question bank can support needs specification and discussions connecting user questions with candidate explanation methods for user-centered AI.
Takeaways & Limitations
The findings reflect design practitioners’ views, focus mainly on high-stakes products, and do not claim a complete analysis of user needs or XAI methods.
Abstract
from arXiv · showhide
A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many recognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for understanding AI remains an open question. By interviewing 20 UX and design practitioners working on various AI products, we seek to identify gaps between the current XAI algorithmic work and practices to create explainable AI products. To do so, we develop an algorithm-informed XAI question bank in which user needs for explainability are represented as prototypical questions users might ask about the AI, and use it as a study probe. Our work contributes insights into the design space of XAI, informs efforts to support design practices in this space, and identifies opportunities for future XAI work. We also provide an extended XAI question bank and discuss how it can be used for creating user-centered XAI.
Q. Vera Liao IBM Research AI Yorktown Heights, NY, USA
The paper addresses a gap between algorithm-centric XAI research and the practical work of designing explainable AI products. It develops an algorithm-informed question bank to represent user needs as questions about AI.
- INTRODUCTION: XAI research often prioritizes simplified descriptions of model logic, but explanations may fail to satisfy users when their motivations are ignored.The paper frames explanation as having utility beyond merely describing how a model reaches an output.
- INTRODUCTION: A flu-symptom example illustrates that feature-based explanations may not satisfy doctors or add substantial value to clinical decision support.
- INTRODUCTION: The study investigates challenges faced by UX and design practitioners in creating explainable AI products and identifies gaps between algorithmic work and practice.
- INTRODUCTION: The XAI question bank represents explainability needs as prototypical questions users might ask about an AI system.
- INTRODUCTION: The paper contributes design insights, identifies opportunities for future XAI work, and presents an extended question bank for user-centered XAI.
BACKGROUND
Prior XAI work spans interpretability, explanation evaluation, human-centered explanation principles, and HCI studies of AI interaction. This paper focuses on UX and design practitioners as the bridge between algorithm-centric XAI and user-centered explanations.
- BACKGROUND: XAI research commonly seeks to make the reasons behind machine-learning decisions comprehensible, while broader definitions include information that makes models transparent and understandable.
- BACKGROUND: Explanation evaluation asks whether explanations are good enough and how alternatives compare, supporting the development of interactive or personalized explanations.
- BACKGROUND: Human-centered XAI work highlights contrastiveness, selectivity, social interaction, and attention to abnormal events as characteristics underemphasized by algorithmic approaches.
- BACKGROUND: HCI research has examined explainable interactions across contexts including context-aware applications, recommender systems, and debugging tools.
- BACKGROUND: The paper studies UX and design practitioners rather than end users to explore the undefined design space of XAI and challenges in creating explainable AI products.
- BACKGROUND: Existing AI usability guidelines identify capabilities such as explaining system behavior but offer limited actionable guidance for realizing them in products.
- BACKGROUND: The paper examines how design work connects user needs with increasingly available XAI techniques and how practitioners assess technique suitability.
Question-driven explanations
The paper uses questions as a user-centered representation of explainability needs and maps them to explanation methods supported by current XAI techniques. The resulting framework covers nine categories of questions about AI systems.
- Question-driven explanations: The question-driven approach treats explanations as answers to user- and context-dependent questions that express particular purposes or goals.
- Question-driven explanations: The authors build an XAI question bank from a taxonomy of machine-learning explanation techniques rather than from question taxonomies unrelated to ML.
- Question-driven explanations: The taxonomy focuses on explanation forms presented to users, such as rules, decision trees, and feature importance, rather than underlying model characteristics.
- Question-driven explanations: The framework maps How, Why, Why not, and What if questions to explanation methods, then adds How to be that and How to still be this.
- Question-driven explanations: Table 1 distinguishes questions primarily addressed by an XAI method from questions addressed only in some cases.A global decision-tree approximation may address Why, Why not, and What if for individual instances without covering every instance.
- Question-driven explanations: The framework adds Input, Output, and Performance questions, producing nine explainability-needs categories for understanding AI systems.
STUDY DESIGN
The study interviewed 20 UX and design practitioners about users’ needs for understanding AI across IBM products. An XAI question bank scaffolded discussions of needs independently from existing product capabilities.
- STUDY DESIGN: The study conducted semi-structured interviews with 20 UX and design practitioners working across multiple IBM product lines.Participants included design leads, practitioners on commercial and internal products, and systems in testing.
- STUDY DESIGN: About two thirds of products, 68.8%, provided descriptive explanations about data or algorithms, while 37.5% provided explanations for individual decisions.
- STUDY DESIGN: The question bank scaffolded discussions by enumerating explainability categories and grounding conversations in user questions rather than technical details.
- STUDY DESIGN: Participants reviewed cards for question categories and, when time permitted, prioritized them and explained their ordering.
- STUDY DESIGN: Interviews focused on products that participants knew well and explored when, where, and why users needed to understand the AI.
Analysis
The study analyzed interview data using an XAI question bank to identify covered and uncovered user questions, then extended the bank with additional question forms.
- Analysis: Approximately 1,000 minutes of interviews yielded 607 passages broadly relevant to explainability, coded through open and axial coding.The iterative process produced 24 axial codes.
- Gap analysis: Covered questions were grouped by shared intent to identify new phrasings or forms absent from the original examples.For instance, training-data source questions were distinguished from questions about sample size.
- Gap analysis: Of 46 uncovered questions, 22 were excluded as nongeneralizable, while the remaining 24 produced five additional question forms in the Others category.The remaining questions were iteratively grouped and coded by intent.
RESULTS
Interviews showed that explainability serves diverse user and product goals beyond transparency, including decision support, capability assessment, interaction adaptation, feedback, and ethical responsibility.
- Motivations and utility: Users seek explanations to gain evidence, assess unexpected judgments, improve decision confidence, form causal hypotheses, and sometimes mitigate decision biases.Explanations can connect an AI output to a concrete follow-up action.
- Motivations and utility: Users evaluate AI capability through explanations, including data quality, transferability, alignment with domain knowledge, and operational limitations.These evaluations inform system adoption and ongoing use.
- Motivations and utility: Explanations help users adapt interaction behaviors, such as note-taking, focusing on likely errors, or formulating chatbot questions.The desired adaptation depends on how users work with the AI system.
- Motivations and utility: Explanations can support a feedback loop in which users debug, manipulate data, or correct instances to improve AI performance.This role appears in both debugging tools and systems where users can revise data or labels.
- Motivations and utility: Practitioners also framed explainability as an ethical responsibility when AI tools create risks for individuals or society.This motivation concerns creators’ responsibility to provide an appropriate level of explainability.
- Design implications: The study argues that explanation methods should follow users’ motivations and downstream actions rather than transparency alone.For insight-seeking users, example-based explanations may be more useful than feature-based accounts of model logic.
- Design implications: Practitioners criticized treating AI as a decision-maker whose rationale is sufficient, emphasizing discussion around outputs and support for users’ goals.Clinical decision-support users may need help with what happens around a recommendation rather than how the model derived it.
In quest for human-like explanations
Practitioners compared algorithmic explanations with human communication and emphasized selective, social, adaptive, and interactive explanations tailored to users’ questions and evolving understanding.
- Human-like explanations: Algorithmic explanations can diverge from human explanations because AI features, learned patterns, and confidence measures do not always match human decision practices.Informants viewed numerical confidence as difficult to interpret in ordinary interaction.
- Human-like explanations: Some practitioners modeled explanations on domain experts’ reasoning to align users’ perceptions of AI with established decision-making mental models.One clinical system mirrored how doctors explain information extracted from medical literature.
- Desirable properties: Human explanations were described as selective, balancing sufficient trustworthy information against overwhelming users and acknowledging that AI may not be fully explainable.Selectivity focuses attention on a limited number of causes.
- Desirable properties: Human explanations were also described as social, adapting to users’ backgrounds and to changing questions as understanding and trust develop.Users may move from broad questions to highly specific follow-ups over time.
- Interactive explanations: The selective and social nature of explanation motivates interactive or conversational XAI that supports follow-up questions and closes understanding gaps.The paper proposes a question-driven framework as a viable path to interactive explanations.
XAI: challenges and needs in design practices
Designing explainable AI products requires balancing user needs, technical capabilities, workflow and business constraints, and the complexity of interconnected system components.
- Design challenges: Explainability can conflict with proprietary-data protection, legal or marketing concerns, seamless workflow integration, and users’ reactions to problematic rationales.Greater transparency may expose explanations users find wrong, oversimplified, or outdated.
- Design challenges: Explainable AI products require a holistic approach linking multiple algorithms and system components rather than explaining one model and one output.Users may be uncertain about the broader system context surrounding a model.
- Design challenges: Practitioners face tensions among explainability, system and business goals, along with teamwork, skill, time, and resource constraints.Successful design requires pairing user-centered ideas with what available tools and algorithms can support.
- Support for design practices: The question bank could support explainability-needs specification by laying out prototypical questions users may ask to understand AI systems.The authors call for further study of factors shaping question variability and appropriate answers.
- Analytic approach: The analysis examines explainability-needs variability and gaps between algorithmic explanations and user needs through category codes and design-challenge passages.The study uses the question bank and interview coding to guide this analysis.
- Analytic approach: Decision context and user type are factors associated with the kinds of explainability users may need.Decision context includes criticality, time sensitivity, and complexity; user type includes AI knowledge, domain knowledge, attitude, role, and responsibility.
- Analytic approach: The paper treats these factors as sensitizing concepts rather than conclusions about how user needs vary, underscoring the difficulty of predefining explainability needs.The authors support empirically identifying application-specific needs.
Input/data
Users seek input and data explanations to evaluate AI capabilities, especially during onboarding, and to adapt usage or improve system performance. They also ask about data limitations, biases, missingness, compliance, and transferability.
- Understanding training data helps users evaluate AI capabilities, particularly during onboarding and in decision-making or quality-control roles.
- Data explanations support users who can manipulate inputs to adapt AI usage or improve AI performance.
- Users ask about sample size, bias, critical subgroups, missing data, regulatory compliance, and model transferability to gauge AI limitations.
- Input and output explanations are static explanations that arise mainly during early system use rather than frequent transaction-to-transaction interactions.
- Users want to understand how best to use AI outputs, including their capability scope and effects on other system components, within downstream workflows.
Performance
Users prioritize actionable understanding of AI performance and mistakes over isolated performance metrics. They also need explanations calibrated to decisions, user expertise, and the level of global or local detail required.
- Users often rank performance explanations low, especially without AI backgrounds or in less-critical decisions, because metrics may be difficult to interpret or mismatch real-world performance.
- Performance explanations should clarify whether AI is good enough for a use case and expose limitations in actionable terms.
- The precision-recall trade-off is a deliberate limitation whose relevance depends on the use case and can change users’ actions.
- Global explanations help users evaluate capabilities and build mental models for interacting with or improving AI, especially in quality-control or model-adjustment roles.
- Global explanations must balance detail and workflow fit: users may need confirmation, top features, or feature usage rather than a complete account of overall model logic.
- Users frequently seek explanations of surprising decisions, while common feature-based methods may explain why a prediction occurred without explaining why an expected alternative did not.
What if, How to be–inspecting counterfactual
Counterfactual and system-change questions are less established in products but can help users inspect scenarios, understand capability boundaries, and adapt interactions. The question bank supports needs discovery and mapping user questions to candidate explanation methods.
- What if, How to be–inspecting counterfactual: What-if explanations let users test scenarios, inspect impacts on decisions or costs, and understand the boundaries of system capabilities.
- What if, How to be–inspecting counterfactual: Counterfactual explanation techniques are growing, but feature-influence methods remain concentrated in data-science tools while contrastive and example-based methods are newer.
- What if, How to be–inspecting counterfactual: Users also ask how AI changes and how they can change it, alongside follow-up and terminology questions outside the algorithm-informed categories.
- DISCUSSION: The study identifies gaps in real-world user needs and practitioner support despite extensive XAI research and calls for responsible, transparent AI.
- DISCUSSION: Interactive and human-centered XAI should address questions such as multilevel How explanations, contrastive Why-not explanations, and how or why systems change.
- DISCUSSION: The question bank can guide product heuristics, user research, and card sorting while accommodating product-specific questions and usage contexts.
- DISCUSSION: Designers need shared mappings between user questions and algorithmic capabilities to support technical discussions and select suitable explanation methods with data scientists.
Limitations
The study’s conclusions are bounded by its practitioner-centered sample, high-stakes product focus, rapidly changing XAI field, and single-organization setting. It therefore frames its findings as incomplete while identifying directions for broader, shared work.
- Scope of evidence: The analysis reflects design practitioners’ views rather than a complete account of end-user explainability needs.The authors suggest studying additional AI-product roles, such as data scientists, in future work.
- Scope of evidence: The product sample emphasizes high-stakes tasks, where explainability needs may be greater and XAI practice more advanced.
- Changing technical context: The rapidly advancing XAI field limits claims about the completeness of the methods discussed.Greater practitioner access to XAI techniques may also change the demands and concerns identified in the study.
- Organizational scope: Because all informants worked for one organization, design practices may differ in other companies or organizational settings.The informants nevertheless represented diverse product lines and locations within that organization.
- Future directions: The authors position question-dependent, user-centered explanation design and shared knowledge repositories as opportunities for future XAI collaboration.