PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-Centered Explainable AI (XAI): From Algorithms to User Experiences

Q. Vera Liao, Kush R. Varshney

arXiv:2110.10790v5cs.AIcs.HC

TL;DR

As XAI techniques grow, human-centered approaches address the need to align explainability with people's understanding and use of AI. The chapter surveys technical and HCI work to show how these approaches guide technical choices, expose pitfalls, and develop frameworks for human-compatible XAI.

  • Problem

    The growing collection of XAI techniques has no one-fits-all solution, while assumptions underlying explanations can disconnect from people's actionable understanding.

  • Method

    The chapter overviews XAI algorithms and selectively surveys HCI research that designs, evaluates, and develops conceptual and methodological tools for XAI.

  • Results

    Human-centered approaches help navigate, assess, and expand the XAI toolbox, including by revealing that contrastive explanations can help people identify individual-fairness concerns.

  • Takeaways & Limitations

    Technical XAI choices should reflect users' explainability needs, while empirical research can uncover pitfalls in existing methods and inform new ones.

  • Takeaways & Limitations

    The area remains nascent, and relevant theories from social, behavioral, and information sciences remain to be explored.

Abstract

from arXiv · show

In recent years, the field of explainable AI (XAI) has produced a vast collection of algorithms, providing a useful toolbox for researchers and practitioners to build XAI applications. With the rich application opportunities, explainability is believed to have moved beyond a demand by data scientists or researchers to comprehend the models they develop, to an essential requirement for people to trust and adopt AI deployed in numerous domains. However, explainability is an inherently human-centric property and the field is starting to embrace human-centered approaches. Human-computer interaction (HCI) research and user experience (UX) design in this area are becoming increasingly important. In this chapter, we begin with a high-level overview of the technical landscape of XAI algorithms, then selectively survey our own and other recent HCI works that take human-centered approaches to design, evaluate, and provide conceptual and methodological tools for XAI. We ask the question "what are human-centered approaches doing for XAI" and highlight three roles that they play in shaping XAI technologies by helping navigate, assess and expand the XAI toolbox: to drive technical choices by users' explainability needs, to uncover pitfalls of existing XAI methods and inform new methods, and to provide conceptual frameworks for human-compatible XAI.

1 INTRODUCTION

As AI spreads into high-stakes domains, XAI must address human understanding rather than merely provide technical transparency. This chapter surveys how human-centered approaches help select, evaluate, and extend XAI methods.

  • Motivation: AI’s deployment in high-stakes domains increases the need for people to understand its uncertain and potentially hazardous behavior.The text identifies healthcare, finance, transportation, and criminal justice as examples.
  • Problem: Despite many XAI algorithms and developer toolkits, successful real-world XAI applications remain relatively scarce.The chapter presents this gap as a central challenge for applying existing techniques.
  • Human-centered challenge: Explainability depends on recipients’ perception, knowledge, goals, and ability to use information, so technical transparency alone is insufficient.Making a model fully transparent does not guarantee that people can make sense of the information without becoming overwhelmed.
  • Human-centered challenge: Human-centered XAI centers technical development on people’s explainability needs and defines success through human experience, empowerment, and well-being.The chapter characterizes XAI as both a design challenge and an algorithmic challenge, creating opportunities for HCI and UX contributions.
  • Roles of human-centered approaches: Human-centered approaches help navigate the XAI toolbox by using user research to drive technical choices according to users’ explainability needs.The chapter rejects a one-size-fits-all solution and highlights HCI methods for understanding application-specific needs.
  • Roles of human-centered approaches: Empirical user studies reveal pitfalls in existing XAI methods, while cognitive and behavioral theories provide conceptual tools for human-compatible computational and design frameworks.The chapter also notes that theory-driven XAI remains nascent, with relevant social, behavioral, and information-science theories still underexplored.

2 WHAT IS EXPLAINABLE AI AND WHAT ARE THE TECHNIQUES?

XAI encompasses technical means for making AI understandable, spanning interpretable models and post-hoc explanations for opaque models. Techniques address global behavior, individual outcomes, and counterfactual changes, but each involves different approximations and scope choices.

  • XAI broadly includes technical means for making AI understandable, while scholars continue to debate the boundaries among explainability, transparency, interpretability, and related terms.
  • XAI techniques either use directly interpretable models or apply post-hoc methods to complex opaque models.Interpretable examples include decision trees, rule-based models, and linear regression; opaque examples include deep neural networks and large tree ensembles.
  • Directly interpretable models can achieve comparable performance to opaque models in many contexts, especially with well-structured datasets and meaningful features.
  • Post-hoc explanations commonly target global model logic, individual prediction outcomes, or counterfactual behavior under changed inputs.They may use feature-based explanations to illuminate model internals or example-based explanations to support case-based reasoning.
  • Global explanations approximate opaque models with simpler models or representative examples, while outcome explanations estimate feature importance or retrieve similar cases.Examples include distillation, LIME, SHAP, saliency methods, and example-based justifications.
  • Counterfactual explanations identify changes, often minimizing the changes needed to obtain a different prediction, especially when people seek remedy or recourse.

3 DIVERSE EXPLAINABILITY NEEDS OF AI STAKEHOLDERS

Stakeholders differ substantially in their explainability needs because their knowledge, objectives, tasks, and usage contexts vary. Human-centered approaches therefore use user research and question-driven design to connect XAI techniques with specific needs.

  • There are no one-size-fits-all XAI solutions because stakeholders’ explainability needs vary with their goals and contexts.
  • Stakeholders include model developers, administrators, decision-makers, impacted groups, and regulators, whose objectives range from debugging and adoption to recourse and compliance.
  • User-centered approaches characterize explainability needs through stakeholder knowledge, objectives, and application- or interaction-specific research.Participatory design can compare users’ current mental models with expert-prescribed models to identify what needs explanation.
  • Interviews with 20 designers across 16 machine-learning applications produced an XAI Question Bank containing more than 50 user questions organized into nine categories.
  • The question categories cover global logic, individual predictions, alternative and desired predictions, input changes, performance, data, and output use.
  • Question-driven design uses user research to select XAI techniques and iteratively design experiences, including mappings from questions such as Why or How to Be That to explanation methods.
  • The mapping guidance links user-question categories to methods available in open-source toolkits and includes broader needs concerning performance, data, and output scope.

4 PITFALLS OF XAI: MINDING THE GAPS BETWEEN ALGORITHMIC EXPLANATIONS AND ACTIONABLE UNDERSTANDING

Human-centered XAI research exposes gaps between algorithmic explanations and the actionable understanding people need in realistic contexts. It identifies pitfalls in user goals, cognitive processes, and sociotechnical use, while informing context-sensitive evaluation and design.

  • User goals and evaluation: XAI methods are difficult to assess because their effectiveness depends on users’ diverse goals, contexts, and downstream tasks.Proxy tasks such as simulation tests may not predict effectiveness in realistic tasks such as decision-making.
  • User goals and evaluation: Simulation performance can diverge from appropriate reliance, which requires judging when to trust an AI recommendation and when to remain cautious.Users may need to recognize flawed reasoning, not merely reproduce model predictions.
  • Implications for XAI design: Human-centered evaluation should study interactions in context and develop methods that account for actual user needs in downstream settings.The chapter frames explainability as both a technical and design problem, including communication choices and sociotechnical factors.
  • Context-sensitive explanation needs: Research suggests that explanation needs vary across use contexts, from multi-level model diagnosis to global onboarding and local or counterfactual inspection.Development and debugging may require explanations spanning different levels of model behavior, while unexpected predictions prompt more local inspection.
  • Context-sensitive explanation needs: Contrastive explanations can help people identify individual fairness concerns involving similar individuals from different protected groups.This finding comes from comparisons of four explanation types for fairness judgments.
  • Cognitive pitfalls: Explanations can produce overconfidence, illusory understanding, information overload, and reduced ability to detect model mistakes.Feature-importance explanations hindered mistake detection in one controlled experiment, while explanations lacking useful information could evoke trust similar to real explanations.

5 THEORY-DRIVEN HUMAN-COMPATIBLE XAI

Theory-driven XAI uses theories of human explanation, reasoning, cognition, and interaction to make explanation technologies more human-compatible. These theories motivate interactive and conceptual frameworks, but translating them into computational and design systems remains an open challenge.

  • Theoretical foundations: Human-centered theories can provide conceptual and methodological tools for designing XAI around how people reason, interact, and pursue explanatory goals.The chapter presents theoretical analysis as a way to inspire new computational and design frameworks.
  • Theoretical foundations: Human explanations are often contrastive, selective, social, and shaped by the explainer’s goals and beliefs about the explainee.People select causes using heuristics such as abnormality, intentionality, necessity, sufficiency, and robustness.
  • Theoretical foundations: Malle’s theory separates explanation generation into an information process and an impression-management process involving social goals.Current technical XAI approaches primarily address the information process, leaving impression management comparatively underexplored.
  • Interactive XAI: The social nature of explanation supports interactive XAI in which users ask follow-up questions until they achieve actionable understanding.Conversational and dialogue models offer foundations for explanations relevant to questions and argumentative exchanges.
  • Design frameworks: A human-reasoning framework links XAI methods to explanation goals, reasoning processes, causal explanation types, rational-choice elements, and cognitive biases.Designers can use the framework to identify methods that support users’ goals or address gaps in current XAI approaches.
  • Open challenges: Theory-driven XAI remains nascent because many cognitive, social, and behavioral theories are unexplored and difficult to operationalize computationally.The chapter identifies cross-disciplinary collaboration among AI, HCI, and social sciences as a likely requirement.

6 SUMMARY

XAI is rapidly expanding across technical and human-centered research, while real-world successful examples remain relatively scarce. Because explainability is human-centric and XAI is an interaction problem, the chapter surveys human-centered work that helps researchers navigate, assess, and expand the XAI toolbox.

  • 6 SUMMARY: XAI has rapidly expanded through algorithms, open-source toolkits, industry efforts, public interest, and HCI and social-science research.The chapter situates XAI as a fast-growing area spanning both technical development and human-centered machine learning.
  • 6 SUMMARY: Explainability is a human-centric property, so XAI must be studied as an interaction problem.This motivates the growing role of HCI and UX research in XAI.
  • 6 SUMMARY: A human-centered XAI research community has emerged within a techno-centric field whose technical foundations are already substantially developed.HCI work often needs to challenge existing technical assumptions while building on available AI progress.
  • 6 SUMMARY: The chapter provides a selected overview of this community’s insights and available resources for researchers and practitioners.It focuses on works that use human-centered approaches to design, evaluate, and conceptualize XAI.
  • 6 SUMMARY: The chapter encourages research that bridges design practices with state-of-the-art XAI techniques, uncovers pitfalls, challenges algorithmic assumptions, and develops human-compatible XAI.It also connects these directions to broader challenges in human-centered machine learning.
Loading 2110.10790v5…