Source-linked AI summary
Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach
Upol Ehsan, Mark O. Riedl
TL;DR
Complex, consequential AI systems create a need for explanations that are understandable and appropriate to the people using them. The paper introduces HCXAI and a reflective sociotechnical approach, illustrated through rationale-generation studies showing co-evolution between technical development and human-factor understanding. It concludes by identifying research directions and design practices for addressing social context and intellectual blind spots.
Problem
Existing XAI must better account for who the human is, including users’ values, interpersonal dynamics, and socially situated contexts.
Method
The paper develops HCXAI through a reflective sociotechnical framework and a case study of natural-language rationale generation for non-technical end-users.
Results
The case study shows that technical advancements and understanding of human factors co-evolve, while generated rationales achieved the highest user satisfaction among the evaluated agents.
Takeaways & Limitations
Reflective HCXAI can support value-sensitive design and evaluation while opening research spaces beyond narrowly technical, one-to-one human-computer interactions.
Takeaways & Limitations
Reflective sociotechnical work requires sustained engagement with partner communities and practitioners able to bridge multiple domains.
Abstract
from arXiv · showhide
Explanations--a form of post-hoc interpretability--play an instrumental role in making systems accessible as AI continues to proliferate complex and sensitive sociotechnical systems. In this paper, we introduce Human-centered Explainable AI (HCXAI) as an approach that puts the human at the center of technology design. It develops a holistic understanding of "who" the human is by considering the interplay of values, interpersonal dynamics, and the socially situated nature of AI systems. In particular, we advocate for a reflective sociotechnical approach. We illustrate HCXAI through a case study of an explanation system for non-technical end-users that shows how technical advancements and the understanding of human factors co-evolve. Building on the case study, we lay out open research questions pertaining to further refining our understanding of "who" the human is and extending beyond 1-to-1 human-computer interactions. Finally, we propose that a reflective HCXAI paradigm-mediated through the perspective of Critical Technical Practice and supplemented with strategies from HCI, such as value-sensitive design and participatory design--not only helps us understand our intellectual blind spots, but it can also open up new design and research spaces.
1 Introduction
The paper frames explainability as an HCI problem requiring attention to who the human is and to AI systems’ social context. It proposes HCXAI as a reflective sociotechnical approach combining technical and social elements.
- 1 Introduction: XAI provides human-understandable justifications for AI outputs, while explanation generation is a post-hoc interpretability approach focused on accessible information rather than fully elucidating model operation.The distinction separates model interpretability from explanations designed for practitioners and users.
- 1 Introduction: Explainability requirements depend on who the human user is, affecting data collection and how an AI system’s rationale should be described.Engineers and riders, for example, may require different explanations from a self-driving car.
- 1 Introduction: HCXAI centers technology design on a holistic account of the human, including values, interpersonal dynamics, and the socially situated nature of AI systems.The approach responds to the paper’s concern that technical XAI discourse can lose the human side of the problem.
- 1 Introduction: The paper advocates reflection on implicit values and practices by incorporating both social and technical elements into HCXAI design and evaluation.Reflection is intended to make unconscious values and practices consciously actionable for designers and stakeholders.
- 1 Introduction: A case study examines how technological development and understanding of non-expert users’ perceptions of automatically generated rationales evolve together.The paper uses the case study to motivate future sociotechnical research directions and a reflective HCXAI paradigm.
2 Case Study: Rationale Generation
The case study traces rationale generation as technical development and understanding of human factors co-evolve. Across two phases, generated rationales progressed from technical feasibility toward human-centered evaluation of perception, preference, and mental-model formation.
- Approach: Rationale generation produces natural-language explanations for an agent’s behavior, translating internal states and actions into real-time rationales.The approach treats rationale generation as post-hoc explanation generation and uses neural machine translation for fast responses.
- Implications: The case study presents technology development and human-factors understanding as mutually co-evolving, motivating a turn toward sociotechnical HCXAI research.The authors position the case study as a foundation for new research directions.
- Phase 1: Technological Feasibility & Baseline Plausibility: Phase 1 established technical feasibility using a semi-synthetic corpus of 225 action-rationale pairs from 12 participants.Participants provided think-aloud data, which was combined with procedurally generated sentences grounded in gameplay behavior.
- Phase 1: Technological Feasibility & Baseline Plausibility: The Phase 1 system significantly outperformed random and majority-classifier rationales across environments with different obstacle densities.Evaluation combined procedural BLEU scoring with human assessment; 53 participants rated the system’s rationales highest in satisfaction.
- Phase 1: Technological Feasibility & Baseline Plausibility: Phase 1 human evaluation identified explanatory power, relatability, and understandability as important components of user satisfaction and confidence.These findings informed the design of better human-centered evaluations in the next phase.
- Phase 2: Technological Evolution & Human-centered Plausibility: Phase 2 varied neural-network input configurations to generate different rationale styles and conducted user studies comparing their plausibility and perception.The studies examined confidence, human-likeness, adequate justification, understandability, and contextual preferences.
- Phase 2: Technological Evolution & Human-centered Plausibility: Users perceived the intended differences between rationale styles and, when context permitted, preferred detailed rationales for forming a stable mental model of agent behavior.The findings connected upstream neural configuration changes with user-perceived rationale differences and preferences.
3 What’s Next: Turn to the Sociotechnical
The case study motivates sociotechnical research that examines how users’ backgrounds and collaborative social settings shape XAI perception and use. It also cautions that controlled studies provide formative insights, while Frogger findings require further empirical and theoretical work before transfer to real-world systems.
- Scope and formative value: The Frogger case study provides formative understanding of technical and human factors, but its findings require further work before transfer to real-world systems.The authors frame controlled evaluation as a first step for consequential sociotechnical systems.
- Research agenda: The identified research areas are preliminary rather than exhaustive, and their depth became visible through multiple formative phases of the case study.The paper highlights perception differences and social signals as challenges and opportunities revealed by the case study.
- Perception differences due to users’ backgrounds: Future research should examine how professional and epistemic backgrounds affect perception of the same XAI system and preferences for its explanations.The proposed method compares related groups receiving the same explanation, such as engineers and lay drivers of self-driving cars.
- Social signals and explanations: HCXAI research should extend beyond isolated one-to-one interactions to social signals and explanations in team-based collaborative decision-making.The paper notes that consequential AI systems are commonly embedded in organizational settings involving teams.
- Social signals and explanations: A proposed between-subject study would compare technical explanations alone with technical plus social signals and measure confidence in acting on VM right-sizing recommendations.The scenario involves costly errors: underestimating can overload and crash systems, while overestimating wastes resources.
- Sociotechnical framing: Ignoring socially situated systems yields only a partial account of the human, because organizational culture, assumptions, and biases can shape confidence beyond technical explainability.The authors argue that understanding these social factors may matter to explanation adoption as much as the technology itself.
4 Human-centered XAI, Critical Technical Practice, and the Sociotechnical Lens
The paper frames HCXAI as a reflective sociotechnical approach that brings human factors, values, and marginalized perspectives into XAI design. Critical Technical Practice and HCI strategies help question assumptions, expand design spaces, and expose practical challenges.
- Reflective HCXAI: A reflective HCXAI must critically examine implicit assumptions and practices while remaining value-sensitive to users and designers.Reflection creates intellectual space for progress through conceptual and technical impasses.
- Critical Technical Practice: Critical Technical Practice questions XAI’s core metaphors and assumptions, identifies marginalized aspects, and generates new questions and hypotheses.It supports reflection that brings unconscious experience into conscious awareness.
- Critical Technical Practice: Applying CTP shifts explainability beyond model-centered analysis by asking whether explanatory ability belongs to the model, the human, or their interaction.This reframing can reveal overlooked design possibilities in which meaning is co-created during action.
- Reflective HCXAI: CTP brings human-centered XAI to the foreground, supports new ways to understand human factors, and can give users interaction capabilities that amplify their voices.The paper uses trust as an example: explanations might support either acceptance or reasonable skepticism, depending on context.
- Operationalizing HCXAI: Participatory design, value-sensitive design, reflection-in-action, and ludic design provide complementary ways to operationalize reflective HCXAI without privileging one tradition.Participatory design foregrounds marginalized perspectives, while value-sensitive design examines stakeholder values, tensions, and political realities.
- Operationalizing HCXAI: In consequential systems, such as AI-mediated radiology task lists, explanations raise a design question about privileging user acceptance or user reflection.Answering this requires engagement with relevant communities and attention to stakeholder values and tensions.
- Challenges: Reflective HCXAI depends on sustained engagement with partner communities and on practitioners who can translate across multiple domains.Community engagement is resource- and time-intensive, requires organizational buy-in, and demands researchers with depth in at least two areas.
5 Conclusions
The conclusion presents HCXAI as a human-centered approach that treats consequential AI as socially situated and advocates reflective sociotechnical design. The paper connects this paradigm to CTP, participatory design, and value-sensitive design to question dominant XAI metaphors and open research and design spaces.
- Conclusion: HCXAI places humans at the center of technology design by examining values, interpersonal dynamics, and the socially situated nature of AI systems.The paper advocates incorporating social and technical elements within the design space.
- Conclusion: The case study shows that technical development and understanding of human factors co-evolve together.The paper also identifies open questions about refining who the human is and extending beyond one-to-one human-computer interactions.
- Conclusion: A reflective HCXAI paradigm uses Critical Technical Practice alongside participatory and value-sensitive design to question dominant XAI metaphors and open new research and design spaces.This proposal is presented as a way to recognize the socially situated nature of consequential AI systems.