Source-linked AI summary
Explainability in Human-Agent Systems
Avi Rosenfeld, Ariella Richardson
TL;DR
Human-Agent Systems need explainability, but the field lacks consistent definitions and must account for varied users, goals, and interaction contexts. The paper proposes definitions and an integrated Why, Who, What, When, and How taxonomy, alongside an evaluation framework. Its central conclusion is that no single explanation type fits all systems, and explanation evaluation remains difficult because important goals such as user understanding lack quantifiable measures.
Problem
Machine-learning research uses explainability and interpretability inconsistently, while Human-Agent Systems need explanations suited to their interactions and goals.
Method
The paper defines key concepts and systematically organizes explainability around Why, Who, What, When, and How, including objective and subjective evaluation.
Results
The framework concludes that explanation choice and evaluation must vary with the user, timing, explanation type, and system requirements.
Takeaways & Limitations
There is no one explanation type for every Human-Agent System; requirements and goals should be mapped before selecting and evaluating explanations.
Takeaways & Limitations
Important explanation goals, including user understanding, still lack quantifiable measures, making their evaluation difficult.
Abstract
from arXiv · showhide
This paper presents a taxonomy of explainability in Human-Agent Systems. We consider fundamental questions about the Why, Who, What, When and How of explainability. First, we define explainability, and its relationship to the related terms of interpretability, transparency, explicitness, and faithfulness. These definitions allow us to answer why explainability is needed in the system, whom it is geared to and what explanations can be generated to meet this need. We then consider when the user should be presented with this information. Last, we consider how objective and subjective measures can be used to evaluate the entire system. This last question is the most encompassing as it will need to evaluate all other issues regarding explainability.
1 Introduction
The paper frames explainability as a central information-sharing issue in Human-Agent Systems and develops an integrated taxonomy addressing why, who, what, when, and how. It argues that explanation requirements depend on system motivation and user role, while explanations can support persuasion, trust, adoption, and legal needs.
- Human-Agent Systems require decisions about the type and quality of information shared between agents and human users.
- The paper defines explainability and related concepts, then organizes them through a taxonomy of Why, Who, What, When, and How.
- Explainability requirements should be considered together because the needed explanation depends directly on the system’s motivation.
- In human-centric systems, explanations may persuade users to choose actions by presenting arguments about an agent’s decision, policy, or presentation.
- In agent-centric systems, explanations can help convince human participants that decisions are correct, support technology adoption, and build trust; legal considerations may also require them.
2 Definitions of Explainable Systems and Related Terms
The paper distinguishes explainability from related concepts whose meanings are often conflated, defining explainability as human understanding of an agent’s logic and interpretability and transparency as properties that support explanations.
- Theoretical foundations: Explanations can be understood as answers to why-questions that contrast a selected possibility with alternatives.This framework requires an explanation to help a user choose or understand why S_0 was preferred over S_1...S_n.
- Theoretical foundations: Communication-based theories treat explanations as responses to a recipient’s information request, so a complete explanation is unnecessary when the user does not require it.This account emphasizes the recipient’s requested knowledge rather than a universally complete account of the system.
- Terminological foundations: The literature uses explainability, interpretability, and transparency inconsistently, with interpretability lacking an agreed formal technical meaning.Artificial intelligence research often emphasizes explainable systems, whereas machine learning research commonly focuses on how algorithms make decisions.
- Formal definitions: The paper formalizes machine-learning explanations around training records R, features F, targets T, and algorithm L.The training set is represented as R × F, and L is created from the training records to fit labeled targets.
- Formal definitions: Explainability is the ability of a human user to understand an agent’s logic, while interpretability and transparency describe aspects of that logic.The paper treats interpretability and transparency as contributing to, but not constituting, explainability.
- Relationship among concepts: Interpretability includes transparent models and non-transparent tools such as model or outcome tools, feature analysis, visualization, and prototype analysis.The paper uses a Venn diagram to show overlaps among these approaches and their relationship to explainability.
3 Why a Human-Agent System should be Explainable?
The need for explainability depends on why it is needed, how much control the agent has, and whom the explanation serves. The paper classifies necessity as not helpful, beneficial, or critical, with trust, safety, acceptance, fairness, and oversight among the motivating purposes.
- Necessity: The paper’s taxonomy classifies explainability as not necessary, beneficial, or critical according to the importance of understanding the agent’s logic.The classification is tied to the reason an explanation is needed and its objective for the user or another stakeholder.
- Autonomy and cooperation: The required explanation depends on the agent’s autonomy: a fully human-controlled agent may need no explainability, whereas agents supporting human tasks may require transparency or faithful explanations.Adjustable autonomy links explainability needs to the degree of cooperation between agent and human user.
- Critical need: When an agent supports a human task, explanations can be critical for guidance, negotiation, training, argumentation, and life-or-death decisions.Examples include step-based hints in intelligent tutoring systems and arguments that present a negotiation agent’s position.
- Beneficial need: Explanations are beneficial rather than critical when they help but are not required for optimal or peak-efficient system behavior.This category separates helpful explanatory information from explanations necessary for the system to operate effectively.
- Motivations: Explainability may support acceptance, safety, trust, fairness, ethical decisions, legal compliance, debugging, and scientific discovery.The paper lists these as proposed reasons for making human-agent systems explainable.
- Target audiences: Explanations may target end-users, experts, researchers, system designers, or legal and policy specialists, producing different objectives and potentially different explanatory forms.For non-end-users, explanations can reveal gaps between an agent’s formulation or implementation and its observed performance.
4 Who is the Target of the Explanation?
The target of an explanation may be a regular user, an expert user, or an external entity, and the required detail depends on the target’s expertise and the system’s purpose.
- The needed explanation detail depends on why the system requires understanding and how the explanation was generated.
- Legal purposes may require detailed explanations directed to legal experts rather than regular users.
- The same explanation may help a developer but be useless to a regular user because target expertise affects explicitness.
- External regulatory entities can receive explanations governed by their guidelines, whereas developers may use user-usability guidelines.
5 What Interpretation can be Generated?
The paper identifies six approaches for generating interpretations, ranging from transparent model-based methods to post-hoc tools, and characterizes them by faithfulness and explicitness.
- Six interpretation approaches include transparent algorithms, feature analysis, post-hoc model and outcome tools, visualizations, and prototypes.
- The six approaches differ in faithfulness and explicitness, with trade-offs varying across implementations.
- Transparent algorithms generate interpretations directly from the learning algorithm and can provide them before, during, or after task execution.
- No single algorithm provides both high accuracy and explicitness, motivating algorithms that incorporate explainability into learning.
- Feature selection or analysis can generate interpretations independently of, or alongside, a machine-learning model, but may lose faithfulness when disconnected from it.
- Outcome explanations are localized to a given instance and prediction, with examples including LIME, SP-LIME, and Shapley-based methods.
6 When Should Information be Presented?
Explanations may be presented before, during, or after task execution, and agents may use multiple timings for different purposes.
- Explanations can be presented before, during, or after a task.
- Before execution, explanations may justify an intended action, support conceptualization, or provide evidence of fairness.
- During execution, explanations may be especially important when an agent fails so users can trust it to correct the error.
- After execution, explanations can support retrospective reports.
- Timing is not exclusive: agents may supply different explanations before, during, and after the task.
- When explainability is critical for beginning operation, information should be presented at the task’s beginning so users can decide whether to accept the recommendation.
7 How can Explanations be Evaluated?
The paper argues that evaluating explainability in Human-Agent Systems requires a general framework that addresses interconnected goals, users, explanation properties, and task contexts. It proposes measuring agent performance, interpretation quality, and human understanding while accounting for trade-offs and unresolved measurement challenges.
- Evaluation challenge: A general evaluation framework remains difficult because explanation detail, user type, timing, content, and interface are interconnected.The authors note that experts and regular users may need different explanations at different times and through different interfaces.
- Evaluation challenge: Existing evaluations often use binary judgments or task taxonomies but do not quantify the effectiveness of individual explainability components.The paper identifies explicitness, faithfulness, and transparency as components requiring direct evaluation.
- Evaluation framework: The proposed evaluation assigns separate scores to agent prediction performance, the interpretation presented to the user, and the user’s acceptance of that interpretation.The three elements correspond to the learning agent, its interpretation, and human understanding or acceptance.
- Trade-offs: Explanation detail can improve user confidence yet consume processing time, creating a trade-off between explanation benefits and time-sensitive task performance.The paper uses online stock purchasing to illustrate how lengthy explanations may prevent users from responding to quickly passing market fluctuations.
- Evaluation framework: The overall utility combines multiple complementary or contradictory goals using importance weights and goal grades, with hard-constraint failures reducing utility to zero.The framework includes agent metrics such as accuracy and recall, interpretation properties such as explicitness and faithfulness, and user outcomes such as improved performance.
- Open challenges: The framework assumes that users, designers, or external organizations can assign goal importance and grades, but these values are difficult to quantify in real-world applications.The paper distinguishes easier-to-measure prediction metrics from less established measures for interpretation properties and user experience.
8 Discussion
The discussion argues that explainability in Human-Agent Systems must be analyzed as an integrated set of questions rather than as explanation generation alone. It identifies practical trade-offs and open problems in evaluating explanation quality across users, purposes, methods, and settings.
- Integrated taxonomy: The paper’s taxonomy maps prior work across explainability’s fundamental questions, but no reviewed study addresses all aspects.The mapping is presented as a categorization framework rather than an exhaustive literature list.
- Integrated taxonomy: Why, who, when, and how are crucial because they heavily affect what explanation should be generated.The paper emphasizes that explanation requirements are interwoven and context-dependent.
- Method and evaluation trade-offs: Black-box and white-box methods should not be judged solely by agent performance, because legal motivations may require greater transparency, explicitness, or faithfulness.For external legal entities, feature analysis may be sufficient to address concerns about feature misuse.
- Method and evaluation trade-offs: Evaluation may need to change fundamentally with the explanation’s target user, presentation timing, and explanation type.These elements can vary substantially across Human-Agent Systems.
- Open issues: Open questions include measuring explicitness and faithfulness and developing canonical tasks for comparing explanation algorithms.The paper also asks whether one interpretation method is clearly better for particular tasks.
- Open issues: The discussion questions whether black-box interpretability can equal white-box interpretability and whether justification can outperform more explicit, faithful interpretable models.These questions remain unresolved in the paper’s analysis.
9 Conclusion
The paper proposes a framework for comparing and evaluating explainability in machine-learning-based Human-Agent Systems. It defines key concepts and organizes explainability around five questions, while concluding that explanation choice must follow each agent’s requirements and goals.
- Conclusion: The framework is designed to enable comparison and evaluation of explainability in diverse and complex Human-Agent Systems.The authors reject a universal explanation type.
- Conclusion: The paper proposes consistent definitions for explainability-related terms and systematically addresses Why, Who, What, When, and How.The sequence begins with why explanation is needed, then determines its target, type, timing, and evaluation.
- Conclusion: The framework is intended as a basis for future studies and further analysis or extension of new algorithms.The paper identifies additional open issues for the emerging field.