Source-linked AI summary
What Does Explainable AI Really Mean? A New Conceptualization of Perspectives
Derek Doran, Sarah Schulz, Tarek R. Besold
TL;DR
Explainable AI lacks a single shared meaning across research communities and system designs. The paper uses corpus analysis to compare these perspectives, then distinguishes opaque, interpretable, and comprehensible systems and proposes truly explainable systems centered on automated reasoning.
Problem
Explainability and interpretability are used without a clear, widely shared definition, while high-stakes decisions require understandable rationales.
Method
The paper analyzes normalized explanation-term frequencies and contexts in ACL, NIPS, ICCV/ECCV, and COGSCI proceedings, then develops conceptual system categories.
Results
The analysis finds field-specific explainability patterns and distinguishes opaque systems, mathematically analyzable interpretable systems, and symbol-emitting comprehensible systems.
Takeaways & Limitations
Truly explainable systems should make automated reasoning central to producing crafted explanations rather than leaving final explanation generation to human analysts.
Takeaways & Limitations
Completeness is not treated as an explicit or necessarily desirable trait, because simplified explanations may better suit users’ needs.
Abstract
from arXiv · showhide
We characterize three notions of explainable AI that cut across research fields: opaque systems that offer no insight into its algo- rithmic mechanisms; interpretable systems where users can mathemat- ically analyze its algorithmic mechanisms; and comprehensible systems that emit symbols enabling user-driven explanations of how a conclusion is reached. The paper is motivated by a corpus analysis of NIPS, ACL, COGSCI, and ICCV/ECCV paper titles showing differences in how work on explainable AI is positioned in various fields. We close by introducing a fourth notion: truly explainable systems, where automated reasoning is central to output crafted explanations without requiring human post processing as final step of the generative process.
1 Introduction
High-stakes AI decisions require more than outputs: users need understandable rationales to assess whether decisions follow rational and ethically acceptable reasoning. The paper therefore asks what qualifies as an explanation.
- Motivation: High-stakes machine decisions can affect financial, safety, security, personal, and ethical outcomes.Examples include predictions about health, employment, crime, and terrorism risk.
- Motivation: Blindly trusting such decisions is difficult without a deep understanding of the system’s decision making.
- Motivation: AI systems should provide human-understandable explanations expressing the rationale behind their outputs.
- Motivation: Analysts can use explanations to evaluate whether decisions rely on rational arguments and avoid reasoning that conflicts with ethical or legal norms.
- Problem: Because “explainable” lacks a clear definition, the paper questions whether current systems genuinely provide explanations.Candidate examples include learned feature rules, rich visualizations, and accompanying text.
2 Existing Perspectives in Explainable AI
The paper combines corpus analysis with conceptual distinctions to show that explainability is approached differently across AI communities. It distinguishes opaque, interpretable, and comprehensible systems by the insight and user work they provide.
- Corpus analysis: Normalized frequencies are comparable between years and conferences, while word clouds summarize terms appearing near explanation-term mentions.Important context words occur within a 20-word window and have frequencies highly above average.
- Corpus findings: COGSCI shows substantially higher explainability-term frequency, while NIPS shows emphasis around 2008 and a slight increase in 2016.The paper attributes possible differences to Cognitive Science’s focus on explaining the mind and its processes.
- Corpus findings: ACL links explainability with features, examples, and words; NIPS links it with methods, algorithms, and results; ICCV/ECCV combines data and features with algorithmic use of images.
- System notions: Opaque systems hide the mechanisms mapping inputs to outputs, functioning as prediction oracles without indicating how or why predictions are made.Closed-source systems and genuine black-box approaches are examples.
- System notions: Interpretable systems let users mathematically study how inputs map to outputs, requiring transparency and understanding of technical mapping details.Regression weights and linear-classifier decision boundaries illustrate this notion.
- System notions: Comprehensible systems emit symbols such as words or visualizations that users relate to inputs and outputs, with comprehension depending on users’ own knowledge and reasoning.Comprehensibility is graded because users differ in how easily they compile and understand those symbols.
3 Defining Notions of Explainability
The paper distinguishes opaque, interpretable, and comprehensible systems by the insight they provide into algorithmic mechanisms and decisions. These notions also depend on the user, and complete explanations are not always desirable.
- Interpretable systems: Interpretable systems let users study how inputs are mathematically mapped to outputs, requiring transparency into technical model details.Examples include comparing regression covariate weights or examining linear-classifier decision boundaries.
- Comprehensible systems: Comprehensible systems may remain opaque internally but emit symbols that users compile and reason over to relate inputs to outputs.Comprehensibility is graded by the relative ease or difficulty of this user-driven compilation and comprehension.
- Relations among notions: Comprehensible and interpretable systems are both improvements over opaque systems, but interpretation requires transparent mechanisms whereas comprehension does not.The preferable type depends on the user’s background and purpose.
- User-dependent traits: Explainable-system traits depend on both the learning machine and the user, so confidence and trust vary with user capability, biases, and prior experience.The paper specifically links confidence in an interpretable system to the user’s ability to understand its input/output mapping.
- User-dependent traits: Completeness is not an explicit trait and may be undesirable when simplified explanations better fit the user’s needs or treatment context.The doctor example illustrates explanations that omit details or use incomplete preconceptions.
4 Truly Explainable AI Should Integrate Reasoning
The paper argues that interpretable and comprehensible models enable explanations but do not generate a complete reasoning account themselves. It proposes integrating automated reasoning with emitted symbols and domain knowledge to produce explanations directly.
- Motivation: Interpretable and comprehensible models do not themselves yield explanations because human analysts must formulate the reasoning connecting model decisions to human-understandable input features.Different analysts may deduce different explanations from the same decision, depending on their background knowledge.
- Reasoning integration: Neural-symbolic integration seeks automated reasoning over model properties and decision factors by extracting symbolic rules from connectionist models.Combining this with research on human-comprehensible representations and reasoning approaches might support systems that provide their own full explanations.
- Reasoning integration: A reasoning engine can combine symbols emitted by a comprehensible model with a domain-specific knowledge base to derive logical relations relevant to its decision.The knowledge base encodes relationships between concepts represented by the symbols.