Source-linked AI summary
The Pragmatic Turn in Explainable Artificial Intelligence (XAI)
Andrés Páez
TL;DR
XAI lacks a well-defined goal when it treats explanation as sufficient without first specifying what it means to understand a model or decision. The paper develops a pragmatic and naturalistic account of understanding, arguing that interpretative or approximation models are central to understanding opaque models and post-hoc interpretability, while functionalist approaches fall short.
Problem
XAI seeks understandable models and decisions, but explanatory strategies lack a well-defined goal without an account of what it means for agents to understand them.
Method
The paper examines different types of understanding and alternative paths to understanding in AI, including objectual, post-hoc, and functional understanding.
Results
Interpretative or approximation models provide the best route to objectual understanding of machine-learning models and are necessary for post-hoc interpretability.
Takeaways & Limitations
XAI should pursue objective understanding of opaque models and their decisions through pragmatic methods and devices, rather than relying exclusively on traditional explanations.
Takeaways & Limitations
The paper focuses on unambiguously black-box models, and functional approaches to post-hoc interpretability leave the black box intact.
Abstract
from arXiv · showhide
In this paper I argue that the search for explainable models and interpretable decisions in AI must be reformulated in terms of the broader project of offering a pragmatic and naturalistic account of understanding in AI. Intuitively, the purpose of providing an explanation of a model or a decision is to make it understandable to its stakeholders. But without a previous grasp of what it means to say that an agent understands a model or a decision, the explanatory strategies will lack a well-defined goal. Aside from providing a clearer objective for XAI, focusing on understanding also allows us to relax the factivity condition on explanation, which is impossible to fulfill in many machine learning models, and to focus instead on the pragmatic conditions that determine the best fit between a model and the methods and devices deployed to understand it. After an examination of the different types of understanding discussed in the philosophical and psychological literature, I conclude that interpretative or approximation models not only provide the best way to achieve the objectual understanding of a machine learning model, but are also a necessary condition to achieve post-hoc interpretability. This conclusion is partly based on the shortcomings of the purely functionalist approach to post-hoc interpretability that seems to be predominant in most recent literature.
1. Introduction
The paper argues that XAI should be reformulated around a pragmatic, naturalistic account of understanding. Without clarifying what it means to understand models and decisions, explanatory strategies lack a well-defined goal.
- XAI explanations aim to make models or decisions understandable to stakeholders, but the relevant understanding remains underspecified.
- The paper analyzes human understanding in machine learning and black-box models to characterize XAI’s theoretical goal.
- Black-box models challenge the assumed inseparability of explanation and understanding because traditional explanations may be unavailable.
- Post-hoc interpretability and model transparency are presented as different levels of the same kind of understanding, while functional understanding often falls short.
- Because stakeholders differ in purposes and background knowledge, no single interpretative strategy should be expected to work equally well in every case.
- The paper develops distinctions among understanding-why, objectual understanding of model workings, and functional understanding.
2. Why not settle for AI-explanations?
The paper argues that understanding opaque machine-learning models should not depend on traditional AI explanations. Factivity requirements and limited epistemic access make a pragmatic focus on understanding more suitable.
- Current AI explanations concern model outputs or workings, unlike logic-based explanations that support adding inputs to a belief set or database.
- Traditional scientific explanations are factive, but opaque models block the epistemic access needed to explain specific decisions or computations truthfully.
- General knowledge of a deep neural network’s structure cannot explain specific decisions, because successes and failures cannot be traced to particular hidden-layer causal paths.
- Replacing explanation with understanding relaxes factivity and permits methods that are not entirely faithful yet provide indirect epistemic access to otherwise inaccessible facts.
- A pragmatic focus also clarifies that explanation and trust have no simple correlation, because contextual factors can foster or hinder trust.
3. Alternative Paths to Understanding
The paper surveys nontraditional routes to understanding opaque models, including manipulation, analogy, exemplification, and models or idealizations. These devices need not faithfully represent their objects, because adequate fit is pragmatic.
- Examples, analogies, diagrams, maps, models, idealizations, and simulations can support understanding by emphasizing selected features for pragmatic purposes.
- Useful representations need to be sufficiently accurate without being too accurate, as illustrated by subway maps.
- The adequate fit between an interpretative model and its object is pragmatic, so a representation may be “true enough” for particular purposes.
- Causal information can arise from observation, experimentation, manipulation, and inference without being delivered as a propositional explanation.
- Manipulating inputs and observing effects provides implicit causal and modal information about how a system works.
- The ability to manipulate a system into desired states and think counterfactually is presented as a sign of understanding.
4. Types of Understanding in AI
The paper distinguishes understanding-why from objectual understanding, then argues that understanding decisions requires localized objectual understanding of the model. It concludes that interpretative models provide objectual understanding, whereas functional approaches provide a more limited kind.
- Understanding-why and objectual understanding: Understanding-why concerns why a model produced a result, whereas objectual understanding concerns grasping a system’s inner workings.The paper treats understanding-why as a type of objectual understanding at a local level.
- Understanding-why and objectual understanding: Answering counterfactual questions and making predictions depends on objectual understanding of the broader body of knowledge surrounding the specific object.The paper illustrates this with atmospheric structure, chemistry, and behavior in reasoning about global warming.
- Understanding decisions: Understanding a model’s decision requires some degree of objectual understanding of the model, not merely manipulation of parameters and observation of outputs.Parameter tinkering yields functional generalizations with a weak inductive base rather than genuine counterfactual reasoning.
- Factivity and machine learning: The factivity of understanding-why is difficult to defend when analysis relies on idealized or literally false theories and models, as in machine learning.The paper argues that factivity can be defended only in simple scenarios where all relevant causal variables are available.
- Implications for XAI: Transparency and post-hoc interpretability are varieties of objectual understanding, while functional approaches provide a different and more limited kind.The paper states that interpretative models provide the kind of understanding described in the philosophical literature.
5. Conclusion
The conclusion reframes XAI around objective understanding, favoring interpretative models while emphasizing fit to both opaque systems and their intended users.
- XAI should shift from an uncertain search for explanatory devices toward studying objective understanding of opaque models and decisions.
- Interpretative models are presented as the best available route to understanding models through transparency and post-hoc interpretability.
- The conclusion rejects a purely functional approach to post-hoc interpretability because leaving the black box untouched can undermine XAI’s purpose.
- Interpretative models must balance fidelity to the original black-box model with comprehensibility for intended users.
- Designing understandable interpretative tools requires empirical study of user backgrounds, expertise, cognitive biases, and model complexity.
- XAI should also consider interdisciplinary design of user-friendly, accessible, and engaging interpretative tools and interfaces.