Source-linked AI summary
The Challenge of Crafting Intelligible Intelligence
Daniel S. Weld, Gagan Bansal
TL;DR
AI systems are increasingly used in mission-critical settings even though complex models can be difficult to understand, test, and control. This survey examines inherently interpretable models and explanatory mappings for inscrutable systems, and concludes that intelligibility supports debugging, control, insight, acceptance, and accountability while remaining difficult to define and generalize across arbitrary models.
Problem
Complex AI decisions are difficult to understand and test in mission-critical settings, while the meaning of intelligibility itself remains hard to specify.
Method
The survey reviews inherently interpretable models, mappings from inscrutable systems to simpler explanatory models, and interactive explanation and control approaches.
Results
The survey identifies intelligibility as important for revealing problematic behavior, supporting control and insight, and addressing acceptance and legal accountability.
Takeaways & Limitations
Interpretable models can expose unintuitive behavior even when test accuracy is high, while explanations for inscrutable models must address differences from their underlying operation.
Takeaways & Limitations
A general method for mapping user control actions from explanatory models back to arbitrary inscrutable source models seems unlikely.
Abstract
from arXiv · showhide
Since Artificial Intelligence (AI) software uses techniques like deep lookahead search and stochastic optimization of huge neural networks to fit mammoth datasets, it often results in complex behavior that is difficult for people to understand. Yet organizations are deploying AI algorithms in many mission-critical settings. To trust their behavior, we must make AI intelligible, either by using inherently interpretable models or by developing new methods for explaining and controlling otherwise overwhelmingly complex decisions using local approximation, vocabulary alignment, and interactive explanation. This paper argues that intelligibility is essential, surveys recent work on building such systems, and highlights key directions for research.
1 INTRODUCTION
AI systems now perform many circumscribed tasks and are entering mission-critical roles, but their complex, context-specific behavior can be difficult to test and understand. The survey examines interpretable models and explanatory mappings as approaches to intelligible AI.
- AI systems are increasingly deployed in mission-critical settings despite decisions based on vast action spaces, neural networks, and many factors.Examples include credit scoring, recidivism prediction, news selection, and self-driving cars.
- Imperceptible input changes can cause deep neural networks to make erratic predictions, illustrating unintelligible behavior.Figure 1 shows an image classification changing from “panda” to “gibbon.”
- Intelligible behavior should reveal factors causing an action, support prediction of alternatives, and enable effective user control.The paper notes a central tension between explanations that are concise and those that are accurate.
- The survey considers inherently interpretable models and simpler explanatory models mapped from inscrutable systems.Interpretable models offer transparency and veracity, whereas explanatory mappings can preserve the performance of complex techniques but may differ from their actual operation.
- Building intelligible AI requires communicating complex computation to people through expertise spanning HCI, AI, machine learning, philosophy, and psychology.Figure 2 organizes the survey’s approaches and indicates where they are discussed.
2 WHY INTELLIGIBILITY MATTERS
Intelligibility matters because high performance alone may not reveal flawed objectives, inadequate features, deployment failures, or unacceptable decisions. Explanations can support debugging, user control, acceptance, human insight, and legal accountability.
- Perfect performance can still be insufficient when the objective is flawed or incomplete, especially when balancing multiple utility attributes is difficult.The paper uses the paperclip-maximization example to illustrate this concern.
- Correlated or omitted features can produce problematic models, making intelligibility useful for identifying and correcting unintuitive behavior.The paper describes asthma appearing negatively correlated with pneumonia mortality because of correlated treatment variables.
- Intelligibility helps users detect when deployed models fail to generalize under distributional drift.The deployment distribution may differ from training data or change over time, particularly in adversarial domains.
- Explanations can improve user control and acceptance by clarifying undesired actions and accompanying algorithmic decisions.Understanding an AI’s rationale can help users issue instructions that improve future behavior, while explanations can increase acceptance.
- Intelligible models facilitate scientific insight and help audit decisions for legal liability, including AI-specific errors.The paper cites medical machine learning, AlphaGo, GDPR explanation rights, and self-driving cars as examples.
3 DEFINING INTELLIGIBILITY
The paper defines intelligibility through what users can predict and control, especially how feature changes alter outputs, rather than merely whether they can simulate a model. Explanations must also balance accuracy, simplicity, user expectations, and the relevant alternative outcome.
- Human simulatability asks whether a user can easily predict a model’s output for a given input.
- The paper proposes answering counterfactual “what-if” questions as a better intelligibility criterion than simulatability alone.
- A model is intelligible when users can predict how changing a feature alters its output and reliably modify that response curve.
- Because explanations are contrastive, systems must identify the relevant foil, which can vary with the user and becomes harder for multi-class classifiers.
- Human preferences for simple explanations and susceptibility to cognitive biases complicate whether explanations should persuade, educate, or protect users from error.
4 INHERENTLY INTELLIGIBLE MODELS
The section presents GAMs and GA2Ms as interpretable models whose feature contributions can be inspected, visualized, and sometimes corrected by domain experts. GA2Ms retain competitive performance for moderate-dimensional semantic domains, but face scope and complexity limits.
- Model structure: GAMs predict by summing single-feature shape-function terms, while GA2Ms add pairwise interaction terms for greater expressiveness.Shape functions may be nonlinear, including splines and decision trees.
- Empirical value: For moderate numbers of semantic features, GA2M models achieve performance competitive with random forests and neural networks while remaining intelligible.Among GA2M learning methods, bagged shallow regression-tree shape functions learned via gradient boosting achieved the highest accuracy.
- Model structure: GA2M contributions can be visualized, enabling users to inspect how individual features and feature pairs affect predictions.In the pneumonia example, contributions are expressed as changes in log odds, including age, asthma, and age–cancer interactions.
- Empirical value: The pneumonia model learned that asthma decreased risk, a counter-intuitive pattern attributed to correlated treatment variables that could cause severe underestimation for untreated patients.The example illustrates how high test accuracy can coexist with clinically misleading behavior.
- Human control: Domain experts can correct erroneous GA2M patterns by zeroing a term’s weight or redrawing its line graph through a GUI.Adding a correlated pulmonologist-visit feature and retraining could also make asthma itself increase predicted pneumonia risk.
- Limitations: GA2Ms expose missing-feature effects and support skepticism and debugging, but million-feature settings remain beyond human simulation and may overwhelm users.With many features, the number of visualizations may grow quadratically, and semantic grouping or importance ordering may be needed.
- Limitations: GA2Ms are restricted to binary classification; proposed multiclass extensions may work technically, but their intelligibility and editability remain uncertain.One-vs-rest or hierarchical classifiers are possible extensions, but preservation of usable shape-function editing is unknown.
5 UNDERSTANDING INSCRUTABLE MODELS
Inscrutable models can be explained by mapping them to simpler models, but useful explanations must balance comprehensibility, fidelity, and an appropriate explanatory vocabulary. Local approximation methods such as LIME provide instance-specific explanations, while vocabulary alignment and model-specific mappings address important limitations.
- Inscrutable models may be inaccessible as black boxes or too complex for users to interpret, even when their parameters are available.
- The Comprehensibility / Fidelity Trade-Off: Explanations must balance comprehensibility against fidelity because complete traces of complex models are difficult to understand, while simplification can sacrifice faithfulness.
- Local Explanations: Local explanations simplify a model relative to one input query, unlike global models that describe behavior across the model.
- Locally-Approximate Explanations: LIME samples nearby instances, obtains black-box predictions, weights them by proximity, and trains a simpler classifier over semantically meaningful features.
- Locally-Approximate Explanations: LIME’s local classifier is intended to approximate the black-box decision boundary near the instance being explained, not globally.
- Choice of Explanatory Vocabulary: Choosing an explanatory vocabulary is difficult: pixels lack semantic meaning, while segmentation regions offer visual interpretability without necessarily matching the classifier’s decision process.
- Choice of Explanatory Vocabulary: Jointly training classifiers with image-captioning systems can align hidden variables with meaningful concepts, although some generated explanations include details absent from the image.
- Facilitating User Control with Explanatory Models: Mapping an inscrutable model into an explanatory model does not generally provide a way to map user edits back to the original model.
6 TOWARDS INTERACTIVE EXPLANATION
The paper envisions explanation as an interactive process tailored to the user, because an initial explanation may not address every concern. Users should be able to ask follow-up questions, inspect rationales, test counterfactuals, change vocabulary, and adjust models.
- An ideal explanation depends on the audience’s technical background and the user’s purpose, such as accepting a decision versus debugging a system.
- Explanation systems should be interactive because users are likely to have follow-up questions that an initial explanation cannot answer.
- Figure 7 illustrates a dog/fish classifier dialog combining LIME explanations with influential training examples retrieved using influence functions.
- Interactive explanations can redirect the answer by changing the comparison class or request more detail within a restricted region of feature space.
- Users can ask for a decision’s rationale, including influential labeled training examples identified through influence functions or nearest-neighbor methods.
- Users can test model sensitivity by asking which minimal feature perturbations would produce a different output.
- Interactive systems may let users change the explanatory vocabulary or adjust the underlying model through new examples, corrected labels, features, or shape functions.
7 EXPLAINING COMBINATORIAL SEARCH
Deep-lookahead planning systems face intelligibility problems because their action models and goals contain assumptions that may be incomplete or incorrect. Local explanations can simplify cognitively overwhelming plans, but vocabulary selection remains important.
- Planning algorithms may produce formally sound plans whose action models and goal specifications contain unspoken, incomplete assumptions.
- For large search spaces, local explanations can simplify plans that would otherwise overwhelm human cognition, while explanatory vocabulary remains a central issue.
8 FINAL THOUGHTS
The paper concludes that intelligibility is needed alongside robustness for understanding, controlling, learning from, and assigning responsibility for deployed AI. It surveys interpretable models and post-hoc mappings, and advocates interactive, multidisciplinary research.
- Intelligibility can help identify mistakes caused by distributional drift or incomplete representations of goals and features.
- Intelligibility can facilitate human control in collaborative human–AI teams, support learning from AI, and help assign liability when AI systems err.
- Understanding complex AI may require either inherently interpretable models or post-hoc explanations that map inscrutable models to simpler models using currying and local approximation.
- Research should address explanatory vocabulary, mappings from user edits back to inscrutable models, and interactive explanations supporting follow-up actions.
- The paper calls for collaboration among multiple disciplines to advance explanation systems.