Source-linked AI summary
The Mythos of Model Interpretability
Zachary C. Lipton
TL;DR
Interpretability research lacks a shared definition despite models’ increasing use in consequential settings. This paper critically analyzes its motivations and proposed model properties, finding that interpretability comprises distinct, sometimes competing ideas, including transparency and post-hoc explanation.
Problem
Interpretability lacks a precise, shared definition even as machine-learning models enter critical domains, making claims about model interpretability difficult to assess.
Method
The paper critically analyzes literature on interpretability’s motivations, model properties, and explanatory techniques.
Results
The analysis finds that interpretability is not monolithic, with human transparency and post-hoc explanations representing competing notions.
Takeaways & Limitations
Interpretability claims should be understood in relation to the distinct motivations and concepts they invoke.
Takeaways & Limitations
Local post-hoc explanations may mislead because small input changes can produce substantially different explanations.
Abstract
from arXiv · showhide
Supervised machine learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? We want models to be not only good, but interpretable. And yet the task of interpretation appears underspecified. Papers provide diverse and sometimes non-overlapping motivations for interpretability, and offer myriad notions of what attributes render models interpretable. Despite this ambiguity, many papers proclaim interpretability axiomatically, absent further explanation. In this paper, we seek to refine the discourse on interpretability. First, we examine the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant. Then, we address model properties and techniques thought to confer interpretability, identifying transparency to humans and post-hoc explanations as competing notions. Throughout, we discuss the feasibility and desirability of different notions, and question the oft-made assertions that linear models are interpretable and that deep neural networks are not.
1. Introduction
The introduction argues that interpretability is ill-defined: its motivations and technical meanings are diverse and sometimes discordant. It frames the paper’s investigation around supervised learning, competing notions of transparency and post-hoc explanation, and the gap between model objectives and real-world goals.
- Motivations and definitions: Interpretability is presented as an ill-defined concept whose motivations and technical descriptions vary across the literature.The paper argues that claims about interpretability may therefore exhibit a quasi-scientific character.
- Scope and questions: The paper focuses on supervised learning and asks what interpretability is and why it matters, including the claim that linear models are more interpretable than deep neural networks.Reinforcement learning and interactive learning are outside the paper’s main scope.
- Motivations and definitions: Interpretability is motivated by diverse desiderata, including trust, performance, robustness, mechanistic understanding, and useful information.The introduction questions whether trust means faith in performance, robustness, or another property of model decisions.
- Motivations and definitions: Supervised-learning objectives can diverge from real-world goals when error minimization replaces causal discovery or training data fails to represent a changing deployment environment.The introduction gives medical research and product recommendation as examples of these divergences.
- Interpretability notions: The paper distinguishes transparent models, whose mechanisms can be understood, from post-hoc interpretations that explain predictions without revealing those mechanisms.Human verbal explanations and neural-network saliency maps exemplify post-hoc interpretations, creating tension between the two notions.
2. Desiderata of Interpretability Research
Interpretability research addresses real-world objectives that supervised-learning metrics alone do not capture, especially when deployment costs, human judgment, or changing environments matter. The literature therefore links interpretability to diverse aims, including trust, scientific hypothesis generation, robust generalization, useful information, and contestable decisions.
- Deployment objectives: Interpretability is sought when test-set predictive performance does not capture the real-world costs or objectives of deployment.Standard supervised-learning metrics generally require only predictions and ground-truth labels, whereas deployment objectives may include ethics, legality, or changing environments.
- Trust: Trust is an ambiguous motivation: it may mean confidence in performance, subjective comfort, or confidence that models will not perpetuate harmful biases.A model can be accurate yet raise concerns about racial bias and over-policing, so trust is not reducible to predictive accuracy.
- Scientific understanding: Model interpretation can support scientific discovery by generating hypotheses, but learned associations are not guaranteed to be causal and may depend on strong prior assumptions.Interpretations may suggest relationships for experimental testing, while unobserved common causes can explain observed associations.
- Generalization and robustness: Interpretability matters when models must generalize beyond familiar data, particularly in nonstationary or adversarial environments where accurate i.i.d. performance can be insufficient.CNNs can be induced to misclassify imperceptibly perturbed images, and credit-rating systems can be gamed without changing borrowers’ underlying ability to repay.
- Information for decision-makers: Interpretations may provide useful information to human decision-makers without revealing model internals, such as by identifying similar cases that support a diagnosis.In some applications, the model’s real purpose is exploration or assistance, with supervised training serving only as weak supervision.
- Explanation and contestability: Explanations for algorithmic decisions raise unresolved requirements of correctness, clear reasoning, and contestability under proposed European Union regulations.The regulations’ right to explanation leaves open what explanations should look like and how to distinguish valid explanations from merely appeasing ones.
3. Properties of Interpretable Models
Interpretable-model techniques and properties broadly divide into transparency, which concerns how a model works, and post-hoc explanations, which concern what else it can tell us. The distinction is organizational rather than absolute.
- 3. Properties of Interpretable Models: Interpretable-model techniques broadly fall into transparency and post-hoc explanations.Transparency asks how the model works, whereas post-hoc explanations ask what else the model can tell us.
- 3. Properties of Interpretable Models: The transparency–post-hoc distinction is useful but not absolute.Post-hoc analysis can attempt to uncover the significance of model parameters, overlapping with transparency concerns.
3.1. Transparency
Transparency concerns understanding how a model works at the levels of the whole model, its components, and its training algorithm. The paper argues that neither model families nor intuitive parameters are intrinsically interpretable, because transparency depends on complexity, human cognitive limits, and feature representation.
- Transparency: Transparency can be assessed through simulatability of the whole model, decomposability of its components, and transparency of the training algorithm.These levels capture understanding the model’s mechanism, individual inputs or parameters, and how training behaves.
- Transparency: Simulatability treats interpretability as a person’s ability to contemplate the entire model and step through its calculations in reasonable time.This notion supports viewing simple models as interpretable and sparse linear models as more interpretable than dense models on the same inputs.
- Transparency: For decision trees, simulatability may depend either on total model size or on the computation required for inference.The total number of nodes can grow much faster than the root-to-leaf inference path.
- Transparency: Because reasonable complexity is subjective and human cognition is limited, linear models, rule systems, and decision trees are not intrinsically transparent.High-dimensional linear models, unwieldy rule lists, and deep trees may be less transparent than comparatively compact neural networks.
- Transparency: Component-level intelligibility requires each input, parameter, and calculation to admit an intuitive explanation, but engineered features and apparently intuitive weights can undermine it.Linear-model weights may be fragile to feature selection and preprocessing, and this notion requires individually interpretable inputs.
- Transparency: Algorithmic transparency is stronger for linear models, whose error surfaces and convergence can be analyzed, than for deep learning’s heuristic optimization procedures.Deep learning methods are powerful, but their operation is poorly understood and their success on new problems cannot currently be guaranteed a priori.
3.2. Post-hoc Interpretability
Post-hoc interpretability extracts useful information from opaque models without necessarily revealing precisely how they work or sacrificing predictive performance. Common approaches include verbal explanations, visualizations, local saliency analyses, and example-based explanations, each offering partial insight into model behavior.
- Post-hoc Interpretability: Common post-hoc approaches include natural-language explanations, visualizations of learned representations or models, and explanations by example.These approaches differ from directly exposing the model’s internal operation.
- Post-hoc Interpretability: Post-hoc interpretations can provide useful information to practitioners and end users even when they do not explain precisely how a model works.This approach permits after-the-fact interpretation of opaque models without sacrificing predictive performance.
- Post-hoc Interpretability: Separate models can generate verbal explanations for another model’s predictions, such as mapping a reinforcement learner’s state representation onto explanations of strategy.Related recommender-system work jointly trains rating-prediction and review-topic models whose latent factors are connected through topic distributions.
- Post-hoc Interpretability: Visualization methods probe learned representations by projecting them into two dimensions or altering inputs to enhance selected hidden-layer activations.Examples include t-SNE visualizations, gradient-based activation enhancement that produced dog faces, and image recovery from high-level AlexNet representations.
- Post-hoc Interpretability: Local explanations can be misleading because saliency maps may change substantially after moving a single pixel, unlike linear models’ global input-output relationships.Saliency maps highlight input regions whose changes would most influence an output, but they describe only local behavior.
- Post-hoc Interpretability: Example-based explanations use learned representations to identify other examples that a model considers most similar alongside its prediction.This method can be applied after training deep neural networks or latent-variable models for discriminative tasks.
4. Discussion
Interpretability is not a monolithic concept: claims about it must specify whether they concern transparency or post-hoc interpretation and provide appropriate evidence. The paper cautions that transparency may sacrifice predictive power, post-hoc explanations may mislead, and future work should better align objectives and metrics.
- Linear and deep models: Linear models can lose simulatability or decomposability with high-dimensional or heavily engineered features, so their interpretability depends on the chosen notion.The paper characterizes linear models as uncontroversially transparent algorithmically, but not necessarily simulatable or decomposable in every feature setting.
- Linear and deep models: Choosing between linear and deep models can trade algorithmic transparency against decomposability because deep models use meaningful raw features while linear models often require hand-engineered features.Deep networks may support sensible post-hoc reasoning from raw or lightly processed features, whereas comparable linear performance may require heavily engineered representations.
- Linear and deep models: Deep neural networks can have an advantage for post-hoc interpretation because their learned representations can be visualized, verbalized, or clustered.Although linear models appear to have a better track record for studying the natural world, the paper gives no theoretical reason this must remain true.
- Interpretability as a specific objective: Interpretability claims are meaningful only when they specify a definition, directly demonstrate transparency, or establish that a post-hoc explanation achieves a clear objective.The paper treats transparency and post-hoc interpretability as distinct notions requiring different forms of evidence.
- Risks and trade-offs: Transparency demands can conflict with improving complex-task performance, while subjective post-hoc objectives can encourage misleading but plausible explanations.The paper urges justification for sacrificing predictive power and warns against blindly optimizing explanations to placate subjective demands.
- Future directions: Future work should address mismatches between real-life and machine-learning objectives through richer loss functions and performance metrics, and extend the analysis to reinforcement learning.Examples include sparsity-inducing regularizers and cost-sensitive learning.
5. Contributions
The paper argues that interpretability lacks an agreed meaning and that many existing techniques are incompatible with pressing real-world machine-learning problems. It proposes a taxonomy and calls for critical problem formulation and shared responsibility for machine learning’s societal impact.
- Interpretability has no agreed-upon meaning, despite machine-learning papers often using the term in a quasimathematical way.
- Many currently investigated interpretability techniques are incompatible with pressing problems facing machine learning in the wild.
- The paper takes a first step toward a comprehensive taxonomy of interpretability desiderata and methods.
- Critical problem formulation is necessary because flawed formulations cannot be repaired by algorithms or experiments alone.
- Practitioners and researchers share responsibility for accounting for machine learning’s societal impact and alignment with societal desiderata.