Source-linked AI summary
Towards A Rigorous Science of Interpretable Machine Learning
Finale Doshi-Velez, Been Kim
TL;DR
Interpretability research lacks consensus on what interpretability is and how it should be measured, especially when problem formulations are incomplete. The paper defines interpretability, identifies when explanations are needed, and proposes evaluation approaches matched to the claimed contribution. It concludes that interpretability evaluation should use application-grounded, human-grounded, or functionally-grounded methods and that shared taxonomies can support comparison.
Problem
Interpretability lacks consensus about its meaning and measurement, while explanations are not necessary for every ML system.
Method
The paper defines interpretability, analyzes incompleteness scenarios, proposes three evaluation approaches, and organizes interpretability factors into a shared taxonomy.
Results
The paper concludes that evaluation should match the claimed contribution through application-grounded, human-grounded, or functionally-grounded methods.
Takeaways & Limitations
A common taxonomy can support rigorous evaluation, citation, and comparison of interpretability research across applications and methods.
Takeaways & Limitations
Interpretability is not required for systems with insignificant consequences or problems sufficiently validated in real applications, and constructing a full evaluation matrix is expensive.
Abstract
from arXiv · showhide
As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.
1 What is Interpretability?
Interpretability is defined as the ability of an ML system to explain or present its outputs in understandable terms to a human. The paper frames explanations as useful for qualitatively assessing desiderata such as fairness, privacy, reliability, robustness, causality, usability, and trust.
- Interpretability is the ability of an ML system to explain or present its outputs in understandable terms to a human.
- The paper proposes data-driven operational definitions and evaluations of explanations because formal definitions of explanation remain elusive.
- Interpretability can qualitatively help assess whether systems meet criteria including fairness, privacy, reliability, robustness, causality, usability, and trust.
- An explanation may be feasible yet fail to correspond to a causal structure, exposing a potential concern.
2 Why interpretability? Incompleteness
The paper argues that interpretability is needed when incomplete problem formalizations create unquantified gaps that hinder optimization or evaluation. Explanations can make the effects of those gaps visible, while some systems do not require explanation when consequences are insignificant or validation is strong.
- Not all ML systems require interpretability when unacceptable results have no significant consequences or the problem is well-studied and validated in applications.
- Interpretability is needed when incompleteness in problem formalization creates a fundamental barrier to optimization and evaluation.
- Unlike uncertainty, which can be quantified, incompleteness can produce unquantified bias from factors such as domain knowledge in model selection.
- Safety: Safety concerns arise because complex end-to-end systems are rarely completely testable across all possible inputs and failure scenarios.
- Incomplete objectives, ethical constraints, and multi-objective trade-offs can leave important effects or decisions insufficiently specified.
- Explanations help make the effects of gaps in problem formalization visible.
3 How? A Taxonomy of Interpretability Evaluation
The paper proposes application-grounded, human-grounded, and functionally-grounded evaluation, ranging from task-specific human experiments to formal proxies without human experiments. Evaluation should match the specificity and type of the claimed contribution.
- Evaluation claims should match their evidence: application claims require application or closely related human evaluation, while model-class claims require functional metrics.
- Interpretability evaluation comprises application-grounded, human-grounded, and functionally-grounded approaches.These approaches range from task-relevant evaluation to general evaluation.
- Application-grounded evaluation: Application-grounded evaluation tests explanations with human experiments in the real application and its intended task.Examples include doctors using explanations while diagnosing patients.
- Application-grounded evaluation: Application-grounded studies may use the exact application task or a simpler partial task, with human-produced explanations as an important baseline.
- Human-grounded evaluation: Human-grounded evaluation uses simpler human-subject experiments that preserve the target application's essence and can test general explanation quality.
- Human-grounded evaluation: Human-grounded tasks include forced-choice comparisons, forward simulation of model outputs, and counterfactual simulation of changes needed for desired predictions.
- Functionally-grounded evaluation: Functionally-grounded evaluation uses formal interpretability proxies without human experiments, but proxy selection remains a central challenge.
4 Open Problems in the Science of Interpretability, Theory and Practice
The paper proposes linking real-world, simplified human, and functional evaluations while investigating task- and method-related factors that shape explanation needs. It outlines data-driven repositories, latent-dimension analysis, and human-grounded experiments as a path toward more rigorous interpretability science.
- Three evaluation types must inform one another: real-world task needs should shape simplified tasks, while functional proxies should reflect real-world performance.
- Data-driven factors: A data-driven matrix could compare methods across real-world tasks, then reveal latent dimensions associated with interpretability.Rows represent specific tasks, columns represent methods, and entries record end-task performance.
- Data-driven factors: Creating human-input repositories is harder than building standard datasets because each case requires human assessment, though crowdsourcing may address this challenge.
- Data-driven factors: Constructing the task-method matrix is expensive, and interpreting its latent dimensions requires iterative hypotheses and validation.Less expensive human-grounded evaluations on simulated tasks can test proposed hypotheses.
- Task-related dimensions: Task-related explanation needs may vary with global versus local goals, incompleteness severity, time constraints, and user expertise.Global explanations support scientific understanding or bias detection, whereas local explanations justify specific decisions.
- Method-related dimensions: Method-related factors include cognitive-chunk form and number, compositionality, interactions among chunks, and how people understand uncertainty or stochasticity.A prototype may contain more information than a feature, while compositional structure can reduce what users process at once.
5 Conclusion: Recommendations for Researchers
The paper recommends aligning interpretability claims with evaluation type and using a shared taxonomy to make applications and methods comparable. It presents these practices as groundwork for more rigorous interpretability research while acknowledging unresolved links among applications, human understanding, and machine-learning regularizers.
- The paper lays groundwork for rigorously defining and evaluating interpretability, but formal links among applications, human understanding, and machine-learning regularizers remain open.
- Interpretability research should choose evaluations that match the specificity and focus of its claims.Application-focused contributions call for application-grounded or closely related human-grounded evaluation, while model-class optimization calls for functionally-grounded metrics.
- Researchers should categorize applications and methods with a common taxonomy to support evaluation, citation, and comparison.The paper illustrates this with distinctions between local and global explanation needs and application-level versus general evaluation.
1. How is the problem formulation incomplete? (Section 2)
The proposed taxonomy organizes interpretability evaluation by level and by task- and method-related factors. It is intended to replace vague claims with shared terms that clarify what is being evaluated and why.
- Evaluation level can be application-based, a general user study, or a proxy measure.
- Task-related factors include global versus local scope, incompleteness severity, user expertise, and time constraints.
- Method-related factors include cognitive-chunk form and number, compositionality, monotonicity, and uncertainty.
- A shared and evolving taxonomy can move research away from vague claims about a model’s interpretability toward common application and method categories.