Source-linked AI summary
Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
Cynthia Rudin
TL;DR
High-stakes black-box models can produce consequential decisions while their posthoc explanations may be unfaithful or insufficient. The paper advocates inherently interpretable models, presents applications and algorithms that support them, and reports cases where transparent models match black-box accuracy. It also notes practical limitations involving proprietary software and the computational hardness of constructing optimal models.
Problem
Black-box models are used in high-stakes decisions despite risks involving errors, limited accountability, and explanations that may not faithfully represent their computations.
Method
The paper distinguishes posthoc explanations from inherently interpretable models and examines interpretable applications, model forms, and optimization methods.
Results
The FICO Explainable ML Challenge found no performance difference between interpretable and explainable models, and a globally interpretable additive model won its Recognition Prize.
Takeaways & Limitations
Examples in criminal justice, air-quality prediction, and medicine support considering transparent models for high-stakes applications.
Takeaways & Limitations
Counterfactual explanations generally do not suffice for high-stakes black-box decisions, and proprietary interpretable software is not clearly better than publicly available alternatives.
Abstract
from arXiv · showhide
Black box machine learning models are currently being used for high stakes decision-making throughout society, causing problems throughout healthcare, criminal justice, and in other domains. People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to \textit{explain} black box models, rather than creating models that are \textit{interpretable} in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward -- it is to design models that are inherently interpretable. This manuscript clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision.
1 Introduction
High-stakes ML is increasingly used in healthcare, criminal justice, and other domains despite black-box models lacking human-understandable explanations. The paper argues for inherently interpretable models, whose explanations faithfully reflect their computations.
- Motivation: Black-box models increasingly influence high-stakes predictions that deeply affect human lives.Reported consequences include incorrect parole denials, poor bail decisions, unsafe pollution predictions, and inefficient resource allocation.
- Motivation: Posthoc Explainable ML creates a second model to explain a black box, but such explanations can be unreliable and misleading.The paper contrasts this approach with models that are interpretable by construction.
- Motivation: Inherently interpretable models provide explanations that are faithful to what the model actually computes.
- Interpretability: Interpretability is domain-specific and may involve useful model forms or structural constraints such as monotonicity, causality, additivity, sparsity, or physical knowledge.For structured data, sparsity can help people understand how variables interact jointly.
- Scope: The manuscript discusses Explainable ML problems before addressing challenges in Interpretable ML, focusing mainly on high-stakes decisions and model troubleshooting.
2 Key Issues with Explainable ML
The paper identifies several problems with explaining black-box models: explanations may be unfaithful, unintelligible, incompatible with contextual information, and operationally error-prone. It also rejects an assumed accuracy–interpretability trade-off, arguing that interpretable models can often match complex models, especially on structured data.
- Accuracy and interpretability: The paper calls the accuracy–interpretability trade-off a myth, noting that simple and complex classifiers often show no significant performance difference on structured data with meaningful features.Uninterpretable algorithms may still support knowledge discovery, but are not generally the final goal.
- Accuracy and interpretability: Figure 1 presents an unquantified fictional accuracy–interpretability curve rather than evidence from data.The paper describes this smooth trade-off as atypical in data science.
- Accuracy and interpretability: The assumed trade-off has discouraged interpretable modeling, while researcher training and standard ML toolkits provide limited support for interpretable methods.The author reports not having encountered a high-stakes application requiring a complete black box across several domains.
- Unfaithful explanations: Black-box explanations cannot be perfectly faithful, so they may inaccurately represent the original model in parts of the feature space.A perfectly faithful explanation would equal the original model and make the original unnecessary.
- Unfaithful explanations: Explanation models can achieve similar predictions while using different features, making them misleading accounts of how the black box computes.The paper uses recidivism prediction and COMPAS to illustrate this distinction.
- Insufficient explanations: Saliency maps often show where an image network looks without explaining why it assigns a class, and similar maps can appear across classes.Showing explanations only for the correct class can therefore create false confidence in the explanation and the black box.
- Context and human judgment: Black boxes make it difficult to combine database variables with outside information that should raise or lower a risk estimate.The paper identifies this as a recurring issue in high-stakes decisions, including recidivism assessment.
3 Key Issues with Interpretable ML
The paper identifies barriers to replacing black boxes with interpretable models, including proprietary incentives, computational and expertise demands, and limits of posthoc explanations. It also presents cases where transparent models match black-box performance or avoid serious deployment failures.
- Proprietary incentives: Proprietary incentives can preserve black-box systems even when accurate transparent alternatives exist.COMPAS is described as equally accurate to a three-rule CORELS model, while companies can profit from licensing proprietary predictions.
- Empirical comparisons: COMPAS and CORELS have similar true and false positive and negative rates on Broward County data.CORELS uses only age and number of past crimes in a three-rule interpretable model.
- Deployment risks: BreezoMeter predicted dangerously poor wildfire air quality as good, whereas the EPA’s free air-quality index would have provided a reliable result.The proprietary model’s failure illustrates risks of opaque high-stakes prediction systems.
- Limits of explanations: Counterfactual explanations may fail because the minimal input change can differ across individuals and combine information outside the database.The paper therefore argues they generally do not suffice for high-stakes black-box decisions.
- Construction challenges: Interpretable models require application-specific constraints, computational effort, and domain expertise.Constrained optimization can be computationally hard, although the paper argues that high-stakes applications justify the additional cost.
- Interpretability and accuracy: Transparent models may uncover important predictive patterns without relying on black-box complexity.The paper connects this possibility to the belief that black boxes uniquely discover subtle hidden patterns.
4 Encouraging Responsible ML Governance
The paper argues for governance that favors interpretable models when they perform as well as black boxes and requires transparency about interpretable alternatives. It also cautions that mandates applied too broadly could reduce industrial participation.
- Regulatory limits: Current “right to an explanation” rules may not require explanations to be accurate, complete, or faithful to the underlying model.The paper warns that inadequate explanations can undermine such policies.
- Policy proposals: A proposed mandate would prohibit black-box deployment for certain high-stakes decisions when an equally performing interpretable model exists.Organizations could be held accountable for selling a black box when an equally accurate transparent alternative is available.
- Incentives: The paper suggests compensation structures such as lump-sum payments could encourage interpretable models for public-good applications.Such arrangements would reduce reliance on opacity and recurring licensing revenue.
- Scope and limitations: The proposed mandate would not solve all problems and, if applied too broadly, could reduce industrial participation in beneficial machine-learning applications.Its supported scope is limited to cases where accurate interpretable alternatives can be constructed.
- Policy proposals: A weaker proposal would require organizations introducing black boxes to report interpretable-model accuracy.This would make claimed accuracy–interpretability trade-offs easier to evaluate and encourage careful use of interpretable methods.
5 Algorithmic Challenges in Interpretable ML
Interpretable machine learning faces computational and domain-specific challenges, but optimization advances and suitable model designs can produce accurate, sparse, inherently interpretable models across applications.
- Interpretable models can potentially replace black boxes while exposing flaws in models and data and avoiding misleading posthoc explanations.
- Challenge #1: Constructing optimal logical models: Logical-model construction requires optimizing predictive error and model size, rather than relying on heuristics that may be inaccurate, unnecessarily large, or uninterpretable.
- Challenge #1: Constructing optimal logical models: CORELS demonstrates that theoretically hard logical-model problems can be solved for practical datasets using combined theoretical and systems-level techniques.
- Challenge #2: Construct optimal sparse scoring systems: Scoring systems are sparse linear models with integer coefficients, but finding accurate, compact systems requires solving computationally hard integer-lattice optimization problems.
- Challenge #2: Construct optimal sparse scoring systems: RiskSLIM and related cutting-plane methods can solve these scoring-system problems to optimality or near-optimality for large samples and moderate numbers of variables within minutes.
- Challenge #3 Define interpretability for specific domains and create methods accordingly, including computer vision: Prototype networks incorporate image prototypes and similarity computations directly into predictions, while remaining approximately as accurate as their original deep neural networks.
- Challenge #3 Define interpretability for specific domains and create methods accordingly, including computer vision: The FICO challenge found no performance difference between interpretable and explainable models, and a globally interpretable additive model won its Recognition Prize.
6 Conclusion
The commentary urges a shift from assuming black boxes are necessary toward developing inherently interpretable models for high-stakes decisions. It argues that policy and regulatory efforts should prioritize interpretable models to improve safety and trust.
- The paper seeks to shift machine learning away from assuming that black boxes are necessary for accurate predictions.
- It encourages policymakers to require significant attempts at interpretable models before accepting black-box models.
- Awareness of current challenges in interpretable machine learning could help policymakers demand greater safety and trust in high-stakes models.
- Without this shift, unclear standards for explanations could allow unsafe black boxes and contribute to poor decisions across criminal justice, air-quality guidance, and lending.
A On the Two Types of Black Box
Black boxes arise either because their functions are too complicated for humans to comprehend or because they are proprietary. These types differ in what can be inspected, though their consequences are related.
- One type of black box is too complicated for a human to comprehend, while the other is proprietary.
- For a complicated but nonproprietary model, users can still inspect its variables and model form.
B Performance Comparisons
For structured problems with meaningful covariates, no single machine learning algorithm consistently dominates, supporting the possibility that interpretable models can perform well. Publication practices can obscure this comparison by selectively reporting algorithms, datasets, and baselines.
- Structured-data algorithms generally perform similarly, with no algorithm clearly dominating the others.
- Variation from tuning one algorithm can exceed variation between different algorithms.
- The Rashomon Set argument suggests that interpretable models might perform well when no algorithm dominates.
- Selective publication encourages omitting accurate baselines, poorly performing datasets, and well-tuned baseline methods.
C Counterfactual Explanations
Counterfactual explanations offer sufficient feature changes for switching a black-box prediction, but they may not identify the lowest-cost action for the individual. Determining that action requires user-specific cost information.
- Counterfactual explanations state a sufficient, but not necessary, feature change for switching a prediction to another class.
- Multiple counterfactuals may differ in practical difficulty, so users would ideally receive the lowest-cost action according to their own cost metric.
- Identifying the lowest-cost counterfactual requires eliciting cost information from the user.
D Interpretable Models that Provide Smaller-Than-Global Explanations
Models can provide explanations smaller than their full global structure, allowing individual decisions to be justified with only a few conditions. Disjunctive rule models and falling rule lists offer globally or locally concise explanations.
- Smaller-than-Global Explanations: A complicated global model can still produce a sparse individual explanation requiring only one or two conditions.The credit-risk example focuses on justifying loan denials rather than approvals.
- Smaller-than-Global Explanations: Disjunctive normal form models deny loans when any one of several conjunctions of conditions is satisfied.Only the satisfied conjunction needs to be shown as the client’s defining reason.
- Smaller-than-Global Explanations: Decision rules, rule sets, and associative classifiers are alternative names for disjunctive normal form models.These models have been studied extensively and can be generated as globally interpretable models with few conjunctions.
- Smaller-than-Global Explanations: Falling rule lists use a few logical conditions to identify important decisions, then add conditions to distinguish among lower-risk groups.For patient mortality, the initial rules can identify a high-risk group before further categorizing low-risk patients.
E Algorithm Stability
Algorithmic instability can arise when many nearly equally accurate models exist, but this paper treats that flexibility as useful for expert customization and interpretability. Regularization improves stability while reducing users’ freedom to select among alternatives, and this view is not universally shared.
- Algorithm Stability: Small training-data changes can produce different decision trees, while correlated features can produce equally accurate but different linear models.The instability problem therefore concerns both tree-based and linear modeling settings.
- Algorithm Stability: The paper hypothesizes that instability reflects the Rashomon effect, in which many nearly equally good predictive models exist.Regularization increases stability but limits the user’s ability to choose among models.
- Algorithm Stability: For purely predictive applications, variable importance does not establish causality, so domain experts are needed to choose a single best model.The model’s predictive role does not imply that it represents how outcomes are generated.
- Algorithm Stability: Instability can let domain experts impose additional constraints or select the most interpretable model without sacrificing accuracy.This advantage applies when the alternative models are nearly or equally accurate.
- Algorithm Stability: Not all interpretability researchers agree that algorithmic instability is advantageous.The paper presents its positive assessment as a contested position.