Source-linked AI summary
Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
Christoph Molnar, Giuseppe Casalicchio, Bernd Bischl
TL;DR
Interpretability matters for deploying ML in products, decisions, and research, yet its evaluation and meaning remain unsettled. This paper reviews IML’s history and methods, concluding that the field has reached a first state of readiness while challenges remain in uncertainty, causality, feature dependence, and definition.
Problem
Interpretability is difficult to evaluate because no accepted ground-truth explanation or straightforward measure of interpretability exists.
Method
The paper surveys IML’s historical foundations, interpretation methods, evaluation approaches, and outstanding challenges.
Results
IML has reached a first state of readiness, with maturing methods, open-source implementations, and growing industrial adoption.
Takeaways & Limitations
Progress in IML should draw on statistics and rule-based ML while incorporating sensitivity analysis, causal inference, and social-science perspectives.
Takeaways & Limitations
Important challenges remain in statistical uncertainty, causal interpretation, dependent features, and defining interpretability.
Abstract
from arXiv · showhide
We present a brief history of the field of interpretable machine learning (IML), give an overview of state-of-the-art interpretation methods, and discuss challenges. Research in IML has boomed in recent years. As young as the field is, it has over 200 years old roots in regression modeling and rule-based machine learning, starting in the 1960s. Recently, many new IML methods have been proposed, many of them model-agnostic, but also interpretation techniques specific to deep learning and tree-based ensembles. IML methods either directly analyze model components, study sensitivity to input perturbations, or analyze local or global surrogate approximations of the ML model. The field approaches a state of readiness and stability, with many methods not only proposed in research, but also implemented in open-source software. But many important challenges remain for IML, such as dealing with dependent features, causal interpretation, and uncertainty estimation, which need to be resolved for its successful application to scientific problems. A further challenge is a missing rigorous definition of interpretability, which is accepted by the community. To address the challenges and advance the field, we urge to recall our roots of interpretable, data-driven modeling in statistics and (rule-based) ML, but also to consider other areas such as sensitivity analysis, causal inference, and the social sciences.
1 Introduction
Interpretability can determine whether ML is used in products, decisions, or research. The paper reviews IML’s historical foundations, interpretation methods, and remaining challenges.
- Interpretability can influence whether an ML model is used in a product, decision process, or research.
- IML methods can discover knowledge, debug or justify models and predictions, and control or improve models.
- The paper reviews IML’s historical building blocks and methods for interpreting models.
- The authors argue that IML has reached a state of readiness while important challenges remain.
2 A Brief History of IML
IML builds on long-standing traditions in regression, rule-based modeling, and machine learning. Its recent expansion followed the rise of deep learning and increased attention to interpretability.
- Linear regression modeling dates to the early 19th century and developed into tools including generalized additive models and elastic net.
- ML increasingly adopted nonlinear, non-parametric algorithms whose complexity is controlled by hyperparameters selected through cross-validation.
- This flexibility often produced models with good predictive performance but reduced interpretability.
- Random-forest feature importance was an important IML milestone before the field accelerated around 2015.
- The random forest paper had over 60,000 Google Scholar citations by September 2020, alongside many frequently cited improvements to its importance measure.
- Rule-based ML methods function both as standalone algorithms and as building blocks for many IML approaches.
3 Today
IML has reached a first state of readiness through methodological consolidation, clearer understanding of weaknesses, available open-source implementations, and growing institutional and industrial attention.
- IML is maturing through surveys, consolidation of terminology and knowledge, and work defining and evaluating interpretability.
- Researchers have identified weaknesses in general IML methods and in methods including permutation importance, Shapley values, counterfactuals, partial dependence plots, and saliency maps.
- Open-source implementations include iml and DALEX for R, and Alibi and InterpretML for Python.
- Regulation, trustability, transparency, and fairness have intensified discussion of interpretability needs, while IML has entered industry.
4 IML Methods
The paper groups IML methods into component analysis, sensitivity analysis, and surrogate models. These approaches differ in model dependence, locality, and how they connect inputs, predictions, and interpretable structures.
- Analyzing Components: Component-based methods assign meaning to decomposable model parts, but their interpretations are tied to the model structure.
- Analyzing Components: Linear regression interprets feature weights, while decision trees and rules trace predictions through learned structures.
- Analyzing Components: High-dimensional linear models and deep trees become less interpretable; LASSO and pruning reduce the parts requiring interpretation.
- Analyzing Components: Component analysis can suit well-understood algorithms but does not combine well with cross-validation across broad model classes.
- Analyzing Sensitivity: Sensitivity methods are mostly model-agnostic: they manipulate inputs, analyze predictions, and provide local or global explanations.
- Explaining Individual Predictions: Local methods explain individual predictions using approaches such as Shapley values, counterfactual explanations, and CNN-specific saliency maps.
- Explaining Global Behavior: Global methods explain expected behavior through feature importance, which ranks relevance, and feature effects, which describe changes in predicted outcomes.
- Surrogate Models: Surrogate methods train interpretable models to copy black-box behavior, then analyze the surrogate’s components; LIME applies this locally near a target point.
5 Challenges
IML faces unresolved challenges in statistical uncertainty, causal interpretation, feature dependence, evaluation, and broader sociotechnical deployment. The field needs clearer assumptions, stronger evaluation foundations, and more dynamic, interdisciplinary perspectives.
- Statistical Uncertainty and Inference: IML explanations are often reported without quantified uncertainty, although research is beginning to address uncertainty for feature importance, relevance propagation, and Shapley values.Both models and explanations are computed from data and are therefore subject to uncertainty.
- Statistical Uncertainty and Inference: Structural or distributional assumptions underlying IML analyses should be stated clearly and tested with better diagnostic tools.The paper warns that insufficient rigor could reproduce statistical problems such as p-hacking and multiple-testing errors.
- Causal Interpretation: Causal interpretation requires models to reflect the underlying causal structure, but predictive performance and causality can conflict when confounders are omitted.The paper uses wet ground and tomorrow’s weather to illustrate why predictive usefulness does not alone justify causal interpretation.
- Dependent Features: Dependent features complicate attribution and extrapolation because correlated variables share information and feature permutation can break their association.Random-forest importance may attribute higher importance to correlated features, while permutation-based methods can create unrealistic data points.
- Definition of Interpretability: Interpretability lacks a widely accepted rigorous definition, making evaluation harder than predictive assessment because no ground-truth explanation or straightforward interpretability measure is available.Objective and human-centered evaluations instead assess aspects such as sparsity, interaction strength, fidelity, perturbation sensitivity, and simulatability.
- More Challenges Ahead: IML research must move beyond static analyses of fixed models and data toward the full process of data collection, explanation, and use across diverse people and institutions.The paper identifies human-computer interaction, psychology, sociology, statistics, and computer science as relevant areas for addressing these challenges.