PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, Chudi Zhong

arXiv:2103.11251v2cs.LGstat.ML

TL;DR

Interpretable machine learning matters for high-stakes decisions, troubleshooting, and practical adoption, while black-box explanations can be misleading. This survey organizes a confusing literature into a pathway covering fundamental principles and technical challenges, concluding that interpretability need not generally reduce accuracy. All conclusions remain sensitive to the chosen model space and to information lost when dimensionality reduction emphasizes local structure.

  • Problem

    Black-box models can create serious societal problems, while posthoc explanations may be misleading and make troubleshooting difficult.

  • Method

    The survey provides a pathway through interpretable machine-learning topics by reviewing principles, methods, and open technical problems.

  • Results

    The survey concludes that interpretable models do not generally sacrifice accuracy and identifies important technical challenges across interpretable machine learning.

  • Takeaways & Limitations

    Interpretability supports troubleshooting and practical use, but methods should preserve relevant global structure and account for the assumptions defining a model space.

  • Takeaways & Limitations

    Rashomon ratios depend on the chosen hypothesis space and may not be directly comparable across different model spaces.

Abstract

from arXiv · show

Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the "Rashomon set" of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.

Introduction

The survey frames interpretability as essential for trustworthy, usable, and troubleshootable ML, while distinguishing interpretable ML from a confusing explainability literature. It organizes the field around fundamental principles and technical challenges spanning classical and modern problems.

  • Interpretability helps people troubleshoot and use ML models in practice, especially when black-box failures affect health, freedom, racial bias, or safety.
  • The survey provides fundamental principles and examines 10 technical grand challenges in inherently interpretable machine learning.
  • These challenges range from sparse logical models and scoring systems to disentanglement, dimensionality reduction, causal or physical constraints, Rashomon sets, and reinforcement learning.
  • Black-box models can be difficult to troubleshoot and may predict correctly for the wrong reasons, producing poor real-world performance or serious societal consequences.
  • The literature combines interpretability and explainability in ways that can obscure distinctions, practical use-cases, and precise arguments.

General Principles of Interpretable Machine Learning

Interpretable machine learning uses domain-specific constraints to make model reasoning understandable, especially for high-stakes decisions, while avoiding a presumed accuracy penalty. The survey distinguishes inherently interpretable models from explanations of black boxes and organizes major principles around trust, accuracy, iterative refinement, and technical challenges.

  • Definition: Interpretability is defined by domain-specific constraints that make a model, its predictions, or its data more understandable to humans.The relevant constraints can differ substantially across domains.
  • Trust: Interpretable models allow users to decide whether to trust them, but do not necessarily create trust and may instead enable distrust.They provide more information about reasoning than black boxes, supporting scrutiny of possible generalization beyond the dataset.
  • Accuracy: Interpretability is not generally opposed to accuracy: across practical data-science processes and static datasets, interpretable models have not proven less accurate.Interpretability can support troubleshooting, which the survey states can lead to better accuracy.
  • Accuracy: The accuracy–interpretability dichotomy is especially misleading because simple models often perform well on tabular data, while interpretable neural networks can retain accuracy on raw data.The survey contrasts meaningful tabular features with raw pixels, sound, or text, where visual or analogous explanations may be required.
  • Research agenda: The survey frames interpretability as an iterative, problem-specific process and identifies technical challenges ranging from sparse models and additive constraints to disentanglement, causal constraints, Rashomon sets, and reinforcement learning.Performance and interpretability metrics may both be refined during the full data-science process.
  • High-stakes use: For high-stakes decisions, inherently interpretable models should be used when possible rather than black boxes supplemented by explanations.The survey distinguishes interpretable ML from XAI approaches that approximate or explain black-box models.

1 Sparse Logical Models: Decision Trees, Decision Lists, and Decision Sets

Sparse logical models use human-readable logical statements, while a central challenge is optimizing predictive performance and sparsity together under difficult computational constraints.

  • Interpretability and sparsity: Sparsity is commonly used to improve interpretability for tabular logical models because people can handle only 7±2 cognitive entities simultaneously.More sparsity does not always mean more interpretability when complex relationships require several coordinated components.
  • Model forms: Logical models express predictions through human-understandable if-then, or, and and statements.Decision trees branch on conditions, decision lists apply ordered rules, and decision sets predict positively when any conjunction is satisfied.
  • Challenge 1.1: Sparse decision trees: Full decision-tree optimization is NP-complete, so heuristic greedy splitting and pruning have historically been the dominant approach.Recent methods instead directly optimize performance and tree sparsity, often measuring sparsity by the number of leaves.
  • Challenge 1.1: Sparse decision trees: GOSDT produces a 9-leaf Monk 2 tree that is optimal for balancing accuracy and sparsity, compared with CART’s 16-leaf tree.Optimization formulations penalize tree complexity through a trade-off parameter multiplied by the number of leaves.
  • Challenge 1.1: Sparse decision trees: Scaling optimal trees remains difficult because current methods can scale exponentially with the number of dimensions and generally target medium-sized datasets.Variable screening, compact mathematical-programming formulations, lower bounds, and search scheduling are proposed routes to better scalability.
  • Challenge 1.2: Continuous variables: Continuous variables remain challenging because preprocessing into indicators prevents jointly optimizing split variables, thresholds, and overall tree structure.Selecting thresholds and structures together is identified as an important unresolved problem for decision-tree optimization.

2 Scoring systems

Scoring systems make predictions from a few small integer point values, but jointly optimizing accuracy, sparsity, integrality, and user constraints is computationally difficult.

  • Purpose and structure: Scoring systems use a few small integer coefficients so users can calculate predictions by adding, subtracting, and multiplying without a computer.They support quick decisions in high-stakes medical and criminal-justice settings and can facilitate counterfactual reasoning.
  • Optimization formulation: A generic scoring-system objective minimizes loss plus a penalty on nonzero terms, with user-specified trade-offs and additional domain constraints.Coefficients may be restricted to small integers such as −10 through 10.
  • Optimization difficulties: Rounding real-valued coefficients can eliminate signal from many variables and reduce predictive performance when their contributions are jointly significant.A vector such as [5.3, 6.1, 0.31, 0.30, ...] can become [5, 6, 0, 0, ...] after rounding.
  • Solution approaches: Exact optimization becomes difficult for larger problems, especially when continuous covariates are discretized into many threshold-based dummy variables.Using too few thresholds can lose accuracy, whereas using all possible thresholds can substantially enlarge the problem.
  • Solution approaches: Mixed-integer programming can solve scoring systems directly when the problem is not too large and the loss is discrete or linear.For nonlinear logistic loss, RiskSLIM uses cutting planes within a branch-and-bound framework with solver callbacks.
  • Human-guided model development: A practical workflow can combine user constraints, model exploration, manual adjustment, and validation on data not used for training.The example includes constraints on age dependence and error rates before clinical adoption as a decision aid.

3 Generalized Additive Models

Generalized additive models extend linear and additive models with flexible feature-wise functions while retaining a transparent additive structure. The survey highlights challenges in controlling their sparsity, smoothness, monotonicity, and usefulness for troubleshooting complex data.

  • Model structure: GAMs generalize linear and additive models by assigning potentially nonlinear univariate component functions to individual features.With identity or logistic link functions, the same form describes additive regression or generalized additive classification models.
  • Model structure: GAM component functions can be visualized individually, including the relationship between two-hour plasma glucose concentration and diabetes risk.Continuous features produce informative curves, while bivariate components can be shown as heatmaps for pairwise interactions.
  • Model structure: Integer weights with few nonzero indicator terms turn a GAM into a scoring system, while directional indicators and nonnegative coefficients can enforce monotonicity.Spline basis functions provide another representation for component functions.
  • Challenges: Boosted stumps or trees make GAMs powerful and reliable, but boosting adds terms iteratively, making sparsity and smoothness difficult to control.The survey connects boosting with strong AUC and accuracy while identifying interpretability-control challenges.
  • Challenges: The survey identifies controlling GAM simplicity and using GAMs to troubleshoot complex datasets as two central technical challenges.Relevant concerns include sparse component selection, smoothness, monotonicity, missing confounders, biased records, and reconciling data sources.
  • Challenges: GAM-based troubleshooting can expose counterintuitive medical-record patterns and motivate progressively simpler models for clinical use.The survey describes raw medical records with missing measurements, unobserved variables, and ambiguous insurance codes as a motivating setting.

4 Modern case-based reasoning

Modern case-based reasoning classifies new instances by comparing them with similar examples or learned prototypes, extending this human-like strategy to structured and raw data. Neural prototype methods provide visual evidence for predictions, but handling complex data such as video remains an open challenge.

  • Foundations: Case-based reasoning solves new problems using similar past cases and includes nearest-neighbor and prototype-based techniques.Nearest neighbors retrieve similar training instances, whereas prototypes represent characteristic examples or parts.
  • Foundations: Case-based reasoning applies to tabular data, images, and other raw data by separating feature extraction from prediction when humans can verify extracted concepts.The approach can also support matching treatment and control units in observational causal inference.
  • Nearest neighbors: Learned distance metrics and latent-space mappings are mathematically equivalent perspectives for making nearest-neighbor classification effective.The mapping transforms the original metric so Euclidean distance in latent space supports k-nearest-neighbor prediction.
  • Prototypes: Prototype-based methods address cases where different parts of an observation resemble different references, motivating part-based prototypes for images and recipes.Whole-observation comparison can be inappropriate when meaningful similarities are localized to distinct components.
  • Deep case-based reasoning: Deep prototype networks compare inputs with learned prototypes or prototypical parts and provide prediction evidence through similar visual patterns.Li et al. used decoded digit prototypes, while ProtoPNet used training-image patches and achieved accuracy comparable to non-interpretable black-box models on CUB-200-2011.
  • Open challenges: Efficient case-based reasoning for video remains an open challenge because videos are high-dimensional ordered sequences of frames.Although dynamic prototype networks learn prototypical video patches, efficiently comparing videos and finding similar examples remains unresolved.

5 Complete supervised disentanglement of neural networks

Complete supervised disentanglement aims to align neural-network information flow with researcher-specified, human-interpretable concepts. The survey identifies open challenges spanning whole-network coverage, concept selection, continuous concepts, and interpretable prediction mappings.

  • Concept-based representations: A disentangled latent space routes information about each concept through a corresponding neuron, making predictions easier to understand.The ideal representation aligns latent-space axes with concepts such as lamps or beds and supports concept-based reasoning.
  • Concept-based representations: Standard neural networks can mix unrelated concepts on the same neuron, so disentanglement is not guaranteed without architectural or training constraints.Post hoc analyses have found neurons activated by unrelated concepts such as dining tables and Greek-style buildings.
  • Open challenges: Current methods do not yet make every neuron or every layer of a deep neural network simultaneously interpretable.Existing approaches generally target one layer, leaving semantic meanings elsewhere in the network unknown.
  • Open challenges: Supervised disentanglement still requires choosing useful concepts and converting continuous concepts into suitable representations.Applications may need domain-expert concept selection and thresholds for variables such as age or tumor size.
  • Open challenges: Making the latent representation interpretable does not by itself make the mapping from concepts to outputs interpretable.The decision process includes both x → c and c → y mappings, while current methods primarily target c.

6 Unsupervised disentanglement of neural networks

Unsupervised disentanglement seeks interpretable information flow when concepts are unknown, numerous, or incompletely labeled. The survey emphasizes unresolved challenges in discovering meaningful concepts across domains and evaluating them under labeling bias.

  • Motivation and scope: Unsupervised disentanglement is needed when concepts are unknown or too numerous to parameterize, including material patterns and unlabeled scene information.It aims to make network information flow easier to understand and interact with without predefined concepts.
  • Existing approaches: Concept discovery remains difficult on realistic data: SCAE and related systems perform poorly beyond simple datasets, including SVHN-style digit images.The cited figure contrasts SVHN samples with stroke-like templates discovered by SCAE.
  • Open challenges: Unsupervised disentanglement must address both unknown concepts and known concepts whose labels are incomplete or biased.The survey distinguishes these as two domains requiring different evaluation and modeling strategies.
  • Evaluation and domain transfer: Materials science lacks ground-truth labels for key patterns, making it difficult to choose inductive biases and evaluate discovered concepts quantitatively.The survey uses metamaterial band-gap prediction as an example where useful patterns can be discovered without supervision.
  • Evaluation and domain transfer: Human-annotation-based evaluation can be problematic when labels cover objects but omit information such as lighting and furniture style.An unsupervised model may discover important concepts absent from the annotation set.
  • Open challenges: Compositional constraints developed for natural images may not transfer directly to materials science, where constituent patterns can jointly determine physical properties.The survey specifically questions how to redefine image-based compositional assumptions for other domains.

7 Dimension Reduction for Data Visualization

Dimension-reduction methods project high-dimensional data into low-dimensional spaces for human visualization, trading off preservation of global distances against local neighborhoods. The survey highlights computational, hyperparameter, and interpretability challenges in these projections.

  • Purpose and formulation: Dimension reduction maps high-dimensional data into usually 2D or 3D spaces so people can inspect structure, form hypotheses, and design features.The low-dimensional mapping itself is treated as an interpretability constraint.
  • Method development: Recent methods differ substantially in their visual results because each chooses different distance, neighborhood, initialization, and loss-function strategies.The survey uses isotropic-Gaussian projections to illustrate these differences and the effects of loss design.
  • Global and local structure: Global methods preserve pairwise distances, whereas local methods emphasize nearest-neighbor structure and can better preserve clusters while losing overall layout.On MNIST, PCA preserves scale differences but fails to separate digits, while t-SNE separates digits but loses scale information.
  • Method development: t-SNE addressed the crowding problem by transforming high-dimensional distances, but it remains sensitive to hyperparameters and weak at preserving global structure.Its variants target slow runtimes and related shortcomings through graph-based approximations and acceleration.
  • Practical limitations: Modern dimension-reduction methods often have many data-dependent hyperparameters that are difficult to tune without knowing the original structure.Poor choices can produce misleading visualizations, as illustrated for t-SNE and UMAP on the Mammoth dataset.
  • Open challenges: A major open problem is making the high-to-low-dimensional transformation itself interpretable through explanatory visualizations or constrained methods.PCA can sometimes reveal dominant original dimensions, but modern mappings are typically more complex.

8 Machine learning models that incorporate physics and other generative or causal constraints

Physics-guided machine-learning models incorporate physical laws or other domain constraints into learning, making them alternatives to purely data-driven models and numerical solvers. The survey focuses on PINN training, constraint integration, uncertainty reduction, and extensions beyond neural networks.

  • Motivation and scope: Physics-guided models train against ordinary or partial differential equations, so their predictions are constrained to follow supplied physical laws.This can provide interpretable alternatives to traditional numerical methods for solving PDEs.
  • Physics-informed neural networks: A PINN can approximate a differential-equation solution without conventional labeled pairs by minimizing residual error at selected input points.Network derivatives are obtained through back-propagation, allowing training from the equation itself.
  • Physics-informed neural networks: PINNs have been extended with alternative formulations and convolutional or graph-neural-network backbones for scientific applications.Reported applications include fluid-mechanics modeling and other scientific domains.
  • Physics-informed neural networks: For the heat equation, the PINN output must satisfy the equation together with initial and Dirichlet boundary conditions, whose violations contribute to the loss.Observed data can additionally be included through a mean-squared-error term.
  • Open challenges: PINN training remains challenging because numerical stiffness can destabilize gradient backpropagation across competing loss terms.Dynamic balancing of loss terms has been proposed, but improving PINN training remains open.
  • Open challenges: Physics-guided learning also faces uncertainty from scarce data and unresolved questions about integrating constraints into non-neural models.The survey calls for co-designing experiments or simulations with training and exploring models such as Gaussian processes.
  • Beyond physical laws: Knowledge-infused deep learning extends constraint incorporation beyond physics by guiding neural networks with relationships encoded in knowledge graphs.Knowledge graphs represent relationships among entities, including common-sense knowledge.

9 Characterization of the “Rashomon” set of good models

The Rashomon set is the collection of nearly equally accurate models for a dataset, and characterizing it may reveal simpler, fairer, or otherwise preferable models among accurate alternatives. The section examines measures of its size while emphasizing that parameter-space volumes and ratios depend on representation choices.

  • Definition: The Rashomon set contains models whose training loss is within a threshold of a reference model’s loss.The threshold is controlled by the Rashomon parameter ϵ or a percentage of the reference loss.
  • Why multiplicity matters: A large Rashomon set must contain a simple model under weak assumptions when simpler functions can approximate the relevant complex function class.The argument uses a large region of complex functions and an approximating class of simpler functions such as decision trees.
  • Why multiplicity matters: Similar performance across diverse algorithms can serve as a rule of thumb suggesting a large Rashomon set and the possibility of an equally accurate simpler model.The suggested algorithms include boosted trees, support vector machines, neural networks, random forests, and logistic regression.
  • Why multiplicity matters: Rashomon-set models can also vary in properties such as fairness and monotonicity, making the set useful for analyzing modeling choices beyond interpretability.The set is therefore relevant to selecting models with multiple desired properties.
  • Characterization and measurement: There is no settled choice among Rashomon-set size measures, and current research has only begun characterizing and using model multiplicity.The pattern Rashomon ratio addresses some parameter-space issues but does not eliminate the broader measurement question.
  • Characterization and measurement: Parameter-space volume, sampling, and pattern ratios provide possible size measures, but different representations and denominators can change the resulting values.Pattern Rashomon ratios count unique prediction patterns, while parameter-space measures can be altered by parameterization or duplicated variables.

10 Interpretable reinforcement learning

Interpretable reinforcement learning seeks policies whose actions and intent humans can understand, despite RL’s exploration, delayed rewards, and complex state spaces. The section surveys policy representations and identifies the challenge of preserving performance while simplifying policies or states.

  • RL setting: RL agents repeatedly observe states, choose policy-driven actions, and receive rewards that determine subsequent states.In diabetes management, states include glucose and patient measurements, actions are insulin injections, and rewards depend on sugar levels.
  • RL setting: RL is harder to interpret than supervised learning because agents collect data through exploration and may receive delayed feedback after action sequences.These properties complicate training, understanding, and debugging.
  • Motivation: Interpretability could reduce harmful actions and support troubleshooting in applications such as long-term medical treatment and human-robot collaboration.Understanding choices and intent may help constrain the search space.
  • Interpretable policy representations: Decision trees and rule lists can represent interpretable policies or value functions using logical conditions and action distributions at leaves.Policy trees may be grown incrementally when estimated discounted future reward improves sufficiently.
  • Interpretable policy representations: Symbolic states, hierarchical skills, and human-described task decompositions provide alternative assumptions for constructing understandable policies.A task such as stacking a blue block can be decomposed into finding, getting, and placing the block.
  • Open challenges: No general, well-performing interpretable method for deep RL currently provides transparent actions or intent, while post-hoc explanations can be incorrect or incomplete.Existing approaches include tree-based explanations, reward decomposition, attention methods, and saliency maps used for exploration rather than explanation.
  • Open challenges: Interpretable RL must determine which constraints preserve accuracy, whether interpretable policies can match black-box performance, and how to simplify complex state spaces.A rule-based PIRL policy experienced a performance drop, while state-space reductions have enabled faster planning in a clinical setting.

11 Problems that weren’t in our top 10 but are really important

The survey highlights additional open problems involving preprocessing, uncertainty, case difficulty, model design, visualization, causal matching, deployment, legal explanations, and broader classes of interpretable models. These omissions underscore that interpretable ML extends beyond the ten featured challenges.

  • Additional open problems: Preprocessing remains an open problem because methods such as PCA can improve computational tractability while combining original features and damaging interpretability.The section asks for preprocessing that preserves predictive power while retaining interpretable features.
  • Additional open problems: Other open questions concern conveying uncertainty, assigning simpler models to easier cases, and co-designing models with visualizations.The survey also points to interactive visualization tools and case-dependent model allocation.
  • Model design: The motivation for interpretable neural networks on tabular data is unclear because neural networks generally offer little benefit there and are harder to optimize reliably.Unlike some optimal methods, neural networks do not generally provide a known distance from a globally optimal solution.
  • Causal inference: Interpretable matching in observational causal inference can create overlapping treatment and control data to support interpretable models for treatment effects or policies.Sparse decision trees and other supervised interpretable methods can be applied to matched data.
  • Practice and scope: The practical use of explanations, legal requirements, and interpretable model families remains unresolved, including whether a general legal “Right to Explanation” exists under the GDPR.The survey notes that explanations are often used internally for troubleshooting rather than shown to users.

12 Conclusion

The conclusion presents the survey as an entry point into a confusing and interdisciplinary interpretable-ML literature. It emphasizes both the practical importance of interpretable models and the mathematical elegance of simple, accurate solutions.

  • Conclusion: The survey aims to guide readers through important interpretable-ML topics amid confusing terminology and blurred distinctions between interpretability and black-box explanation.It distinguishes designing interpretable models from explaining black boxes.
  • Conclusion: The authors frame interpretable models as socially important while also emphasizing the beauty and mathematical elegance of simple, accurate models.The conclusion links practical significance with appreciation of model structure.
Loading 2103.11251v2…