PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, Chudi Zhong

arXiv:2103.11251v2cs.LGstat.ML

TL;DR

Interpretability matters for practical and high-stakes ML, but the growing interpretable/XAI literature can be confusing. This survey offers an entry point to important topics and reports no scientific evidence for a general accuracy–interpretability tradeoff across full data-science pipelines.

  • Problem

    The growing interpretable/XAI literature is confusing, despite interpretability's importance for practical use and high-stakes ML.

    Source: Introduction · p. 1, §12 · p. 55

  • Method

    The paper surveys important interpretable-ML topics to provide readers a pathway into the field.

    Source: §12 · p. 55

  • Results

    The survey reports no scientific evidence for a general accuracy–interpretability tradeoff across full data-science pipelines, and says interpretable models do not generally lose accuracy on fixed datasets and metrics.

    Source: General Principles of Interpretable… · pp. 5, 7

  • Takeaways & Limitations

    The survey offers an entry point to important interpretable-ML topics, an area the authors identify as key to model use in practice.

    Source: §12 · p. 55

    The Rashomon ratio depends on the considered hypothesis space, so ratios across different model spaces may not be directly comparable.

    Source: §9.1 · p. 47

Abstract

from arXiv · show

Interpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the "Rashomon set" of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.

Introduction

The survey argues that inherently interpretable models are important for trust, troubleshooting, and practical use, and presents guiding principles and 10 technical challenges in interpretable machine learning. It focuses on technical challenges while setting deployment, human-computer-interaction, robustness, and fairness topics aside.

Introduction

  • Interpretable models are easier to troubleshoot and use in practice, while inscrutable black boxes can cause serious harms and conceal errors, bias, and domain-shift failures.The authors argue that explanations of black boxes can instead mislead or add unwarranted authority, motivating models that are inherently interpretable.Source: Introduction · pp. 1–2
  • The survey offers general guiding principles to clarify interpretability’s distinction from explainability and help readers avoid problematic assumptions and practices.It notes that much literature assumes black boxes should be explained without considering whether an equally accurate interpretable model exists.Source: Introduction · p. 2
  • The 10 technical challenges span sparse trees and scoring systems, constrained additive models, case-based reasoning, neural-network disentanglement, dimensionality reduction, physical or causal constraints, Rashomon sets, and interpretable reinforcement learning.The survey includes both classical and modern challenges, some of which are difficult to solve or formulate; it sets sociotechnical deployment, HCI, robustness, and fairness interactions aside.Source: Introduction · pp. 2–3
  • Table 1 offers a rule of thumb for matching dataset types to supervised-learning algorithms, distinguishing clean, tabular, and raw data.Tabular data has categorical or real-valued features that are meaningful predictors individually; raw data is unprocessed and complex, such as images, medical records, or time series.Source: Table 1 · p. 3, Introduction · p. 3

General Principles of Interpretable Machine Learning

Interpretable ML constrains models using domain-specific requirements that make their reasoning understandable, enabling users to assess trust rather than guaranteeing it. The paper argues accuracy and interpretability are not generally opposed and recommends inherently interpretable models for high-stakes decisions when possible.

General Principles of Interpretable Machine Learning

  • An interpretable model obeys domain-specific constraints; suitable constraints vary, with tabular tasks often favoring fully understandable sparse models and visual tasks favoring case-based or disentangled networks.Formulating interpretable models can involve computationally hard optimization, data troubleshooting, and refinement of the chosen definition of interpretability.Source: General Principles of Interpretable… · pp. 3–5
  • Interpretability does not itself create trust; it gives users information to decide whether to trust a model, while black boxes make it harder to assess generalization.Source: General Principles of Interpretable… · p. 5
  • There is no general accuracy–interpretability tradeoff: troubleshooting can improve accuracy in full data-science workflows, and static-dataset studies find interpretable methods need not sacrifice accuracy.The paper distinguishes this from the accuracy cost of sparsity in very small models.Source: General Principles of Interpretable… · pp. 5–7
  • Across the data-science process, performance and interpretability metrics should be refined iteratively, with domain experts comparing constrained models and explaining their preferences.Source: General Principles of Interpretable… · p. 8
  • For high-stakes decisions, use inherently interpretable models when possible rather than posthoc explanations of black boxes, which can be misleading and dangerous.Interpretable ML creates a predictive model that is not a black box; it is distinct from approximating or explaining a black box after the fact.Source: General Principles of Interpretable… · p. 8
  • Posthoc explanations can mislead: saliency maps show which pixels are used but not how they support a prediction, and different methods can produce inconsistent maps.Source: General Principles of Interpretable… · pp. 8–9

1 Sparse Logical Models: Decision Trees, Decision Lists, and Decision Sets

Logical models use human-understandable if-then, and, and or statements to explain predictions. The section reviews sparse-model optimization and identifies scalability, continuous-feature handling, and global constraints as major challenges.

1 Sparse Logical Models: Decision Trees, Decision Lists, and Decision Sets

  • Decision trees, ordered if-then-else lists, and unordered OR-of-AND decision sets provide understandable prediction rules and are closely related representations.Logical models suit categorical data with complex interactions and multiclass problems, and are robust to outliers and able to handle missing data.Source: §1 · pp. 10–11
  • Greedy tree induction can leave accuracy and interpretability on the table because it does not repair poor splits; optimal tree learning is NP-complete and balances loss against leaf count.Methods include mathematical programming, stochastic search, and customized dynamic programming with branch-and-bound; decision-list and decision-set construction presents parallel optimization challenges.Source: §1 · pp. 11–12

1.1 Can we improve the scalability of optimal sparse decision trees?

  • Optimal sparse decision trees remain difficult to scale: current methods handle medium-sized datasets, while runtime can grow exponentially with the number of dimensions.For mathematical-programming methods, formulation is an open challenge: smaller formulations can speed LP relaxations, while stronger relaxations can improve bounds and prune the search.Source: §1 · p. 13

1.2 Can we efficiently handle continuous variables?

  • Optimizing continuous-feature splits jointly with tree structure is challenging: exhaustive threshold indicators greatly expand the search space, while approximating thresholds may sacrifice optimality.Support bounds have been computationally expensive to check, leaving open how to use them more effectively or preserve near-optimality with fewer indicators.Source: §1 · pp. 13–14

1.3 Can we handle constraints more gracefully?

  • Global domain constraints are difficult to enforce in greedy tree construction, and although optimization methods can accommodate them, they can make the problem harder.Falling-probability constraints can make rule lists more interpretable and useful while increasing optimization difficulty.Source: §1 · p. 14

2 Scoring systems

Scoring systems make predictions through simple point-based arithmetic, but designing sparse, accurate models that satisfy user constraints is computationally challenging. The section reviews optimization approaches and identifies scalability and constraint elicitation as open challenges.

2 Scoring systems

  • Scoring systems use a few small point values for human-computable predictions and can support counterfactual reasoning, though linear models handle interactions and multiclass problems poorly.Threshold-based systems are assessed by false-positive and false-negative rates; risk scores convert totals to probabilities and may be assessed by AUC and calibration.Source: §2 · p. 15
  • The design objective trades off predictive loss against the number of nonzero coefficients while requiring small integer values and potentially user-defined constraints.Ideally, the user specifies the loss, sparsity tradeoff, and domain constraints, while a computer performs the optimization.Source: §2 · pp. 15–16
  • Rounding real-valued coefficients can erase the combined predictive signal of many small coefficients, while strong ℓ1 regularization can further degrade sparse solutions.The example rounds [5.3, 6.1, 0.31, 0.30, 0.25, 0.25, 0.24, ..., 0.05] to [5, 6, 0, 0, ..., 0].Source: §2 · p. 16
  • Approximate linear-programming methods trade computational efficiency for uncertainty about closeness to the desired solution, while rounding and expert-in-the-loop tools offer practical but constraint-limited alternatives.Sequential Rounding evaluates rounding each coefficient up or down, then Discrete Coordinate Descent polishes the integer solution; design interfaces let experts adjust models directly.Source: §2 · pp. 16–18
  • Exact mixed-integer optimization handles discrete losses and user constraints, while RiskSLIM addresses logistic loss; these methods can struggle with large problems and continuous covariates.RiskSLIM uses cutting planes in a branch-and-bound framework with MIP-solver callbacks; discretizing continuous covariates can greatly increase problem size or lose accuracy if too few thresholds are used.Source: §2 · pp. 16–17

2.1 Improve the scalability of optimal sparse scoring systems

  • Optimal sparse scoring systems remain difficult to scale: state-of-the-art methods handle thousands of samples and tens of variables in about an hour, making repeated constrained redesign slow.The challenge is to scale to larger datasets and reduce solving time, especially with complicated constraints.Source: §2 · p. 18

2.2 Ease of constraint elicitation and handling

  • Scoring-system design needs better ways to elicit and incorporate expert constraints and preferences, and to let experts explore similarly accurate models without sacrificing predictive performance.The physician example illustrates adding constraints on age dependence and error rates, exploring candidate models, and validating the selected model on unseen data.Source: §2 · pp. 18–19

3 Generalized Additive Models

GAMs extend linear models with interpretable univariate feature functions, while this section frames challenges in controlling their complexity and using them to troubleshoot complex datasets.

3 Generalized Additive Models

  • GAMs extend GLMs with possibly nonlinear univariate feature functions whose plots expose each feature’s contribution; the model hierarchy includes additive, linear, and scoring models.For example, a component plot relates post-test plasma glucose concentration to diabetes risk; pairwise interactions can be heatmapped, though GAMs typically support few such terms and handle multiclass problems unnaturally.Source: §3 · pp. 19–20
  • Backfitting trains component functions on residuals, while boosting aggregates single-feature trees; boosted GAMs can be powerful but are not naturally sparse.AdaBoost maximizes convex proxies for classification error and AUC simultaneously, which the paper connects to boosted models’ high AUC and accuracy.Source: §3 · p. 21

3.1 How to control the simplicity and interpretability for GAMs

  • Challenge 3.1 is to control component-function sparsity, smoothness, and monotonicity, since convex penalties can bias sparse fits, reduce accuracy, or erase meaningful patterns.When component functions combine splines or indicators, penalties can control smoothness and sparsity, but choosing a smooth function plus an indicator for a meaningful jump remains hand-designed; boosting also makes sparsity and smoothness difficult to control.Source: §3 · p. 21

3.2 How to use GAMs to troubleshoot complex datasets

  • Challenge 3.2 is whether GAMs can expose missing confounders and data-quality problems in complex medical records while supporting useful prediction.Asthma patients appearing to fare better may reflect higher natural risk and better care; missing measurements, biased billing codes, and inconsistent records complicate interpretation.Source: §3 · pp. 21–22

4 Modern case-based reasoning

Case-based reasoning predicts by comparing new inputs with similar past cases, offering a potentially interpretable approach that applies to tabular and raw data. The section describes nearest-neighbor and prototype methods, including part-based approaches, and identifies open challenges for extending and improving prototype learning.

Modern case-based reasoning

  • Nearest-neighbor methods classify using similar training cases, while prototype methods compare inputs with learned representative cases rather than the full training set.Prototype learning includes LVQ updates and discrete optimization approaches such as set-cover and squared-maximum-mean-discrepancy objectives.Source: §4 · pp. 23–24
  • Adaptive distance metrics and learned latent-space mappings are mathematically equivalent ways to make nearest-neighbor comparisons better suited to classification.Deep models can learn the latent space, and some approaches use neighbors from multiple hidden layers or weighted latent-space distances.Source: §4 · pp. 23–24
  • Part-based methods compare subsets of observations, offering more flexibility and interpretability when whole-case comparisons are unhelpful, though averaged latent prototypes may look unrealistic for images.Convex combinations of training examples can suit tabular data, but averaging latent positions may not produce a realistic image.Source: §4 · pp. 24–25
  • Chen et al. (2019)'s ProtoPNet classified CUB-200-2011 birds at accuracy comparable to black-box models and explained predictions by matching image parts to class prototypes.Prototype projection maps each learned part to a nearby encoded training patch of the same class, improving visualization quality over decoder-based prototypes.Source: Figure 7 · p. 25, §4 · p. 26

4.1 How to extend the existing case-based reasoning approach to handling more complex data, such as video?

  • Extending case-based reasoning to video remains open: comparing high-dimensional videos efficiently is difficult, though their ordered frames may support prototypes built from neighboring frames or frame parts.Trinh et al. (2021) developed a dynamic prototype network for deep-faked and real videos, but efficient video comparison remains a challenge.Source: §4 · pp. 26–27

4.2 How to integrate prior knowledge or human supervision into prototype learning?

  • Prototype learning rarely incorporates expert supervision, despite the potential for doctors to select or design prototypes and thereby improve accuracy, interpretability, and data efficiency.Experts could also specify regions of interest for prototypes to focus on.Source: §4 · p. 27

4.3 How to troubleshoot a trained prototype-based model to improve the quality of prototypes?

  • A remaining challenge is to replace invalid prototypes systematically without harming performance or retraining from scratch, while avoiding class imbalance caused by simply removing them.Invalid prototypes can capture nonrepresentative features, such as text in medical images that improves training accuracy but not test accuracy.Source: §4 · p. 27

5 Complete supervised disentanglement of neural networks

The challenge is to fully disentangle a neural network so that neurons represent human-interpretable concepts and information about each concept follows its corresponding path. Supervised methods use researcher-selected concepts, but important problems remain in disentangling whole layers and networks, choosing concepts, and interpreting how concepts produce outputs.

5 Complete supervised disentanglement of neural networks

  • The goal is a latent space where each concept’s information follows its corresponding neuron, unlike standard DNNs where unrelated concepts can share neurons.Separate concept classifiers can capture irrelevant features and reduce performance; supervised methods instead align latent axes with predefined concepts using labeled data, producing purer representations without hurting accuracy.Source: §5 · pp. 28–29

5.1 How to make a whole layer of a DNN interpretable?

  • Concept classification can score well yet leave concept activations correlated; current single-layer approaches leave residual neurons unconstrained and require samples for every target concept.Training with random subsets of concepts or combining unsupervised concept detection with human validation are suggested avenues, but the passages note these have not been tried in current methods.Source: §5 · p. 30

5.2 How to disentangle all neurons of a DNN simultaneously?

  • Disentangling all neurons across layers requires a comprehensive concept set and an appropriately placed hierarchy, since low-level concepts arise earlier and high-level concepts deeper.Manually specifying thousands of concepts and their hierarchy is difficult, while automating that hierarchy remains a challenge.Source: §5 · p. 30

5.3 How to choose good concepts to learn for disentanglement?

  • Choosing useful concepts is difficult: existing medical attributes may omit concepts used by a model, and continuous concepts such as age require thresholds for discrete representations.Active learning could help domain experts create and refine concepts.Source: §5 · pp. 30–31

5.4 How to make the mapping from the disentangled layer to the output layer interpretable?

  • Even with interpretable concepts, the concept-to-output mapping remains a black box; linear mappings may lack expressiveness, while interpretable nonlinear models are potential alternatives.Neural additive models (Agarwal et al., 2020) and neural decision trees (Yang et al., 2018) could model this mapping; linear mappings are also less effective for user interventions on concept values (Koh et al., 2020).Source: §5 · p. 31

6 Unsupervised disentanglement of neural networks

Unsupervised disentanglement aims to make neural-network information flow interpretable when concepts are unknown or too numerous to label, potentially enabling discovery in uncharted domains. The section surveys approaches and highlights evaluation, transfer, and realistic-scene challenges.

6 Unsupervised disentanglement of neural networks

  • Without supervision, a disentangled latent space could align neurons with undiscovered material patterns, helping researchers relate those patterns to physical properties and inform material design.Figure 10 illustrates patterns discovered in metamaterial unit cells for predicting whether they support band gaps.Source: §6 · p. 32, Figure 10 · p. 33
  • Generative models can disentangle factors such as age, pose, and identity through latent-feature independence, but current methods work only on relatively simple imagery.Learning a scene decomposition into object groups remains beyond these approaches; co-occurring objects and repeated instances are additional difficulties.Source: §6 · pp. 32–33
  • Compositional architectures use capsules, object constituents, or iterative attention to discover parts, but realistic images remain difficult for these methods.On SVHN, the Stack Capsule Autoencoder discovers stroke-like structures rather than digit concepts; GLOM has been proposed, but no working system has been developed.Source: §6 · pp. 33–34, Figure 11 · p. 34
  • Some prototype and convolutional-filter methods implicitly learn disentangled object parts, but have not generalized to nonlocalized scene concepts such as room style or weather.Prototype diversity is encouraged by a separation cost, while filter methods use mutual-information regularization to promote category-specific and spatially localized activations.Source: §6 · pp. 34–35

6.1 How to quantitatively evaluate unsupervised disentanglement?

  • Known concepts can support quantitative evaluation, but unknown concepts lack ground-truth labels, while biased annotations may overlook concepts an unsupervised model discovers.For known concepts, evaluation can test whether the learned representation disentangles them; in domains such as materials science, neither ground-truth labels nor reliable human evaluation may be available, and image annotations often omit lighting or style.Source: §6.1 · p. 35

6.2 How to adapt neural network architectures designed with compositional constraints to other domains?

  • Compositional architectures built for objects and parts in images may not transfer to domains such as materials science, where constituent patterns jointly determine physical properties.The assumptions behind Capsule Networks (Sabour et al., 2017) and Slot Attention (Locatello et al., 2020)—including negligible interference between objects—may not fit other domains.Source: §6.1 · pp. 35–36

6.3 How to learn part-whole disentanglement for more complicated patterns in large vision datasets?

  • Part-whole segmentation works on simple synthetic datasets, but no work has successfully learned part-whole relationships on realistic datasets such as ImageNet.Real-world object interactions may require new techniques.Source: §6.1 · p. 36

7 Dimension Reduction for Data Visualization

Dimension reduction maps high-dimensional data into usually 2D or 3D space to help people understand data structure, form hypotheses, and spot bias or noise, but projections can lose relationships. Its algorithms trade off preserving distances and local neighborhoods, and their choices can produce markedly different visualizations.

7 Dimension Reduction for Data Visualization

  • On MNIST, PCA preserves scale information but fails to separate digits, t-SNE separates them but loses scale, while UMAP and PaCMAP separate digits and retain some global information.The figure compares these methods in an abstract 2D projection; its axes are not quantified.Source: §7 · p. 37, Figure 12 · p. 37

7.1 How to capture information from the high dimensional space more accurately?

  • Recent DR methods emphasize nearest neighbors, but may lose non-local information and rely on distance metrics that do not accurately represent sample relationships.Global-aware initialization and selectively preserving distances between non-neighbor samples can mitigate global-information loss, but the metric assumption may still fail, such as for neural-network weights or activations.Source: §7.1 · p. 39

7.2 How should we select hyperparameters for DR?

  • Because unsupervised DR offers no known structure for tuning dozens of hyperparameters, poor choices can create misleading projections, spurious clusters, and lost connectivity or detail.Tuning may require many time- and power-consuming runs and expertise; hyperparameters remain data-dependent, motivating better automatic selection. Misleading clusters can be mistaken for new cell subtypes in single-cell RNA-sequencing data.Source: §7.2 · pp. 39–41, Figure 14 · p. 40

7.3 Can the DR transformation from high- to low-dimensions be made more interpretable or explainable?

  • The DR mapping is typically complex; PCA can reveal dominant original dimensions, while post-hoc visualizations or constrained DR could explain how low-dimensional axes derive from high-dimensional features.The paper identifies these as possible ways to help users understand the final two or three dimensions.Source: §7.2 · pp. 40–41

8 Machine learning models that incorporate physics and other generative or causal constraints

Physics-guided models incorporate physical laws and other domain constraints, making them alternatives to purely data-driven models and interpretable through their adherence to those constraints. The section presents physics-informed neural networks (PINNs) and identifies open challenges in training them, choosing data, and incorporating differential constraints into other models.

Motivation

  • Physics-guided models constrain learning to follow supplied physical laws, may require little or no data, and can offer alternatives to numerical PDE solvers.The paper describes these models as interpretable because they are constrained to obey the provided laws of physics.Source: §8 · p. 41

PINNs

  • PINNs approximate PDE solutions by minimizing equation and initial- and boundary-condition residuals, and can be trained without labeled solution data; observed data can add a mean-squared-error term.Raissi et al. (2019) developed a general PINN framework; the model’s loss reflects how closely its output obeys the heat equation and its conditions.Source: §8 · pp. 41–42

8.1 How to improve training of a PINN?

  • PINN training can suffer from numerical stiffness in gradient backpropagation; dynamically balancing loss terms is one proposed mitigation, but improving training remains open.Wang et al. (2020a) proposed dynamically balancing the different loss terms.Source: §8 · p. 43

8.2 How do we combine PINNs with optimal experimental/simulation design to quickly reduce uncertainty in approximating physical models?

  • Co-designing experimental or simulation data collection with PINN training remains open and could improve efficiency and reduce uncertainty where data are scarce or PDE parameters must be learned.Training data are often collected before PINN training, which can leave large uncertainty bands in sparsely sampled regions.Source: §8 · p. 43

8.3 Are there other ways of incorporating PDE (or other) constraints into models other than neural networks?

  • Gaussian processes have also been used to approximate differential-equation solutions, but incorporating differential constraints into nonparametric models remains an open challenge.Prior approaches can rely on finite-difference approximations or be nondifferentiable at observation points.Source: §8 · pp. 42–43

Other constraints

  • Beyond PDEs, knowledge-infused deep learning uses domain knowledge represented in knowledge graphs to guide neural-network learning, though effective incorporation remains a challenge.Knowledge graphs encode relationships between entities, including common-sense knowledge.Source: §8 · p. 43

9 Characterization of the “Rashomon” set of good models

The Rashomon set comprises models with similarly low loss, creating an opportunity to choose accurate models for interpretability and other desired properties. Characterizing, visualizing, and navigating this set remain open challenges.

9 Characterization of the “Rashomon” set of good models

  • The Rashomon set contains models whose training loss is within ϵ of a reference model’s loss, with ϵ balancing predictive-performance tolerance against flexibility in model choice.The reference may be an empirical risk minimizer, optimal model, or another model; ϵ can be an absolute threshold or a percentage of the reference loss.Source: §9 · p. 44
  • A large Rashomon set must, under weak assumptions, contain a simpler model if simpler functions can approximate the complex functions it contains (Semenova et al., 2019).For example, decision trees can approximate polynomials, so a set containing a ball of such polynomials must also contain a simpler function.Source: §9 · pp. 44–45
  • The set can contain models with fairness or monotonicity, and its size reflects learning-problem complexity; Model Class Reliance bounds variable importance across its models.Fisher et al. (2019) define Model Class Reliance using the minimum and maximum importance across models in the Rashomon set.Source: §9 · p. 45

9.1 How can we characterize the Rashomon set?

  • Proposed size measures include ϵ-flatness, ϵ-sharpness, the Rashomon ratio, and the pattern Rashomon ratio, but none is a universal, simple-to-calculate characterization.Parameter-space volumes depend on parameterization, and Rashomon ratios depend on the chosen hypothesis-space denominator; pattern ratios use unique prediction patterns but remain difficult to compute and sensitive to data changes.Source: §9.1 · pp. 46–48
  • Visualizations can reveal the set’s layout, size, sparse models, and model-selection stability, while variable-importance diagrams project its models into importance space.Model Class Reliance summarizes the range of importance for each variable across all models in the set; wider tree-based selection graphs indicate more available models (Kissel and Mentch, 2021).Source: §9 · p. 45, §9.1 · p. 48
  • Interactive selection can help users navigate accurate models toward desired properties, but BEAMES searches for constraint-compatible models only for linear regression classifiers.The proposed interaction mechanisms include changing visualization dimensions, weighting data points, and providing ongoing user feedback.Source: §9.1 · p. 49

10 Interpretable reinforcement learning

Interpretability is harder in reinforcement learning because agents explore, face delayed rewards, and make interdependent state-action decisions, while deep policies are often opaque. The section presents interpretable policies as a way to reduce search, avoid harmful actions, and improve troubleshooting, then identifies challenges in policy constraints, performance, and state-space simplification.

Interpretable policy approaches

  • Interpretable policy approaches include decision trees or rule lists, relational regression trees for symbolic states, and hierarchical policies built from human-described skills.Policy trees can use logical conditions at nodes and action-probability estimates at leaves; Roth et al. (2019) describe growing them when estimated discounted future reward increases sufficiently.Source: §10 · pp. 50–51

Current limitations

  • The section reports no general, well-performing transparent method for deep reinforcement learning; existing explainable methods are post-hoc, and PIRL (Verma et al., 2018) has lower performance and domain limitations.The text cautions that post-hoc explanations can be incorrect or incomplete and that saliency maps should support exploration rather than explanation (Atrey et al., 2020).Source: §10 · p. 51

10.1 What constraints would lead to an accurate interpretable policy?

  • Challenge 10.1 asks what constraints can yield accurate, understandable policies, including deep-network methods, sparsity, and explanations of long-term action consequences.Tree-based relational methods are limited to symbolic domains without a way to extract interpretable relations.Source: §10 · pp. 51–52

10.2 Under what assumptions does there exist an interpretable policy that is as accurate as the more complicated black-box model?

  • Challenge 10.2 asks when an interpretable policy can match a black-box model and how to test whether a well-performing sparse policy exists before optimizing for one.The performance drop in PIRL (Verma et al., 2018) could reflect post-hoc approximation or the absence of a strong sparse rule-based policy.Source: §10 · p. 52

10.3 Can we simplify or interpret the state space?

  • Challenge 10.3 asks whether simplifying state spaces can improve learning efficiency and human understanding; Zhang et al. (2021) report smaller state spaces and faster planning by computing policies at clinical decision points.Zahavy et al. (2016) found hierarchical structure in deep Q-learning activations using t-SNE, while the text notes this structure had not previously been used to simplify the state space.Source: §10 · pp. 52–53

11 Problems that weren’t in our top 10 but are really important

The omitted agenda extends interpretability beyond the ten challenges to preprocessing, uncertainty, model and visualization design, causal matching, variable importance, deployment, legal questions, and other model families.

Preprocessing

  • General preprocessing that preserves or improves predictive power without obscuring original features remains an open problem, since PCA combines those features and can damage interpretability.The paper notes preprocessing may also help address computational difficulty or overfitting in some interpretable modeling problems.Source: §11 · p. 53

Conveying uncertainty

  • Uncertainty quantification remains important to interpretable ML, with work addressing interpretable uncertainty estimation (Tomsett et al. (2020); Antor`an et al. (2020)).Source: §11 · p. 53

Dividing easier and harder cases

Interpretable neural networks for tabular data

  • Interpretable neural networks for tabular data warrant scrutiny because neural networks generally offer little benefit over other models and lack optimal methods’ global-optimality guarantees.The paper also says this makes neural networks harder to implement reliably, as well as less interpretable.Source: §11 · pp. 53–54

Co-designing models with visualizations

  • Co-designed visualizations can make non-sparse models interpretable, helping users focus on prediction-relevant information that a sparse loan model might omit.The cited loan-decision examples are Gomez et al. (2020) and Chen et al. (2021a); the paper notes loan decisions may require many types of applicant credit-history information.Source: §11 · p. 54

Interpretable matching in observational causal inference

  • Almost Matching Exactly learns important variables from training data to prioritize matches; matched data support interpretable estimates of individual treatment effects or treatment policies.Matching treatment and control units on important variables creates overlapping distributions to which supervised interpretable models can be applied.Source: §11 · p. 54

Variable importance and other interpretable models

  • Inherently interpretable models expose each variable’s contribution to a prediction, while general or model-independent variable importance measures remain open directions.The paper also notes that applied Bayesian statistics may qualify as interpretable ML when constrained by an interpretable generative process, among many model families omitted from the survey.Source: §11 · pp. 54–55

Deployment and legal requirements

  • For high-stakes decisions, Rudin (2019) proposes prohibiting black-box models unless no similarly accurate interpretable model exists.The paper notes that explanations are often used internally for troubleshooting rather than shown to users (Bhatt et al. (2020)); human experiments have also produced sometimes nonintuitive results (Kaur et al. (2020); Poursabzi-Sangdeh et al. (2018)). It also reports that the EU GDPR does not establish a “Right to Explanation” (Wachter et al. (2017)).Source: §11 · p. 55

12 Conclusion

The survey offers an entry point into interpretable machine learning by introducing fundamental principles and connecting important areas to real problems. It also emphasizes that interpretability matters in practice and that interpretable models can be mathematically elegant.

12 Conclusion

  • The survey introduces fundamental principles and connects important areas of interpretable machine learning to real problems, offering readers an entry point into a confusing field.The authors note that misleading terminology and failure to distinguish explaining black boxes from designing interpretable models make the literature difficult to follow.Source: §12 · p. 55

Related papers

Loading 2103.11251v2…