PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable machine learning: definitions, methods, and applications

W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, Bin Yu

arXiv:1901.04592v1stat.MLcs.AIcs.LGstat.AP

TL;DR

Interpretable machine learning lacks a common definition and shared criteria for relating, evaluating, and selecting its many interpretation methods. The paper defines the field, organizes methods into model-based and post hoc categories, and introduces the PDR framework. It concludes that predictive accuracy, descriptive accuracy, and human-judged relevancy provide a common vocabulary, while interpretations also require stability and appropriate data.

  • Problem

    Interpretation methods have proliferated without a well-formed definition or common concepts for relating, evaluating, and selecting them for particular problems and audiences.

  • Method

    The paper defines interpretable machine learning, places it in the data-science life cycle, categorizes methods as model-based or post hoc, and introduces the PDR framework.

  • Results

    The paper presents predictive accuracy, descriptive accuracy, and human-judged relevancy as desiderata for evaluating and constructing interpretations, illustrated through real-world examples.

  • Takeaways & Limitations

    The framework provides a common vocabulary for practitioners and researchers to evaluate and select interpretation methods according to problem, data, and audience.

  • Takeaways & Limitations

    Interpretations are constrained by biases in collected data, which can restrict conclusions about the population of interest.

Abstract

from arXiv · show

Machine-learning models have demonstrated great success in learning complex patterns that enable them to make predictions about unobserved data. In addition to using models for prediction, the ability to interpret what a model has learned is receiving an increasing amount of attention. However, this increased focus has led to considerable confusion about the notion of interpretability. In particular, it is unclear how the wide array of proposed interpretation methods are related, and what common concepts can be used to evaluate them. We aim to address these concerns by defining interpretability in the context of machine learning and introducing the Predictive, Descriptive, Relevant (PDR) framework for discussing interpretations. The PDR framework provides three overarching desiderata for evaluation: predictive accuracy, descriptive accuracy and relevancy, with relevancy judged relative to a human audience. Moreover, to help manage the deluge of interpretation methods, we introduce a categorization of existing techniques into model-based and post-hoc categories, with sub-groups including sparsity, modularity and simulatability. To demonstrate how practitioners can use the PDR framework to evaluate and understand interpretations, we provide numerous real-world examples. These examples highlight the often under-appreciated role played by human audiences in discussions of interpretability. Finally, based on our framework, we discuss limitations of existing methods and directions for future work. We hope that this work will provide a common vocabulary that will make it easier for both practitioners and researchers to discuss and choose from the full range of interpretation methods.

1. Introduction

The paper addresses confusion about interpretable machine learning by defining the field, organizing interpretation methods, and introducing a framework for evaluating them. It emphasizes that relevant interpretations depend on the human audience and domain context.

  • Interpretations extract knowledge about domain relationships for uses including medicine, policy-making, science, auditing, regulation, and fairness.
  • The absence of a well-formed definition has produced confusion about what interpretation means, how methods relate, and how to choose among them.
  • It distinguishes model-based and post hoc interpretation methods and introduces the Predictive, Descriptive, Relevant framework.
  • The PDR framework evaluates interpretations by predictive accuracy, descriptive accuracy, and relevancy judged relative to a human audience.
  • The paper defines interpretable machine learning as using machine-learning models to extract relevant knowledge about domain relationships contained in data.
  • Interpretation outputs and useful information vary with context and audience, such as a doctor diagnosing one patient versus an engineer auditing racial discrimination.

2. Interpretation in the data science life cycle

Interpretability is situated within a data-science life cycle in which problem, data, model, audience, and post hoc analysis determine how interpretations are produced and used. The paper separates model-based methods from post hoc methods, with a trade-off between simplicity and predictive performance.

  • Data-science life cycle: Interpretation mainly occurs during modeling and post hoc analysis, while the problem, data, and audience provide context for selecting methods.
  • Problem, data, and audience: Problem context and audience help determine which relationships matter and which interpretation methods are appropriate.
  • Problem, data, and audience: Biases in collected data can manifest in the model and restrict interpretations about the population of interest.
  • Iteration: After post hoc analysis, practitioners may finish or update the problem, data, or model and iterate through the life cycle.
  • Model-based and post hoc methods: Model-based interpretability constrains models to provide useful insight, potentially lowering predictive accuracy when the underlying relationship is complex.
  • Model-based and post hoc methods: Post hoc interpretability extracts information from a trained model and is especially useful when complex black-box models are needed for reasonable predictive accuracy.

3. The PDR desiderata for interpretations

The PDR framework guides the selection and evaluation of interpretation methods through predictive accuracy, descriptive accuracy, and relevancy. Trustworthy interpretations should faithfully represent both the underlying data relationships and what the model learned.

  • The PDR framework proposes predictive accuracy, descriptive accuracy, and relevancy as desiderata for selecting interpretation methods.
  • Accuracy: Predictive accuracy concerns how well the model approximates underlying data relationships, while descriptive accuracy concerns how well an interpretation represents the model’s learned relationships.
  • Accuracy: Trustworthy interpretations should maximize both predictive and descriptive accuracy, with external validation especially important when accuracy is not high.
  • Predictive accuracy: Predictive accuracy should account for prediction distributions and stability under reasonable data and model perturbations, not only average accuracy.

A.2. Descriptive accuracy.

Descriptive accuracy measures whether an interpretation objectively captures relationships learned by a machine-learning model. Model simplicity can improve descriptive accuracy but may reduce predictive accuracy on complex datasets.

  • Descriptive accuracy is the degree to which an interpretation method objectively captures relationships learned by machine-learning models.
  • Model-based methods can provide consistently high descriptive accuracy, whereas complex models may provide higher predictive accuracy but be harder to analyze.

A.3. A common conflict: predictive vs descriptive accuracy.

Interpretation methods must balance predictive and descriptive accuracy with relevancy to the particular human audience and domain problem. The appropriate trade-off depends on context, such as auditing fairness or improving feature engineering.

  • Interpretation methods should consider predictive accuracy, descriptive accuracy, and relevancy together.
  • Relevant interpretations provide insight for a particular audience into a chosen domain problem.
  • Relevancy can prioritize descriptive accuracy when interpretations audit predictions for fairness, or predictive accuracy when they support feature engineering.
  • Model-based methods can trade predictive accuracy for descriptive accuracy, whereas post hoc methods extract information without affecting predictive accuracy.

4. Model-based interpretability

Model-based interpretability constructs models that expose learned relationships, aiming for a balance between human understanding and predictive fit. The paper illustrates this through sparse, simulatable, and auditable models across medical and climate applications.

  • 4. Model-based interpretability: Model-based interpretability constructs models that readily provide insight into learned relationships, balancing simplicity against predictive fit.
  • Sparsity: Sparsity can improve all three interpretation desiderata when the underlying sparse structure is correctly incorporated.Sparse models reduce the number of parameters and can improve understanding, predictive accuracy, and relevancy, but require data-specific structural knowledge.
  • Sparsity: Sparse genomic methods reduce millions of potential feature interactions to manageable candidates for statisticians and biologists.
  • Simulatability: Simulatable models let intended users reason through the entire decision process, but generally require few features and simple relationships.
  • Simulatability: A stroke-risk rule-list classifier used seven if-then statements to achieve competitive accuracy while remaining understandable to non-technical practitioners.
  • Model auditing: Interpretable modules exposed that asthma was associated with lower modeled pneumonia mortality despite higher real-world risk, enabling error correction.
  • Model auditing: Three simple climate-image features enabled quadratic discriminant analysis to achieve high predictive accuracy and transparency against expert labels.

A.1. Interaction and feature importances.

Interpretation methods can extract interactions as well as individual feature importances, providing more complete descriptions of nonlinear models. In genomics, interaction extraction targets biologically meaningful combinations of regulatory factors or genes.

  • A.1. Interaction and feature importances.: Interaction-extraction methods increase descriptive accuracy by identifying relationships among features in nonlinear models.
  • A.1. Interaction and feature importances.: High-order interactions among regulatory factors or genes help define cell-type-specific behavior, making them important biological targets.
  • A.1. Interaction and feature importances.: Iteratively re-weighted random forests can search predictive decision paths for stable combinations of features that frequently co-occur.
  • A.1. Interaction and feature importances.: Interaction methods can produce statistical measures in addition to raw feature values.

A.2. Statistical feature importances.

Statistical feature importances can quantify associations between predictors and outcomes, but their interpretation depends on model assumptions and does not establish causation.

  • A.2. Statistical feature importances.: Linear and logistic regression can compute confidence intervals and hypothesis tests for coefficients and their linear combinations under data-generating-process assumptions.These statistics help assess the statistical significance of observed coefficients.
  • A.2. Statistical feature importances.: In the Harvard admissions example, Asian status had coefficient -0.418 with p<0.001, indicating a significant negative association with admission probability.The result concerned applicants who were not low income.
  • A.2. Statistical feature importances.: The lawsuit’s opposing analyses reached different conclusions after accounting for additional variables, illustrating sensitivity to modeling choices and assumptions.One report found the effect no longer significant after adjustment, while the other characterized race as unfairly used.
  • A.2. Statistical feature importances.: Regression significance tests establish association at best, not causation, so using statistical feature importances inappropriately can produce misleading practical consequences.The paper identifies this limitation in the Harvard admissions example.

A.3. Visualizations.

Visualizations help practitioners inspect what models learn, from patterns driving individual responses to prediction errors and training-data influence, while stability is necessary for scientific interpretation.

  • A.3. Visualizations.: Maximal activation images can identify image patterns that drive the responses of brain cells after neural networks predict those responses.The neural networks fit responses accurately but require further analysis to reveal the driving patterns.
  • A.3. Visualizations.: Post hoc visualizations intended to answer scientific questions should remain stable under reasonable perturbations such as changing the model.A study fit eighteen models and used stable optimization to produce a consensus visualization.
  • A.3. Visualizations.: Residual plots reveal heterogeneity, outliers, large prediction errors, and trends across model predictions.These views help identify particular data points for further analysis.
  • A.3. Visualizations.: Influence functions can locate training examples that contribute most to individual predictions, including mislabeled points.This can identify mislabeled training data without examining the entire dataset.
  • A.3. Visualizations.: Prediction-level interpretation focuses on explaining individual model predictions and can sometimes be aggregated into dataset-level insights.This category includes methods that identify features contributing to a particular prediction.

B.1. Feature importance scores.

Feature importance scores explain individual predictions and can support fairness audits, but prediction-specific scores vary across examples and may miss learned interactions.

  • B.1. Feature importance scores.: Prediction-level feature importance scores assign positive or negative contributions to individual features for a particular prediction.They are often displayed as heat maps highlighting important features.
  • B.1. Feature importance scores.: Because nonlinear models can contain feature interactions, a feature’s importance may differ across examples, making prediction-level scores more informative than dataset-level scores.The difference arises from heterogeneity in the model’s learned relationships.
  • B.1. Feature importance scores.: Variable importance scores can isolate individual contributions among correlated variables and support transparency reports for sensitive predictions.The cited application concerns fairness assessment in predictive policing.
  • B.1. Feature importance scores.: In the arrest-risk example, race was the largest feature-importance value, indicating that the classifier discriminated based on race in that instance.The figure concerns the prediction that Mr. Z was likely to be arrested in the future.
  • B.1. Feature importance scores.: Feature importance scores cannot capture interactions between variables, motivating methods that explicitly display neural-network interactions or use textual explanations.The paper describes these as limitations and directions for related work.

6. Future work

The paper frames future work around evaluating interpretation methods, measuring descriptive accuracy, and improving relevance to real audiences and problems.

  • 6. Future work: The PDR framework organizes open problems around interpretation desiderata, model-based interpretability, and post hoc interpretability.These correspond to the paper’s three main sections on future challenges.
  • 6. Future work: No community consensus exists for evaluating interpretation methods, causing evaluation standards to vary and complicating progress measurement and method selection.The paper identifies evaluation as a central unresolved issue.
  • 6. Future work: Cherry-picked interpretations that appear reasonable provide limited and unfalsifiable evidence about descriptive accuracy.Such evaluations cover only the displayed examples and may not generalize to the dataset.
  • 6. Future work: Simulation studies offer a partial solution by generating data from a known process and testing whether a sufficiently powerful model recovers it.The approach assumes an appropriate simulation, adequate training data, and a model capable of recovering the generative process.
  • 6. Future work: Interpretation methods should improve relevance for a particular audience or problem rather than focusing only on novel output formats.Examples of formats include feature heatmaps, rationales, feature hierarchies, and important training elements.

A.2. Demonstrating relevancy to real-world problems.

The paper evaluates interpretability by its relevancy to real-world problems and identifies distinct challenges for model-based and post hoc methods.

  • Demonstrating relevancy: Directly solving a domain problem provides the strongest evidence that an interpretation is relevant.Human studies can also assess relevance, but crowdsourced participants may not represent domain-specific audiences.
  • Model-based interpretability: Model-based methods often fail to achieve reasonable predictive accuracy, forcing practitioners to seek more accurate models.The paper calls for models that improve predictive accuracy while maintaining descriptive accuracy and relevance.
  • Model-based interpretability: More informative features can let simpler models achieve comparable predictive accuracy, broadening model-based interpretation use.Interactive analysis, visualization, data exploration, clustering, and matrix factorization are cited as routes to more useful features.
  • Post hoc interpretability: Post hoc interpretation still faces open questions about suitable output formats and how to use them for real-world problems.Existing formats include feature heatmaps, feature hierarchies, and important training examples, but may not capture complex model behavior fully.
  • Post hoc interpretability: Post hoc interpretations can reveal incorrect relationships learned by models, but using such findings to improve predictive accuracy remains limited.The paper presents correcting learned relationships through prior knowledge as a potential future research avenue.
Loading 1901.04592v1…