PaperlayerView exact revision on arXiv

Source-linked AI summary

Principles and Practice of Explainable Machine Learning

Vaishak Belle, Ioannis Papantonis

arXiv:2009.11698v1cs.LGcs.AIstat.ML

TL;DR

The paper addresses how to understand and trust decisions from increasingly complex machine-learning systems. It surveys and organizes explainable ML literature for practitioners, then argues that different explanation approaches answer different questions and no single approach suits every scenario.

  • Problem

    Increasingly complex and widely deployed ML models are difficult to understand, while stakeholders face concerns about trust, drawbacks, biases, and choosing among explainability approaches.

  • Method

    The report surveys and distills explainable ML literature, organizing perspectives and frameworks around transparency, evaluation criteria, explanation types, and practical tool selection.

  • Results

    Different explanation types address different aspects of explainability, so no single approach suits every scenario and satisfactory understanding generally requires multiple questions.

  • Takeaways & Limitations

    Practitioners should match explainability approaches to the questions and scope they need to address rather than rely on one universal method.

  • Takeaways & Limitations

    The survey primarily covers mainstream ML statistical explanations and excludes explanation generation, causality-focused approaches, and broader human-comprehension perspectives.

Abstract

from arXiv · show

Artificial intelligence (AI) provides many opportunities to improve private and public life. Discovering patterns and structures in large troves of data in an automated manner is a core component of data science, and currently drives applications in diverse areas such as computational biology, law and finance. However, such a highly positive impact is coupled with significant challenges: how do we understand the decisions suggested by these systems in order that we can trust them? In this report, we focus specifically on data-driven methods -- machine learning (ML) and pattern recognition models in particular -- so as to survey and distill the results and observations from the literature. The purpose of this report can be especially appreciated by noting that ML models are increasingly deployed in a wide range of businesses. However, with the increasing prevalence and complexity of methods, business stakeholders in the very least have a growing number of concerns about the drawbacks of models, data-specific biases, and so on. Analogously, data science practitioners are often not aware about approaches emerging from the academic literature, or may struggle to appreciate the differences between different methods, so end up using industry standards such as SHAP. Here, we have undertaken a survey to help industry practitioners (but also data scientists more broadly) understand the field of explainable machine learning better and apply the right tools. Our latter sections build a narrative around a putative data scientist, and discuss how she might go about explaining her models by asking the right questions.

1 Introduction

Explainable machine learning addresses how to understand and trust decisions from increasingly complex ML systems, while helping stakeholders evaluate models, biases, robustness, and deployment concerns.

  • Explainability can help stakeholders understand model decisions and investigate correctness, robustness, bias, improvement, transferability, and human comprehensibility.
  • Business owners, model-risk analysts, regulators, consumers, and data scientists seek different forms of insight into model fit, reliability, impacts, debugging, and performance improvement.
  • ML models are increasingly deployed across businesses, raising concerns about model drawbacks, data-specific biases, and differences among explainability approaches.
  • The report surveys explainable machine learning to help practitioners understand available approaches and apply tools beyond defaulting to industry standards such as SHAP.

2 Development & Contributions

The report synthesizes explainable machine learning literature into perspectives and frameworks intended to clarify what insights different approaches provide and where they apply.

  • The survey distills notions of transparency, explainability criteria, and explanation types from existing literature and available approaches.
  • It develops a taxonomy and additional frameworks covering transparent versus opaque models, model-specific versus model-agnostic methods, and explainability developments.
  • The overall aim is to help practitioners disentangle different notions of explanation and their intended scopes of application.

3 Scope

The report focuses on statistical explanations extracted from mainstream ML and pattern-recognition models for classification or prediction, rather than natural-language explanation generation.

  • The scope centers on data-driven ML and pattern-recognition methods whose primary goals are classification or prediction based on statistical association.
  • The report studies extracting interpretations of model behavior and decision boundaries, not generating explanations through natural-language understanding.
  • The survey provides a high-level perspective and expects readers to have familiarity with classification and prediction methods.

4 Perspectives on Explainability

The report frames explainability through transparency, evaluation criteria, and post-hoc explanation types, emphasizing that different approaches answer different questions.

  • Transparency: Transparency concerns human-level understanding of a model’s inner workings and includes simulatability, decomposability, and algorithmic transparency.
  • Transparency: Transparent-versus-opaque is not binary in practice, although decision trees and linear regression are commonly treated as simpler than random forests and deep learning.
  • Evaluation Criteria: Explainability methods can be evaluated by comprehensibility, fidelity, accuracy, scalability, and generality.
  • Evaluation Criteria: These concepts are difficult to quantify rigorously and serve primarily as guiding intuition for future developments.
  • Types of Explanations: Post-hoc explanations for opaque models include text, visual, local, example-based, simplification-based, and feature-relevance explanations.
  • Types of Explanations: Feature-relevance explanations rank input variables by influence, but importance scores alone may not provide a complete explanation.

5 Exploring XAI

The section organizes explainable machine learning through frameworks that compare model transparency, explanation capabilities, and redesign alternatives. It also provides tables, a taxonomy, and a question-driven strategy for practitioners.

  • Five frameworks distill the XAI literature by highlighting deployed-model limitations, available explanations, and explainability-oriented model alternatives.
  • The frameworks adapt prior discussions, analyze popular XAI techniques, and include a recommended cheat-sheet strategy and pipeline.
  • Figure 2 arranges models by the kinds of explainability they enable, supporting distinctions between transparent and opaque models.
  • Tables compare models and XAI approaches according to transparency dimensions, explanation types, model agnosticism, and input-transformation requirements.
  • The practitioner strategy uses a putative data scientist who explains models by asking questions that motivate particular explanation types.

6 Transparent Models

Transparent models expose aspects of their architecture or procedures, but their practical explainability depends on complexity, feature understandability, and the audience. The section surveys linear models, trees, KNN, rules, GAMs, and Bayesian networks while emphasizing transparency trade-offs.

  • Transparent models have intrinsic architectures satisfying at least one defined transparency dimension.
  • Linear and logistic regression are transparent because their targets are modeled as linear combinations of predictor variables, though visualization may aid non-experts.
  • Decision trees are understandable when small, but overfitting can harm generalization; tree ensembles improve generalization while typically losing transparency.
  • GAMs and Bayesian networks can be transparent, but non-parametric functions, complex features, or complicated graph topologies can reduce interpretability.
  • Transparency is not sufficient for ready explainability because increasing model complexity can make internal operation difficult to understand.

7 Opaque Models

Opaque models use complex decision boundaries or structures that can improve predictive accuracy at the expense of explainability. The section introduces random forests, support vector machines, and multilayer neural networks as examples.

  • Opaque models trade explainability for higher accuracy by using complex decision boundaries.
  • Random forests combine multiple trees trained on different data subsets to reduce variance and improve generalization over single trees.
  • Support vector machines identify a separating hyperplane with maximum margin in binary classification.
  • Multilayer neural networks achieve state-of-the-art performance across applications, but their complex topology obscures variable interactions and learned high-level features.
  • Neural networks connect inputs to outputs through successive layers whose nodes aggregate preceding outputs and apply activation functions.
  • Increasing the number of neural-network layers makes interpretation harder, while an overly simple network may be simulatable but practically uninteresting.

8 Explainability Approaches

The report reviews post-hoc explainability for opaque models, emphasizing Random Forest-specific techniques alongside model-agnostic methods. Approaches include simplification, feature relevance, representative examples, counterfactuals, visualizations, and local surrogates.

  • Random Forest explanations: Random Forest explainability methods are divided into model-specific techniques and model-agnostic approaches.The review focuses on Random Forests because of their popularity and uses them to illustrate model-specific explainability.
  • Model simplification: Random Forest simplification methods approximate the ensemble with mixtures, compact decision trees, or extracted rules.These approaches reduce thousands of learned rules to a smaller interpretable representation.
  • Representative examples: Representative-example methods inspect training data to identify data points that typify their corresponding classes.The number of examples can be user-specified, and the approach includes theoretical guarantees about example quality.
  • Feature relevance: Feature-relevance methods estimate importance through permutation, accuracy loss after feature removal, feature subsets, interactions, or importance thresholds.Counterfactual methods additionally assess importance by measuring which feature changes produce an opposite prediction.
  • Model-agnostic approaches: Model-agnostic approaches operate through input-output behavior and include simplification, feature relevance, visualizations, LIME, anchors, G-REX, and SHAP.LIME uses a local linear model or decision tree and an interpretable representation, while anchors use sparse if-then rules.

9 Brief Overview of Deep Learning Models

The deep-learning overview covers rule extraction, model distillation, and feature-relevance methods for explaining multilayer neural networks. It distinguishes decompositional, pedagogical, and eclectic strategies while noting that deeper networks make simplification harder.

  • Motivation: Multilayer neural networks achieve strong performance but use complex computational pipelines that are not transparent beyond simple cases.The review therefore considers multiple explanation strategies for these models.
  • Rule extraction: Decompositional rule extraction operates at neuron level, producing rules during training or representing extracted rules with decision trees.Examples include methods based on if-else rules and tree representations.
  • Rule extraction: Pedagogical methods treat the entire neural network as a black-box function and generate explanations without inspecting individual neurons.TREPAN uses decision trees with querying and sampling, while HYPINV uses network inversion.
  • Rule extraction: Eclectic techniques combine internal neural-network information with rule extraction by clustering hidden units or using artificial immune system algorithms.These methods connect inputs to clustered hidden representations through logical rules.
  • Alternative explanations: Model distillation approximates a neural network with a simpler interpretable model, while feature-relevance methods propagate or compare activations to assign feature importance.Examples include Interpretable Mimic Learning, neuron-wise decomposition, DeepLIFT, and Integrated Gradients.

10 Views and Suggestions

The report argues that explainability requires matching methods to questions rather than relying on a universal technique. It recommends combining complementary approaches while recognizing unresolved pipeline design and method-specific limitations.

  • Views on explainability: Explainability methods address different aspects of a model, so no single approach suits every scenario.Multiple questions and explanations may be needed to obtain a satisfactory understanding.
  • Views on explainability: Transparent models can improve interpretability but may compromise performance enough to hinder use in crucial applications.The report therefore frames transparent-model selection as a performance-versus-explainability consideration.
  • Method limitations: Local explanations describe behavior near one instance, but their conclusions may not reflect global behavior and can vary with small perturbations.Defining locality is also difficult in high-dimensional spaces.
  • Method limitations: Model simplification offers flexible, audience-adaptable representations, but its usefulness depends on approximation quality.Feature relevance can omit interactions, especially when features are strongly correlated, while visualizations may rely on assumptions such as independence.
  • Practical suggestions: Combining multiple approaches provides a more cautious explanation because relying on one technique can produce only a partial picture.The report notes that no established pipeline dictates how techniques should be combined.
  • Practical suggestions: A suggested workflow starts with transparent models when explainability is essential, then uses opaque models and complementary explanation techniques when performance is insufficient.Feature relevance, model simplification, local explanations, and visualization can answer different questions.

11 Jane, the Data Scientist

Jane’s case study illustrates an iterative explainability workflow for a high-performing Random Forest loan model. She combines SHAP, ICE plots, counterfactuals, anchors, deletion diagnostics, and inTrees to answer instance-level, local, training-data, and global questions.

  • Model selection: Jane frames model selection as a trade-off between predictive performance and explainability, choosing an opaque model after transparent models perform inadequately.Random Forests provide the best performance among the opaque candidates she tests.
  • Instance and feature behavior: SHAP explains a specific decision through feature-importance estimates but cannot answer how predictions change under altered inputs.Jane therefore adds ICE plots to examine behavior as salary changes while other features remain fixed.
  • Counterfactual explanations: Counterfactual analysis identifies missed payments as the factor distinguishing rejection from an otherwise accepted application.The method answers whether a slightly different application would receive a different decision.
  • Local explanations: Anchors provide local, easy-to-understand if-then rules describing the opaque model’s behavior for similar instances.The example rule links salary above £20k and no missed payments to loan approval.
  • Training-data influence: Deletion diagnostics show that removing noisy training points would not affect performance while identifying points capable of significantly altering the decision boundary.This helps stakeholders understand which training examples influence the model.
  • Global explanations: inTrees extracts robust rules from the Random Forest structure to support global inspection of the model’s behavior.The case study contrasts this model-specific method with model-agnostic SHAP, anchors, and counterfactuals.
  • Workflow implications: Using a larger set of XAI methods enables deeper inquiry, although model-specific tools may be less widely available than model-agnostic libraries.Model-agnostic methods offer consistency across models but may introduce assumptions and approximations.

12 Future Directions

The survey identifies unresolved challenges in explainable machine learning and outlines research directions for improving how explanations are combined, implemented, specialized, and trusted. It also highlights hybrid models and causal analysis as promising avenues for future work.

  • XAI remains a developing field with many unresolved research and operational problems.The survey points readers to possible future research avenues while emphasizing that the field is still emerging.
  • Different explanation types often address separate questions, but there is no clear way to combine them into a complete explanation.This lack of integration hinders automated explanation pipelines and agreement about what a complete explanation should contain.
  • Only a few XAI approaches have efficient implementations, which impedes deployment in large-scale applications.The survey relates this limitation to the field’s youth and emergence.
  • Stronger model-specific approaches could improve fidelity and enable analysis of models’ inner workings rather than only their outcomes.The survey further suggests that avoiding costly approximations could facilitate efficient algorithmic implementations.
  • Hybrid models combine the expressiveness of opaque models with the clear semantics of transparent models.The survey presents this direction as a way to bridge opaque and transparent models and support explainable models with state-of-the-art performance.
  • Future XAI approaches should incorporate causal analysis as the field matures.The survey connects causality with future work on explanations and with fundamental concerns such as fairness and bias.
Loading 2009.11698v1…