PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey Of Methods For Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Dino Pedreschi, Fosca Giannotti

arXiv:1802.01933v3cs.CYcs.AIcs.LG

TL;DR

Opaque decision systems are difficult to understand, creating practical, ethical, safety, and liability concerns. This paper surveys and classifies explanation methods, finding that important questions about defining and measuring explanations remain unresolved.

  • Problem

    Existing research lacks a clear classification that simultaneously organizes explanation problems, explanator types, and black box system types.

  • Method

    The paper formally defines four explanation problems and classifies literature methods across explanation problem, explanator type, and black box dimensions.

  • Results

    The survey provides a comprehensive overview and classification of methods for explaining decision systems based on opaque machine learning models.

  • Takeaways & Limitations

    Important scientific questions remain unanswered, including the lack of agreement on what constitutes an explanation and how to measure it.

  • Takeaways & Limitations

    The literature has not established a general formalism defining explanations and the properties they should guarantee.

Abstract

from arXiv · show

In the last years many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness sometimes at the cost of scarifying accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, delineating explicitly or implicitly its own definition of interpretability and explanation. The aim of this paper is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.

1 Introduction

Opaque machine-learning decision systems raise ethical, safety, and liability concerns, while emerging explanation rights and diverse definitions of interpretability motivate a systematic classification of methods for opening black boxes.

  • Motivation: Black-box systems use sophisticated machine-learning models to predict potentially sensitive individual information, including credit scores, insurance risk, and health status.These models map user features to outcomes or decisions through a learning process based on digital traces.
  • Motivation: The GDPR introduced, to some extent, a right for individuals to obtain meaningful explanations of the logic involved in automated decision-making and profiling.The regulation was adopted by the European Parliament and was scheduled to become law in May 2018.
  • Motivation: Opaque systems built from massive datasets create risks because their decisions are difficult to understand, affecting ethics, safety, and industrial liability.The paper highlights safety-critical applications such as self-driving cars, robotic assistants, and personalized services.
  • Motivation: Explanation is central to responsible, open data science, but scientific communities approach machine-learning explanation from different perspectives and assign it different meanings.Most existing work comes from machine learning and data mining communities.
  • Contribution: The literature contains scattered ad-hoc results and lacks a systematic organization and classification of methodologies for interpreting black-box systems.Open questions concern interpretability, explanation, comprehensibility, explanatory methods, affected data, and the trade-off between prediction accuracy and interpretability.
  • Contribution: The paper proposes a clear classification that simultaneously organizes the problems, data, explanation forms, and model types involved in opening and understanding black boxes.It addresses limitations of general overviews and works focused only on representation formats or particular interpretability issues.

2 Needs for Interpretable Models

Black-box decisions can be critical because hidden logic may create discrimination and undermine trust. Examples include biased classifications, inconsistent credit scores, spurious accuracy, and DNN predictions that humans cannot recognize or explain.

  • Motivation: Black boxes can create discrimination and trust issues when decisions cannot be interpreted.The section identifies these risks as central reasons for requiring interpretable models and explainable predictions.
  • Discrimination: Historical decision data can conceal endemic prejudices, allowing classifiers to reproduce them as apparently general rules.The paper links this danger to a growing “black box society” governed by secret or obfuscated algorithms.
  • Inconsistency: 29% of consumers received credit scores differing by at least fifty points among three major US banks in a study of 500,000 records.The banks were Experian, TransUnion, and Equifax; such differences might mean tens of thousands of dollars over a mortgage’s life.
  • Spurious Accuracy: Accurate black-box classifiers may exploit accidental training artifacts, such as weather differences correlated with enemy and friendly tank photographs.The tank classifier performed poorly in the field despite high test-set accuracy because enemy images were overcast and friendly images were sunny.
  • Neural-Network Failure: DNNs can misclassify images after human-undetectable alterations, despite strong performance on visual and text classification tasks.One reported example caused a tomato image to be labeled as a dog.
  • Neural-Network Failure: DNNs can assign 99.99% confidence to images that are completely unrecognizable to humans.This illustrates a substantial difference between DNN performance and human-recognizable evidence.

3 Interpretable, Explainable and Comprehensible Models

The section defines interpretability as making a model or prediction understandable to humans and distinguishes global from local explanations. It frames interpretability alongside accuracy, fidelity, user context, and model complexity, while noting that complexity is usually approximated by model size.

  • Definitions: Interpretability is the ability to explain a model or prediction in understandable, self-contained terms for a human.The literature also uses comprehensibility as a synonym for interpretability.
  • Interpretability Dimensions: Global interpretability concerns the whole model logic, whereas local interpretability concerns only the reasons for a specific prediction or decision.The section treats these as distinct dimensions of predictive-model interpretability.
  • Interpretability Dimensions: Interpretability depends on user time and expertise: urgent decisions favor simple explanations, while domain experts may prefer larger, more sophisticated models.User background and experience shape how interpretable a model is perceived to be.
  • Model Desiderata: Accuracy measures performance on unseen instances, while fidelity measures how accurately an interpretable model imitates a black-box predictor treated as an oracle.Both can use measures such as accuracy score and F1-score, but fidelity is evaluated against black-box outcomes.
  • Complexity and Limitations: The literature commonly approximates model complexity through size, such as tree depth, number of rules, or conjunctive terms, although this remains difficult to evaluate.Reviewed works rarely study explanation time or user expertise experimentally and often leave global-versus-local scope implicit.

4 Open The Black Box Problems

The survey distinguishes reverse engineering from designing explanations and refines these into three black-box explanation problems plus transparent-box design. These problems target interpretable, accurate predictive models while differing in whether they explain the model globally, explain individual outcomes locally, inspect behavior, or provide transparency.

  • Problem taxonomy: The literature’s problem taxonomy separates reverse engineering from explanation design and identifies three black-box problems plus transparent-box design.The three black-box problems are model explanation, outcome explanation, and inspection; all are formalized as classification problems targeting interpretable and accurate predictive models.
  • Black box model explanation: Black box model explanation seeks a globally interpretable predictor that mimics the black box and yields explanations such as decision trees or rule sets.The comprehensible global predictor provides the global explanation through a decision tree or set of rules.
  • Black box outcome explanation: Black box outcome explanation provides a locally interpretable prediction together with reasons for that individual outcome, without explaining the entire model logic.Examples of local explanations include a decision-tree path or an association rule.
  • Black box inspection: Black box inspection represents how the model works or why it produces some predictions more likely than others through visual or textual outputs.Its formal output is a visual representation of black-box behavior.
  • Interpretation of opening the black box: Opening the black box means explaining the model, explaining outcomes, inspecting it internally, or providing a transparent solution.These four statements summarize the survey’s problem definitions.

5 Problem And Explanator Based Classification

The survey classifies black-box explanation methods by the problem addressed, explanator, black-box model, and input-data type. It organizes papers by problem and solution, while also distinguishing globally versus locally interpretable predictors and generalizable reverse-engineering approaches.

  • Classification criteria: The classification records the problem faced, explanator type, black-box model opened, and input-data type.These features define how the reviewed approaches are organized and compared.
  • Classification structure: Papers are grouped first by problem definition, then by explanator solution and black-box type, with input-data type specified for each work.Sections collect shared problem definitions, subsections collect different explanators, and papers within subsections target the same black-box type.
  • Problem types: Global model-explanation methods produce interpretable predictors that mimic the black box, whereas outcome-explanation methods provide local explanations for individual records.The survey distinguishes globally interpretable predictors from locally interpretable predictors tied to a given record.
  • Explanator types: The explanator taxonomy includes single trees, decision rules, feature importance, salient masks, sensitivity analysis, partial dependence plots, prototype selection, and neuron activation.These explanators respectively support interpretable models, weighted feature explanations, visual highlighting, uncertainty inspection, reduced-space relationships, similar examples, and neural-network inspection.
  • Further classification: Reverse-engineering approaches are classified as generalizable when they query the black box with varied inputs and learn a comprehensible predictor without exploiting internal peculiarities.The survey notes that a further classification emerges from analyzing these papers, beyond classifications explicitly stated in the literature.

6 Solving the Black Box Model Explanation Problem

This section surveys methods that explain black-box models by constructing globally interpretable models that mimic black-box behavior while remaining understandable to humans. It groups approaches by comprehensible predictor, including decision trees and rule-based models.

  • Section scope: The surveyed methods approximate black boxes with globally interpretable predictors that serve as human-understandable explanations.The section distinguishes approaches using decision trees, rules, and other predictor forms.
  • Decision-tree explanations: Decision-tree approaches represent the global explanation with a decision tree and, in this section, operate on tabular data.These methods implement the explanation function f using a decision tree as the comprehensible global predictor cg.
  • Decision-tree explanations: Trepan queries a neural network as an oracle and induces an approximating decision tree by maximizing gain ratio while estimating model fidelity.Trepan was among the earliest single-tree approximations for neural networks, first presented in 1996 by Craven et al.
  • Decision-tree explanations: Prototype-based querying can produce smaller, more understandable decision trees by generating class prototypes, selecting the best ones, and using them to induce the tree.Krishnan et al. constrain prototypes with the input feature dataset and begin from smaller datasets.
  • Decision-tree explanations: Although developed for neural networks, oracle-based tree explainers can potentially generalize to arbitrary black boxes because they do not use neural-network-specific properties.The black box is treated only as an oracle whose inputs and outputs guide construction of a single explanatory tree.
  • Rule-based explanations: Rule-based approaches provide global interpretability by returning decision rules that describe the logic behind a black-box model.The section introduces rules as another commonly used interpretable and easily understandable comprehensible global predictor.

7 Solving the Black Box Outcome Explanation Problem

This section surveys methods for explaining individual black-box predictions through locally interpretable models. It covers model-specific saliency approaches for neural networks and model-agnostic methods that return comprehensible local predictors.

  • Section overview: Outcome-explanation methods provide locally interpretable models that explain individual black-box predictions in terms understandable to humans.The section characterizes this local perspective as increasingly studied in recent years.
  • Saliency masks: Attention-based CNN–RNN models explain image-caption predictions by visualizing the image areas attended to while generating words.The model combines CNN feature extraction with an RNN containing LSTM nodes.
  • Saliency masks: CAM and Grad-CAM methods use network activations to visualize discriminative image regions associated with a particular outcome label.CAM uses global average pooling in CNNs, while Grad-CAM is described as its relaxed generalization.
  • Saliency masks: For text, rationale-based methods learn a short, coherent subset of words that alone suffices for the original prediction and indicates its reason.The rationale is incorporated into the black-box learning process and serves as the local explanation.
  • Model-agnostic explanations: Model-agnostic approaches, including LIME and MES, explain diverse black boxes with comprehensible local predictors or decision-rule explanations.LIME is tested on decision trees, logistic regression, nearest neighbors, SVMs, and random forests, while MES uses Monte Carlo scoring and is tested on logistic regression and SVMs.

8 Solving the Black Box Inspection Problem

The black box inspection problem seeks representations that explain how a classification model works or why it produces particular predictions. The surveyed solutions use sensitivity analysis, partial dependence visualizations, and DNN-specific inspection tools, with some approaches restricted to particular data types or neural networks.

  • Problem definition: Black box inspection represents either how a classification model works or why it returns some predictions more likely than others.The survey frames inspection as an explanation problem for classification systems and notes visualization surveys for DNNs.
  • Sensitivity analysis: Sensitivity analysis explains tabular black boxes by relating output uncertainty to input uncertainty and producing feature-importance or instance-level attribution information.Methods include explanation vectors, Quantitative Input Influence measures, Integrated Gradients, and visualization of feature effects.
  • Partial dependence plots: Partial Dependence Plots provide black-box-agnostic visualizations of predictor–response relationships in reduced feature spaces for tabular datasets.Related approaches visualize feature interactions with Variable Interaction Networks and individual conditional expectations.
  • DNN-specific approaches: DNN-specific tools inspect intermediate-layer computations, neuron activations, learned features, and image patches responsible for particular activations.These tools visualize live layer activations, optimize features in image space, and backtrack network computations.
  • DNN-specific approaches: Information Plane visualizations show that each stochastic-gradient-descent epoch is followed by slow representation compression, while a single trained neuron can match strong-baseline sentiment-analysis performance.The surveyed DNN approaches are described as not generalizable beyond DNN black boxes.

9 Solving the Transparent Box Design Problem

This section surveys transparent-box methods that produce inherently interpretable predictors and global explanations, primarily for tabular data. The approaches use rule-based predictors, prototypes, or other interpretable models, while some address limitations in prototype-based explanations.

  • Rule-based predictors: Rule-based methods learn comprehensible global predictors using predictive association rules, Bayesian falling rule lists, decision lists, Boolean rules, decision sets, rule sets, or sparse clause rules.These approaches vary in their learning mechanisms and in how they balance accuracy, sparsity, rule structure, and domain-specific interpretability.
  • Prototype-based predictors: Prototype-based methods explain predictions through representative instances, including prototype selection for handwritten digits and Bayesian Case Models that pair cluster prototypes with important feature subspaces.The Bayesian Case Model’s global explanation returns prototypes together with their fundamental features.
  • Prototype-based predictors: Prototype explanations for deep neural networks can modify image representations using original-image information and a generic natural-image prior to construct artificial image prototypes.The surveyed method is mainly related to image reconstruction but aims to understand the example associated with a DNN prediction.
  • Prototype-based predictors: Single representative prototypes for deep neural networks may trigger black-box artifacts, including misclassification from nearly invisible perturbations or abstract synthetic images.The section identifies artifact susceptibility as a risk of perturbation, deletion, preservation, and similar prototype-finding approaches.
  • Other transparent approaches: Other approaches explain individual black-box predictions through feature contributions or learn a global decision tree with sparse oblique splits and local sparse additive experts.The feature-contribution method is black-box agnostic and tested on tabular data, whereas OT-SpAMs partitions feature space into regions with leaf-level experts.

10 Conclusion · A Supplementary Materials

The conclusion summarizes the survey’s classification and formalization of black-box explanation problems while emphasizing that explanation standards and human-comprehensibility measures remain unsettled. It also identifies open directions involving hidden features, recommender systems, and learning predictors directly from explanations.

  • 10 Conclusion: The survey provides a comprehensive overview of methods for explaining decision systems based on opaque and obscure machine-learning models.It identifies components of the explanation-problem family and formally defines each problem’s particular characteristics.
  • 10 Conclusion: Machine-learning research primarily describes how black boxes work, whereas data-mining research more often explains decisions without detailing opaque-system mechanisms.
  • 10 Conclusion: No agreement exists on what constitutes an explanation, with prior works variously using rules, decision trees, or other forms.The conclusion identifies formal explanation frameworks and human-comprehensibility measures as needed to improve practical applicability.
  • 10 Conclusion: Important scientific questions remain unanswered despite the many proposed approaches to explaining black boxes.The paper notes that additional open questions are emerging and require further investigation.
  • 10 Conclusion: Explanations should address information used by black boxes beyond the features explicitly requested from users.A system may augment user data by linking it with different data sources before making predictions.
  • 10 Conclusion: Recommender systems remain an open area where suggestions should be accompanied by reasons for the recommendation.A cited case-based reasoning approach generates recommendations while providing explanations of both the process and the recommendations.
  • 10 Conclusion: Explanations can be valuable independently of predictions, since predictors may be learned directly from them.A cited study trained a software agent to simulate Mario Bros using explanations rather than logs of previous plays.
Loading 1802.01933v3…