PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on the Explainability of Supervised Machine Learning

Nadia Burkart, Marco F. Huber

arXiv:2011.07876v1cs.LGcs.AIstat.ML

TL;DR

Accurate machine-learning models can be opaque, creating a need for understandable decision-making in sensitive applications. This survey defines explainable supervised machine learning, reviews and classifies its approaches, illustrates them with a use case, and discusses future directions. It concludes with a broad overview while identifying the need for standardized procedures to measure and compare explainability.

  • Problem

    Accurate models can be difficult for humans to understand, especially in sensitive domains where decision-making must be more transparent, accountable, and understandable.

  • Method

    The paper defines explainable SML, formalizes explanation approaches, reviews corresponding literature, and classifies approaches across the explanation chain.

  • Results

    The survey provides a broad overview of explainable SML approaches, including interpretable models, surrogate fitting, and local or global explanation generation.

  • Takeaways & Limitations

    The paper presents explainability as a field spanning multiple approaches and emphasizes that explanations should be assessed through systematic comparison.

  • Takeaways & Limitations

    A standardized procedure to measure, quantify, and compare explainability is still missing.

Abstract

from arXiv · show

Predictions obtained by, e.g., artificial neural networks have a high accuracy but humans often perceive the models as black boxes. Insights about the decision making are mostly opaque for humans. Particularly understanding the decision making in highly sensitive areas such as healthcare or fifinance, is of paramount importance. The decision-making behind the black boxes requires it to be more transparent, accountable, and understandable for humans. This survey paper provides essential definitions, an overview of the different principles and methodologies of explainable Supervised Machine Learning (SML). We conduct a state-of-the-art survey that reviews past and recent explainable SML approaches and classifies them according to the introduced definitions. Finally, we illustrate principles by means of an explanatory case study and discuss important future directions.

1. Introduction

Explainable supervised machine learning is motivated by the opacity of accurate models, especially where decisions affect people, and is organized through definitions and classifications of explanation approaches.

  • Motivation: Accurate AI models can remain opaque to humans, creating a need for deeper understanding in sensitive domains such as healthcare.The paper links explainability particularly to applications where understanding the model and its outputs is crucial.
  • Motivation: Explainability is especially relevant when problems contain incompleteness that cannot be sufficiently encoded into a model.The paper distinguishes incompleteness from uncertainty, which can be formalized and handled mathematically.
  • Motivation: Automated decisions may require explanations when outcomes affect insurance, employment, education, advertising, or medical treatment, particularly after rejection.The paper notes that people are more likely to seek reasons when an automated decision conflicts with expectations.
  • Survey scope: The survey classifies explainable supervised learning into interpretable models, surrogate model fitting, and explanation generation.Explanation generation is further divided into local and global explainability.
  • Survey scope: Its contributions include formalizing explanation approaches, reviewing reasons and domains for explainability, examining data-related aspects, presenting a continuous use case, and discussing future directions.The contribution list spans the explanation chain for classification and regression and includes data quality and ontologies.

2. Reasons for Explainability and Demanding Domains

The paper presents explainability as serving trust, accountability, fairness, adjustment, and deployment needs, while its necessity varies across domains and decision consequences.

  • Reasons for explainability: Explainability supports trust because understanding a model’s strengths and weaknesses is described as a prerequisite for deployment.The paper also connects explainability with human acceptance of prediction models.
  • Reasons for explainability: Explainability can support fair and ethical decision-making by making automated results comprehensible to affected people.The paper frames explanations as relevant to perceiving conformity with ethical standards and exercising a right to explanation.
  • Reasons for explainability: Explanations help domain experts compare predictions with domain knowledge and adjust models by incorporating that knowledge.The paper also describes explainability as supporting better decisions by domain experts.
  • Demanding domains: Explainability is not required in every domain, particularly where systems operate without human interaction or mistakes have no direct consequences.The paper gives aircraft collision avoidance and some recommendation systems as examples of settings where black-box models may be used.
  • Demanding domains: Healthcare, judicial, banking, automotive, marketing, elections, agriculture, and military training are presented as domains where explanations can clarify decisions or guide action.Examples include explaining cancer-risk screening, credit denial, accidents, customer preferences, voting behavior, harvest decisions, and training goals.

3. Concepts of Explainability

The paper describes explanation approaches ranging from whole-model interpretation to inherently understandable replacements, learning-time steering, and explanations of individual predictions.

  • Explanation approaches: Some approaches explain the model as a whole or replace it with an inherently understandable model such as a decision tree.These approaches target global understanding rather than only individual predictions.
  • Explanation approaches: Other methods steer a model toward greater explainability during learning or explain single predictions by highlighting important features.The passage contrasts learning-process interventions with local feature-based explanations.

3.1 Problem Definition and Dimensions

This section formalizes supervised machine learning as learning a model from feature-target pairs by minimizing prediction error. It distinguishes black-box and interpretable models and introduces global versus instance-focused explanation settings.

  • Problem definition: Supervised machine learning learns a mapping from feature vectors x to targets y using labeled training data D.Classification predicts discrete labels, whereas regression predicts continuous target values.
  • Model types: Black-box models and interpretable models are separate hypothesis spaces, exemplified by neural networks and depth-3 decision trees.Both map inputs from X to outputs in Y, but interpretable models are intended to be human-comprehensible.
  • Optimization: Training minimizes an error measure over the training instances, producing a model h* that can predict on unseen inputs.For parametric models, optimization is expressed in terms of a parameter vector θ.
  • Optimization: Many learning objectives cannot be solved analytically, so numerical methods such as gradient descent commonly produce sub-optimal solutions.Deep neural networks determine their parameters using gradient descent.
  • Prediction pipeline: The learning pipeline applies the resulting model to an unseen instance x to obtain a prediction y = h*(x).The distinction between the learning algorithm and the resulting model is central to this pipeline.

3.2 Five Ways to Gain Interpretability

The paper organizes explainability into model/global and instance/local approaches, then distinguishes direct explanators, interpretable models, and surrogate models. These approaches differ in when interpretability is obtained, what scope it covers, and whether it depends on a particular model or data.

  • Global and local explanations: Explanation approaches are global when they describe an entire model and local when they explain one prediction and its nearby inputs.Local explanations are valid only for the selected instance and its close vicinity.
  • Explanation generation: An explanator takes a learned model and data or an instance as input and produces a human-comprehensible explanation.Explanators support understanding but are not predictors themselves.
  • Explanators: Global explanators directly interpret a model, whereas local explanators explain the prediction for a particular test input.Partial dependency plots are given as an example of global explanation, while local examples include counterfactuals and Shapley values.
  • Interpretable models: Interpretable or white-box models build interpretability into model creation, an approach also called ante-hoc interpretability.Typical examples include small decision trees and linear regression models.
  • Surrogate models: Surrogate fitting translates a black-box model into an approximate interpretable model, either globally over representative data or locally around a test input.A fidelity score measures agreement between surrogate and black-box predictions; local surrogates include LIME and SHAP.
  • Summary: Interpretable model learning and surrogate fitting are mutually exclusive with respect to ante-hoc versus post-hoc explainability.Surrogates are created after a black-box model exists and require data to optimize fidelity or construct local explanations.

3.3 Interpretable Model Types

The paper surveys model families that provide interpretability by nature, including linear, tree-based, rule-based, probabilistic, neighbor-based, interactive, and Bayesian models. Their explanations arise from structures such as feature weights, decision paths, rules, similarities, feedback, or probabilistic dependencies.

  • Linear models: Linear models assign weights to input features, with each weight indicating that feature’s contribution to the prediction.They can support both classification and regression, including scoring systems.
  • Decision trees: Decision trees explain predictions through paths from a root node to a leaf using feature-based splits.Leaves contain class labels for classification or averaged values for regression.
  • Rule-based models: Rule-based models express predictions through conditional structures such as IF condition THEN label ELSE other label.Rules can be organized into lists, sets, tables, or m-of-n forms.
  • Probabilistic models: Naive Bayes simplifies probabilistic classification by assuming that feature effects are independent of one another.The assumption rarely matches reality but significantly reduces the problem’s complexity.
  • Nearest neighbors: Nearest-neighbor models classify or regress by comparing a test instance with similar training instances.Classification uses majority voting, while regression averages neighbors’ values.
  • Interactive models: Interactive models incorporate end-user feedback into learning, allowing users to review outputs, make corrections, and provide feedback.They are treated as human-in-the-loop algorithms.
  • Bayesian networks: Bayesian networks represent variables as nodes and conditional relationships as directed edges to model causal relations probabilistically.Their outputs are probabilistic rather than logical.

3.4 General-Purpose Techniques for Interpretability by Design

Interpretability can be designed into models using constraints and representations that make predictions easier for humans to understand. The surveyed techniques include feature restrictions, monotonic relationships, and ontologies, while simplicity must be balanced against oversimplification.

  • Interpretability and accuracy: Interpretability and accuracy do not necessarily require a trade-off when both objectives are explicitly considered during model generation.The paper notes that claims about relative interpretability depend on the features and data structure involved.
  • Interpretability by design: Interpretability can be enhanced through constraints such as sparsity, monotonicity, limited rules, or restricted tree size.These techniques apply across linear, Bayesian-network, rule-based, and tree-based models.
  • Sparsity: Sparsity limits the number of active elements to accommodate human cognitive capacity while supporting local or global explanations.Models that are too simple may oversimplify the relationship between inputs and outputs.
  • Monotonicity: Monotonicity constrains input-output relationships so that increasing an input consistently increases or decreases the output.The constraint can be incorporated as either a hard or soft condition.
  • Ontologies: Ontologies structure or simplify possible explanations for interpretable models, surrogate models, and their data.They can support either local or global explanation outputs.

3.5 The Explanation

Explanations vary by content, communication form, scope, and audience. The survey distinguishes local and global explanations, several communication modalities, and user groups with different explanatory needs.

  • Explanations can differ in content, communication, and target group, requiring choices about what to explain, how to communicate it, and whom to address.
  • Explanation scope: Local explanations describe an individual prediction or its neighborhood, whereas global explanations describe dependencies and the model’s overall behavior.
  • Explanation scope: Counterfactual explanations explain a decision, provide grounds to contest it, and indicate changes that might produce a preferred outcome.
  • Explanation scope: Prototype explanations use similar examples, while criticism identifies what a prototypical example failed to capture.
  • Communication: Explanations may be textual, graphical, or multimedia, and their appropriate form depends partly on the available data type.
  • Target groups: Non-experts seek simple reasons for decisions, while domain experts, system developers, and AI developers pursue deeper understanding for trust, functionality, debugging, or improvement.

3.6 Assessment of Explainability

The survey presents human-centered criteria and several evaluation strategies for explainability. Explanations should be concise, contrastive, selected, socially appropriate, and assessed through human tasks or proxy measures.

  • Human-centered explanations are expected to be contrastive, concise, selected, social, and focused on causality rather than probabilities.
  • Assessment levels: Explainability can be assessed with real-world human tasks, simple experimental human tasks, or proxy tasks and metrics.
  • Model metrics: Rule-based model complexity can be measured through rule overlap, average rule conditions, tree size, or tree depth.
  • Application metrics: Recommender-system explainability can use neighborhood similarity, including a score ranging from zero to one.
  • Case study: The survey illustrates its categories with classification examples on the IRIS dataset, including a local instance from the virginica class.

4. Interpretable Model Learning

Interpretable model learning includes models interpretable by nature and models interpretable by design. The survey reviews tree-, rule-, additive-, and region-based approaches, with a use case showing shallow trees remain comprehensible.

  • Interpretable-by-nature algorithms produce understandable models without explicitly optimizing training for interpretability, which is treated as an inherent by-product.
  • Interpretable by nature: 1R and ant-based methods generate single rules or ordered and unordered rule sets for classification.
  • Interpretable by nature: Generalized additive models visualize feature–target relationships through shape functions, but standard GAMs do not model feature interactions.
  • Use case: Decision trees of depth two or three remain comprehensible in the survey’s illustrative use case.
  • Interpretable by design: Interpretable-by-design approaches incorporate interpretability into training so the degree of interpretability can be controlled or increased.
  • Interpretable by design: Two-level Boolean rules and Bayesian or optimization-based rule lists trade off accuracy, sparsity, rule size, or model size during learning.

5. Surrogate Models

Surrogate methods approximate black-box models or predictions with interpretable representations. The survey covers global and local surrogates, rule extraction strategies, and post-hoc feature-attribution methods.

  • A surrogate model approximates an otherwise uninterpretable black box, separating prediction from explanation through an interpretable model.
  • Surrogate fitting: Global surrogates learn interpretable models that mimic black-box predictions, while local approaches explain individual model predictions.
  • Global surrogates: sp-LIME combines explanations for representative instances, while k-LIME clusters black-box data into local regions.
  • Surrogate models: Decision-tree extraction methods generate data labels from a black box and train an approximating tree, using procedures such as clustering.
  • Rule extraction: Rule extraction may be pedagogical, decompositional, or eclectic, depending on whether it treats the black box externally or uses its internal structure.
  • Rule extraction: Rule extraction can label feature combinations absent from training data because the underlying black box supplies predictions for those combinations.
  • Rule extraction from neural networks: ANN and DNN rule-extraction methods include BRIANNE, VIA, BIO-RE, genetic programming, SUBSET, MofN, Knowledgetron, and NeuroRule.
  • Feature attribution: SHAP is a local post-hoc method that explains a selected prediction through feature contributions derived from a bias-plus-contributions representation.

6. Explanation Generation

Explanation generation approaches directly infer local or global explanations from a black-box model, rather than fitting surrogate models. The survey organizes these approaches in an overview of global explanation methods.

  • Explanation generation directly infers local or global explanations from a black-box model.
  • The survey catalogs global explanation generation methods in an overview table.
  • Global explanations are presented as generated explanations associated with the model-level explanation pipeline.

6.1 Global Explanation Generation

Global explanation generation methods reveal properties of a black-box model independently of particular predictions. They include feature-importance, sensitivity, interaction, dependence, and visualization approaches, with complementary strengths and limitations.

  • Global explanators reveal black-box properties independently of particular predictions.
  • Sensitivity and interaction methods vary or randomize features to measure output changes and identify feature interactions.GSA varies input features across their ranges, while Golden Eye and ASTRID identify interaction groupings without relying on the classifier’s internal structure.
  • Feature-importance methods assign overall influence scores, but global feature importance does not describe the direction or effect of those features.
  • The global feature-importance example identifies sepal-length and sepal-width as the model’s most decisive features.

6.2 Local Explanation Generation

Local explanation generation methods explain individual predictions within a specific instance’s vicinity. The survey covers feature attribution, gradients, perturbation, attention, counterfactual, and tree-based approaches.

  • Local explanations are valid only near a particular prediction.
  • Feature attribution methods quantify each feature’s contribution, while local gradients encode influence direction by sign and magnitude by absolute value.
  • Perturbation-based methods estimate feature influence by removing, omitting, or changing feature values and measuring prediction differences.LOCO compares models with and without a feature; QII randomly perturbs feature values and measures changes in the prediction.
  • Attention-based explanations highlight promising input parts, but studies disagree about whether attention weights meaningfully explain predictions.
  • For a setosa sample, treeinterpreter contributions sum to 1 because the random forest predicts the class with complete certainty.Petal length and petal width have positive impact, while sepal features are almost not considered.

7. Data and Explainability

Explainability depends on data quality, visualization, and domain knowledge. The survey discusses visualization techniques and ontologies as ways to inspect data, formalize knowledge, and improve explanations.

  • 7.1 Data Quality: Data quality is essential for explainability because incomplete and noisy training data can produce poor results.
  • 7.1 Data Quality: Data quality is context-dependent and includes completeness, uniqueness, timeliness, validity, accuracy, and consistency.
  • 7.2 Data Visualization: Data visualization supports exploratory analysis by reducing entire datasets to two or three dimensions and complementing prediction models.
  • 7.2 Data Visualization: Visualization approaches include feature-weight displays, topology-preserving maps, glyphs, correlation graphs, residual plots, PDPs, ICE plots, and heat maps.
  • 7.3 Ontologies and Explainability: Expert-approved ontologies can support data consistency checks and improve explainability by incorporating domain knowledge before training or after explanation generation.
  • 7.3 Ontologies: Ontologies formalize domain types, properties, and relations, enabling knowledge exchange and inference from relational concepts.

8. Discussion and Open Challenges

The discussion frames explainability as an unsettled field requiring clearer objectives, standardized evaluation, user-centered validation, and further investigation of trust and ontology-based explanations. It also highlights a continuing debate over interpretable models versus black-box systems and the practical similarity of SHAP and LIME for users.

  • Discussion: Explainability research remains divided over whether every machine-learning system needs to be understandable, although regulated areas are broadly recognized as requiring understandable models.The discussion contrasts proponents who prioritize understandable models with opponents who question explainability for every purpose.
  • Discussion: SHAP and LIME produce marginally different feature values, but their explanations are presented as practically similar to users.Both methods provide a general direction and identify significant contributions despite differing production approaches.
  • Open Challenges: Feature-importance explanations may make intuitive sense to statisticians while remaining poorly aligned with how lay users want decisions explained.This concern is identified as a challenge for the intrinsic quality of explanations.
  • Open Challenges: Interpretable models that match black-box accuracy, along with user-centered experiments, are presented as priorities for advancing trustworthy explanations.The discussion separately emphasizes comparing interpretable and black-box models and expanding experiments on user trust and interpretability.
  • Open Challenges: The field lacks a standardized procedure and metrics for measuring, quantifying, and comparing explainability across approaches.The paper calls for explainability measures analogous to accuracy, recall, and F1-score.
  • Open Challenges: Future work should combine ontologies with explanations in practical use cases and evaluate their advantages through user studies.The discussion identifies both practical applications and empirical examination as necessary.

9. Conclusion

The paper surveys explainable supervised machine learning approaches and shows their relevance across domains, while emphasizing that model explanations can provide only human-graspable approximations of decision processes.

  • Explainable machine learning is relevant across many domains, including medical applications such as learning more about COVID-19.
  • The paper defines explainable SML problems and categorizes and reviews past and recent approaches within each field.
  • One surveyed approach uses a local linear approximation to identify influential features and provide an interpretable representation of a complex model’s prediction.
  • Model explanations can approximate decision processes in a human-graspable form, but they do not capture the coherent, contextual stories humans use to explain decisions.
  • Ethical dilemmas in automated decision-making, including choices faced by self-driving cars, remain an important explainability-related challenge.
Loading 2011.07876v1…