PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Artificial Intelligence: a Systematic Review

Giulia Vilone, Luca Longo

arXiv:2006.00093v4cs.AIcs.LG

TL;DR

XAI addresses the difficulty of explaining complex machine-learning models, especially deep-learning systems, amid a rapidly expanding and scattered literature. The paper systematically reviews this research and organizes it into a hierarchical classification. It identifies four main categories and emphasizes unresolved challenges involving explanation representations, structures, and human users.

  • Problem

    Complex, nonlinear machine-learning and deep-learning models are difficult for people to interpret, while explainability research has become abundant and scattered.

  • Method

    The paper conducts a systematic literature review and maps the selected XAI research into a hierarchical classification.

  • Results

    The classification identifies four main categories: reviews, notions, new explainability methods, and evaluation of explainability methods.

  • Takeaways & Limitations

    The review frames XAI as a broad field requiring methods adaptable to different contexts, applications, and end-users.

  • Takeaways & Limitations

    Human involvement in creating and using artificial explanations has not been sufficiently studied.

Abstract

from arXiv · show

Explainable Artificial Intelligence (XAI) has experienced a significant growth over the last few years. This is due to the widespread application of machine learning, particularly deep learning, that has led to the development of highly accurate models but lack explainability and interpretability. A plethora of methods to tackle this problem have been proposed, developed and tested. This systematic review contributes to the body of knowledge by clustering these methods with a hierarchical classification system with four main clusters: review articles, theories and notions, methods and their evaluation. It also summarises the state-of-the-art in XAI and recommends future research directions.

1. Introduction

XAI research has expanded rapidly, creating a scattered body of domain-dependent methods for interpreting machine-learning models and generating human explanations. This review organizes the field and defines broader boundaries for XAI.

  • Research motivation: XAI research has grown significantly, producing a large and scattered body of explainability knowledge.The review considered about 350 articles selected through keyword searches and bibliography examination.
  • Research motivation: Black-box machine-learning and deep-learning models are difficult for laypeople to interpret because their structures are complex and nonlinear.The resulting opacity motivates transparent models, human interaction, and trustworthy inferences.
  • Research gap: Prior reviews often focused on specific model families, explanation formats, or application areas rather than XAI as a whole.Examples include reviews of neural and Bayesian networks, SVM rules, and visual explanations for deep learning.
  • Paper aim: This paper systematically reviews XAI research, organizes its methods and concepts, and proposes a classification system with broader scope.The paper also outlines its research-methods, classification, review, and concluding sections.

2. Research methods

The review defines explicit scope constraints and uses a two-phase search strategy combining keyword searches with backward bibliography checks. This process narrowed several thousand results to a corpus of peer-reviewed XAI publications.

  • Scope constraints: The review excludes explainability studies outside AI and computer science, non-peer-reviewed reports, and methods not designed specifically to produce explanations.Methods for general data visualization or feature selection were discarded when explanation was not their direct purpose.
  • Search strategy: The systematic review was conducted in two phases.The phases consisted of database searching followed by bibliographic expansion.
  • Search strategy: Google Scholar searches used the terms “explainable artificial intelligence”, “explainable machine learning”, and “interpretable machine learning”.The searches returned several thousand results, with the first ten pages treated as the likely source of relevant articles.
  • Corpus construction: The initial searches produced almost two hundred peer-reviewed publications, after which bibliographies were checked thoroughly to identify further studies.The supplied passage states that this second phase led to selection of one hundred articles whose bibliographic details continue beyond the excerpt.

3. Classification of scientific articles on explainability

The paper classifies XAI literature into four main categories: reviews, notions, new explainability methods, and method evaluation. These categories form a hierarchical literature map whose branches can overlap and whose relationships reflect the field’s development.

  • Main categories: The classification contains four main categories: reviews, notions, new explainability methods, and evaluation of explainability methods.The categories respectively cover literature organization, conceptual requirements, novel methods, and performance studies.
  • Hierarchical structure: The classification is represented as a tree whose root contains the four categories and whose leaves represent scientific articles.Branches can have different depths, reflecting more detailed organization within the literature.
  • Hierarchical structure: A paper may appear in multiple branches because it can address multiple dimensions of XAI.The classification therefore allows overlap rather than assigning every paper to one exclusive category.
  • Literature distribution: The map highlights how research effort is distributed across categories, with development of explainability methods receiving particular emphasis.Figure 2 part b displays the percentage of articles grouped by category.
  • Category relationships: Reviews depend on relevant notions, methods, and evaluation approaches, while evaluation generally follows the creation of explainability methods.The stated relationships connect conceptual requirements, method development, and assessment within the field.

4. Reviews of the XAI literature

Reviews of XAI literature are organized by application field, construction approach, theory, output format, and problem type. They provide entry points into a rapidly growing field, while unresolved challenges remain in representing model inference and structuring explanations effectively.

  • Review categories: Review articles summarize explainability solutions for specific systems, techniques, application fields, output formats, or problem types.Examples include textual explanations, neural networks, visual explanations, rules, regression, and classification.
  • Role of reviews: Reviews help researchers enter the rapidly growing explainability literature and identify techniques, strengths, and limitations.They organize information across specialized classes of solutions and AI techniques.
  • Review categories: Construction-approach reviews distinguish data-driven methods from knowledge-driven methods.Data-driven approaches extract knowledge from trained models without prior expert knowledge, whereas knowledge-driven approaches capture expert knowledge and logic.
  • Review categories: Generic reviews cover broad ranges of data- or knowledge-driven models and their explainability methods.They are used when a review cannot be placed in another category.
  • Theories and explanation design: Explanation goals depend on the application domain, underlying model, and end-users’ preferences and preconceptions.The reviewed theories include explaining how and why an answer was reached, clarifying terms, and teaching users about the domain.
  • Open challenges: Learning effective representations of the inferential process of data-driven models remains an open problem.The literature also reports unresolved challenges in visually inspecting data-driven models and explaining their functioning.

5. Notions related to the concept of explainability

The review organizes explainability notions around the attributes, types, and structures of explanations, while linking them to user trust, causal reasoning, and practical purposes. It also emphasizes that universal criteria and effective explanation structures remain unresolved.

  • Conceptual organization: The proposed notion clusters cover attributes of explainability, types of explanation, and the structure of explanations.Structure concerns components such as causes, context, and consequences, including their ordering.
  • Purposes of explainability: Trust is identified as a principal reason for producing explanations because it can increase users’ confidence and comfort with a system.
  • Attributes of explainability: Causality is treated as a fundamental attribute, but data-driven associations do not guarantee causal relationships and causal inference depends strongly on prior knowledge.The review distinguishes generating hypotheses from explainability methods themselves.
  • Purposes of explainability: Explanations are described as serving justification, control, improvement, and discovery.These purposes respectively concern decision justifiability, debugging and flaw identification, model accuracy and efficiency, and extracting novel knowledge.
  • Open issues: Universal objective criteria for building and validating explanations remain unresolved despite extensive cross-disciplinary work on how explanations are defined, generated, selected, evaluated, and presented.
  • Types of explanations: Explanation classifications vary with users, behaviors, and problems, commonly distinguishing why and how questions and traced-based from reconstructive explanations.
  • Structure of explanations: The structure of explanations remains an open problem because its ability to accommodate novel information within prior beliefs can affect learning and generalization.

6. Development of new methods for explainability

The review classifies explainability methods by the scope and stage of explanation generation, then adds problem type, input data, and output format. Figure 4 uses these categories to organize methods and article distributions, while distinguishing model-agnostic from model-specific post-hoc approaches.

  • Classification criteria: Methods are first discriminated by explanation scope: global methods explain a model’s overall inference process, whereas local methods explain individual inferences.
  • Classification criteria: Methods are also classified by stage: ante-hoc approaches build explainability into model development, while post-hoc approaches explain an unchanged trained model at testing time.
  • Additional criteria: The review adds problem type, input data, and output format as criteria for describing explainability methods.Problem type includes classification or regression; inputs may be numerical/categorical, pictorial, textual, or time series; outputs may be numerical, rules, textual, visual, or mixed.
  • Classification overview: Figure 4 classifies each method across stage, scope, problem type, input data, and output format, and shows the distribution of articles across these branches.
  • Model dependence: Post-hoc methods are divided into model-agnostic approaches applicable to any black-box model and model-specific approaches restricted to particular model classes.Ante-hoc methods are generally intrinsically model-specific because they make model functioning transparent during development.
  • Thematic organization: The review groups the large set of identified methods into five thematic classes and reports their classification characteristics in accompanying tables.

6.1. Output formats

XAI methods communicate model behavior through visual, argumentative, and numerical formats, with visual aids offering an intuitive way to expose model functioning.

  • Visual explanations use graphical tools to communicate information and illustrate a model’s inner functioning.Heat-maps can highlight image regions or text words that most influence an inference.
  • Argumentative explanations organize antecedents and consequents into arguments linked by attacks to represent conflicts.This structure is presented as having explanatory power for data-driven models.
  • Numerical explanations represent model behavior as crisp values, vectors, matrices, or tensors.Probe and Concept Activation Vectors are examples of methods using numerical formats.

6.2. Model agnostic methods for explainability

Model-agnostic explainability methods support multiple output formats, but their applicability remains constrained by the input types and explanation goals of the technical problem.

  • Methods designed for any learning technique may still be limited by the technical problem’s input types and intended explanation.
  • Model-agnostic methods produce numerical, rule-based, visual, and mixed explanations.The reviewed examples include feature-contribution scores, decision trees, graphs, heat-maps, plots, interactive interfaces, and prototypes.
  • Numeric explanations: Numerical methods commonly quantify the contribution of an input variable or variable group.Distill-and-Compare trains a simpler student model to mimic a complex teacher, using GAMs to assess feature contributions numerically.
  • Rule-based explanations: Rule-based methods extract interpretable rules, including logical formulas represented as decision trees.One approach combines split predicates along input-to-prediction paths into conjunctions and groups paths for output classes into disjunctions.
  • Visual explanations: Visual methods highlight influential inputs with salient masks or represent explanations through graphs, restricted support regions, heat-maps, and plots.Sensitivity Analysis uses local gradients to indicate how a data point must move to change its predicted label.
  • Mixed explanations: Mixed methods combine numerical or visual representations with text, interactive exploration, prototypes, or adversarial examples.Prototypes provide correctly predicted positive examples, while adversarial examples support contrastive explanations.

6.3. Model-specific methods for explainability based on neural networks

Model-specific neural-network methods predominantly explain deep neural networks through visual representations, while also using rules, numerical formats, text, and combinations of these outputs.

  • Most reviewed neural-network explainability methods produce visual explanations, especially salient masks and scatter-plots.Other methods generate alternative visual aids, rules, textual or numerical explanations, or combinations of formats.
  • Visual explanations: Attention and heat-map methods identify image regions or input areas associated with predictions.CLEAR back-propagates output activation values to return attentive regions, while SWAF uses GradCam heat-maps for visual question answering.
  • Visual explanations: Saliency methods attribute predictions to input features using inversion, relevance propagation, gradient accumulation, or contribution decomposition.DeepLIFT can propagate importance when gradients are zero and separately represent positive and negative contributions.
  • Visual explanations - miscellaneous: Neural-network explanations also use case retrieval, clustering, scatter-plots, and interventions on semantic units.GAN Dissection adds or removes identified semantic objects to observe how a GAN reacts.
  • Visual explanations - miscellaneous: Graph-based methods visualize network structure, part-pattern activations, co-activation relationships, or symbolic reasoning over knowledge graphs.Symbolic Graph Reasoning combines local-to-semantic voting, graph reasoning, and semantic-to-local mapping.
  • Rule-based explanations: Rule-based neural-network methods include decompositional extraction from neuron activations and pedagogical extraction from model-generated instances.DIMLP extracts rules after max-pooling by determining relevant discriminative hyperplanes, while REFNE extracts rules from neural-network ensembles.

6.4. Model-specific methods for explainability related to rule-based systems

Rule-based explainability methods represent model decisions with human-inspectable IF-THEN rules, using both ante-hoc rule learning and post-hoc expert-system interpretation.

  • ACO generates unordered IF-THEN rule sets through sequential covering and compares iterations using predictive accuracy and rule count.Experiments on thirty-two public datasets reported the best results for both predictive accuracy and model size among the compared induction algorithms.
  • AntMinter+ constructs monotonic rulesets with domain knowledge represented by a directed acyclic graph.It adds rules iteratively until a training-coverage threshold is reached or further rules stop improving classifier performance.
  • Rule-based explanations can also be presented as combinations of visual and textual examples or as miscellaneous explanation formats.
  • Interpretable Decision Set and Bayesian rule methods learn unordered IF-THEN rules while modeling accuracy, interpretability, or posterior label probabilities.Interpretable Decision Set emphasizes short, non-overlapping rules, whereas BRL evaluates probabilities over rule permutations.
  • Linear Programming Relaxation learns human-interpretable two-level Boolean rules in conjunctive or disjunctive normal form.
  • Mycin exemplifies post-hoc explainability through a knowledge base of expert IF-THEN rules, contextual facts, an inference engine, and a natural-language interface.

6.5. Other model-specific methods for explainability

The review covers explainability methods tailored to ensembles, SVMs, and Bayesian networks, including techniques that modify inputs, extract rules, simplify models, or expose inference structure.

  • Ensemble methods: Ensemble-specific methods interpret models by modifying input features, generating textual explanations, merging distributed trees, or simplifying tree ensembles.FAB estimates parameters and the optimal number of input-space regions to derive a simpler model with suitable complexity and prediction accuracy.
  • Support Vector Machines: ExtractRule converts hyperplane-based classifiers such as SVMs into compact, non-overlapping symbolic rules that expose the classifier’s inferential process.The review also describes prototypes and feature-tweaking methods that help users understand or alter SVM predictions.
  • Support Vector Machines: Weighted Linear Classifier produces visualizable linear SVM or random-hyperplane models whose accuracy is comparable to nonlinear SVM classifiers.The resulting projections on separating hyperplanes and decision surfaces serve as explanations.
  • Bayesian networks: Explaining Bayesian network Inferences generates decision-tree rules showing how influential variables in a target node’s Markov blanket interact to produce predictions.The method works backward from the target node and derives intermediate nodes, considering parents supported by evidence.
  • Bayesian networks: ASPIC+ represents Bayesian-network reasoning through structured arguments that can attack one another and receive inferential strength from incremental or posterior-probability measures.The cited framework distinguishes evidence-weight measures such as likelihood ratios from strengths based on posterior probabilities.

6.6. Self-explainable and interpretable methods

Self-explainable methods generate naturally interpretable models during modeling rather than explaining opaque models afterward. The review includes prototype-, shape-function-, region-, circuit-, and feature-selection-based approaches.

  • Self-explainable methods: Ante-hoc, or naturally interpretable, models are designed to be explainable through their architecture and input format.Their output format depends on how the model is structured and what inputs it receives.
  • Prototype-based models: Bayesian Case Model jointly infers cluster labels, representative prototypes, and important features for explainable case-based reasoning, classification, and clustering.The learned prototypes correspond to observations that best represent dataset clusters.
  • Additive models: Generalized Additive Models and GA2Ms use shape functions trained on one or up to two features, making individual feature contributions easy to understand and visualize.The review describes bar and line charts as visualization formats for these contributions.
  • Circuit-based models: Probabilistic Sentential Decision Diagrams represent conditional probabilities with logical circuit nodes and can be visualized as interpretable binary trees.Nodes are logical AND gates with two inputs or logical OR gates with an arbitrary number of inputs.

7. Evaluation of methods for explainability

The review divides explainability evaluation into objective, automated assessments and human-centred evaluations involving end-users. Studies examine robustness, completeness, model complexity, productivity, and users’ ability to understand models.

  • Evaluation approaches: The review identifies two main evaluation approaches: objective evaluations using automated metrics and human-centred evaluations involving end-users’ feedback or judgement.This dual categorization structures the evaluation literature summarized in Figure 17.
  • Objective evaluations: Objective comparisons test sensitivity to input perturbation, sensitivity to model-parameter randomization, and explanation completeness.Completeness measures how extensively an explanation describes the underlying model’s inferential process, including affected input features.
  • Objective evaluations: Most formal comparison studies evaluate visual explanations for neural-network image or text classification using heat maps or feature maps.Comparisons measure changes in these maps before and after perturbing inputs or model parameters.
  • Objective evaluations: Saliency-map studies compare explanations with untrained or label-permuted networks, measure Spearman rank correlations, and test image occlusion based on relevance levels.Other work compares completeness across seven methods using derivatives, input perturbations, and network weights.
  • Human-centred evaluations: Human-centred studies involve lay users or domain experts who interact with explanatory systems and report whether they understand or can debug model behaviour.Examples include two-week interaction studies, diagnostic interfaces for patient-risk models, and interactive why-question tools for email classification.
  • Human-centred evaluations: Displaying model internal parameters can hamper users’ ability to notice unusual inputs and correct inaccurate predictions.The review contrasts this with users’ easier simulation of models containing few features.

8. Final remarks and recommendations

The review positions XAI as a broad AI field whose literature is organized into a hierarchical structure and whose future depends on better integration of users, explanators, and evaluation.

  • Scope and contribution: XAI concerns methods that explain the inner logic of learning algorithms, induced models, or knowledge-based inference systems.The review uses XAI to emphasize applicability beyond the narrower term Interpretable Machine Learning.
  • Scope and contribution: The systematic review organizes scattered explainability research into four clusters and extends earlier reviews that focused on specific aspects of the field.Its stated contribution is a structured and hierarchical organization of a wider body of literature.
  • Recommendations: The review recommends greater attention to end-users and interactive explanatory interfaces, drawing on Human-Computer Interaction.It states that human involvement in creating and using existing explainability methods has not been sufficiently studied.
  • Recommendations: The review identifies an inverse relationship between model accuracy and interpretability or explainability as a design trade-off for future work.It suggests combining modern learning techniques with reasoning approaches as one possible direction.
  • Frameworks: The current XAI framework typically moves from input data to model formation, explanation, and occasional objective or human-centred evaluation.The envisioned framework instead places explanators at the center of development because end-users ultimately interact with them.

Appendix A. Appendix

Appendix A classifies explainability methods and evaluation studies across output formats, construction stages, problem types, scopes, input data, and evaluation approaches.

  • Method classifications distinguish output formats, ante-hoc or post-hoc stage, classification or regression problems, global or local scope, and input-data types.The input categories include numerical/categorical, pictorial, textual, and time-series data.
  • Neural-network methods are separately catalogued for visual, rule-based, textual, numerical, and mixed explanations.
  • Evaluation studies are classified by human-centered measurement, comparative methodology, and comparisons of visual explanations across input-data types.Human-centered evaluations distinguish qualitative or quantitative measurement and explanation format.
Loading 2006.00093v4…