PaperlayerView exact revision on arXiv

Source-linked AI summary

From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, Christin Seifert

arXiv:2201.08164v3cs.AI

TL;DR

Evaluating explanations is not self-evident, and accuracy alone can leave models non-transparent. This survey organizes explanation quality into twelve properties and reviews evaluation practices, finding frequent reliance on anecdotal evidence and user studies.

  • Problem

    Accuracy alone can leave models non-transparent, while evaluating explanations presents various pitfalls.

  • Method

    The survey introduces twelve properties of explanation quality and aggregates evaluation practices across 312 papers introducing explainable AI methods.

  • Results

    1 in 3 papers evaluate exclusively with anecdotal evidence, and 1 in 5 papers evaluate with a user study.

  • Takeaways & Limitations

    The Co-12 properties provide an aggregated view of what to evaluate, while the survey responds to calls for automated and quantifiable evaluation metrics.

  • Takeaways & Limitations

    A change in predictive model f is necessary but not sufficient for correctness, and user validation can unintentionally combine evaluation of explanation properties.

Abstract

from arXiv · show

The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented as a subjectively validated binary property, we consider it a multi-faceted concept. We identify 12 conceptual properties, such as Compactness and Correctness, that should be evaluated for comprehensively assessing the quality of an explanation. Our so-called Co-12 properties serve as categorization scheme for systematically reviewing the evaluation practices of more than 300 papers published in the last 7 years at major AI and ML conferences that introduce an XAI method. We find that 1 in 3 papers evaluate exclusively with anecdotal evidence, and 1 in 5 papers evaluate with users. This survey also contributes to the call for objective, quantifiable evaluation methods by presenting an extensive overview of quantitative XAI evaluation methods. Our systematic collection of evaluation methods provides researchers and practitioners with concrete tools to thoroughly validate, benchmark and compare new and existing XAI methods. The Co-12 categorization scheme and our identified evaluation methods open up opportunities to include quantitative metrics as optimization criteria during model training in order to optimize for accuracy and interpretability simultaneously.

1 INTRODUCTION

The survey frames XAI evaluation as a multi-faceted, quantitatively assessable problem and systematically reviews evaluation practices and methods across hundreds of papers. It identifies widespread reliance on anecdotal evidence while organizing explanation quality and quantitative metrics into practical guidance.

  • Motivation: Traditional indicators measure prediction accuracy and computational complexity, but interpretability remains difficult to quantify and lacks standardized evaluation metrics.Evaluation choices may depend on the application domain, explanation and data types, user knowledge, and the question being answered.
  • Conceptual framework: Interpretability is treated as a multi-faceted characteristic whose properties can be measured quantitatively rather than as a binary property.The survey proposes a multi-dimensional view of how strongly explanation-quality properties are satisfied.
  • Systematic review: The survey systematically collected 606 papers published from 2014–2020, including 312 papers that introduced an XAI method.The review analyzes research activity and evaluation practices in a structured manner.
  • Findings: 1 in 3 papers evaluated exclusively with anecdotal evidence, while 1 in 5 papers evaluated with a user study.The survey also found feature importance to be the most common explanation type and single-prediction explanations to predominate over global model insights.
  • Conceptual framework: The Co-12 properties categorize explanation quality, including Completeness, Correctness, and Compactness, and structure the analysis of quantitative evaluation methods.The framework makes explicit that explanation quality comprises multiple aspects.
  • Implications: The collected evaluation methods provide tools to validate and compare XAI methods and could support incorporating explanation-quality metrics into model training.The survey identifies optimizing accuracy and interpretability simultaneously as a research opportunity.

2 RELATED WORK: ANECDOTAL EVIDENCE, FUNCTIONAL EVALUATION AND USER STUDIES

Prior XAI evaluation work distinguishes plausibility from correctness and considers evaluation with or without users. The review highlights pitfalls of anecdotal evidence, functional proxies, and human studies, motivating structured quantitative evaluation across multiple explanation properties.

  • XAI evaluation literature centers on the distinction between plausibility and correctness, and between evaluation with and without users.
  • Evaluating Plausibility or Correctness: Anecdotal inspection cannot reliably distinguish errors in the predictive model from errors in the explanation method.An unreasonable explanation may reflect flawed training data, model reasoning, or the explanation procedure.
  • Evaluating Plausibility or Correctness: Visual judgments that explanations look reasonable may assess black-box accuracy rather than explanation faithfulness and can be misleading.Saliency maps resembling edge detectors illustrate how human observers may mistake plausible patterns for model-sensitive explanations.
  • Functional Evaluation: Functional evaluation uses computational proxies such as explanation size or input perturbation, avoiding human experiments while improving scalability.These measures can save time and costs and suit unethical user studies or methods not yet mature for user evaluation.
  • User Studies: User studies can produce subjective results, ethical concerns, limited replicability, and bias toward simpler explanations aligned with user expectations.Human validation may also combine evaluation of explanation correctness with evaluation of predictive-model correctness.
  • Discussion: The review frames explanation quality as multifaceted, introducing twelve properties and organizing quantitative evaluation methods around them.It argues that explanation aspects should be evaluated independently because knowledge about good explanations is scattered.

3 METHODOLOGY

The survey systematically collected XAI literature, categorized methods across six dimensions, and evaluated explanation quality using the Co-12 properties and a structured review protocol. It distinguishes user-based from non-user evaluation and focuses on metrics assessing explanations rather than predictive performance or construction cost.

  • Method categorization: Each paper was categorized along six dimensions, including input data, explanation type, explanation problem, predictive model, task, and explanatory method.Categories are non-exclusive, so papers can occupy multiple categories within a dimension.
  • Method categorization: The review distinguishes post-hoc explanations, interpretability built into predictive models, and supervised explanation training.These categories describe whether explanation follows model training, is integrated into the model, or is learned from ground-truth explanations.
  • Evaluation protocol: The evaluation protocol records whether studies use users, whether evidence is anecdotal, and which qualitative or quantitative property-specific methods they apply.User studies are further classified as application-grounded with domain experts or human-grounded with lay participants.
  • Evaluation protocol: The review excludes task accuracy, prediction computation time, explanation construction overhead, and analyses that use explanations to inspect model properties rather than evaluate explanations.The scope centers on the explanation method and its produced explanations.
  • Co-12 properties: The Co-12 framework identifies twelve desired explanation properties as a non-binary basis for evaluating explanation quality.The properties aggregate concepts from prior literature while minimizing semantic overlap and include aspects such as correctness and other quality dimensions.

4 OVERALL STATISTICS OF INCLUDED PAPERS

The number of included XAI papers increased steadily from 2016 onward, with especially notable growth in 2018 and substantial increases in 2019 and 2020 for papers applying or evaluating existing methods. The review notes differing growth patterns across conferences.

  • Publication trends: The number of included XAI papers shows a steady increase since 2016, with a significant rise in 2018.The dataset does not show the exponential growth reported by some earlier work, although ACL and ICML exhibit exponential increases.
  • Publication trends: NeurIPS has the most papers on explainable AI and interpretable machine learning in the dataset, with striking increases in 2018 and 2020.AAAI also experienced a substantial increase in explainability papers over the period.
  • Evaluation trends: The number of papers applying or evaluating existing XAI methods increased substantially in 2019 and again in 2020.The authors suggest this may indicate growing awareness of XAI evaluation and comparison, and possibly increasing maturity.

5 STATISTICS ON XAI METHODS AND THEIR EVALUATION

Among papers introducing XAI methods, evaluation increasingly uses quantitative measures while anecdotal-only evaluation declines, but user studies remain near one-fifth of papers. Quantitative evaluations commonly cover only one or two Co-12 properties, especially Coherence and Output-completeness.

  • Overall evaluation practice: 33% of papers introducing XAI methods evaluated exclusively with anecdotal evidence, while 58% applied quantitative evaluation.The review analyzed 312 papers introducing an explanation method.
  • Overall evaluation practice: 22% of papers evaluated with human subjects in a user study, and 23% of those studies used domain experts.The remaining user studies were not application-grounded with domain experts according to the review’s classification.
  • Evaluation scope: Some papers supported interpretability claims with mathematical theory instead of explicitly applying quantitative evaluation metrics.Other papers argued that their model architecture was inherently interpretable and therefore did not explicitly evaluate interpretability.
  • Evaluation trends: The fraction of papers using quantitative evaluation slightly increased over time, whereas anecdotal-only evaluation decreased.The authors characterize this pattern as evidence that XAI evaluation practice is maturing.
  • Co-12 coverage: 73 papers quantitatively evaluated exactly one Co-12 property, often Coherence, and most quantitative studies evaluated one or two properties.Coherence and Output-completeness were evaluated most often, followed by Correctness, Compactness, and Covariate complexity.

6 QUANTITATIVE EVALUATION METHODS FOR XAI

The review organizes automated, quantitative XAI evaluation methods by Co-12 properties and recommends combining methods to obtain a multi-dimensional view of explainability.

  • The collection focuses on functionally grounded methods without user studies, while user-study methods are summarized separately.
  • The review identifies quantitative evaluation methods in 361 papers and groups them by the Co-12 properties.
  • Table 4 links evaluation methods to the Co-12 properties they can measure, without implying that every paper evaluated every linked property.
  • A thorough evaluation should select multiple methods covering as many Co-12 properties as possible, producing a multi-dimensional view of explainability.
  • Table 3 describes each automated evaluation method, its main explanation types, and the papers applying it.
  • Correctness: Model-parameter randomization checks whether explanations change after perturbing or reinitializing model parameters, but sensitivity is necessary rather than sufficient for correctness.

6.2 Functionally Evaluating Output-Completeness

Functionally evaluating output-completeness compares how explanations affect model outputs when features are deleted, preserved, or used to construct surrogate explanations.

  • Deletion and preservation checks: Incremental Deletion and Preservation Checks measure changes in model accuracy or confidence after important features are removed or retained.
  • Output-completeness asks whether an explanation contains enough information to explain the predictive model’s output.
  • Incremental deletion: Incremental Deletion evaluates correctness, output-completeness, and compactness by comparing feature-removal effects with random deletions.
  • Deletion and preservation checks: For the Deletion Check, accuracy should drop substantially, whereas accuracy after the Preservation Check should remain similar.
  • Fidelity: Fidelity measures agreement between model and explanation outputs using decision agreement, approximation error, divergence, entropy, or correlation.
  • Fidelity: Fidelity does not guarantee correctness because matching outputs do not show that the explanation follows the model’s reasoning.

6.3 Functionally Evaluating Consistency

Consistency evaluates whether explanations behave deterministically across identical inputs or across models with equivalent input-output behavior.

  • Consistency evaluates whether identical inputs receive identical explanations and can quantify how deterministic an explanation method is.
  • Implementation invariance: Implementation Invariance requires models with identical outputs for all inputs to produce the same explanations, regardless of internal implementation.
  • Implementation invariance: One evaluation computes Jaccard similarity between feature-importance scores across random model initializations.
  • Reference inputs: Reference-input analyses evaluate explanation sensitivity for DeepSHAP, including the use of a plain black image as a standard image-classification reference.

6.4 Functionally Evaluating Continuity

Continuity evaluates explanation behavior under small input changes and in counterfactual settings, while contrastivity evaluates whether explanations distinguish targets or model outputs.

  • Continuity: Continuity addresses explanation generalizability through similarity under slightly perturbed inputs or fidelity across nearby samples.
  • Continuity: Explanation similarity under perturbations can be measured with rank correlation, top-k intersection, cosine similarity, rule match, normalized distance, or SSIM.
  • Counterfactual continuity: Counterfactual connectedness measures whether generated counterfactuals lie near training samples and have a continuous path to them.
  • Counterfactual continuity: Connectedness is intended to avoid counterfactual explanations that are outliers or artifacts of non-robust models.
  • Contrastivity: Target Sensitivity evaluates whether explanations differ for different targets or outputs, with large explanation differences desired.
  • Contrastivity: Visual plausibility can favor class-insensitive saliency maps, including maps resembling edge detectors, so target sensitivity requires explicit checking.
  • Contrastivity: Target Discriminativeness tests whether another classifier or clustering method can recover the target from the explanation.
  • Contrastivity: Data Randomization Checks compare explanations from models trained on original versus randomized labels, and are model-agnostic sanity checks.

6.6 Functionally Evaluating Covariate Complexity

Covariate complexity evaluates whether explanation features use human-understandable concepts and represent them consistently. The review catalogs quantitative tests based on concept alignment, homogeneity, disentanglement, and regularity.

  • Covariate complexity concerns using human-understandable concepts to explain features.
  • Covariate Homogeneity measures how consistently a covariate represents a predefined human-interpretable concept.For images, concepts can include labeled object parts such as legs, beaks, and tails.
  • Concept alignment can be quantified with Intersection over Union, distance to interpretable concepts, predictive power, semantic-ground-truth distance, or cluster purity.
  • Disentanglement: Disentanglement evaluates whether each latent dimension corresponds to exactly one interpretable factor or attribute.Examples include object color and the presence of a smile; mutual information and controlled variation provide quantitative tests.
  • Disentanglement: Disentanglement tests vary one dimension while holding others fixed, then measure output variance, factor-index prediction accuracy, or residual variance.
  • Covariate Regularity: Covariate Regularity uses Shannon entropy of rule feature distributions or feature-importance scores to assess memorability or attribution noise.

6.7 Functionally Evaluating Compactness

Compactness evaluates how much information or change an explanation requires, including ordinary explanation size, redundancy, and counterfactual distance. The section also distinguishes user-context measures from purely input-based compactness.

  • Explanation compactness is commonly evaluated through absolute size or sparsity.Measures include feature count, decision-tree path length, reduction relative to the complete sample, and the number of decision rules.
  • Redundancy evaluates overlap within an explanation, with lower overlap indicating higher interpretability.Information gain and overlap ratio are reported measures.
  • Counterfactual Compactness: Counterfactual Compactness measures how much must change to obtain a different prediction.It can use input-to-counterfactual distance or the number of required transformations, such as changed features.
  • Counterfactual Compactness: A counterfactual distance of 0 implies zero contrastivity and target sensitivity because nothing changes for the different prediction.
  • Pragmatism: Pragmatism quantifies how difficult it is for an individual to act on counterfactual suggestions using domain knowledge.Unlike Counterfactual Compactness, it incorporates the user’s context and feature-change costs, which may be infinite.
  • User-oriented evaluation includes simulated studies of prediction trust, model selection, and bias identification, while only two papers explicitly evaluated confidence information.

6.11 Functionally Evaluating Coherence

Coherence measures whether explanations align with domain knowledge, expected explanations, or user interests, but such alignment does not establish correctness. The review therefore recommends evaluating coherence and correctness independently alongside other properties.

  • Alignment with Domain Knowledge: Alignment with Domain Knowledge compares explanations with annotated ground truth, such as bounding boxes, segmentation masks, landmarks, or human attention maps.
  • Alignment with Domain Knowledge: Image explanation alignment can be quantified with Intersection over Union, relevance ratios, localization error, pointing-game accuracy, or rank correlation.
  • Alignment with Domain Knowledge: Textual explanation alignment commonly uses BLEU and ROUGE against a ground-truth explanation, while real-valued explanations can use rank correlation.
  • Kendall’s tau can mislead at distribution tails, and Spearman correlation has limitations for global rankings.
  • XAI Methods Agreement: XAI Methods Agreement compares explanations with established methods when ground truth is unavailable.
  • Coherence reflects agreement with expectations rather than correctness with respect to predictive model f, so the two properties must be evaluated independently.A coherent explanation can be incorrect, and an explanation highlighting snow may correctly expose a bad classifier’s reasoning despite low location coherence.

7 IMPLICATIONS AND RESEARCH OPPORTUNITIES

The review frames explainability as multi-faceted and argues for comprehensive, quantitative evaluation using the Co-12 properties. Its catalog supports standardized assessment, multidimensional optimization, and further research into under-addressed properties and trade-offs.

  • The Co-12 framework treats explainability as a multi-faceted concept whose properties should preferably be quantified for informed trade-offs.
  • The collected methods provide concrete tools for evaluating each Co-12 property with unified terminology, contributing to standardization and comparison.
  • Quantitative evaluation methods exist for every Co-12 property, although evaluation has focused mainly on Coherence, Completeness, Compactness, and Correctness.
  • The review identifies research opportunities to develop methods for insufficiently addressed properties and adapt existing methods to different data and explanation types.
  • A method need not score well on every Co-12 property because trade-offs can arise between Coherence and Correctness or between Completeness and Compactness.
  • Co-12 properties can support multidimensional interpretability optimization, but current interpretability optimizers usually involve only one to two properties.
  • Evaluation methods could be incorporated into predictive-model training to tune the accuracy-interpretability trade-off rather than analyzing it only afterward.

SUPPLEMENTARY MATERIAL

The supplementary analysis catalogs XAI methods, explanation types, model tasks, and evaluation practices across a structured corpus. It finds strong concentration in outcome explanations, feature-importance methods, neural-network applications, and anecdotal evaluation.

  • Explanation types: Feature importance methods dominate the explanation types, with standard feature-importance scores used by 27% of papers.Feature importance is used across image, text, tabular, and other data types; heatmaps are especially associated with image data and textual explanations with textual input.
  • Predictive tasks: Classification is the main predictive-model task, representing 63% of papers, although the authors caution that venue selection and task difficulty may influence these statistics.The concentration of classification tasks is associated with the use of outcome explanations for particular classification decisions.
  • Explanation scope: Outcome explanations address 64% of papers, while model inspection accounts for 30% and provides global explanations about predictive-model properties.Outcome explanations focus on individual predictions, whereas model inspection includes global feature importance.
  • XAI method types: Post-hoc and built-in interpretability are the dominant method types, while 15% of papers introduce model-agnostic XAI methods.Built-in approaches include intrinsic interpretability, architectural restrictions, attention mechanisms, regularizers, and combined prediction-explanation architectures; supervised explanation-training methods produce outcome explanations.
  • Evaluation practices: One in three papers evaluate exclusively with anecdotal evidence, and one in five use a user study.Application-oriented venues use anecdotal evidence in roughly half of cases, compared with around 20%-30% at more theoretical venues; user-study usage shows no clear trend.
  • Paper collection: 361 papers were included after applying the inclusion criteria to an initial set of 494 papers.Two reviewers agreed on 68 of 81 inclusion decisions, with Cohen’s kappa κ=0.625 and Matthews correlation coefficient MCC=0.647.
Loading 2201.08164v3…