PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking and Survey of Explanation Methods for Black Box Models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, Salvatore Rinzivillo

arXiv:2102.13076v1cs.AIcs.CYcs.LG

TL;DR

Black-box models achieve strong predictive performance while obscuring how decisions are reached, motivating explanation methods for examining potential biases and practical or ethical issues. This paper categorizes explainers by explanation type and data format, compares representative methods qualitatively and quantitatively, and identifies strategy trends across tabular, image, and text data.

  • Problem

    Black-box models can hide the logic of their internal processes despite their widespread use and high accuracy, motivating methods that reveal how they reach decisions.

  • Method

    The paper surveys recent explainers, organizes them by explanation nature across tabular data, images, and text, and benchmarks faithfulness, stability, robustness, and running time.

  • Results

    Feature importance dominates tabular explanations, saliency maps dominate image explanations, and stochastic synthetic-neighborhood methods show low stability.

  • Takeaways & Limitations

    Rule-based explanations can provide deeper understanding of model decisions, while counterfactual approaches and saliency maps offer prominent alternatives across data types.

  • Takeaways & Limitations

    Text explanations remain at an early stage, with many methods assigning word-level scores that are useful for robustness checks but not for inexpert users.

Abstract

from arXiv · show

The widespread adoption of black-box models in Artificial Intelligence has enhanced the need for explanation methods to reveal how these obscure models reach specific decisions. Retrieving explanations is fundamental to unveil possible biases and to resolve practical or ethical issues. Nowadays, the literature is full of methods with different explanations. We provide a categorization of explanation methods based on the type of explanation returned. We present the most recent and widely used explainers, and we show a visual comparison among explanations and a quantitative benchmarking.

1 Introduction

Black-box models deliver strong performance but obscure their internal logic, creating concerns about bias, ethics, safety, and trust. This survey categorizes explainers, compares their outputs, and benchmarks them across key evaluation criteria.

  • Black-box models, including deep learning and ensemble models, can hide logic and potential issues inherited from biased or unfair training data.
  • XAI methods aim to make model internals and outcomes accessible and understandable to humans.
  • The survey categorizes explanations by their nature across tabular data, images, and text.
  • It presents recent explainers alongside qualitative and quantitative comparisons of faithfulness, stability, robustness, and running time.

2 Related Works

Related work offers taxonomies based on problem type, model properties, and responsible AI concepts. This survey instead organizes explainers from a data-type and returned-explanation perspective, addressing a stated gap in systematic comparison.

  • The literature includes books and surveys covering general machine-learning interpretability and explanations for deep neural networks.
  • Earlier surveys distinguish intrinsic from post-hoc interpretability and classify black-box explanations as model, outcome, or inspection explanations.
  • Related taxonomies define explanation, interpretability, and explainability while linking explainability to responsible AI, fairness, and accountability.
  • This survey rewrites an earlier taxonomy from a data-type perspective and identifies systematic categorization and explanation comparison as missing.

3 Explanation-Based Categorization of Explainers and Evaluation Measures

The paper categorizes explainers by returned explanation, data type, intrinsic or post-hoc status, locality, and model dependence. It evaluates explanations qualitatively and quantitatively using model- and task-oriented criteria.

  • The proposed guide maps black-box models to compatible explanation methods and evaluation measures.
  • The survey covers explanations for tabular data, images, and text, grouping methods by explanation type in its tables.
  • Categorization: Explainable-by-design methods expose decision reasons through transparent models, whereas post-hoc methods explain non-interpretable models after decisions are made.
  • Categorization: Global explainers target overall model logic, while local explainers target the reasons for an individual decision.
  • Categorization: Model-agnostic methods apply to any black-box type, whereas model-specific methods apply only to a specific model type.
  • Evaluation Measures: Qualitative evaluation considers usability and explanation properties, including form, element count, ordering, monotonicity, uncertainty, and stochasticity.
  • Evaluation Measures: Functionally grounded metrics use formal proxies, application-grounded metrics rely on task experts, and human-grounded metrics assess understandability with non-experts.
  • Evaluation Measures: Quantitative evaluation measures how closely an explainer approximates the black box or satisfies a task-specific criterion.

4 Explanations for Tabular Data

The paper organizes tabular-data explainers by returned explanation type and illustrates feature-importance, rule, prototype, and counterfactual approaches across adult and german datasets. It compares explanations qualitatively and evaluates selected methods using fidelity and stability.

  • Tabular explainers are grouped into Feature Importance, Rules, Prototype, and Counterfactual explanation types, then applied to adult and german datasets with LG, XGB, and CAT models.
  • Feature Importance: Feature-importance explanations assign each feature a signed value whose magnitude represents its contribution to the black-box prediction.A zero value indicates no contribution, while positive and negative signs indicate contribution direction.
  • Feature Importance: LIME samples records around the explained instance, whereas SHAP computes additive feature-attribution values using model-specific explainers.The experiments use LinearExplainer for LG, TreeExplainer for XGB, and KernelExplainer for CAT.
  • Feature Importance: On adult, LIME identified similar important features across LG and CAT, while on german the models selected different features despite both predicting correctly.The paper reports that CAT had higher prediction confidence on german, which may relate to the explanation differences.
  • Feature Importance: SHAP force plots show features pushing predictions above or below a base value, while decision plots include all input features and interaction values across records.In one adult example, the base value was 0.18 and the model output was 0.79; another output was 0.0.
  • Feature Importance: DALEX decomposes predictions into feature contributions and provides local and global exploratory tools, while EBM exposes global feature functions and local explanations.DALEX plots encode features on the y-axis and positive or negative contribution on the x-axis; EBM illustrates education-number effects and records from both classes.

4.2 Rule-based Explanation

Rule-based explanations represent predictions through human-readable conditions, either approximating black-box behavior globally or explaining individual decisions locally. The section covers rule extraction, filtering, visualization, and class-specific examples.

  • Rule definition: Decision rules use conjunctive feature conditions as premises that imply a prediction, and a rule covering an instance can serve as a local explanation.A locally faithful interpretable predictor supports treating the rule as an explanation of the black-box decision.
  • Local rule explainers: ANCHOR generates model-agnostic rules by perturbing an instance, then uses bandit sampling and search to find anchors exceeding a user-defined precision threshold.Anchors are intended to preserve the outcome when other feature values change.
  • Examples: Figure 7 presents SkopeRules explanations for XGB on adult, with one rule for class > 50k and one for class < 50k.The reported rules are selected by highest precision and recall for each class.
  • Global rule explainers: Global rule explainers include decision-tree approximations and clustering-based summaries for neural networks, random forests, and tree ensembles.TREPAN maximizes gain ratio and fidelity, while MSFT clusters similar forest trees and retrieves an archetype per cluster.
  • Global and local rule explainers: SkopeRules extracts rules from bagging or gradient-boosted trees, removes redundant rules, and weights the remainder by precision rather than L1 regularization.It can describe the input dataset or output rules used for prediction.

4.3 Prototypes

Prototype-based explanations use representative records to make dataset structure or model reasoning concrete. Methods differ in how they select prototypes, identify criticisms, weight examples, or adapt prototypes to a model's representation.

  • Prototype concept: A prototype is a record, cluster centroid, or synthetic example representing similar records and serving as an explanatory example.Users interpret model reasoning by examining records similar to the case under analysis.
  • Dataset-based methods: MMD-CRITIC selects prototypes near the dataset distribution and diverse criticisms from underrepresented regions, without analyzing the model.Prototypes summarize general dataset behavior, whereas criticisms identify points poorly represented by them.
  • Dataset-based methods: ProtoDash extends MMD-CRITIC by assigning non-negative importance weights to prototypes, allowing it to reflect complicated data structures.
  • Privacy-preserving methods: Privacy-Preserving Explanations produces privacy-protected prototypes and shallow trees using micro-aggregation to balance privacy and comprehensibility.
  • Interpretable prototype models: Prototype Selection uses set-cover optimization to represent the data, assigns each record to a prototype, and trains a nearest-neighbor rule classifier.
  • Model-specific methods: Tree Space Prototype finds class-specific prototypes in an ensemble's tree space using proximity measures, with variants selecting different numbers per class.

4.4 Counterfactuals

Counterfactual explanations describe minimal changes that would produce an opposite prediction, connecting a decision to altered feature values. The surveyed methods differ in how they generate, constrain, diversify, and make counterfactuals feasible.

  • Counterfactual concept: Counterfactuals specify feature changes under which the black-box would return the opposite prediction while other variables remain constant.An ideal counterfactual changes values as little as possible to reach the alternative outcome.
  • Counterfactual categories: Counterfactual explainers are divided into exogenous methods that generate examples synthetically and endogenous methods that derive them from existing data.The passage introduces these categories as distinctions among counterfactual-generation approaches.
  • Contrastive explanations: CEM produces contrastive explanations through Pertinent Positives, which are minimally sufficient factors, and Pertinent Negatives, which are minimally absent factors.
  • Feasibility and diversity: DICE optimizes counterfactuals under proximity, sparsity, and user-defined constraints to ensure feasibility and diversity.Diversity provides different ways to change the outcome class.
  • Feasibility and diversity: FACE returns achievable counterfactuals by finding shortest feasible paths through a graph whose distances use density-weighted metrics.The generated counterfactuals are coherent with the input data distribution.
  • Model-specific methods: CFX identifies pivotal variables in Bayesian Network Classifiers whose removal would produce a different classification.

4.5 Transparent methods

Transparent methods expose interpretable behavior directly through feature contributions or rules rather than explaining an opaque model after the fact. The section includes additive models, correlated explanation outputs, rule learners, and benchmark metrics for explanation quality.

  • Overview: Transparent tabular methods are organized into models that output feature importance and methods that output rules.
  • Feature-importance methods: EBM is a generalized additive model variant that learns nonlinear feature functions through round-robin boosting, making each feature's contribution inspectable.On adult, Maritial Status, Relationship, and Age are reported as the most important features in that order.
  • Feature-importance methods: TED requires training data pairing each record with an explanation and can use any multilabel-capable ML model to associate records with explanations.Its explanations can be rules or feature importance.
  • Rule learners: SLIPPER, LRI, MlRules, RuleFit, and IDS generate interpretable rules using constrained boosting, adaptive induction, probability estimation, tree paths with L1 weighting, or independent decision sets.
  • Evaluation: Table 3 compares explanation methods using fidelity and faithfulness, reporting means and standard deviations over 50 test records.
  • Evaluation: Table 4 compares stability using means and standard deviations over 30 test records.

4.6 Quantitative Comparison

The quantitative comparison evaluates tabular explainers on fidelity, faithfulness, stability, and runtime. Fidelity is generally high, while runtime differs systematically between feature-importance and rule-based methods.

  • Fidelity and faithfulness: Fidelity values are relatively high across LIME, SHAP, Anchor, and LORE, indicating that the local surrogate models mimic their black-box models well.SHAP has lower fidelity for CAT models on German and Adult, while Anchor also has lower fidelity for German CAT.
  • Fidelity and faithfulness: LIME shows higher fidelity than SHAP, especially on Adult, whereas SHAP has lower fidelity for both CAT models.The comparison suggests differences in fidelity across explainer and black-box-model combinations.
  • Fidelity and faithfulness: No model was monotonic overall, so the authors do not report monotonicity results.The comparison also reports relatively low faithfulness for Adult, especially for LIME.
  • Runtime: Feature-importance explainers are faster than rule-based explainers overall, with SHAP fastest and LIME next.LORE runtime depends on the number of genetically generated neighbors, while Anchor and Skoperule require a minimum precision setting.

Runtime Analysis

Tabular explanation methods commonly use feature importance values, while rule-based explanations provide a more logical structure. The passage contrasts their audiences and notes that feature-importance explanations may be difficult for common end-users.

  • Feature-importance explanations: Feature-importance explainers such as LIME and SHAP assign an importance value to each input feature.The passage describes feature-importance explanations as suitable for domain experts who understand the employed features.
  • Feature-importance explanations: Feature-importance explanations may be difficult for common end-users to understand, especially when obtaining the importance values is complex.
  • Rule-based explanations: Rule-based explanations are presented as an alternative to feature-importance explanations for communicating model decisions.The supplied passage begins this contrast but does not provide the subsequent details about rule-based methods.

Discussion

The discussion notes an evaluation caveat for SHAP, summarizes tabular runtime reporting, and organizes image explainers by explanation type and model properties. It also identifies efficiency, accuracy, and domain constraints as open challenges.

  • Evaluation caveat: SHAP fidelity is evaluated by training a classifier on the sum of SHAP values because SHAP does not train a local surrogate.
  • Runtime reporting: Tabular-explainer runtime is reported in seconds as an order-of-magnitude comparison across datasets and black-box explainers.
  • Image explainers: Image explainers are grouped into Saliency Maps, Concept Attributions, Counterfactuals, and Prototypes, with annotations for data type, intrinsic versus post-hoc status, locality, and model dependence.
  • Open challenges: Open challenges include improving explanation efficiency and accuracy and incorporating domain constraints.The discussion also notes that post-hoc rule and prototype methods generally require more time than feature-importance methods, while some prototype approaches lack code.

5 Explanations for Image Data

For image data, the paper categorizes explainers into saliency maps, concept attribution, prototypes, and counterfactuals, then compares their explanations and performance across datasets. Saliency-map methods differ in mechanism, visual output, runtime, and agreement about which image regions support predictions.

  • Explanation types: Image explainers are organized into Saliency Maps, Concept Attribution, Prototypes, and Counterfactuals.The experiments use MNIST, CIFAR-10, and ImageNet with CNNs for MNIST and CIFAR and VGG16 for ImageNet.
  • Saliency Maps: A saliency map represents each pixel’s saliency as brightness or color, with larger values indicating greater salience and red or blue indicating positive or negative contribution.The map is modeled as a matrix whose dimensions match the image.
  • Saliency Maps: Saliency-map methods use distinct mechanisms: LIME segments images, ϵ-LRP redistributes relevance backward, Grad-CAM uses gradients in the last convolutional layer, and RISE aggregates masked-image predictions.These mechanisms produce local post-hoc explanations, with Grad-CAM being model-specific and RISE model-agnostic.
  • Saliency Maps: Integrated gradients can produce more uniform pixels than ϵ-LRP, but arbitrary baselines can distort feature importance, such as reducing the importance of black pixels with a black baseline.The baseline choice may require domain expertise for related methods such as DeepLIFT.
  • Comparison: Qualitative comparisons reveal conflicting regions across explainers: for an ImageNet hockey image, LIME highlights ice, XRAI and Grad-CAM++ the stick, Grad-CAM the fans, and RISE the player.For another ImageNet image, all methods capture the same pattern, while LIME performs poorly on small images and XRAI produces clearer regions.
  • Quantitative and runtime comparison: RISE is best in three of five insertion or deletion experiments, followed by DeepLIFT and ϵ-LRP, while segmentation-based methods struggle on low-resolution images.Grad-CAM and Grad-CAM++ are the fastest methods, whereas segmentation slows substantially for high-resolution images and prototype, counterfactual, and concept-attribution methods are much slower than saliency maps.
  • Discussion: Saliency maps are widely used but expose confirmation bias and describe explanations in pixel-level terms that are mainly useful to expert users assessing black-box robustness.The paper therefore considers multiple explanation types and evaluation dimensions rather than relying on saliency maps alone.

6 Text

The text section organizes explainers by explanation type and surveys saliency, attention-based, and other methods across NLP tasks. It compares their visual forms and evaluates faithfulness, stability, robustness, and runtime.

  • Overview: Text explainers are grouped into Saliency Maps, Attention-Based methods, and Other Methods.The section focuses especially on text classification because NLP tasks are diverse and complex.
  • Saliency Maps: Sentence Highlighting assigns each word a score representing its positive or negative contribution to a classification.It adapts saliency-map explanations to text and can be visualized through highlighted words.
  • Saliency Maps: LIME, Integrated Gradients, DeepLIFT, and L2X adapt existing explainers to text, using word perturbations, gradients, or word groups.LIME replaces words with spaces or similar words; L2X treats groups of words as patches.
  • Saliency Maps: Integrated Gradients and LIME produced more meaningful highlighted-word explanations, while DeepLIFT remained close to the Gradient × Input baseline.The comparison also reports deletion and insertion metrics for Sentence Highlighting.
  • Attention-Based Methods: Attention-based explanations reuse model attention weights as word scores or represent dependencies in an N × N attention matrix.Attention matrices connect each word to every other word, while BERT visualizations encode pairwise attention through line thickness.
  • Runtime Analysis: Attention Matrix methods are instant, whereas other NLP explainers generally require about ten seconds, independently of dataset size.NLP models are large, which contributes to poor runtime performance.
  • Discussion: Text explanations remain at an early stage and mostly provide low-feature word scores that help assess model robustness more than serve inexpert end users.Other approaches include anchors, natural-language explanations, exemplars, counterexamples, and shapelet-based rules.

7 Explanation Toolboxes

The paper reviews popular Python toolkits for machine-learning explanation and summarizes the explainer families they support. AIX360 spans multiple explanation settings, while CaptumAI targets PyTorch models.

  • Toolboxes: The paper surveys popular Python explanation toolkits and briefly describes the explanation models each provides.The toolboxes are presented as practical resources for machine-learning explanation.
  • AIX360: AIX360 supports intrinsic, post-hoc, local, and global explainers across all input-dataset types.Its local post-hoc methods include LIME, SHAP, CEM, CEM-MAF, and ProtoDash; TED provides intrinsic local and rule-based global explanations.

8 Conclusion

The paper surveys XAI methods by data type and explanation strategy and benchmarks them qualitatively and quantitatively. It identifies strategy trends while noting limited comparisons and unresolved human-grounded evaluation.

  • Conclusion: The survey categorizes recent XAI methods according to data types and explanation strategies, then evaluates selected techniques through qualitative and quantitative benchmarks.The benchmarks are intended to support comparison across explanation techniques.
  • Conclusion: Feature importance dominates tabular explanations, saliency maps dominate image explanations, and text explanations remain comparatively limited.Rule-based, counterfactual, concept-attribution, and prototype approaches are also described as emerging or gaining attention in relevant domains.
  • Conclusion: Synthetic-neighborhood generation is a recurring strategy, but the paper links it to low stability in the reported benchmarks.Surrogate models learned from partial or neighborhood-generated data aim to bring intrinsic-method benefits to black-box explanation.
  • Conclusion: Comparing explanation methods remains difficult because few studies compare them and no unifying efficacy metric is established, especially for human-grounded evaluation.The paper points toward greater attention to human–machine interaction and alignment with users’ cognitive models.
Loading 2102.13076v1…