PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenXAI: Towards a Transparent Evaluation of Model Explanations

Chirag Agarwal, Dan Ley, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, Himabindu Lakkaraju

arXiv:2206.11104v5cs.LGcs.AI

TL;DR

Post hoc explanation methods lack systematic benchmarking, despite their use in settings where explanation reliability matters. OpenXAI addresses this gap with an extensible framework combining datasets, models, methods, synthetic ground truths, and 11 metrics; it supports broad, transparent, reproducible comparisons and reports Integrated Gradients as a strong performer on selected stability metrics.

  • Problem

    Systematic evidence comparing post hoc explanations across faithfulness, stability, fairness, methods, models, and datasets remains limited.

  • Method

    OpenXAI combines six feature attribution methods, 11 reliability metrics, diverse datasets and models, and SynthGauss synthetic data with reliable ground-truth explanations in an extensible benchmarking pipeline.

  • Results

    OpenXAI rigorously benchmarks six feature attribution methods across eight datasets, 16 predictive models, and 11 metrics; Integrated Gradients averages 1.95 RIS and 1.86 ROS log scores across datasets for logistic regression.

  • Takeaways & Limitations

    The framework unifies and standardizes evaluation workflows, enabling systematic benchmarking of existing and new explanation methods through public APIs and leaderboards.

Abstract

from arXiv · show

While several types of post hoc explanation methods have been proposed in recent literature, there is very little work on systematically benchmarking these methods. Here, we introduce OpenXAI, a comprehensive and extensible open-source framework for evaluating and benchmarking post hoc explanation methods. OpenXAI comprises of the following key components: (i) a flexible synthetic data generator and a collection of diverse real-world datasets, pre-trained models, and state-of-the-art feature attribution methods, and (ii) open-source implementations of eleven quantitative metrics for evaluating faithfulness, stability (robustness), and fairness of explanation methods, in turn providing comparisons of several explanation methods across a wide variety of metrics, models, and datasets. OpenXAI is easily extensible, as users can readily evaluate custom explanation methods and incorporate them into our leaderboards. Overall, OpenXAI provides an automated end-to-end pipeline that not only simplifies and standardizes the evaluation of post hoc explanation methods, but also promotes transparency and reproducibility in benchmarking these methods. While the first release of OpenXAI supports only tabular datasets, the explanation methods and metrics that we consider are general enough to be applicable to other data modalities. OpenXAI datasets and models, implementations of state-of-the-art explanation methods and evaluation metrics, are publicly available at this GitHub link.

1 Introduction

OpenXAI addresses the limited systematic evidence on post hoc explanation reliability by providing an open-source framework for transparent, reproducible benchmarking across metrics, methods, models, and datasets.

  • Motivation: Post hoc explanations are increasingly used in high-stakes domains, making their reliability important for deciding when to rely on model predictions.Prior work studies faithfulness, stability, and fairness, but broader generalization remains unclear.
  • Framework: OpenXAI unifies datasets, models, explanation methods, and evaluation metrics through an accessible framework for systematic and reproducible benchmarking.The framework is designed as an end-to-end ecosystem with APIs and public benchmarking infrastructure.
  • Related work: Compared with prior tools, OpenXAI extends beyond method libraries or narrow SHAP and synthetic evaluations by including broad stability and fairness benchmarking.Captum emphasizes implementations, Quantus emphasizes selected metrics, SHAP Benchmark focuses on SHAP faithfulness, and XAI-Bench has limited synthetic-data scope.
  • Metrics: OpenXAI provides 11 quantitative metrics covering faithfulness, stability, and fairness, including seven metrics not previously implemented in related libraries or benchmarks.The newly implemented metrics include FA, RA, SA, SRA, PRA, RRS, and ROS.
  • Synthetic data: Its synthetic data generator creates datasets with independent features, unambiguous local neighborhoods, and clearly specified feature influence to support reliable ground-truth explanations.These properties theoretically guarantee that accurate models trained on the generated data adhere to the underlying ground-truth explanations.
  • Empirical benchmarking: OpenXAI benchmarks six feature attribution methods across 11 metrics, eight real-world or synthetic datasets, and 16 predictive models.The benchmarking evaluates which methods are effective with respect to different reliability notions across diverse settings.

2 Overview of OpenXAI Framework

OpenXAI provides an end-to-end environment for benchmarking post hoc explanations using synthetic and real-world datasets, predictive models, explainers, and quantitative evaluation metrics.

  • Datasets and Predictive Models: OpenXAI combines synthetic and real-world datasets to support both ground-truth evaluation and realistic benchmarking of explanation methods.Synthetic data enables constructed ground-truth explanations, while real-world data broadens practical evaluation.
  • Synthetic Datasets: SynthGauss generates clustered data with independent features, unambiguous local neighborhoods, and clearly specified feature influence for reliable ground-truth explanations.Clusters are separated by setting intracluster distances below intercluster distances, while identity covariance matrices ensure feature independence.
  • Real-world Datasets: OpenXAI includes seven diverse real-world datasets spanning finance, lending, healthcare, criminal justice, dataset sizes, dimensionalities, class imbalance, and feature types.The release focuses on tabular data and provides data loading and preprocessing capabilities.
  • Dataloaders: Its dataloaders load existing or custom datasets into train and test sets for model training and local explanation generation.Users can customize train-test splits when predetermined splits are unavailable.
  • Explainers: The framework provides six feature-attribution explainers, a random baseline, and interfaces for generating explanations and incorporating custom methods.Included explainers are LIME, SHAP, Vanilla Gradients, Gradient x Input, SmoothGrad, and Integrated Gradients.
  • Evaluation Metrics: OpenXAI implements twenty-two quantitative metrics covering faithfulness, stability, and fairness, including ground-truth similarity and predictive faithfulness measures.Ground-truth faithfulness metrics compare selected features between post hoc and ground-truth explanations, while PGI measures prediction changes after perturbing influential features.

3 Benchmarking Analysis

OpenXAI benchmarks explanation methods across faithfulness, stability, and fairness using standardized datasets, models, metrics, and evaluation procedures. The reported results reveal distinct method strengths and subgroup disparities.

  • Experimental Setup: OpenXAI benchmarks six feature attribution methods and a random baseline on the first 1000 test instances of each dataset.The experiments use the openxai.Evaluator module and include logistic regression and neural network models.
  • Faithfulness: Vanilla Gradients, Integrated Gradients, and SmoothGrad achieve perfect scores on six ground-truth faithfulness metrics across all datasets.The metrics are PRA, RC, FA, RA, SA, and SRA; LIME approaches their performance in many cases.
  • Faithfulness: Vanilla Gradients, SmoothGrad, and LIME are most faithful on average across datasets and models under the PGI metric.Integrated Gradients is on par with the leading methods under PGU.
  • Stability: Integrated Gradients tends to outperform other attribution methods on stability, with log-average RIS and ROS scores of 1.95 and 1.86.Vanilla Gradients and Gradient x Input score 3.16 and 3.37 for RIS, and 2.91 and 4.49 for ROS; LIME, SHAP, SmoothGrad, and Random score roughly 10–15.
  • Fairness: Fairness analysis finds subgroup disparities in explanation faithfulness, especially for Gradient x Input and SHAP on Adult Income.Larger gaps between subgroup metric values indicate higher disparities and are undesirable.

4 Conclusion

OpenXAI provides an open-source ecosystem for evaluating post hoc explanations in high-stakes applications. It supports benchmarking new methods and aims to promote transparency, collaboration, and continued development.

  • Conclusion: OpenXAI combines datasets, explanation methods, evaluation metrics, leaderboards, and documentation in an extensible open-source ecosystem.The framework can incorporate new explanation methods, datasets, and metrics.
  • Conclusion: Systematic and efficient benchmarking with OpenXAI can inform and accelerate research on post hoc explanations.The framework is intended to support transparency and collaboration around evaluations.

A Evaluation Metrics

OpenXAI evaluates post hoc explanations through faithfulness, stability, and fairness metrics, covering both agreement with ground truth and sensitivity to perturbations. It standardizes metric computation across explanation methods and datasets.

  • Evaluation Metrics: OpenXAI evaluates explanation reliability through faithfulness, stability, and fairness metrics.Faithfulness concerns agreement with model behavior or ground truth; stability concerns changes under perturbations; fairness concerns subgroup disparities.
  • Faithfulness: Ground-truth faithfulness compares generated explanations with logistic-regression coefficients using Feature, Rank, Sign, Signed Rank, and Rank Correlation Agreement metrics.These metrics assess shared features, ranking, attribution direction, and rank correlation.
  • Faithfulness: Predictive faithfulness uses PGI and PGU to measure prediction changes after perturbing important or unimportant features, respectively.PGI perturbs top-k features while holding others fixed; PGU perturbs non-top-k features while preserving the top-k features.
  • Stability: Stability is measured with RIS, RRS, and ROS, which relate explanation changes to changes in inputs, internal representations, and output probabilities.RRS uses the first layer of a two-layer neural network in the experiments.
  • Fairness: Fairness compares averaged faithfulness or stability metrics across majority and minority subgroups, with larger gaps indicating greater disparities.The reported fairness analysis includes PGU and presents subgroup comparisons for logistic-regression and neural-network models.

C Synthetic Dataset

OpenXAI’s synthetic data generator creates clustered datasets with controllable feature dependence, local neighborhoods, and ground-truth feature influences. Under specified structural assumptions, the most accurate model adheres to each neighborhood’s unique ground-truth explanation.

  • Synthetic Dataset: The generator controls cluster membership, feature dependencies, and ground-truth explanation sparsity through parameterized data-generation components.Masking vectors determine influential features, while cluster centers and covariance settings control local structure.
  • Data Generation: Synthetic instances are sampled from cluster-specific Gaussian distributions, then labels are formed from model probabilities and balanced using their median.The procedure randomly selects a cluster, samples x_i from N(μ_k, Σ_k), and assigns binary labels relative to the median probability.
  • Theoretical Guarantee: Theorem 1 states that a maximally accurate model trained under these assumptions adheres to every neighborhood’s unique ground-truth explanation.The proof uses contradiction by replacing a model whose top-T features do not match the ground truth.
  • Assumptions: The synthetic construction assumes feature independence and unambiguous, well-separated local neighborhoods.Distances between clusters should greatly exceed within-cluster distances, while features remain independent.
  • Assumptions: Each local neighborhood has a unique ground-truth explanation generated from a feature subset with a clear relative importance ordering.The influential subset and its weight vector define the neighborhood’s ground-truth attribution structure.

D.1 Real-World Datasets

OpenXAI includes seven real-world benchmark datasets, primarily from high-stakes domains, alongside configurable data-loading and train-test splitting. The collection spans credit, income, criminal-justice, diabetes, and cardiovascular prediction tasks.

  • Dataset Collection: OpenXAI includes seven real-world benchmark datasets and supports multiple, customizable train-test split strategies.Predetermined splits can be loaded when supplied by a dataset.
  • Credit and Finance: German Credit contains 1,000 applicants with demographic, personal, and financial features for good-versus-bad credit-risk classification.The target categorizes applicants according to credit risk.
  • Credit and Finance: HELOC contains 9,871 homeowners’ anonymized financial attributes for predicting repayment within two years.The task uses credit-report information to predict whether an applicant will repay the account.
  • Income: Adult Income predicts whether 45,222 individuals earn more than $50K annually using demographic, education, employment, personal, and financial features.The dataset includes variables such as age, race, gender, degree, occupation, and capital gain or loss.
  • Criminal Justice: COMPAS classifies 6,172 defendants according to whether they are predicted likely to commit a violent crime if released.The records cover defendants released on bail at U.S. state courts during 1990–2009.
  • Health and Financial Risk: Give Me Some Credit predicts two-year financial distress risk for 102,209 individuals using demographic, personal, and financial features; Pima-Indians Diabetes and Framingham Heart Study are also included.The collection therefore covers additional health-related prediction tasks.

D.2 Explanation Methods

OpenXAI benchmarks six feature-attribution methods across configurable tabular-data settings, two pretrained model families, and reproducible default evaluation parameters. The implementation covers perturbation-based and gradient-based explainers with differing training and white-box requirements.

  • Explanation Methods: OpenXAI implements Vanilla Gradients, Integrated Gradients, SmoothGrad, Gradient x Input, LIME, and SHAP.A random explanation baseline is also included in the library.
  • Method Properties: The methods differ in whether they require model-internal gradient access and whether they learn a local approximation model.These axes are summarized in Table 6.
  • Implementations: The implementations build on Captum classes for the gradient-based methods and SHAP, while LIME uses the authors’ Python package.The listed Captum classes include Saliency, IntegratedGradients, NoiseTunnel, InputXGradient, and KernelShap.
  • Hyperparameters: OpenXAI uses author-guided default hyperparameters and fixes the explanation-method seed at 0 for reproducibility.LIME, SHAP, SmoothGrad, and Integrated Gradients each have explicit parameter dictionaries.
  • Data Parameters: The synthetic-data defaults use 1,000 samples, 20 dimensions, 10 clusters, 0.25 sparsity, and a 0.25 test split.The Gaussian covariance defaults to the identity when sigma is None.
  • Evaluation Parameters: Faithfulness evaluations use σ=0.1 with 100 perturbations, whereas stability uses σ=10^-5 and selects 100 prediction-preserving samples from 1,000 candidates.All stability metrics use the p=2 norm.
  • Models: The release provides logistic-regression and deep-neural-network models with supplied weights for all eight datasets in the pipeline.The neural networks use two fully connected hidden layers with 100 nodes each, ReLU activations, and a softmax output.

D.5 Results for Pima-Indians Diabetes and Framingham Heart Study datasets

This section adds healthcare datasets to the benchmark and reports faithfulness and stability results across explanation methods and models.

  • The benchmark includes Pima Indians Diabetes and Framingham Heart Study datasets from healthcare applications.
  • LR models: Tables 8 and 9 report ground-truth and predictive faithfulness on Pima Indians Diabetes and Framingham Heart datasets with LR models.Both tables summarize averages and standard errors across 1000 test instances.
  • ANN models: Tables 10 and 11 report predictive faithfulness on Pima Indians Diabetes and Framingham Heart datasets with ANN models.The tables report average and standard error metric values across 1000 test instances.

D.6 Remaining Results on LR models

This section reports remaining LR-model results for ground-truth and predictive faithfulness, plus stability, across synthetic and real-world datasets.

  • Faithfulness: Table 19 reports ground-truth and predictive faithfulness results on the German Credit dataset with an LR model.
  • Stability: Tables 20–23 report stability results for HELOC, Adult Income, COMPAS, and GMSC datasets with LR models.

D.7 Remaining Results on ANN models

This section reports ANN-model faithfulness, stability, and fairness evaluations across synthetic and real-world datasets.

  • Faithfulness: Tables 24–29 report predictive faithfulness results for synthetic, HELOC, German Credit, Adult Income, COMPAS, and GMSC datasets with ANN models.The tables report average and standard-error metric values across 1000 test instances.
  • Stability: Tables 30–35 report stability results for synthetic, HELOC, German Credit, Adult Income, COMPAS, and GMSC datasets with ANN models.The tables report log-average and standard-error metric values across 1000 test instances.
  • Fairness: Figure 3 evaluates fairness using the PGU metric on German Credit with an ANN model.It compares average and standard-error values for male majority and female minority subgroups.
  • Fairness: Larger majority–minority gaps in the PGU bars indicate higher, undesirable disparities.
  • Fairness: Figure 4 evaluates fairness using the PGU metric on Adult Income with an ANN model.It compares average and standard-error values for male majority and female minority subgroups.

E Choice of XAI methods, datasets, and models

OpenXAI focuses its first release on widely used feature-attribution methods and established datasets, while targeting tabular data despite broader applicability to text and images.

  • Choice of XAI methods, datasets, and models: LIME, SHAP, and Gradient-based methods remain popular post hoc explanation methods in both research and practice.Recent 2022 studies have analyzed these methods and called for further study given their widespread adoption.
  • Choice of XAI methods, datasets, and models: The first OpenXAI release focuses on tabular data, although its metrics and explanation methods are generic enough for text and images.The tabular focus is motivated by explainability needs in high-stakes applications such as loan approvals and disease diagnosis.
  • Choice of XAI methods, datasets, and models: The framework uses datasets widely employed in contemporary XAI and fairness research.These datasets have supported evaluations of new methods and studies of existing methods at ICML, NeurIPS, and FAccT.
Loading 2206.11104v5…