PaperlayerView exact revision on arXiv

Source-linked AI summary

Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI

Leila Arras, Ahmed Osman, Wojciech Samek

arXiv:2003.07258v2cs.CVcs.LGcs.NEeess.IV

TL;DR

The paper addresses the lack of objective, commonly accepted evaluation for neural-network heatmaps. It introduces CLEVR-XAI, a ground-truth-based VQA benchmark with quantitative metrics, and finds that explanation-method comparisons can contradict prior studies, with LRP, Integrated Gradients, and Guided Backpropagation performing most accurately among tested methods.

  • Problem

    Existing heatmap evaluations rely largely on human assessment or proxy tasks, leaving objective comparison of explanation quality unresolved.

  • Method

    The paper builds CLEVR-XAI from CLEVR-style visual question answering with ground-truth masks and evaluates ten explanation methods on a trained Relation Network.

  • Results

    Among tested methods, Layer-wise Relevance Propagation, Integrated Gradients, and Guided Backpropagation were most accurate, while Deconvnet and Grad-CAM were least accurate.

  • Takeaways & Limitations

    CLEVR-XAI provides a benchmark, annotations, and metrics for analyzing and improving neural-network explanation methods.

  • Takeaways & Limitations

    The ground-truth masks omit object shadows even though the VQA model could use them for some questions, such as shape queries.

Abstract

from arXiv · show

The rise of deep learning in today's applications entailed an increasing need in explaining the model's decisions beyond prediction performances in order to foster trust and accountability. Recently, the field of explainable AI (XAI) has developed methods that provide such explanations for already trained neural networks. In computer vision tasks such explanations, termed heatmaps, visualize the contributions of individual pixels to the prediction. So far XAI methods along with their heatmaps were mainly validated qualitatively via human-based assessment, or evaluated through auxiliary proxy tasks such as pixel perturbation, weak object localization or randomization tests. Due to the lack of an objective and commonly accepted quality measure for heatmaps, it was debatable which XAI method performs best and whether explanations can be trusted at all. In the present work, we tackle the problem by proposing a ground truth based evaluation framework for XAI methods based on the CLEVR visual question answering task. Our framework provides a (1) selective, (2) controlled and (3) realistic testbed for the evaluation of neural network explanations. We compare ten different explanation methods, resulting in new insights about the quality and properties of XAI methods, sometimes contradicting with conclusions from previous comparative studies. The CLEVR-XAI dataset and the benchmarking code can be found at https://github.com/ahmedmagdiosman/clevr-xai.

1. Introduction

The paper argues that existing heatmap evaluations lack objective ground truth and may mismatch explanations with what a model actually uses. It introduces CLEVR-XAI, a selective and realistic VQA benchmark with ground-truth masks and quantitative metrics for comparing explanation methods.

  • Motivation: Existing XAI evaluations can fail to establish whether explanations reflect genuinely decisive input features or spurious correlations.The motivation includes concerns about models exploiting dataset artifacts and the need to trace decisions to decisive input regions.
  • Motivation: In computer vision, post-hoc explanations assign pixel-level relevance scores that visualize each pixel’s relative contribution to a prediction.The paper lists methods including Class Saliency Map, Grad-CAM, Integrated Gradients, LRP, Excitation Backpropagation, and Guided Backpropagation.
  • Motivation: Prior proxy evaluations can mismatch an explanation’s goal because they measure perturbation effects or localization rather than the current decision’s decisive regions.This mismatch is especially relevant when decisions depend on image context or dataset biases.
  • Proposed benchmark: CLEVR-XAI evaluates explanations against ground-truth object coordinates in realistically rendered synthetic 3D scenes using a visual question answering task.The VQA setup varies object number, location, and size, while questions select which objects are relevant to each prediction.
  • Contributions: CLEVR-XAI contains 39,761 simple and 100,000 complex questions with corresponding pixel-level ground-truth masks.These datasets are intended as a benchmark for evaluating visual explanations.
  • Contributions: The paper proposes relevance mass accuracy and relevance rank accuracy, then compares ten explanation methods using a trained Relation Network.The study reports new findings about explanation quality and properties that sometimes contradict previous comparative studies.

2. Previous Work on Evaluating Explanations

Earlier explanation evaluations rely on perturbations, localization proxies, randomization, or human assessment, each with important limitations. The paper motivates a synthetic, selective benchmark that controls visual biases and evaluates explanations under the original model and data configuration.

  • Perturbation-based evaluation: Pixel-flipping evaluates relevance orderings by repeatedly altering pixels and measuring the resulting change in model prediction.A larger prediction-performance drop is treated as evidence of more accurate relevance ordering.
  • Perturbation-based evaluation: Randomization tests compare explanations after changing model weights or training labels, but evaluate a modified model or training setup.These tests therefore differ from the deployed model and its original configuration.
  • Limitations: Perturbation-based evaluations can produce misleading comparisons because they introduce discrepancies between original training/test configurations and the evaluation setup.The discrepancy arises from modified inputs, retraining, or parameter randomization.
  • Localization-based evaluation: Weakly supervised localization evaluates heatmaps against object annotations using thresholding and metrics such as Intersection over Union.This approach is commonly applied to real-world image-classification tasks.
  • Benchmark rationale: CLEVR-XAI uses synthetic rendered shapes with a uniform background, multiple objects, varied attributes, and occlusions to control biases while requiring selective reasoning.Its VQA questions modulate which objects matter, unlike standard image classification where one object is typically relevant.
  • Evaluation landscape: Human visual assessment provides end-user insights but cannot replace objective and systematic automatic evaluation of computer-vision XAI.The paper contrasts this approach with its ground-truth benchmark and quantitative metrics.

3. Ground Truth Evaluation of Heatmaps

CLEVR-XAI is a controlled benchmark that evaluates heatmaps against automatically generated, question-specific pixel-level ground truths in synthetic but realistically rendered VQA scenes. It supports both selective single-object questions and complex multi-object reasoning, alongside pooled heatmaps and two quantitative accuracy metrics.

  • Benchmark dataset: CLEVR-XAI builds on CLEVR to provide questions paired with pixel-level ground truth masks for evaluating visual explanations.The benchmark uses CLEVR’s controlled synthetic VQA generation pipeline and adds explanation ground truths.
  • Benchmark dataset: The evaluation set contains 10,000 images and approximately 140,000 questions, including 39,761 simple and 100,000 complex questions.Both subsets use the same images, with approximately four simple and ten complex questions per image.
  • Question design: Simple questions isolate one target object, whereas complex questions involve spatial relations, attribute comparisons, counting, and potentially several relevant objects.Simple queries ask about object attributes; complex queries follow the broader CLEVR question distribution.
  • Ground truth generation: Object segmentation masks and each question’s functional program automatically generate ground truths for the target object, all objects, or multiple program-dependent object sets.For simple questions, GT Single Object isolates the target, while GT All Objects includes every object and excludes the background; complex examples use program outputs to define relevant objects.
  • Heatmap evaluation: Heatmaps are pooled across channels into positive single-channel images, then evaluated with relevance mass accuracy and relevance rank accuracy.The metrics assess whether the major relevance lies inside a ground truth mask and return values from 0 to 1, with higher values indicating more accurate heatmaps.

4. Explanation Methods

The paper compares ten post-hoc explanation methods, converting pixel-level relevances into heatmaps for quantitative evaluation in the CLEVR-XAI setting. The methods include gradient-based, noise-aggregated, modified-backpropagation, Grad-CAM, Integrated Gradients, and related variants.

  • Ten explanation methods are evaluated, with alternative variants and hyperparameters reported for each method.The study allows tuning variants and hyperparameters to identify the best-performing variant, unlike standard train-validation-test tuning.
  • Each method assigns scalar relevance values to input variables, which are pooled across channels into positive H × W heatmaps for evaluation.The target class is the model’s predicted answer, and the resulting heatmaps are evaluated against CLEVR-XAI ground-truth masks.
  • Gradient-based methods: Gradient and Gradient×Input use partial derivatives, while squared-gradient variants replace the simple derivative and require one backward pass.Gradient×Input additionally multiplies the derivative by the input variable’s value.
  • Modified backpropagation and Grad-CAM: Deconvnet and Guided Backpropagation modify ReLU backward passes, whereas Grad-CAM combines last-convolutional-layer gradients with feature maps and upsamples the result.Guided Grad-CAM multiplies Grad-CAM and Guided Backpropagation relevances element-wise; Grad-CAM variants use different layers and discard zero-valued heatmaps.
  • Integrated Gradients: Integrated Gradients approximates an input-to-baseline gradient integral, testing zero, mean-image, and mean-channel-value baselines.The zero-valued baseline is black, while the latter two baselines are grey-valued because CLEVR images have grey backgrounds.

5. Experiments

Experiments evaluate explanation methods with CLEVR-XAI ground truths, showing strong differences across methods, configurations, metrics, selectivity, and model confidence.

  • 5.1. Model: The selected relation network reaches 93.3% CLEVR test accuracy, compared with 95.5% reported by its original authors.
  • 5.2. Simple Questions: CLEVR-XAI-simple: Tuning mass accuracy or rank accuracy selects the same best variant and hyperparameters for every method, with largely consistent method rankings.The finding supports using both metrics to assess explanation quality.
  • 5.2. Simple Questions: CLEVR-XAI-simple: For integrated gradients, mean-channel and mean-image baselines work best, whereas a zero-valued baseline yields mass and rank accuracy of only 0.27.The result emphasizes the importance of baseline selection for IG.
  • 5.2. Simple Questions: CLEVR-XAI-simple: Sampling-based methods are highly sensitive to hyperparameters: using no more than 100 samples and noise of at least 0.10 would reduce mass accuracy by at least 0.10 and rank accuracy by at least 0.08.
  • 5.2. Simple Questions: CLEVR-XAI-simple: Grad-CAM and Deconvnet are by far the worst-performing methods, whereas Guided Grad-CAM performs relatively well mainly because of its Guided Backprop component.The authors attribute Grad-CAM’s weakness partly to gradient dilution through average pooling.
  • 5.2. Simple Questions: CLEVR-XAI-simple: LRP and Guided Backprop perform remarkably well with a single backward pass, while LRP generally outperforms Guided Backprop.
  • 5.2. Simple Questions: CLEVR-XAI-simple: Relevance accuracy rises with model confidence or easier questions while standard deviation decreases, indicating dependence on prediction confidence or question difficulty.

6. Discussion

The discussion argues that CLEVR-XAI produces more reliable comparisons than prior evaluation protocols while exposing important limitations and open questions. Results differ from several earlier comparative studies, and performance depends on methodological choices such as IG’s baseline.

  • The study could not confirm previous findings, and its results sometimes contradict earlier comparative studies.
  • Limitations: The ground-truth masks omit object shadows, although the VQA model could use shadows when answering some shape questions.
  • Limitations: Relevance accuracy below 1.0 may reflect classifier uncertainty, which can diffuse relevance across multiple scene objects.
  • Outlook: The evaluation remains suitable for comparing relative XAI-method performance despite its imperfections.
  • Outlook: Integrated Gradients and SmoothGrad use perturbed inputs that may leave the training-data manifold, risking artifacts and unreliable model behavior.

7. Conclusion

The paper presents CLEVR-XAI as a ground-truth VQA benchmark for analyzing neural-network explanations. In its comparison, LRP, IG, and Guided Backpropagation were most accurate, whereas Deconvnet and Grad-CAM were least accurate.

  • CLEVR-XAI provides a VQA-based evaluation paradigm with ground-truth annotations and metrics for analyzing and improving neural-network explanations.
  • The framework is realistic, selective, controlled, and evaluated on the same model and data configuration used during training.
  • Layer-wise Relevance Propagation, Integrated Gradients, and Guided Backpropagation were most accurate, while Deconvnet and Grad-CAM were least accurate.

Appendix A. Additional CLEVR-XAI-complex example

The appendix illustrates how functional programs determine ground-truth object masks for complex CLEVR-XAI questions. Some masks can be undefined or empty, reducing the number of usable examples for particular ground truths.

  • A functional program identifies which scene objects count as ground truths for a CLEVR-XAI-complex example.
  • For the illustrated tree-structured question, the GT Unique mask is undefined because the program contains no unique function.
  • The GT Unique First-non-empty mask selects the first non-empty object set returned when the program functions are processed in reverse order.
  • Some complex-question ground-truth masks are undefined or empty, so corresponding statistics contain fewer than 100,000 data points.

Appendix B. CLEVR-XAI dataset statistics

Table B.10 reports CLEVR-XAI dataset statistics by ground truth, including question counts, pixel counts, and object counts. Pixel statistics are based on 128×128 inputs, with other mask resolutions also available.

  • The dataset statistics report minimum, maximum, mean, and standard deviation for pixels and objects across ground-truth categories.
  • Program length counts basic functions in a question’s functional program and serves as an indicator of question complexity.
  • Pixel counts are calculated on 128×128 images because that is the neural network’s input size.
  • The dataset release also supports ground truths at 320×480 or another chosen image size.

Appendix C. Neural Network used for the XAI Evaluation

The evaluated model combines convolutional, recurrent, relational, and classification components. Its training uses specified optimization, regularization, preprocessing, and augmentation settings.

  • Architecture: The model uses a four-layer CNN with conv →relu →batchnorm blocks and 24 stride-2, 3×3 kernels per layer.
  • Architecture: Questions are encoded by a unidirectional LSTM with 32-dimensional word embeddings and a 128-unit hidden layer.
  • Architecture: The Relation Network contains four 256-unit fully connected layers with ReLU activations, followed by element-wise summation.
  • Architecture: The classifier has two 256-unit ReLU layers, dropout p = 0.5 in the second, and a 28-unit output layer for CLEVR answers.
  • Preprocessing: Images are resized to 128×128 with pixel values in [0, 1], while questions are lowercased, de-punctuated, and drawn from an 80-word vocabulary.
  • Training: Training uses Adam with batch size 64, learning rate 2.5e-4, gradient clipping at 5.0, l2 regularization of 4e-5, learning-rate decay, and augmentation.Augmentation consists of random cropping and random rotation; training runs for at most 1200 epochs.

Appendix D. Resizing of Ground Truth Masks

Ground-truth masks must be transformed to match the model’s 128×128 input resolution before evaluating explanations.

  • Mask resizing: The model’s 128×128 image input requires resizing the original ground-truth masks to the same size.
  • Mask resizing: The original masks correspond to CLEVR images sized 320×480 before preprocessing.
  • Mask resizing: Mask resizing uses the same operation as standard input-image resizing, starting from masks valued at 1.0 on ground-truth pixels.

Appendix E. XAI methods hyperparameters

The appendix summarizes the tuned hyperparameters and pooling techniques used for each explanation method in the evaluation.

  • Configuration: Hyperparameters and pooling techniques were selected separately for each XAI method.
  • Configuration: The configurations were tuned for each relevance accuracy metric.
  • Configuration: Table E.11 recapitulates the XAI-method hyperparameters used in the evaluation.

Appendix F. Heatmaps

The appendix presents heatmaps for correctly and falsely predicted CLEVR-XAI questions, using the model’s predicted class as the explanation target. The examples show that false predictions can reflect attention to the wrong object or an incorrect interpretation of the right object’s color.

  • Examples: Heatmap examples target the neural network’s predicted class across CLEVR-XAI-simple and CLEVR-XAI-complex questions.
  • Example selection: Examples are selected by highest predicted softmax probability, separately considering correct and false predictions in the simple subset.
  • Visualization: Visualizations include raw channel-pooled heatmaps and overlays on the original images, with relevance values color-coded per heatmap.
  • Interpretation: False predictions may focus on the wrong object, such as a brown cylinder instead of a yellow ball or a blue cube instead of a purple cube.
  • Interpretation: A false prediction can also involve detecting the correct object but misinterpreting its color, as with the small red ball classified as brown.
Loading 2003.07258v2…