Source-linked AI summary
Sanity Checks for Saliency Metrics
Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, Alun Preece
TL;DR
Saliency methods estimate pixel relevance, but existing fidelity metrics may not reliably distinguish accurate explanations. The paper evaluates these metrics with psychometric reliability checks and finds high variance, inconsistent rankings, and low agreement about the underlying notion of fidelity. These findings are illustrated on a single model and task, so users should apply the checks in their own settings.
Problem
Different saliency methods produce different maps, while limited evidence exists about whether fidelity metrics accurately assess pixel-relevance estimates.
Method
The paper evaluates saliency fidelity metrics using psychometric reliability measures and sanity checks, distinguishing method-level from individual-explanation fidelity.
Results
Current metrics showed high variance, inconsistent image-by-image rankings, low internal consistency, and differing underlying fidelity concepts.
Takeaways & Limitations
Comparative rankings from saliency metrics can be untrustworthy, so users should apply sanity checks to metrics for their particular data and model.
Takeaways & Limitations
The results assess one model and task and are an illustration rather than a general statement about saliency-metric consistency.
Abstract
from arXiv · showhide
Saliency maps are a popular approach to creating post-hoc explanations of image classifier outputs. These methods produce estimates of the relevance of each pixel to the classification output score, which can be displayed as a saliency map that highlights important pixels. Despite a proliferation of such methods, little effort has been made to quantify how good these saliency maps are at capturing the true relevance of the pixels to the classifier output (i.e. their "fidelity"). We therefore investigate existing metrics for evaluating the fidelity of saliency methods (i.e. saliency metrics). We find that there is little consistency in the literature in how such metrics are calculated, and show that such inconsistencies can have a significant effect on the measured fidelity. Further, we apply measures of reliability developed in the psychometric testing literature to assess the consistency of saliency metrics when applied to individual saliency maps. Our results show that saliency metrics can be statistically unreliable and inconsistent, indicating that comparative rankings between saliency methods generated using such metrics can be untrustworthy.
Introduction
Saliency maps aim to explain image-classifier outputs by estimating each pixel’s relevance, but different methods produce different maps and require fidelity evaluation. This paper examines whether existing metrics reliably measure that fidelity.
- Saliency maps estimate the relevance of individual image pixels to a DNN’s output.
- Different saliency methods can produce different maps for the same image and classifier output, creating a need to compare their relevance estimates.
- Proposed explanation axioms can be satisfied by useless methods, such as assigning uniform relevance scores to every pixel.
- The paper distinguishes fidelity of a saliency method from fidelity of an individual explanation and proposes sanity checks for saliency metrics.
- Current saliency metrics can have high variance, depend on implementation details, be statistically unreliable for individual maps, and measure different notions of fidelity.
Background
The paper reviews saliency-map fidelity and metrics based on perturbing pixels and observing classifier outputs. It also explains why ROAR cannot assess individual-map fidelity and is computationally intensive.
- Feature attribution methods, called saliency maps for image pixels, estimate each feature’s relevance to a model output.
- Fidelity measures how well an explanation represents the model’s processing and assigns relevance values to input pixels.
- AOPCM evaluates fidelity by sequentially perturbing pixels in relevance order and measuring classifier-output changes under MoRF or LeRF.
- Under MoRF, accurate methods should make classifier output decrease rapidly; under LeRF, accurate methods should make it change slowly.
- AOPCM uses classifier output differences across pixel-deletion steps and averages them over dataset images, with L controlling the deletion amount.
- ROAR perturbs relevant training pixels and retrains a new classifier, but measures only global method fidelity and requires extensive computation.
Evaluating saliency metrics
The paper evaluates saliency-metric reliability for individual explanations using psychometric-testing concepts, because validity cannot be directly tested without ground-truth saliency maps. It examines agreement across images, methods, and metrics, while excluding test-retest reliability for deterministic settings.
- Motivation: Ground-truth saliency maps are unavailable, so the paper assesses reliability as a necessary, though insufficient, condition for metric validity.A valid metric must be reliable, but a reliable metric need not be valid.
- Psychometric framework: The analysis treats saliency methods as tests, the neural network as an agent, and input images as raters administering the tests.This analogy motivates importing reliability statistics from psychometric testing.
- Reliability measures: Inter-rater reliability measures whether the same metric ranks saliency methods consistently across input images.The paper uses this to assess consistency of method rankings between images.
- Reliability measures: Inter-method reliability measures agreement among saliency methods using the same metric, whereas internal consistency compares different metrics applied to the same saliency method.Both forms are assessed through score correlations, using Spearman’s ρ.
- Scope: Test-retest reliability is excluded because the saliency methods and models are deterministic and fixed.The paper notes that repeated administrations do not apply in this setting.
- Inter-rater reliability: Krippendorff’s α quantifies consistency in saliency-method rankings across images, with α = 1 indicating identical rankings and α = 0 indicating random rankings.Negative values indicate systematic disagreement, while low α makes future rankings largely unpredictable.
Experiments
Experiments examine AOPC and faithfulness on a single CNN trained for CIFAR-10, using four saliency methods plus edge detection and two pixel-perturbation schemes. The design prioritizes detailed analysis of metric properties over broad variation across models and datasets.
- Metrics: The study investigates AOPC with MoRF and LeRF ordering and faithfulness F.These metrics require a defined perturbation function for image pixels.
- Study scope: The paper focuses on one model and a few methods so metric properties can be explored in depth without effects from multiple models and datasets.This is an intentional scope choice rather than an exhaustive benchmark design.
- Model and task: The experiments use one CNN trained on CIFAR-10, a 10-class dataset with 32×32 RGB images.The model has three convolutional blocks, batch normalization, max pooling, ReLU activations, and a SoftMax classification layer.
- Saliency methods: Four saliency methods—sensitivity analysis, gradient⊙input, deep Taylor decomposition, and DeepSHAP—are compared with edge detection.The methods were selected as well-known approaches with differing properties and assumptions, rather than as an exhaustive set.
- Perturbations: Perturbations replace selected pixels either with the dataset mean or with uniformly random RGB values.The study uses single-pixel perturbations rather than pixel regions to avoid assumptions about feature scale.
- Evaluation design: Faithfulness is estimated from 100 randomly selected pixel IDs reused for every image, while AOPC receives a random-ordering baseline from 100 random pixel orderings.These sampling choices reduce the computational cost of evaluating every pixel and ordering.
Results
Results show that saliency-metric rankings and scores depend on perturbation details, while reliability analyses find inconsistent rankings, weak agreement among metrics, and low faithfulness across methods. These findings challenge the stability of comparative fidelity assessments.
- AOPC results: MoRF AOPC rankings change between mean and random RGB perturbations, with gradient⊙input and SHAP top-ranked under different perturbation schemes.Deep Taylor decomposition and sensitivity are indistinguishable under mean perturbation, but sensitivity ranks higher under random RGB perturbation.
- AOPC results: LeRF rankings remain consistent across perturbations but are opposite to the MoRF rankings.For LeRF, lower AOPC indicates better fidelity, whereas higher AOPC indicates better fidelity for MoRF.
- Faithfulness results: Faithfulness means are close to zero for every method, with edge detection consistently lowest.Perturbation choice changes the score distributions, means, and method rankings.
- Reliability results: Krippendorff’s α values indicate inconsistent method rankings across images, and random RGB perturbation reduces α for faithfulness and MoRF AOPC relative to mean perturbation.Values below 0.65 are often considered to indicate unreliability, although no strict threshold exists.
- Inter-method reliability: The lowest mean pairwise Spearman correlation between methods is 0.13 for faithfulness with random RGB perturbation and edge detection included.Including edge detection lowers mean pairwise correlation because it does not assess the model.
- Internal consistency: Different metrics generally do not measure the same underlying quantity across saliency methods.Faithfulness is only weakly correlated with MoRF AOPC and slightly anticorrelated with LeRF AOPC, while MoRF–LeRF correlations vary by method.
Discussion
The evaluated saliency metrics showed substantial variance, implementation sensitivity, and inconsistent rankings across images and metrics. These findings make fidelity comparisons difficult, especially without ground-truth saliency maps and when perturbations may move images off the learned data manifold.
- Discussion: Reliability conclusions are illustrative rather than general, because the study assessed metrics on one model and task.The authors recommend applying the sanity checks in the context of each particular model and dataset.
- Discussion: Global saliency metrics had high variance across test images, whereas prior work reported mainly mean metric values.Broad score distributions were observed over all test images, potentially obscuring image-level variability.
- Discussion: Metric scores and saliency-method rankings changed with the perturbation implementation, including mean versus random RGB perturbations.Both faithfulness and AOPC scores were affected by this choice.
- Discussion: Image-by-image rankings had low inter-rater reliability, so global scores may poorly indicate fidelity on future test images.The inconsistency concerns rankings of saliency methods for individual images.
- Discussion: Different fidelity metrics showed generally low and highly variable correlations, indicating they may not measure the same underlying concept.This reflects low internal consistency among metrics applied to the same saliency methods.
- Discussion: Without ground-truth saliency maps, metric variance is difficult to disentangle, and perturbations may move images off the learned data manifold.The appropriate perturbation cannot currently be identified a priori as one that stays on-manifold.
- Discussion: AOPCMoRF and AOPCLeRF produced reversed global saliency-method rankings, partly because LeRF favors edge-detection-like methods.Mean perturbation can change edge contrast without removing the edge, while removing contiguous color regions may remove little discriminative information.
- Discussion: Metric scores depended on image class, with frog images showing distinct AOPCMoRF, AOPCLeRF, and faithfulness patterns.The causes were not fully examined and may depend on class features, classifier representations, and saliency-map quality.
Conclusion
The paper proposes sanity checks and development practices for evaluating saliency metrics more reliably. It emphasizes direct quantitative comparison, analysis of implementation variants and failure modes, and investigation of score variance across models and datasets.
- Conclusion: Saliency-metric sanity checks assess inter-rater reliability, inter-method reliability, and internal consistency when validity cannot be tested against ground truth.The checks adapt reliability measures from psychometric testing to saliency metrics.
- Conclusion: New metrics should be compared directly with earlier metrics measuring related quantities, with their differences made explicit quantitatively.The recommendation targets comparability between competing fidelity measures.
- Conclusion: Metric variants such as mean versus random RGB perturbation should be analyzed under different conditions to reveal implementation effects.The recommendation applies when a metric admits multiple implementation choices.
- Conclusion: Metrics should be tested for contexts in which they can be “tricked,” such as edge detection receiving the best AOPCLeRF scores.This helps identify cases where a metric’s ranking may not reflect intended fidelity.
- Conclusion: Metric developers should encourage users to investigate score variance and its effect on saliency-method choices for their particular model.The recommendation links reliability analysis to downstream method selection.
- Conclusion: Saliency-method developers should use several metrics across varied datasets and models rather than relying on one fidelity metric.The authors warn that a single metric can produce misleading results.