Rethinking Saliency Maps: A Cognitive Human Aligned Taxonomy and Evaluation Framework for Explanations

1 Introduction

• Existing saliency methods face concerns about reliability and unclear goals, while users often seek comparative explanations that conventional methods are not designed to support.Examples include asking why an image is a sports car rather than a convertible, or what distinguishes one dog breed from another.Source: §1 · pp. 1–2
• Without an explicit reference frame and semantic granularity, users may misinterpret a saliency map, and current metrics cannot validate alignment with the intended question.A fur-texture cue may help distinguish dog breeds but be irrelevant to explaining dog rather than cat; existing metrics largely assess pointwise faithfulness.Source: §1 · p. 2
• RFxG organizes saliency explanations by reference frame—pointwise or contrastive—and granularity—fine-grained or coarse-grained—to reflect distinct user questions.The introduction contrasts asking why an image is a husky with asking why it is a husky rather than a shih-tzu.Source: §1 · p. 2
• The paper proposes four faithfulness metrics to assess explanations across both RFxG axes and adds ImageNet group-level labels derived from the WordNet hierarchy.The group labels support evaluation at group granularity.Source: §1 · p. 2
• The evaluation framework covers ten saliency methods, four model architectures, and three datasets.Source: §1 · p. 2

2.1 Saliency-Based Explanation Methods

• Saliency methods include activation-based, perturbation-based, path-integration, and transformer-specific approaches, but prior work lacks a formal framework for classifying and evaluating their explanatory styles.The section positions RFxG as a user-centric framework for that gap.Source: §2.1 · pp. 2–3

2.2 Evaluation Metrics for Saliency Maps

• Most existing metrics emphasize pointwise evaluation, while contrastive metrics such as CAUC and CDROP have limitations and do not cover semantic granularity adequately.CAUC relies on thresholding that can be unstable for dense maps and multiplies prediction scores, which can produce misleadingly low values; CDROP inherits core limitations.Source: §2.2 · p. 3
• The proposed metrics evaluate across RFxG axes, measure contrastiveness through probability differences, and avoid unstable thresholding mechanisms.Source: §2.2 · p. 3

2.3 Critiques of XAI and Motivation for RFxG

• Critiques of underspecified interpretability goals motivate a structured perspective that aligns saliency explanations and evaluation with user questions.The section frames explanation purpose and user context as central to making evaluation coherent.Source: §2.3 · pp. 3–4

3 New Taxonomy for User-Centric Explanations

• RFxG decomposes attribution methods along reference-frame and granularity axes to clarify design decisions, assumptions, and downstream limitations.Source: §3 · p. 4

3.1 Reference Frame: What Is the Explanation Relative To?

• Pointwise explanations highlight evidence supporting the predicted class, whereas contrastive explanations compare it with an alternative class or hypothesis.The two forms answer, respectively, why the model made a prediction and why it chose class A over class B.Source: §3.1 · p. 4

3.2 Semantic Granularity: What Level of Class Detail is Targeted?

• Group-level explanations target features predictive of a superclass, while class-level explanations isolate features specific to a single class.The text contrasts general dog features with distinctive husky traits such as fur pattern or blue eyes.Source: §3.2 · p. 4

4 Evaluating User-Perspective Saliency Maps

• The metrics perturb images by suppressing the top α fraction of salient pixels, using 10% steps from α = 0.1 to 0.9 and AUC to aggregate results.Black-pixel masking is the default; the authors report consistent performance trends across masking alternatives.Source: §4 · pp. 4–5
• CCS measures the AUC of the prediction gap as salient regions are removed, assessing whether those regions distinguish one class from another.A high CCS indicates that removed regions are critical for distinguishing the contrasted classes.Source: §4 · p. 5
• CGC evaluates class-versus-group explanations, while CGS tests whether deleting group-discriminative features lowers one group’s confidence and raises the other’s.CGC aggregates the target-class drop and confidence rises among competing members; CGS captures suppression and promotion across groups.Source: §4 · p. 5
• PGS evaluates pointwise explanations for a semantic group by measuring the average group-confidence drop after salient regions are removed.It captures semantic generality rather than specificity, distinguishing it from class-centric faithfulness metrics.Source: §4 · p. 5
• CCS, CGC, PGS, and CGS form a user-aligned suite for contrastive and group-based faithfulness that complements, rather than replaces, standard metrics.The suite is intended to support a more purpose-driven assessment of saliency explanations.Source: §4 · p. 5

5 Evaluation Framework and Experiments

• The framework’s experiments used an NVIDIA DGX 8xA100 server and PyTorch; reported quantitative comparisons were statistically significant by a t-test at p = 0.05.Additional technical and implementation details and extended analyses are in the Appendix.Source: §5 · p. 5

5.1 Datasets

• The benchmark uses VOC, ImageNet, and COCO validation data, with WordNet-based semantic groups containing at least five subordinate leaf nodes.The resulting image-class-group triplets are intended for public release; sports car, cab, and limousine are grouped under car.Source: §5.1 · p. 5

5.2 Models and Explanation Methods

• Four models and ten saliency methods were evaluated; CNN and Transformer methods were adapted as needed for RFxG settings, with default hyperparameters.The study focuses on methods producing one map per question and excludes concept-based and generative explanations.Source: §5.2 · pp. 5–6

5.3 Metric Computation and Evaluation Setup

• For each image, the setup selects the top predicted class, a second class from its semantic group, and pointwise or contrastive group targets, then applies black-pixel masking.This selection strategy is intended to support meaningful contrastive setups and valid group-level comparisons.Source: §5.3 · p. 6

5.4 Quantitative Results

• Across datasets and architectures, IIA outperforms other methods particularly on CCS, PGS, and CGS, while PGS and CGS generally score higher than CGC and CCS.Table 1 reports the RFxG results across datasets, models, and explanation methods.Source: Table 1 · p. 6, §5.4 · p. 6
• IG underperforms across metrics, especially in contrastive settings, and IIA ranks only third on CDROP despite its overall superiority.The authors suggest CDROP may not fully reflect focused maps’ discriminative power, particularly when sparse attributions are penalized.Source: §5.4 · p. 7

5.5 Qualitative Results

• Figure 2 shows RN saliency maps for a sports car under pointwise class, class-versus-class, and class-versus-group questions.Additional qualitative examples are provided in the Appendix.Source: §5.5 · p. 7

6 Conclusion

• RFxG distinguishes pointwise from contrastive explanations and fine-grained from coarse-grained semantics, addressing ambiguity in saliency-map interpretation.The framework is presented as a user-intent-driven perspective on what saliency maps represent.Source: §6 · p. 7
• The authors identify a gap in existing metrics, which largely prioritize pointwise faithfulness while neglecting contrastive reasoning and semantic granularity.They propose four metrics to assess explanation quality across both RFxG dimensions.Source: §6 · p. 7
• Across ten methods, four architectures, and three datasets, IIA consistently performs well at capturing contrastive evidence and semantic groupings.The authors describe the framework as a foundation for explanations that are faithful to models and aligned with user needs.Source: §6 · p. 7

A Appendix Overview

• The appendix contains additional evaluation details, method adaptations, extended quantitative results including VOC, qualitative examples, and a discussion of limitations and future work.Its sections B through F respectively cover these topics.Source: A Appendix Overview · p. 11

B Evaluation Details

• For VOC, images from classes overlapping with ImageNet were removed, and remaining validation images used ViT-B’s top predicted class as ground truth; COCO follows the VOC protocol.ImageNet evaluation uses original images and their corresponding ground-truth labels.Source: B Evaluation Details · p. 11
• WordNet groups are used when they contain at least five classes; otherwise, neighboring groups are merged until that threshold is met, with labels based on a unique superordinate category or concatenated group names.The construction procedure is described as also using a git repository.Source: B Evaluation Details · p. 11

• The implementation uses Captum’s GradientSHAP, Gradient-Rollout, CNN Contrastive GradCAM, and two contrastive methods for ViTs.Gradient-Rollout incorporates gradients into standard attention rollout.Source: C Method Implementation Details · p. 11
• Methods without native contrastive or group support are adapted by applying their core computation to class differences and group aggregations.This enables consistent application under different reference frames and semantic granularities.Source: C Method Implementation Details · p. 11
• The illustrative Contrastive-GradCAM computes gradients for a contrastive score rather than fA alone, retaining negative contributions in maps distinguishing classes A and B.The passage identifies fA and fB as the pre-softmax logits for the two classes.Source: C Method Implementation Details · pp. 11–12
• For each metric, the reported results use an explanation method suited to that metric’s purpose.The table gives GC on CCS with Contrastive-GradCAM as an example.Source: Table 3 · p. 12

• Table 3 presents additional evaluation results for the VOC datasets.Source: D Additional Quantitative Results · p. 12
• The VOC results follow trends similar to those in Table 1.Source: D Additional Quantitative Results · p. 12
• The section’s additional results concern VOC, complementing the results reported in Table 1.Source: D Additional Quantitative Results · p. 12

• Figure 3 compares maps from GC, IIA, and SC for a sports-car example.Source: E Qualitative Results · p. 12
• The example contrasts a sports car with a cab and uses a Car group that includes sports cars, racers, and other vehicle classes.Source: E Qualitative Results · p. 12
• The figure shows Class Contrastive, Class-Group Contrastive, and Pointwise Group explanations.Source: Figure 3 · p. 12

F Limitations and Future Work

• The authors present the limitations as opportunities to extend RFxG rather than as undermining its core contributions.Source: F Limitations and Future Work · p. 12, §F.4 · p. 13

F.1 Methodological Considerations

• Black-pixel removal could introduce distribution-shift artifacts, though blur and inpainting produced similar RFxG results; future perturbations should preserve local statistics efficiently.Source: §F.1 · p. 12

F.2 Scope and Generalizability

• The implementation currently targets image classification; proposed extensions include recommendation and NLP explanations aligned to contrastive questions and abstraction levels.Source: §F.2 · pp. 12–13

F.3 Integration with Concept-Based Explanations

• RFxG could extend from pixel-space saliency to concept explanations, and its metrics could be adapted to multimodal visual-text settings.Source: §F.3 · p. 13

F.4 Broader Impact and Deployment Considerations

• High-stakes deployment needs domain-specific adaptations; clinicians may prefer organ-level explanations, patients lesion-specific ones, and task utility could complement faithfulness.Source: §F.4 · p. 13