PaperlayerView exact revision on arXiv

Source-linked AI summary

ConEx: Human-Interpretable Saliency Maps via Concept-Aware Attribution

Yehonatan Elisha, Oren Barkan, Ziv Weiss Haddad, Noam Koenigstein

arXiv:2610.04605v1cs.CVcs.AIcs.LG

TL;DR

Pixel saliency highlights important regions but often does not connect them to meaningful concepts, while concept methods may lack spatial grounding. ConEx automatically discovers and grounds concepts, then combines them with model relevance for explanations. Across its evaluations, ConEx consistently outperforms existing saliency and concept-based methods, though its language-model-generated attributes may inherit societal biases.

  • Problem

    Visual explanations often lack either the semantic meaning of concepts or the spatial grounding needed to locate them in an image.

    Source: §1 · p. 1

  • Method

    ConEx uses label-free vision-language priors and zero-shot segmentation to discover and ground concepts, then constructs CAVs and combines concept presence with model relevance.

    Source: §1 · pp. 1–2

  • Results

    ConEx consistently outperforms existing saliency and concept-based methods across multiple datasets and architectures.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results indicate that grounded concepts can support explanations that are human-meaningful, faithful, and robust to spurious correlations.

    Source: §5 · p. 9

    Because automated concept discovery relies on LLM-generated textual attributes, societal biases in vision-language priors could transfer to generated explanations.

    Source: Impact Statement · pp. 9–10

Abstract

from arXiv · show

Many visual explanation methods in computer vision highlight pixel importance but struggle to link these low-level cues to semantically meaningful concepts, limiting their interpretability and trustworthiness. We introduce Concept-based Explanations (ConEx), a novel framework that bridges saliency visualization with concept-based reasoning to provide both faithfulness and interpretability. ConEx automatically discovers class-specific concepts and represents them through concept activation vectors (CAVs), learned without manual supervision using an architecture-specific masking mechanism that reduces noise introduced by the segmentation masks to enhance concept purity. ConEx generates faithful saliency maps that reveal where each concept appears in the image and how it contributes to the prediction. To evaluate the reliability of these learned concepts, we propose two complementary metrics, Vector-Concept Match (VCM) and Concept-Class Match (CCM), that quantify concept alignment and enable direct comparison with existing methods. Extensive experiments across diverse settings demonstrate that ConEx achieves state-of-the-art performance on faithfulness, segmentation, and concept-quality benchmarks. Overall, ConEx advances the field toward truly interpretable and concept-grounded explanations in vision models.

1. Introduction

Pixel saliency and concept-based explanations each leave a gap between spatial localization and semantic meaning, while many concept methods require supervision. ConEx combines spatially grounded concepts with saliency-style explanations and introduces metrics for evaluating concept reliability.

1. Introduction

  • Saliency methods localize model attention but offer limited semantic insight, while concept methods describe attributes without spatial specificity.The disconnect makes it harder to verify whether identified concepts appear in the input and limits explanation trustworthiness.Source: §1 · p. 1
  • ConEx is a fully automatic, post-hoc framework that bridges saliency visualization and concept reasoning without manual annotation or retraining.It uses label-free vision-language priors, zero-shot segmentation, and validated concept segments to construct CAVs.Source: §1 · pp. 1–2
  • Across three datasets and five architectures, ConEx consistently surpasses state-of-the-art saliency and concept-based methods across four evaluation dimensions.The dimensions include faithfulness, concept quality, segmentation accuracy, and human interpretability.Source: §1 · p. 2
  • ConEx introduces architecture-specific embedding strategies to enhance concept purity and support robust CAV construction for CNNs and ViTs.The strategies use layer-wise masking for CNNs and unmasked patch aggregation for ViTs.Source: §1 · p. 2
  • ConEx defines VCM and CCM as complementary metrics for quantitatively assessing concept reliability and alignment.The framework evaluates concepts through their alignment and their relationship to model decisions.Source: §1 · p. 2

2. Related Work

Prior work offers pixel-level localization or semantic concepts, but often lacks the other form of explanation or depends on manual concept curation. ConEx combines automatic concept discovery, spatial grounding, and architecture-specific feature extraction, while its complete spatial pipeline is currently designed for CNNs.

Post-Hoc Attribution Methods

  • Pixel-attribution methods reveal which input regions influence predictions, but do not clarify the semantic concepts driving decisions.The approaches surveyed include gradient-, activation-, perturbation-, game-theoretic, path-integration, and relevance-decomposition methods.Source: §2 · p. 2

Concept-Based Interpretability

  • Concept methods map model behavior to human-interpretable attributes, but existing CAV approaches can require manual curation or coarse image-level supervision.ACE uses unsupervised clustering, while classifier-based CAV training can be sensitive to sample selection and outliers.Source: §2 · p. 2
  • Visual-TCAV bridges saliency and concepts but requires curated examples, user-specified concepts, and full-image CAVs; ConEx instead grounds automatically discovered concepts.ConEx uses vision-language priors and zero-shot segmentation, architecture-specific embedding strategies, and multiplicative fusion with LRP.Source: §2 · pp. 2–3

Architecture-Specific Interpretability

  • ConEx uses layer-wise masking with neighborhood padding for CNNs and patch-token selection and aggregation for ViTs.Its CAV construction and validation are compatible with both architectures, but the complete spatial explanation pipeline is currently designed for CNNs.Source: §2 · p. 3

3. Method

ConEx explains classifier predictions through human-interpretable concepts, with a representation, localization, and attribution for each concept. Its pipeline automatically discovers and validates concepts, constructs architecture-specific CAVs, and produces local and global explanations without manual concept curation or additional training.

3.2. Automatic Concept Discovery and Validation

  • ConEx discovers class-discriminative textual attributes, grounds them with zero-shot segmentation, and retains concepts meeting occurrence-rate and spatial-coverage criteria.The resulting concepts are spatially grounded and frequently occurring within their classes.Source: §3.2 · p. 3

3.3. Concept Activation Vectors

  • Each CAV is the difference between mean embeddings of positive concept samples and negative samples from other concepts, representing the concept’s latent-space direction.The method describes this centroid difference as more robust to perturbations than classifier-based approaches such as SVMs.Source: §3.3 · p. 3

3.4. Architecture-Specific Embedding Strategies

  • CNN embeddings use layer-wise masking with limited first-layer neighborhood padding, while ViT embeddings average segment-overlapping patch tokens from an intermediate layer.These strategies aim to avoid masking artifacts while preserving semantic information.Source: §3.4 · p. 4

3.5.1. CONCEPT LOCALIZATION VIA CHANNEL-WEIGHTED VECTORS

  • ConEx localizes concepts by weighting latent feature maps with channel-weighted vectors derived from pooled CAVs, then normalizing the resulting semantic maps.The maps highlight where the network recognizes a concept, with intensity indicating activation strength; positive-centroid normalization supports cross-concept and cross-image comparison.Source: §3.5.1 · pp. 4–5

3.5.2. CONCEPT-AWARE ATTRIBUTION VIA MULTIPLICATIVE FUSION

  • Multiplicative fusion combines concept-presence maps with LRP relevance, amplifying attributions only where both coincide.This suppresses irrelevant regions for spatially precise, semantically grounded explanations.Source: §3.5.2 · p. 5

3.5.3. CLASS-SPECIFIC SEMANTIC ATTRIBUTION

  • For each concept, ConEx measures the normalized class-probability drop after masking its regions, then aggregates concept scores into a class-specific saliency map.The resulting explanation identifies which concepts drove the prediction and where they were detected.Source: §3.5.3 · p. 5

3.5.4. GLOBAL SEMANTIC ATTRIBUTION

  • Global semantic attribution aggregates concept attributions across examples to identify concepts that consistently contribute to a class’s predictions.These class-level summaries complement instance-specific maps.Source: §3.5.4 · p. 5

3.6. CAV Validation Metrics

  • VCM assesses CAV–concept alignment, while CCM assesses whether a CAV predicts its associated class; together they evaluate semantic fidelity and predictive relevance.The metrics evaluate ConEx using CWVs as representations of concept directions in latent space.Source: §3.6 · p. 5

4. Experiments

The experiments evaluate ConEx across datasets, architectures, faithfulness, concept quality, and human understandability. The reported results show strong performance across these evaluations, with ablations examining its CAV construction and attribution-fusion choices.

4.1. Experimental Setup

  • Experiments span ImageNet, CUB, Stanford Dogs, and ImageNet-Segmentation, using five pretrained CNN and ViT models and saliency- and concept-based baselines.Concept discovery and grounding use GroundedSAM; occurrence-rate and spatial-coverage thresholds are 15% and 20%, respectively.Source: §4.1 · pp. 5–6

4.2.1. FAITHFULNESS

  • Faithfulness is assessed with insertion/deletion metrics and FunnyBirds completeness, correctness, and contrastivity scores using predefined object-part concepts.FunnyBirds uses 500 images across 50 classes; the authors also aggregate Visual-TCAV concepts to enable a fairer comparison.Source: §4.2.1 · p. 6

4.2.2. CONCEPT QUALITY

  • Concept quality is evaluated with CINS/CDEL and held-out VCM/CCM tests, alongside segmentation alignment to human-annotated object regions.The VCM and CCM evaluations use held-out concept segments and sampled class gradients; segmentation alignment is assessed on ImageNet-Segmentation.Source: §4.2.2 · pp. 6–7, §4.2.3 · p. 7

4.2.4. HUMAN EVALUATION PROTOCOLS

  • The understandability study asks participants to match highlighted concepts to candidate explanations, using 300 samples and 58 participants.It compares ACE, MCD, and ConEx across five randomly selected ImageNet classes.Source: §4.2.4 · p. 7

4.3.2. CONCEPT QUALITY AND CAV VALIDATION

  • ConEx achieves the highest CINS, CDEL, VCM, and CCM scores across architectures, while ViT models score lower than CNN counterparts.The authors attribute the lower ViT scores to challenges in operating on patch-level representations.Source: §4.3.2 · p. 7

4.3.3. HUMAN INTERPRETABILITY

  • ConEx scores highest on SA, PPR, and INNS, and lowest on INTS in the human understandability evaluation.The authors interpret these results as evidence that its explanations are more intuitive, consistent, and semantically distinct to users than prior methods.Source: §4.3.3 · p. 7

4.3. Results and Analysis

  • ConEx achieves state-of-the-art results on perturbation-based INS/DEL across datasets and architectures, and on the FunnyBirds part-based benchmark.The authors report that this holds despite ConEx operating at the concept level; FunnyBirds uses predefined concepts and part-based interventions.Source: §4.3.1 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.4. Qualitative Comparison and Ablation Studies

  • ConEx maps are more localized and semantically coherent than GC and FG maps, and ablations favor its centroid-based CAV construction and LRP fusion.The authors report that integrating GC, SHAP, or IIA into ConEx improves faithfulness over those methods used alone.Source: §4.4 · p. 8

5. Conclusion

ConEx is a fully automatic, concept-driven framework that combines semantic interpretability with model faithfulness. Across datasets and architectures, the authors report precise spatial grounding, high semantic fidelity, and performance that consistently surpasses existing saliency and concept-based methods.

5. Conclusion

  • ConEx combines label-free vision-language priors and zero-shot segmentation to discover, ground, and validate class-discriminative concepts without human supervision or retraining.The framework is presented as unifying semantic interpretability and model faithfulness.Source: §5 · p. 8
  • Across multiple datasets and architectures, ConEx consistently outperforms existing saliency and concept-based methods.The conclusion characterizes its results as precise in spatial grounding and high in semantic fidelity.Source: §5 · p. 9
  • The authors conclude that concepts grounded in the model’s representational space can yield explanations that are human-meaningful, faithful, and robust to spurious correlations.Source: §5 · p. 9

Impact Statement

The Impact Statement presents ConEx as a potential aid for auditing high-stakes systems while warning that biases in vision-language priors may enter its generated explanations. The appendix overview maps supplementary implementation details and analyses.

Impact Statement

  • ConEx could assist audits of high-stakes AI decisions, foster trust, and help identify harmful dataset biases or spurious correlations.The statement also warns that biases encoded in vision-language priors may transfer into explanations generated from LLM-produced attributes.Source: Impact Statement · p. 9

A. Appendix Overview

  • The appendix overview points readers to implementation specifications, concept validation, and optional negative-sample refinements.It also lists ViT localization challenges, additional faithfulness and human-evaluation results, ablations, and segmentation and FunnyBirds tests.Source: A. Appendix Overview · p. 14
  • The appendix also identifies further human-evaluation results, an ablation analysis, and additional segmentation and FunnyBirds experiments.Source: A. Appendix Overview · p. 14

B. Implementation Details

The implementation uses architecture-specific segment embeddings to construct concept vectors and generate attributions. Its evaluations report robust faithfulness across masking baselines and better CAV quality from masked ViT averaging than from the standard [CLS] approach.

CAV construction

  • Each concept vector is the difference between mean embeddings from 100 positive segments and 100 negative segments containing other concepts.Source: B. Implementation Details · p. 14

Concept map generation

  • Concept maps use globally pooled channel-weighted vectors, normalization with ϵ = 10^-8, and clipping during aggregation; LRP with ϵ = 0.01 and multiplicative fusion produce importance maps.The masking threshold τ is set to the mean value.Source: B. Implementation Details · p. 14

Masking Strategies

  • Performance differences between methods remained below 2% across masking strategies, while ConEx retained superior faithfulness scores to all baselines.The evaluated baselines included zero, Gaussian noise, uniform noise, Gaussian blur, and random training-data patches.Source: B. Implementation Details · p. 14

ViT Embeddings

  • Masked averaging of intermediate ViT patch embeddings yielded superior CAV quality to the standard [CLS] token approach.The method retains patches overlapping a segment from an unaltered image and averages them, avoiding the global [CLS] representation.Source: B. Implementation Details · p. 15

Embedding extraction

  • For ViTs, segment-restricted intermediate features preserve spatial correspondence and model feature hierarchy; CNN embeddings come from the final convolutional layer using layer-wise masking.The CNN strategy applies 7×7 neighborhood padding at the first convolutional layer to preserve semantic information while avoiding boundary artifacts.Source: B. Implementation Details · pp. 14–15

C. Concept Set Validation

Concept sets are generated from class-discriminative textual attributes, grounded with zero-shot segmentation, and filtered by occurrence and spatial coverage. The section reports threshold-ablation results, dataset-level concept statistics, and example validated concepts.

C. Concept Set Validation

  • GroundedSAM supplies masks for class-specific textual attributes, and concepts are discarded unless they meet occurrence-rate and spatial-coverage thresholds.The stated filtering thresholds are at least 15% occurrence and 20% spatial coverage; coverage measures overlap with class-specific object segments.Source: C. Concept Set Validation · p. 15

Example Discovered Concepts

Statistics for Occurrence Rate and Spatial Coverage

  • CUB yielded 348 concepts, with 39% average occurrence and 68% object-region coverage among concepts exceeding the 0.15 coverage threshold.Source: C. Concept Set Validation · p. 16
  • SD yielded 189 concepts, averaging 42% occurrence and 49% region coverage; IN yielded 3,602 concepts, averaging 31% occurrence and 37% coverage among qualifying concepts.Source: C. Concept Set Validation · p. 16

Concept Validation Thresholds Ablation

  • At 40%/40%, concepts fell from 4,602 to 1,022 and INS/DEL were 57.41/10.59; at 5%/10%, 5,835 concepts yielded INS/DEL of 56.81/11.82.The tested faithfulness performance peaked at the default 15% occurrence-rate and 20% spatial-coverage thresholds; evaluation sampled five images per class on IN with RN.Source: C. Concept Set Validation · p. 16

D.1. Providing rigorous negative sample set

The appendix describes a more rigorous negative-sampling option and a multi-label extension. The negative-sampling approach trades additional computation for reported concept-quality gains.

D.1. Providing rigorous negative sample set

  • Negative samples can be selected to have low cosine similarity to the target concept’s mean embedding, reducing visually or semantically similar confounders.Source: §D.1 · p. 16
  • This stricter negative sampling increases computation time and is reported to consistently improve concept-quality metrics.Source: §D.1 · p. 16

D.2. Extension to Multi-Label Classification.

  • The framework can generate class-wise concept explanations for multi-label predictions without architectural or procedural modifications.The pipeline is applied independently to each predicted class.Source: §D.2 · p. 16

E. Why ViT Localization is a Fundamental Challenge

ViT concept localization remains difficult because patch-token representations lack enforced spatial correspondence and current attribution methods do not separate co-occurring concepts. ConEx nevertheless provides global ViT concept attributions, while spatially localized explanations remain an open challenge.

E. Why ViT Localization is a Fundamental Challenge

  • ViT methods produce prediction-level heatmaps but lack a way to disentangle co-occurring concepts in the same image region.Patch tokenization can fragment fine-grained concepts, and self-attention mixes their embeddings, complicating precise boundary recovery.Source: E. Why ViT Localization is a… · pp. 16–17

F.1. Additional Faithfulness Results

  • Table 8’s additional SD fine-grained faithfulness results follow the same trends as the main paper’s results.The passage presents these findings as further support for the robustness of the reported results, without specifying values here.Source: Table 8 · p. 17, §F.1 · p. 17

E. Why ViT Localization is a Fundamental Challenge

  • ConEx provides global ViT concept attributions that support auditing, bias detection, and debugging, but not spatially localized explanations.The paper frames concept-level ViT localization as a broader open problem requiring faithful attention-flow decomposition, patch-to-pixel refinement, and concept-level validation.Source: E. Why ViT Localization is a… · pp. 16–17

G.1. Understandability - full description

  • The understandability study asked participants to match highlighted concepts to candidate explanations and assess recognizability and description consistency.Fifty-eight participants evaluated 300 samples; the protocol measured selection accuracy, recognizability, and within- versus across-concept description similarity.Source: §G.1 · pp. 17–18

G.2. Class Concept Sets Validation

  • 8.3 for clarity, 7.9 for meaningfulness, and 8.5 for class correspondence were the human ratings of ConEx concept sets.Ratings used a 10-point Likert scale and covered concept sets sampled from 10 IN classes.Source: §G.2 · p. 18

H. Ablation Studies

The ablations examine segment-based CAV construction, layer choice, and sample count, alongside segmentation alignment. Across these tests, the reported results favor localized segment embeddings and the selected layer and sample configurations.

H.1. CAV Construction using Segments vs. Full Images

  • 0.68 VCM and 27.83 CCM for segment-based CAVs exceeded 0.59 VCM and 24.68 CCM for full-image CAVs on average.The comparison used IN with RN, CN, and DN models; the paper attributes the gain to reducing background-related confounds.Source: §H.1 · p. 18

H.2. Model Layer Selection for CAV Segments Representation

  • The selected layer configuration performed best, and layer choice affected ViTs more than CNNs.Table 9 compares segment-representation layers for ViT-B and RN; the authors suggest deeper ViT representations differ more than those in CNNs.Source: Table 9 · p. 19, §H.2 · p. 19

H.3. Sensitivity to the number of samples used for CAV construction

  • N = 50 yielded satisfactory results, while N = 100 was chosen to balance computational efficiency and explanatory performance.N = 10 remained competitive with existing CAV methods despite not being optimal.Source: Table 10 · p. 19, §H.3 · p. 19

I.1. Segmentation Alignment.

  • The segmentation evaluation compared explanation maps with human-annotated object regions using mAP, mIoU, and pixel accuracy on IN-S.The setup included RN, DN, and CN and cautions that strong segmentation performance alone does not guarantee superior explanatory power.Source: §I.1 · p. 19
  • ConEx achieved superior alignment with human-annotated object boundaries on IN-S, despite the more constrained task of concept-decomposed explanations.The authors interpret this result as evidence that discovered concepts align with meaningful object parts.Source: §I.1 · p. 19

I.2. FunnyBirds Evaluation Metrics

FunnyBirds evaluates explanations against object parts through correctness, completeness, and contrastivity measures. The paper also notes limitations in segmentation uncertainty, ViT localization, and runtime, with future work targeting localization, broader domains, and efficiency.

I.2. FunnyBirds Evaluation Metrics

  • FunnyBirds assesses explanations at the part level across Completeness, Correctness, and Contrastivity, with an overall score averaging the three.Part Importance is attribution summed within a part, and a part is important when its score reaches the specified share of total attribution.Source: §I.2 · pp. 19–20
  • Correctness measures the correlation between part-importance scores and logit changes after individual parts are removed.The protocol uses Spearman rank-order correlation; single deletion removes one bird part from the image.Source: CORRECTNESS (COR.) · p. 20
  • Completeness averages CSDC, PC, and DC with distractability D; CSDC checks required parts, while PC checks whether preserving important parts maintains the prediction.The metrics also include removing identified important parts and checking whether irrelevant parts are highlighted.Source: COMPLETENESS (COM.) · pp. 20–21
  • Target Sensitivity compares summed class-part importance for two classes selected to have exactly two non-overlapping common parts.The protocol uses the summed importance of the parts belonging to each of the two classes.Source: CONTRASTIVITY (CON.) · p. 21
  • The FunnyBirds reporting also includes model accuracy and background independence, which measures how often removing background objects reduces the target logit by less than 5%.Accuracy is included because an overly simplified model may be explainable without effectively solving the task.Source: ACCURACY AND BACKGROUND INDEPENDENCE · p. 21

J. Limitations and Future Work

  • ConEx’s segmentation depends on GroundedSAM, which can introduce spatial uncertainty; the framework is also slower than pixel-level methods such as Grad-CAM.The paper also identifies instance-level ViT localization as a current limitation and names localization, non-visual domains, and efficiency as future directions.Source: J. Limitations and Future Work · p. 21
Loading 2610.04605v1…