From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation

Reduan Achtibat; Maximilian Dreyer; Ilona Eisenbraun; Sebastian Bosse; Thomas Wiegand; Wojciech Samek; Sebastian Lapuschkin

Source-linked AI summary

From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation

Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojciech Samek, Sebastian Lapuschkin

arXiv:2206.03208v2Jun 7, 2022cs.LGcs.AI

TL;DR

Existing XAI methods often show where a model focuses without revealing which concepts drive its prediction. CRP combines local and global explanations to provide localized, human-understandable concepts, yielding more detailed insights into model reasoning across datasets and architectures.

Problem
Attribution maps identify where important information occurs but not which concepts the model uses, limiting understanding of its reasoning.
Method
CRP and RelMax are post-hoc techniques that explain individual predictions through localized, human-understandable concepts by connecting local and global XAI.
Results
CRP analyses provided conceptual insights into model reasoning, identified and helped correct Clever Hans concepts, and exposed concept themes in latent filter subspaces.
Takeaways & Limitations
CRP enables more detailed, contextualized explanations that reveal both where a model relies on input information and what that information represents.
Takeaways & Limitations
Reference samples may be difficult to interpret when only a small region expresses the concept.

Abstract

from arXiv · show

The field of eXplainable Artificial Intelligence (XAI) aims to bring transparency to today's powerful but opaque deep learning models. While local XAI methods explain individual predictions in form of attribution maps, thereby identifying where important features occur (but not providing information about what they represent), global explanation techniques visualize what concepts a model has generally learned to encode. Both types of methods thus only provide partial insights and leave the burden of interpreting the model's reasoning to the user. In this work we introduce the Concept Relevance Propagation (CRP) approach, which combines the local and global perspectives and thus allows answering both the "where" and "what" questions for individual predictions. We demonstrate the capability of our method in various settings, showcasing that CRP leads to more human interpretable explanations and provides deep insights into the model's representation and reasoning through concept atlases, concept composition analyses, and quantitative investigations of concept subspaces and their role in fine-grained decision making.

1 Introduction

CRP and RelMax connect local and global XAI by explaining individual predictions through localized, human-understandable concepts, answering both where and what. As post-hoc methods applicable to almost any ML model, they support analyses of representations, concept composition, prediction relevance, and model debugging.

Motivation: Local attribution maps show where information matters but not what characteristics the model extracted or whether overlapping concepts were used.Because attribution maps superpose multiple internal decision subprocesses, they can yield unclear, imprecise, or ambiguous explanations.
Motivation: Global XAI reveals learned concepts in general but typically does not show which concept drives a specific classification or link that concept to an individual output.Existing methods that bridge local and global XAI may remain global and class-specific without explaining individual samples.
Contribution: CRP and RelMax explain individual predictions through localized, human-understandable concepts, answering both the “where” and “what” questions of model inference.The methods connect local attribution maps with global concept-level explanations while providing insights into the model’s reasoning process.
Scope: As post-hoc methods, CRP and RelMax require no extra data, model, or training-process requirements and can be applied to almost any ML model.The paper demonstrates CRP-based analyses across multiple datasets, model architectures, and application domains.
Applications: CRP-based analyses reveal concept representation and composition, quantify concepts’ roles in prediction, and enable model analysis, debugging, and interaction, including for safety-critical applications and scientific investigations.CRP and RelMax are also evaluated in a user study against traditional attribution-map approaches.

2 Methods in Brief

CRP extends Layer-wise Relevance Propagation by conditioning backward relevance flows on model-encoded concepts, producing concept-conditional explanations. RelMax complements CRP by selecting representative samples that maximize latent-feature relevance for predictions.

Concept Relevance Propagation in brief: Concept-conditional explanations use θ to specify the explained output category and selected concepts, while controlled masking operations determine relevance flow during backpropagation.For example, θ can specify the class “dog” and a concept such as “fur.”
Concept Relevance Propagation in brief: Condition sets θ may be chosen by stakeholders or configured automatically by ranking network units by relevance for the current prediction across selected layers.The paper’s automatic configuration ranks units within each layer and samples layer positions from higher, middle, or bottom model regions for illustration.
Relevance Maximization: RelMax replaces ActMax as a relevance-based criterion for selecting representative latent-feature samples that show how features support predictions across outcomes.The method visualizes hold-out image segments where a latent channel becomes maximally relevant for a prediction, such as “fur” for “dog.”
Concept Relevance Propagation in brief: CRP extends LRP by disentangling concept-specific relevance flows through conditional backpropagation, yielding concept-conditional relevance maps.LRP propagates relevance from outputs to inputs and intermediate model elements; CRP conditions this process on concepts encoded by hidden-layer components.

3 Results

CRP combines concept-conditioned localization with reference examples and concept atlases to explain both where and what drives individual predictions. Results show that these explanations support model inspection, reveal non-redundant feature roles and learned artifacts, and improve human detection of prediction-impacting artifacts.

3.1 Concept-based explanations: CRP conditional heatmaps and reference samples localized relevant regions while identifying the concepts the model used, including “red spot” and “black eyes.”This addresses the limitation of attribution maps, which indicate where the model focuses but not which feature or concept drives the decision.
3.1 Concept-based explanations: The concept atlas linked “red spot” and “black eye” to the bird’s head, “dots” to the upper body, and “red spot” to a tail-feather stripe.The atlas ranks concepts within image regions using aggregated channel-conditional relevance over super-pixels.
3.2 Feature ablation and robustness: CRP concept-composition graphs revealed how higher-layer concepts are composed from lower-layer concepts, while artifact analysis showed watermark filters aided predictions without being decisive for correct classification.The watermark case indicates that the model can rely on other non-Clever Hans features when samples lack watermarks.
3.3 Concept subspaces: Correlated filter activations did not necessarily indicate redundant information or identical function; CRP combined with RelMax exposed distinct subtleties encoded by such channels.The analyses examined channels associated with keyboard keys, round buttons and rectangular roofing shingles across classes including “laptop computer” and “remote control.”
3.4 Human evaluation: CRP explanations yielded significantly higher artifact-impact detection accuracy than all other methods, with TPR of (89.1 ± 2.4) % and TNR of (72.6 ± 3.4) %.Participants reliably detected whether predictions were affected by a border artifact; all methods showed higher TPR than TNR.
3.4 Human evaluation: Participants exposed to IG reported the highest confidence at approximately 77% despite worst primary-task performance, whereas CRP confidence was approximately 76%.These results indicate that reported confidence did not directly track measured artifact-impact detection performance.

4 Discussion

CRP combines local attribution with global concept understanding to produce more detailed, contextualized, human-understandable explanations of individual predictions. Experiments show that its relevance-based tools support abstract analysis of model reasoning and broaden the applicability of post-hoc XAI.

CRP explains both where an input matters and what the involved latent representations mean through human-understandable, relevance-based examples.By combining local and global XAI perspectives, CRP provides more detailed and contextualized explanations centered on features important to the prediction.
Users can select relevance-based examples for any output class, not only the ground-truth class, enabling systematic investigation of latent neural-network concepts.The selected examples reflect characteristics that play an important role in the prediction process.
Relevance-selected references, concept heatmaps and atlases, and concept composition graphs enabled abstract investigation of model reasoning in common datasets and end-to-end trained models.These tools provided qualitatively and quantitatively demonstrated value and helped identify Clever Hans concepts and investigate their impact.
The proposed tools and increased semantic detail in sample-specific explanations are intended to extend post-hoc XAI to novel or previously difficult models, problems, and data domains.

5 Methods

The methods introduce CRP, an LRP-based conditional attribution framework that disentangles concept-specific relevance, and RelMax, a relevance-based alternative to Activation Maximization for identifying model-useful concept examples.

Disentangling Explanations with CRP: CRP extends LRP with conditions that selectively propagate relevance through neural elements encoding specified concepts, enabling concept-conditional explanations of individual predictions.Conditions can identify output classes or latent concept neurons; binary masking implements the selection efficiently in a single backpropagation step.
Disentangling Explanations with CRP: CRP’s relevance conservation supports measuring global or region-restricted concept importance and estimating dependencies between concepts across adjacent layers in a prediction context.The approach can also be adapted beyond CNNs, including recurrent and graph neural networks.
Relevance Maximization: RelMax complements Activation Maximization by selecting samples containing concepts relevant to a prediction rather than samples producing maximal neuron activations.Its maximization target can be class-, model-, and concept-specific, yielding reference samples that depict concept facets useful during inference.
Relevance Maximization: RelMax relevance is not strictly correlated with activation because propagated downstream relevance incorporates feature interactions in current and subsequent layers.This distinction addresses the interpretability limitation that Activation Maximization may emphasize only a small region expressing a concept.

Declarations

The study used the publicly available ImageNet [112] dataset and released code, models, samples, and an open-source CRP toolbox for reproducibility. Figure images were licensed for reuse by their respective copyright holders.

Data and materials: The study used the publicly available ImageNet [112] dataset, with additional data and model details provided in Supplementary Note L.Code, models, and user-study samples are available in the CRP human-study repository.
Image licensing: The images shown in the manuscript and supplementary material were licensed for reuse and reproduction by iStock, Shutterstock, Pixabay, and Pexels.
Software availability: The authors released an open-source Python CRP toolbox based on PyTorch [99] and Zennit [6], including CRP and RelMax implementations.The repository is publicly available at the zennit-crp GitHub project.
Software environment: The experiments used Python 3.8, zennit-crp v0.6, Zennit v0.4.6, and PyTorch v1.13.1.

Authors’ Contributions … B.3.3 Hierarchical Composition of Relevant Concepts

The paper surveys local, global, and glocal XAI before introducing CRP as an architecture- and data-agnostic extension of LRP that disentangles concept-specific relevance. CRP supports class-conditional, global, local, and prediction-specific hierarchical explanations of model reasoning.

A Survey of Related Work: Local XAI explains individual samples with attribution maps, global XAI characterizes learned concepts and their interactions, and glocal XAI combines both perspectives to reduce interpretation workload.The survey covers model-agnostic surrogates and perturbation methods, gradient and backpropagation approaches, feature visualization, concept discovery, and self-explainable models.
A.3 Glocal XAI: Unlike related approaches, CRP requires no additional assumptions about model architecture, data or label sets, or latent-space structure, while providing multifaceted human-aligned explanations.This broad applicability also extends to purpose-built self-explaining model variants.
B Concept Relevance Propagation: CRP extends LRP with conditional relevance propagation, using masks to disentangle the contributions of selected concepts, neurons, or filters to a prediction.The method addresses latent attribution uninterpretability and superposition by constraining relevance flow according to output and hidden-layer conditions.
B.3 Possible Analyses with CRP: Class-conditional CRP heatmaps isolate which input features support or contradict a selected output class, while multi-conditional CRP can disentangle the relevance of individual concepts used by the network.This extends explanations from class-specific input localization to questions about the relevance of particular latent concepts.
B.3.2 Local Concept Importance: Localized CRP supports Concept Atlases that visualize the concepts models perceive and use locally for decision making.The approach uses convolutional filter channels as concept-bearing units and can propagate masked relevance to lower layers or input space.
B.3.1 Global Concept Importance: CRP measures global concept importance by masking concept channels during backward propagation; conservation ensures the resulting relevance corresponds to the selected concept combination’s impact on inference.In the example, “fur” is more important than “eye” for the dog prediction by relevance sum, although eye relevance is more spatially concentrated.
B.3.3 Hierarchical Composition of Relevant Concepts: CRP identifies the most relevant lower-layer concepts composing a higher-layer concept, yielding interactions specific to the sample, model outcome, and selected relevance conditions.This goes beyond static per-model or per-class interaction analyses [57] [76].
B.3.3 Hierarchical Composition of Relevant Concepts: The hierarchical compositions can be visualized as an ontology-like semantic structure of sub-concepts tailored to individual predictions.The paper presents this visualization in Supplementary Figure 3d of the main manuscript.

B.3.4 Concept Visualization through Reference Samples … C.2 Activation-based Reference Sample Selection

The paper develops data-based, relevance-aware concept visualization that connects where concepts appear with what they represent, while organizing localized concept evidence into interpretable atlases. It contrasts activation-based reference selection and generative feature visualization with CRP’s prediction-context relevance signals.

B.3.4 Concept Visualization through Reference Samples: CRP selects reference samples by neuron relevance for a prediction rather than activation alone, because high activation may reflect nonrepresentative or adversarial features.This targets manifestations that are actually used in the network’s inference context.
B.3.4 Concept Visualization through Reference Samples: CRP heatmaps refine reference-sample visualization by cropping to receptive fields and highlighting the input components that define a concept.In the illustrated example, orange/red regions become the definite key component after CRP localization.
B.3.5 Concept Atlases: Concept atlases combine channel-conditional relevances over superpixels to show which concepts dominate particular image regions and their regional importance ranking.The most relevant concept determines each region’s color, while subsequent concepts define texture or overlay patterns.
B.3.5 Concept Atlases: For a Green Anole prediction, CRP identified concepts for lizard legs and heads, an eye’s dark spot with bright border, stonewall, vegetation, and grainy texture.These concepts were observed in features.40 of a VGG-16 model with BatchNorm trained on ImageNet.
C Understanding Concepts in Latent Representations: Human-understandable concepts emerge in DNN neurons without explicit concept supervision, ranging from lower-layer textures and edges to higher-layer objects and abstract emotions.Multimodal neurons responding to concepts across modalities have also been reported [104] [18].
C.1 Generative Approaches to Understanding Latent Features: Generative feature visualization commonly maximizes a unit’s activation with gradient ascent, but realistic high-resolution image synthesis remains difficult despite transformation, frequency, preconditioning, and learned-data priors [79] [92].The discussion therefore focuses on data-based approaches.
C.2 Activation-based Reference Sample Selection: Activation-based reference selection ranks existing samples by channel or neuron activation, using summed or maximum activation targets that trade broad concept coverage against localized interpretability [28].Maximum activation can isolate a small region and make concepts harder to grasp, while the paper argues localization can instead be handled with subsample selection and conceptual heatmaps.

C.3 Relevance-based Reference Sample Selection … D Detailed Analysis: Explaining Concepts with Examples

The paper combines relevance-based sample selection, receptive-field cropping, and CRP heatmaps to make learned concepts more interpretable by showing which samples, regions, and features support predictions. These tools address multifaceted concepts, distractors, and imprecise localization across network architectures.

C.3 Relevance-based Reference Sample Selection: Multifaceted neurons are difficult to explain because a single image cannot easily display their potentially conflicting facets, while real-image collections make facet filtering difficult.Existing approaches either combine facets into unintelligible visualizations, show only one facet, or require clustering and other diversity mechanisms for reference selection.
C.3 Relevance-based Reference Sample Selection: RelMax selects reference samples by neuron relevance rather than activation, revealing how and for which samples the model uses a neuron in practice.Relevance-based targets can be class-, model-, and potentially concept-specific, unlike activation-based targets that primarily capture stimulation of a latent filter.
C.3 Relevance-based Reference Sample Selection: Relevance-based selection captures concepts that contribute to predictions even when they produce weaker activations, whereas activation sorting can favor highly excited but unused features.In the illustrated example, blue samples have higher activations but zero downstream output, while red samples have smaller activations yet support the remaining prediction computation.
C.4 Zooming into the Reference Sample by Using Receptive Field Information: Receptive-field cropping masks distracting content and localizes the encoded concept, while providing an upper bound on its true input scale.This approach uses LRP-based receptive-field computation and can restrict reference samples to the pixels contributing to a selected intermediate neuron.
C.5 Finding the Relevant Concepts within the Receptive Field by using CRP: CRP resolves relevant input features within receptive fields through precise heatmaps, overcoming blurry channel up-sampling and applying to convolutional and dense neurons.By conditioning relevance propagation on a channel, neuron, or class output, CRP can identify where a concept activates or where it is used for a specific class prediction.
C.5 Finding the Relevant Concepts within the Receptive Field by using CRP: CRP-based cropping and heatmaps reveal the model’s actual concept focus when full reference images contain salient distractors.Examples expose stripes on a bird’s breast and hair-like sparks on a drilling machine, rather than visually striking but irrelevant regions.
D Detailed Analysis: Explaining Concepts with Examples: The detailed analysis contrasts relevance- and activation-based reference examples and then examines concept understanding through class-conditional relevance selection.This section presents qualitative support for understanding predictions with CRP-based explanations on popular real-world image datasets.

D.1 Activation- vs. Relevance-based Sample Selection … E.1 From Conditional Heatmaps to Concept Atlases

The analyses show that relevance-based sampling more reliably exposes how concepts function in context than activation-based sampling, while conditional reference sets and concept atlases further clarify class-specific and spatially localized model reasoning.

D.1 Activation- vs. Relevance-based Sample Selection: Activation samples show strongest feature manifestations, whereas relevance samples reveal concepts actually used for classification, often producing qualitatively different interpretations.For a polysemantic VGG-16 channel, activation emphasized colorful cars and coarse patterns, while relevance highlighted gray-colored reptile-related examples and classification usefulness.
D.1 Activation- vs. Relevance-based Sample Selection: Relevance Maximization produces reference sets that are more robust than Activation Maximization, whose examples depend strongly on the maximization target and concept size.Activation-based selection can yield spurious edge cases when spatial aggregation mismatches the concept’s extent, whereas relevance-based selection is more stable across computation modalities.
D.2 Handling High and Low Sample Set Variety: Reference-set variety must balance abstraction and interpretability: greater diversity can reveal broader concepts, but excessive variety can obscure their common factor.The paper discusses decreasing variety through output-class-specific selection and increasing it by extending the reference dataset when initially selected examples are nearly identical.
Class-Conditional Reference Sample Selection: Class-conditioned relevance retrieval reveals how a concept supports different prediction targets, enabling target-specific reference examples even without ground-truth labels.CRP scores can be computed with respect to selected output categories, then used to rank classes by their use of the observed concept and retrieve corresponding examples.
Class-Conditional Reference Sample Selection: Class-conditioned examples clarify ambiguous concepts by separating their uses across classes, such as red features for strawberries, ambulances, and hens or thin lines in trousers and bags.For filter 209, jointly viewing examples from its top benefiting classes exposed a regular dot-like texture; Fashion-MNIST conditioning similarly separated concept uses.
E Detailed Analysis: Concept Atlas and Concept Composition: Concept atlases address attribution maps’ limitation that they locate informative regions without identifying the concepts represented there.This motivates combining channel-conditional heatmaps with masked reference samples to connect local evidence to globally interpretable channel concepts.
E.1 From Conditional Heatmaps to Concept Atlases: CRP concept atlases link prediction evidence to interpretable concepts across diverse inputs, including person, skin, hair, and water channels for “swimming trunk” and vegetation-related concepts for a maze.The maze example also shows that additional concepts may be needed when the standard attribution map highlights image regions not covered by the top-ranked concepts.
E.1 From Conditional Heatmaps to Concept Atlases: For Fashion-MNIST, conditional heatmaps localize distinct pullover concepts to spatial parts, including the right side, left sleeve, bottom, shoulder, and lower region.Channels 4 and 13 activate on the right side, channel 2 on the left sleeve, channel 14 on the bottom, and channel 11 on the shoulder and bottom.

E.2 Localized Analysis of Model Predictions … F.1 Computational Workflow

CRP supports localized analysis, hierarchical concept-composition tracing, and glocal explanations, while exposing interpretability limitations and a workflow that separates model preprocessing from instance-specific computation.

E.2 Localized Analysis of Model Predictions: CRP aggregates relevance within selected regions to identify and compare their most relevant channels and concepts, complementing top-down concept atlases with localized model analysis.The approach is demonstrated on regions containing stork beaks and wings, bird bodies and faces, and cat heads and whiskers.
E.3 Understanding Hierarchical Concept Composition: Attribution graphs trace relevance from a selected concept to lower-layer concepts, showing how abstract concepts arise from low-level patterns and compose for a specific prediction.Independent additional backward passes over important filters can disentangle the hierarchical contribution of lower-level concepts.
E.3 Understanding Hierarchical Concept Composition: A single backward pass cannot isolate a lower-layer concept’s sole contribution, because relevance below the immediately preceding layer is sum-aggregated.The proposed iterative masking procedure addresses this limitation by conditioning successive backward passes on selected concepts at each layer.
E.4 Challenging to Interpret Examples: Glocal CRP explanations can be difficult to interpret because concepts may be redundant, abstract or poly-semantic, poorly visualized, or ambiguously localized.Examples include lamp shades, koalas, and Gray Catbirds whose relevant channels are not clearly associated with recognizable features.
E.4 Challenging to Interpret Examples: Interpretability can improve when the reference-image dataset is expanded, since limited reference-sample variety contributes to unclear concepts in some explanations.This limitation is explicitly identified for the Gray Catbird example, whose references consist only of birds.
F Walkthrough Demonstration of CRP-based Glocal Explanation Computation and Interpretation: The glocal CRP and RelMax workflow covers concept-atlas construction, region-specific concept extraction, and step-by-step explanation computation and interpretation.The workflow is introduced as a computational and interpretive walkthrough of the explanation process.
F.1 Computational Workflow: Preprocessing usually takes minutes once per trained model, whereas post-hoc instance analysis takes seconds; exact runtimes are reported in Supplementary Table 2.Preprocessing includes finding reference samples and optionally computing masked CRP heatmaps, with optional heatmap preparation costing O(c · r).
F.1 Computational Workflow: For an analyzed sample, CRP performs class-conditioned inference and relevance propagation, selects the N most relevant concepts, localizes the k strongest concepts, and builds a region-based Concept Atlas.Input space can be partitioned into super-pixels or time windows, while multi-GPU MapReduce can reduce preprocessing time up to linearly.

F.2 Interpretation Workflow

The interpretation workflow combines normalized heatmaps, Concept Atlases, and region-guided local analyses to explain both where concepts occur and how they contribute to a prediction. Users inspect concept relevance, reference examples, spatial prevalence, and locally aggregated rankings to interpret model behavior.

How to Read Heatmap Visualizations: Attributions are normalized to [-1, 1], where red denotes positive relevance, blue negative relevance, and white near-zero or irrelevant attribution.Normalization uses the maximum absolute attribution across all representation-layer dimensions; the bwr colormap maps the resulting range to colors.
How to Read a Concept Atlas: A Concept Atlas walkthrough localizes the most relevant channel, interprets its reference samples, checks its spatial prevalence, and then compares the next concept.For channel 281, conditional heatmaps highlight the cat’s head and body, masked references suggest cat-like features, and the atlas shows highest relevance around the cat.
How to Read Localized Explanations: Localized explanations let users select an image region, aggregate CRP relevance across concepts, rank them, and inspect their reference samples and conditional heatmaps.This bottom-up analysis complements global Concept Atlas views and uses the conservation principle of LRP to obtain each concept’s regional importance.

G Additional Details: Human Evaluation Study · H Detailed Analysis: Clever Hans Detection and Quantitative Results

The human evaluation tested whether explanations reveal a black-border shortcut, with CRP enabling the most accurate and F1-based detection. The accompanying analysis extends CRP from qualitative explanation toward understanding and questioning model decisions through concept-specific evidence.

G Additional Details: Human Evaluation Study: The study trained two ImageNet VGG-16 models: one learned a black-border shortcut from a random subset of 30 classes, while the other saw the border for every class.The artifact was a thick black border around training images, designed either to incentivize class-specific shortcut learning or lessen its impact by making it class-independent.
G Additional Details: Human Evaluation Study: Participants assessed whether predictions were influenced by the border in a between-subjects Amazon Mechanical Turk study with 125 people, 25 per method, viewing 14 randomly selected samples.Each participant evaluated explanations from only one compared method after receiving instructions on how to interpret them.
G Additional Details: Human Evaluation Study: The evaluation compared CRP and RelMax explanations against Integrated Gradients (IG), SHAP [77], Grad-CAM, and LRP [65].IG, Grad-CAM, and SHAP were computed with Captum [66], while LRP used zennit [6] with the LRPε−z+−♭-rule.
G Additional Details: Human Evaluation Study: CRP produced the highest human-study accuracy, (80.9 ± 3.4) %, and F1-score, (82.3 ± 1.8) %, significantly outperforming the other methods.Grad-CAM was the strongest local baseline with (63.7±3.4) % accuracy and (67.4±2.3) % F1-score, whereas IG achieved (51.7±1.9) % accuracy and (52.7±2.9) % F1-score; random guessing corresponds to 50 %.
G Additional Details: Human Evaluation Study: Although CRP led accuracy and F1-score, IG produced the highest decision confidence and LRP the highest explanation clarity.These findings qualify the task-specific advantage of CRP: it was strongest for detecting artifact impact, while other methods led different subjective evaluation criteria.
G Additional Details: Human Evaluation Study: Example explanations show attribution shifts toward the image border when the artifact strongly affects the prediction, with LRP and Grad-CAM visibly concentrating relevance there.The examples used a 9-pixel border on unchanged 224 px×224 px images and included samples with and without strong Clever Hans [70] impact.
H Detailed Analysis: Clever Hans Detection and Quantitative Results: The H analysis builds on CRP’s ability to explain predictions through involved concepts, decompose them into concept-specific explanations, and visualize models and datasets.This section moves from qualitative demonstrations toward quantitative analysis of how explanations can be used to inspect model reasoning.
H Detailed Analysis: Clever Hans Detection and Quantitative Results: Section H uses CRP and example-based explanations to support interaction with models by second-guessing decisions according to the features they used.The analysis begins by asking how many concepts must be observed and understood to comprehend a decision among the hundreds of potentially available filters.

H.1 Quantifying the Foundation of Decision Making … H.4 Assessing Concept Similarity in Latent Space

Across these sections, CRP quantifies how concentrated model decisions are, tests concept causality through latent-space manipulation, exposes Clever Hans artifacts, and groups related concepts to reveal both shared and class-specific representations.

H.1 Quantifying the Foundation of Decision Making: Deactivating filters in descending relevance shows that 58, 74, or 100 filters capture 85%, 90%, or 95% of cumulative relevance, while 14, 16, or 20 filters reduce confidence by 85%, 90%, or 95%.Approximately 300 of 512 filters in VGG-16 features.28 are not involved in predictions on average.
H.2 Investigating the Impact of (Clever Hans) Concepts: Replacing the band-aid concept with skin decreases class-1 output, reversing the effect observed when the band-aid concept is inserted.The replacement is performed in higher-level features.28 using spatially averaged skin activations.
H.2 Investigating the Impact of (Clever Hans) Concepts: In an ImageNet swimming-trunks example, CRP locates artifact channels whose progressive deactivation changes model confidence, showing how Clever Hans features contribute to recognition.The identified artifacts are obtained by locally analyzing the surfer’s skin.
H.3 Explanation-based Image Retrieval for Clever Hans Assessment Beyond Class Boundaries: For ImageNet’s “safe” class, the 20 most relevant watermark filters aid prediction but are not decisive for correct classification.The watermark is localized in the input attribution map, and filters 454, 361, 203, 414, and 486 are identified in features.30.
H.3 Explanation-based Image Retrieval for Clever Hans Assessment Beyond Class Boundaries: Inverse search reveals that filter 361’s watermark artifact contaminates multiple ImageNet classes, indicating a shared Clever Hans feature spanning class boundaries.CRP traces the filters’ relevance for the true classes back to the character-based artifact in input space.
H.4 Assessing Concept Similarity in Latent Space: Clustering activation-based channel similarities with t-SNE reveals semantic neighborhoods, but correlated filters are not necessarily redundant or used for the same purpose.Examples include transitions from cat faces and fur to big-eyed animals, and channels sharing stimuli while receiving class-specific CRP attributions.

I Additional Experiment: Towards XAI for Fairness in ML

CRP and RelMax expose systematic demographic and appearance-related biases in a VGG-16 male-versus-female classifier trained on Adience. The explanations reveal how clothing and hairstyle stereotypes can drive discriminatory predictions and help developers identify dataset and model biases.

Motivation and capability: CRP and RelMax revealed model shortcuts exploiting artifacts at a conceptual, human-understandable level, as quantified through a user study.The techniques expose reasoning processes and shortcuts that may otherwise remain hidden.
Fairness applications: These explanations target real-world, high-stakes applications by exposing discriminatory behavior against demographic groups where biased predictions can harm individuals.The section emphasizes ethical and judicial settings governed by requirements for moral and objective decisions [48].
Clothing-related bias: A VGG-16 model fine-tuned on Adience for male-versus-female face classification associated shirt and suit collars with the “male” outcome.A correctly predicted “female” sample showed negative relevance concentrated around clothing collars, indicating that the model treated them as non-female features.
Hairstyle-related bias: The same model misclassified a male face with long hair as “female,” exposing a hairstyle stereotype in its learned decision process.The explanation provides developers with detailed information about systematic dataset biases learned and used by the model.

J Additional Experiment: Concept Analysis on Time Series Data · K Challenges and Future Work

CRP extends beyond images when data can be visualized interpretably, as demonstrated on ECG arrhythmia time series where concept analyses reveal model-relevant waveform patterns. Despite more informative explanations than attribution-only counterparts, challenges remain for future work, including user choices and algorithmic limitations.

J Additional Experiment: Concept Analysis on Time Series Data: CRP applies to non-image domains when the data can be meaningfully visualized in an interpretable way.
J Additional Experiment: Concept Analysis on Time Series Data: The ECG use case applies CRP to a CNN for arrhythmia detection, using domain-appropriate representations to verify inference concepts and support potential scientific discovery.
J Additional Experiment: Concept Analysis on Time Series Data: The attribution graph shows a higher-layer filter composed of concepts for the upper decline and low-voltage valley, while channel 41 detects the high-amplitude-to-valley pattern relevant to PVC classification.
J Additional Experiment: Concept Analysis on Time Series Data: The experiment demonstrates CRP’s applicability to non-image data and takes initial steps toward testing model plausibility and enabling scientific discovery.
J Additional Experiment: Concept Analysis on Time Series Data: Time dependency between beats remains a limitation because it is not apparent in the shown visualization.
J Additional Experiment: Concept Analysis on Time Series Data: For a detected premature ventricular contraction, the model focuses on a strong decline followed by a rebound, corresponding to clinically meaningful waveform structure.
K Challenges and Future Work: Although CRP- and RelMax-based explanations are more informative than attribution-only counterparts, future work must address user-dependent choices and additional algorithmic difficulties.

K.1 Interpreting Reference Samples … L.1 Datasets and Models

The supplementary sections discuss interpretability limits, user workload, and methodological choices in CRP analyses, then document the datasets, models, and training setups used across experiments. They emphasize that concept clarity depends on architecture and training, while CRP and RelMax remain broadly applicable without imposing training or architectural constraints.

K.1 Interpreting Reference Samples: Reference samples remain user-interpreted, and ActMax can suffer from task-independent activation scores or activations overshooting useful levels; RelMax addresses some of these shortcomings.Concept Whitening [28] and MILAN [55] are discussed as complementary approaches for improving concept clarity or providing more objective language descriptors.
K.1 Interpreting Reference Samples: Concept interpretability varies with architecture, training procedure, dataset size, and layer; concepts are typically clearer in more accurate and VGG-like models than ResNets.Different models may encode similar concepts with different numbers of neurons, increasing robustness and redundancy.
K.1 Interpreting Reference Samples: CRP and RelMax are broadly applicable out-of-the-box, without constraining training procedures, data or label availability, or model architectures.This compatibility allows concept-enhancing methods to be integrated into the CRP framework.
K.2 Degrees of Freedom in Glocal Analyses and the Effects User Workload: Glocal XAI can overwhelm users because it exposes exponentially more parameters and information than traditional local explanations.CRP provides richer model understanding across layers, but its many layers and neurons contributed to relatively low clarity scores in the user study.
K.2 Degrees of Freedom in Glocal Analyses and the Effects User Workload: A practical starting point is to inspect higher-level layers, then follow concept composition hierarchically because these layers contain more specialized, class-specific, and abstract concepts.The proposed hierarchy was introduced in the work but not explored to its full extent.
K.2 Degrees of Freedom in Glocal Analyses and the Effects User Workload: RelMax measures concept utility alongside ActMax’s feature stimulation, while quantitative explanation-quality measures could help tune glocal explanations and reduce user workload.Selecting attribution variants and concept-level analysis parameters remains a central degree of freedom, and explanation-quality measurement is still difficult and under-specified [3].
L.1 Datasets and Models: The experiments use dataset-specific preprocessing, augmentation, optimization, and evaluation protocols, including ImageNet normalization, augmentation for birds, and held-out evaluation for ISIC 2019.The ISIC 2019 model achieved a final test accuracy of 82.15%.

L.2 Implementation Details for Concept Relevance Propagation

CRP computes attribution scores with a composite LRP rule after model canonization, initializes relevance from selected output activations, and optionally applies per-layer normalization for comparing concepts across datasets. The implementation uses zennit with PyTorch, while normalization improves representation of external data but removes model-confidence filtering when labels are unavailable.

Distribution Rules: CRP uses the recommended composite LRPε−z+−♭-rule [65] for ResNets and VGG-16 after model canonization [50] [90].The implementation is provided through zennit [6] for PyTorch [98].
Distribution Rules: The ε-rule stabilizes relevance redistribution, assigns each neuron attribution proportional to its contribution, and produces maps that differ from input-gradient multiplication [65] [90].In ReLU networks, the two methods are highly similar but not identical.
Initialization: Relevance is initialized with the selected output activation, making intermediate relevances proportional to prediction confidence; initializing multiple outputs can yield unclear explanations.RelMax therefore favors high-confidence samples when ranking filter concepts.
Canonization and Practical Implementation: Canonization creates functionally equivalent models suited to rule-based attribution, while the zennit-based CRP and RelMax package modifies backpropagated gradients into attribution values.Canonization commonly merges BatchNorm layers or handles skip connections [50].
Per-layer Relevance Normalization: Per-layer relevance normalization bounds attributions to [−1, 1] and enables class-specific concept comparisons across original and external datasets.Without normalization, RelMax favors high-confidence samples; with it, external ImageNet samples appear more often among references, but confidence-based filtering is lost without matching labels.