PaperlayerView exact revision on arXiv

Source-linked AI summary

Mechanistic understanding and validation of large AI models with SemanticLens

Maximilian Dreyer, Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebastian Lapuschkin, Wojciech Samek

arXiv:2501.05398v1cs.LGcs.AI

TL;DR

Neural networks remain difficult to validate because their component functions and links between data, representations, and predictions are opaque. SEMANTICLENS maps component-level knowledge into a foundation model’s semantic space to support scalable search, analysis, comparison, labelling, and auditing. The framework enables component-level validation and reveals spurious behaviours, while its scope still includes limitations of post-hoc interpretability and a need for further evaluation metrics and generative-model applications.

  • Problem

    Neural-network explanations remain difficult to scale and often fail to connect learned concepts, their use in inference, and their associated training data.

    Source: §1 · p. 2

  • Method

    SEMANTICLENS embeds component-level hidden knowledge and activating examples into the semantic space of a multimodal foundation model, linking components with predictions and data.

    Source: §1 · pp. 2–3

  • Results

    SEMANTICLENS supports comprehensive audits of internal components and reveals spurious behaviours in pretrained models.

    Source: §5 · p. 17

  • Takeaways & Limitations

    Component-level semantic representations support systematic understanding and validation of AI-model internals toward fairness, safety, robustness, and reduced trust gaps.

    Source: §5 · pp. 17–18

    Further meaningful evaluation metrics, applications to generative models, and the limitations of post-hoc versus ante-hoc interpretability remain open issues.

    Source: §5 · p. 18

Abstract

from arXiv · show

Unlike human-engineered systems such as aeroplanes, where each component's role and dependencies are well understood, the inner workings of AI models remain largely opaque, hindering verifiability and undermining trust. This paper introduces SemanticLens, a universal explanation method for neural networks that maps hidden knowledge encoded by components (e.g., individual neurons) into the semantically structured, multimodal space of a foundation model such as CLIP. In this space, unique operations become possible, including (i) textual search to identify neurons encoding specific concepts, (ii) systematic analysis and comparison of model representations, (iii) automated labelling of neurons and explanation of their functional roles, and (iv) audits to validate decision-making against requirements. Fully scalable and operating without human input, SemanticLens is shown to be effective for debugging and validation, summarizing model knowledge, aligning reasoning with expectations (e.g., adherence to the ABCDE-rule in melanoma classification), and detecting components tied to spurious correlations and their associated training data. By enabling component-level understanding and validation, the proposed approach helps bridge the "trust gap" between AI models and traditional engineered systems. We provide code for SemanticLens on https://github.com/jim-berend/semanticlens and a demo on https://semanticlens.hhi-research-insights.eu.

1 Introduction

SEMANTICLENS addresses the difficulty of understanding and validating neural-network components by embedding their hidden knowledge into a semantically structured foundation-model space. This representation connects concepts, components, training data, and predictions, enabling scalable search, description, comparison, interpretability assessment, and audits.

Motivation

  • Neural networks lack semantic descriptions of individual components, making validation difficult in high-stakes settings where faulty or Clever Hans-type behaviours may go unnoticed [1] [2].Existing approaches remain difficult to scale, often depend on human intervention, and do not holistically connect data, representations, and predictions.Source: §1 · p. 2

Method

  • SEMANTICLENS embeds component-level hidden knowledge into the semantically structured multimodal space of a foundation model such as CLIP [23].It collects highly activating concept examples for each component and embeds them as a vector in the foundation model’s semantic space.Source: §1 · pp. 2–3
  • The resulting representation links model components to semantic embeddings, relevant training data, and prediction relevance scores for scalable analysis without a human-in-the-loop [24].CRP identifies components and circuits relevant to individual predictions, while activating examples identify associated data samples.Source: §1 · pp. 2–3

Capabilities

  • Textual or multimodal search can identify components and data samples encoding specified concepts, while systematic annotation describes learned, missing, and used knowledge.The framework also compares concepts across models, architectures, and training procedures.Source: §1 · p. 3
  • SEMANTICLENS audits whether encoded knowledge aligns with human-defined expectations and evaluates component interpretability through clarity, polysemanticity, and redundancy.These operations support validation of model reasoning against requirements such as the ABCDE-rule for melanoma classification.Source: §1 · p. 3

2 Related Work

Related work spans feature visualization, concept labelling, attribution, comparison, discovery, auditing, and interpretability measurement. SEMANTICLENS combines these capabilities while addressing gaps in holistic analysis, especially the use of concepts during inference and the discovery of unexpected or spurious concepts.

Feature Visualization

  • Feature-visualization methods select highly activating examples synthetically or from datasets, while SEMANTICLENS uses original test samples and CRP-based crops to reduce distracting features.Using dataset examples avoids the distributional concerns associated with synthetic concept examples.Source: §2 · pp. 3–4

Concept Labelling

  • Concept-labelling methods range from annotation-dependent activation analysis to foundation-model labelling; SEMANTICLENS operates on maximally activating examples using vision-language representations.CLIP-Dissect also uses multimodal foundation models, but SEMANTICLENS extends the representation toward broader component analysis.Source: §2 · pp. 4–5

Concept Importance Scores

  • Importance scores complement concept identification by measuring how learned concepts or components contribute to outputs, with SEMANTICLENS using CRP for singular components or groups.This distinguishes what a model encodes from how that knowledge is used in inference.Source: §2 · pp. 4–5

Concept Comparison

  • Representation-comparison methods often provide one overall alignment scalar, whereas concept-level approaches can identify shared and differing concepts across models.SEMANTICLENS bases such comparisons on concept-example embeddings.Source: §2 · pp. 4–5

Concept Discovery and Interpretability

  • Concept-discovery research highlights neuron redundancy and polysemanticity, while SEMANTICLENS focuses on the neural basis but can also apply to sparse autoencoders or factorized activations.Its interpretability measures address clarity, polysemanticity, and redundancy.Source: §2 · p. 5

Concept Audits and Explanation Frameworks

  • Existing audits detect expected concepts but may overlook other model regions, while SEMANTICLENS quantifies valid, spurious, and unexpected concepts within a more comprehensive framework.Compared with methods such as CRP, CRAFT, and CLIP-Dissect, it combines search, labelling, comparison, description, and evaluation.Source: §2 · p. 5

3 Methods

SemanticLens describes neural-network components with activating examples, embeds them into a foundation-model semantic space, and links them to predictions and data. This representation supports concept search, labelling, model comparison, alignment audits, and interpretability measures.

3.1 Describing the Role of Neurons via Concept Examples

  • Highly activating, attribution-cropped samples are retrieved from the training database to describe each neuron's role.For each neuron, CRP identifies relevant input regions; top-m maximally activating samples are selected.Source: §3.1 · pp. 5–6

3.2 Transformation into a Semantic Space

  • Each neuron's concept examples are averaged through a foundation model into one semantic vector, with 30 examples yielding converged representations in ImageNet experiments.Averaging reduces noisy background signals and better represents the overall semantic meaning.Source: §3.2 · p. 6

3.3 Concept Search, Labelling and Comparison

  • Cosine-similarity search identifies components encoding a queried concept, while predefined embeddings enable automated labels and quantitative comparisons between models.A null embedding can be subtracted to reduce background influence, and labels are withheld below a similarity threshold.Source: §3.3 · p. 6

3.4 Auditing Concept Alignment

  • Alignment audits compare component semantics with valid and spurious concepts, then incorporate inference relevance to flag potentially harmful or unexpected components.Ideally, relevant components align with valid concepts and not spurious ones; components aligned with neither warrant closer examination.Source: §3.4 · p. 7

3.5.1 Concept Clarity

  • Clarity measures how readily a component's common theme can be understood from its examples using average pairwise semantic similarity.Clarity decreases with distracting background elements or abstract concepts containing apparently unrelated elements; the alternative expression avoids large similarity matrices.Source: §3.5.1 · p. 7

3.5.2 Concept Similarity and Redundancy

  • Concept similarity and redundancy compare semantic representations across arbitrary neuron sets and architectures, but semantic redundancy does not necessarily imply functional redundancy.Similar semantics can correspond to different input features, such as zebra and striped-boarfish patterns.Source: §3.5.2 · pp. 7–8

3.5.3 Concept Polysemanticity

  • A neuron is polysemantic when subsets of its concept examples produce diverging semantic directions, with clustering used to identify those subsets.The experiments use h = 2 clusters; an alternative clarity-based measure performs worse in the user study.Source: §3.5.3 · p. 8

4 Results

The results evaluate SemanticLens for exposing, auditing, and comparing neural-network knowledge across medical debugging and interpretability studies, including identifying valid concepts, spurious correlations, model gaps, and inference circuits. They also examine how training interventions affect latent interpretability, finding differing effects for L1 sparsity regularization and dropout across model architectures.

4.1.1 Search: Finding the Needle in the Haystack

  • SEMANTICLENS retrieves neurons encoding queried concepts, including sensitive attributes, watermarks, and specific knowledge, enabling targeted detection of fairness risks and shortcut features.It uses cosine similarity between image- or text-derived probing vectors and embedded neurons; examples include “hijab,” “dark skin,” watermarks, and bioluminescence.Source: §4.1.1 · p. 9

4.1.2 Describe: What Knowledge Exists and How Is It Used?

  • Semantic embeddings, concept labels, attribution graphs, and activating samples connect what components encode to how they influence predictions, exposing knowledge gaps and spurious training correlations.For Ox, attribution reveals wheel and long-fur/grass circuits, while unlabelled neurons associate Indian person, palm tree, and watermark with affected classes and retrievable training data.Source: §4.1.2 · pp. 9–11

4.2 Audit Alignment: Do Models Reason as Expected?

  • Alignment audits identify valid and spurious concepts, quantify their component-level reliance, and test behavioral impact; combined spurious features yield an AUC of 0.91, while single-feature tests exceed 0.98.Across 26 ImageNet classes, every class uses some spurious or background features, and lower alignment can reflect abstract or polysemantic components.Source: Figure 3 · p. 12, §4.2 · p. 13

4.3.1 ABCDE-Rule for Melanoma Detection

  • For melanoma detection, SEMANTICLENS checks model knowledge against the ABCDE-rule while distinguishing expected lesion concepts from artifacts such as hairs, band-aids, red skin, rulers, and vignetting.The audit includes asymmetry, ragged border, blue-white veil, large lesion, and crusty surface, alongside benign, disease-related, and spurious concepts.Source: §4.3.1 · pp. 14–15

4.3.2 Finding Bugs in Medical Models

  • The medical audit finds valid melanoma concepts alongside neurons encoding measurement scale bars, blue band-aids, and red skin, with the artifacts materially influencing class predictions.Red skin and blue band-aids are strongly used for “other,” while measurement scale bars are slightly more relevant for “melanoma.”Source: §4.3.2 · p. 15

4.3.3 Model Correction and Evaluation

  • Retraining with artifact augmentation reduces sensitivity substantially, whereas pruning 40 of 512 penultimate-layer neurons provides only slight robustness gains; both improve clean melanoma accuracy from 71.4% to 72.8%.For poisoned samples, the pruned model’s non-melanoma accuracy still drops by over 20% when red color is added, while retraining strongly reduces artifact sensitivity.Source: Figure 4 · p. 14, §4.3.3 · p. 15

4.4 Evaluating Human-Interpretability of Model Components

  • Computable clarity, similarity, polysemanticity, and redundancy measures align with human perception above 0.74 correlation, while model architecture and training affect interpretability.ResNets show greater clarity than ViTs; wider models have more redundancy, narrow models more polysemanticity, and sparsity regularization improves interpretability overall.Source: §4.4 · p. 15, §4.4.1 · p. 16, §4.4.2 · p. 17

4.4.2 Rating and Improving Interpretability

  • Dropout makes VGG-13 components more redundant and class-specific, whereas ResNet-34 and ResNet-50 are not strongly affected.Results are based on eight runs, with standard errors shown in Fig. 5c; increased redundancy is expected to improve robustness to component pruning.Source: §4.4.2 · p. 17
  • L1 sparsity regularization improves interpretability across all measured aspects, producing more specific, less polysemantic, and semantically redundant neurons.The experiments apply L1 regularization to neuron activations during training.Source: §4.4.2 · p. 17

5 Discussion

SEMANTICLENS transfers model components into an understandable semantic representation for holistic evaluation, while exposing spurious behaviours relevant to fairness, safety, and robustness. The framework is released as an open-source Python toolbox, but its usefulness remains dependent on component interpretability and further evaluation.

5 Discussion

  • SEMANTICLENS transfers large-model components into an understandable semantic representation for holistic understanding and evaluation.The discussion frames this transfer as the basis for understanding and evaluating model inner workings.Source: §5 · p. 17
  • Its capabilities enable comprehensive audits that reveal spurious behaviours in popular pretrained models, highlighting concerns for fairness, safety, and robustness.Source: §5 · p. 17
  • Interpretation and auditing remain constrained by the interpretability of model components, especially in newer transformer architectures.Source: §5 · p. 18
  • Trust and safety require verification of internal components, motivating holistic approaches intended to quantify latent-component validity and reduce spurious behaviours.Source: §5 · p. 18
  • Further work includes meaningful evaluation metrics, generative-model applications, and limitations of post-hoc versus ante-hoc interpretability.Source: §5 · p. 18

Code Availability

  • The framework is publicly available as an open-source Python toolbox built with PyTorch, Zennit-CRP, and Zennit.Source: Code Availability · p. 18

A Extended Related Work

The related-work review positions SEMANTICLENS as a holistic, foundation-model-based framework that combines semantic component representations with search, labelling, relevance, comparison, and auditing. It also describes the experimental settings and distinguishes the method from approaches focused on isolated interpretability capabilities.

A Extended Related Work

  • SEMANTICLENS combines concept examples, labelling, relevance, comparison, discovery, audits, and interpretability measures within one component-level framework.Source: A Extended Related Work · pp. 26, 28, Table A.1 · p. 28
  • Unlike activation-pattern approaches, SEMANTICLENS embeds cropped concept examples into a semantically structured foundation-model space, while acknowledging the assumption that examples precisely define concepts.Source: A Extended Related Work · p. 26
  • SEMANTICLENS can also operate on sparse autoencoders or factorized activations and supports similarity analysis between individual concepts rather than only scalar space-level alignment.Source: A Extended Related Work · p. 27
  • Existing methods variously provide neuron labels, concept importance, local explanations, prototypes, attribution graphs, or model comparisons, but typically omit other capabilities.Source: A Extended Related Work · pp. 28–29
  • Network Dissection, Net2Vec, CLIP-Dissect, and TCAV support representation analysis or expected-concept evaluation, yet do not fully capture component-level use of unexpected concepts.Source: A Extended Related Work · p. 29
  • Related systems use CLIP or other foundation models for automated labelling, while SEMANTICLENS differs through relevance-focused cropping and averaged image embeddings.Source: A Extended Related Work · pp. 29–30

B Experimental Settings

  • The experiments evaluate pretrained models and use datasets, foundation models, and architectures described in the subsequent experimental settings.Source: B Experimental Settings · p. 30

B.1 Architectures and Models

The experiments cover convolutional, VGG, and vision-transformer architectures with multiple foundation models and datasets. ImageNet supports the main model studies, while ISIC provides a binary melanoma-versus-other medical classification setting.

B.1 Architectures and Models

  • The study evaluates ResNet18, ResNet32, ResNet50, ResNet50v2, ResNet101, ResNet101v2, additional timm ResNets, VGG variants, and three vision transformers.Source: §B.1 · p. 30
  • Because vision-transformer features can be negative or positive, concept examples are generated through both activation maximization and minimization.Source: §B.1 · pp. 30–31
  • Five foundation models are used: CLIP-OpenAI, CLIP-LAION, Mobile-CLIP, DINOv2, and WhyLesionCLIP.Source: §B.1 · p. 31

B.2.1 ImageNet

  • ImageNet-1k contains 1,281,167 training images, 50,000 validation images, and 1000 object classes, and is used throughout the ImageNet experiments.Source: §B.2.1 · p. 31
  • ImageNet inputs are resized to 256 × 256, centre-cropped to 224 × 224, and normalized with ImageNet channel statistics.Source: §B.2.1 · p. 31

B.2.2 ISIC Challenge

  • The medical experiments use 25,331 ISIC 2019 dermoscopic images spanning nine diagnostic categories, grouped into melanoma versus other for binary classification.Source: §B.2.2 · p. 31
  • ISIC inputs are resized to 224 × 224 and normalized using channel means and standard deviations of 0.5.Source: §B.2.2 · p. 31

C Search in Semantic Space

SemanticLens searches component embeddings with multimodal probes, ranks neurons using a null-adjusted similarity score, and supports concept-level description through labels and UMAP structure. The examples show differentiation among related concepts, broad representation diversity, cross-model comparison, and spurious Chinese-text structure.

C Search in Semantic Space

D.1 Concept Set

  • The annotation concept set includes vegetation, food, colours, textures, animals, animal parts, objects, object parts, and scenes.Source: §D.1 · pp. 33–35

D.2 UMAP Embeddings

  • UMAP embeds 50 maximally activating images per neuron with CLIP-Mobile-S2, clusters the embeddings, and labels each cluster using its two most aligned concepts.Source: §D.2 · p. 35, Figure D.1 · p. 36
  • Across models, semantic structure includes vegetation, animals, vehicles, and textures; ResNet50 additionally contains a Chinese-text outlier associated with a known ImageNet spurious correlation.Source: §D.2 · p. 36

D.3 Network Dissection

Network dissection labels and organizes neuron concepts, enabling comparisons of representation structure across layers and models. It also reveals hierarchical concept groupings and spurious-correlated clusters.

D.3 Network Dissection

  • UMAP embeddings organize ResNet50 representations by semantic concept families and expose an outlier cluster associated with Chinese-text spurious correlation.The embedding contains 160 clusters with top labels provided for each cluster.Source: Figure D.2 · p. 37
  • Higher-level concepts dominate later layers, whereas textures and colours are more prevalent in earlier layers.In the ResNet50v2 example, dog is assigned to 146 neurons in the last layer, followed by insect, food, and bird.Source: §D.3 · p. 37
  • Relative concept shares expose representational differences across architectures and training regimes.ResNet50v2 contains more higher-level concepts than ResNet50, while VGG-16 shows more higher-level concepts than VGG-13.Source: §D.3 · p. 37
  • The same semantic embedding analysis is also applied to ResNet50.a1 using the same concept-family colour coding and 160-cluster labelling.Source: Figure D.3 · p. 38
  • Hierarchical parent categories simplify encoded knowledge by grouping specific concepts into broader structures.For ImageNet’s Ox class, three horn-related neurons are grouped under Head, Body Parts, and Physical.Source: §D.3 · p. 38

D.4 Evaluation of Concept Labelling

The concept-labelling evaluation tests whether labels assigned by SEMANTICLENS correspond to neuron activation patterns. It compares the method with established alternatives and examines a normalized faithfulness score.

D.4 Evaluation of Concept Labelling

  • The evaluation compares SEMANTICLENS with INVERT and CLIP-Dissect using synthetic concept examples and neuron activation patterns.The study also includes ablations of prompt templates, concept-example counts, and activation pooling.Source: §D.4 · p. 38

D.4.1 Evaluation Procedure

  • A label is evaluated by whether its generated concept examples strongly activate the corresponding neuron.The procedure generates synthetic images for each candidate concept label and measures neuron responses.Source: §D.4.1 · p. 39
  • The normalized score ϕ(i, k) maps a neuron-label pair between 0 and 1, with higher values indicating more faithful labels.Scores are averaged across neuron-label pairs and reported with standard error.Source: §D.4.1 · p. 39

D.4.2 Benchmark

The benchmark evaluates neuron labels across multiple classifiers and concept datasets. SEMANTICLENS is competitive overall, but its performance depends on architecture, label-space size, and network depth.

D.4.2 Benchmark

  • Benchmark evaluation covers 200 final-layer neurons from five ImageNet classifiers and concept labels drawn from ImageNet, Broden, and Paco.Synthetic concept sets contain 40 Stable Diffusion images per label, while activation-maximizing samples use 30 examples.Source: §D.4.2 · p. 39
  • SEMANTICLENS consistently outperforms the random baseline and performs particularly well on ResNet architectures.On ImageNet, it reaches 0.774 ± 0.022 on ResNet101, closely matching CLIP-Dissect and INVERT.Source: §D.4.2 · p. 39
  • SEMANTICLENS outperforms the other methods on all investigated models except VGG19.Source: §D.4.2 · p. 39
  • The method struggles with larger label spaces such as Broden, where it often scores below CLIP-Dissect and INVERT.Earlier ResNet layers also produce artificially high random-baseline scores because their candidate labels are poorly suited to low-level neurons.Source: §D.4.2 · p. 40

D.4.3 Ablation Study

The ablations show that SEMANTICLENS depends on foundation-model and sampling choices, while the attribution graph links labelled concepts across layers to explain model decisions. Together, these analyses identify effective configurations and remaining robustness challenges.

D.4.3 Ablation Study

  • Prompt-template optimization raises SEMANTICLENS from 0.497 to 0.542 with CLIP-OpenAI, compared with an increase from 0.578 to 0.588 for CLIP-Dissect.The authors suggest further prompt optimization could improve labelling accuracy.Source: §D.4.3 · p. 42
  • Mean pooling slightly improves SEMANTICLENS, increasing its score from 0.537 to 0.542 with CLIP-Mobile-S2 and a prompt template.CLIP-Dissect changes little with pooling strategy.Source: §D.4.3 · p. 42
  • For ResNet models, SEMANTICLENS generally improves with more ActMax samples and stabilizes around 20 samples.CLIP-Dissect instead peaks near 15 samples with some foundation models but continues improving to 50 with CLIP-OpenAI.Source: §D.4.3 · p. 42
  • For VGG models, max pooling degrades SEMANTICLENS performance as sample counts increase, whereas mean pooling keeps performance relatively stable.The authors attribute this pattern partly to noisier semantic embeddings from less interpretable VGG neurons.Source: §D.4.3 · p. 43
  • SEMANTICLENS performs best with CLIP-Mobile-S2 embeddings, prompt templates, and mean pooling, while performance stabilizes above 20 concept samples.The study uses 30 concept samples by default and proposes weighted averaging as a future improvement for less interpretable architectures.Source: §D.4.3 · p. 43

D.5 Attribution Graph

  • The attribution graph propagates relevance across layers to show where concepts occur and how labelled component groups depend on one another.Unaligned components are marked with question marks, highlighting concepts absent from the expected label list.Source: §D.5 · pp. 43–44

E Compare in Semantic Space

This section uses SemanticLens to compare model knowledge in a shared semantic space and audit whether representations encode valid or spurious concepts. It also defines semantic labels for lesion analysis and specifies the sparse-model training setup.

E Compare in Semantic Space

  • SemanticLens quantitatively compares models using concept-assigned neuron counts or set similarity based on average maximal pairwise semantic-embedding similarity.The similarity measure quantifies how much of model M’s encoded semantics also appears in model N and supports comparisons across training recipes, sizes, layers, and architectures.Source: E Compare in Semantic Space · p. 45
  • ResNet50v2 learns more specific, class-linked concepts than ResNet50, whose neurons more often encode abstract concepts shared across classes.For Komondor, ResNet50 uses a mop-like neuron for both Komondor and mop, whereas ResNet50v2 learns a class-specific concept; the models also share bird-related knowledge. [74]Source: E Compare in Semantic Space · p. 44, Figure E.1 · p. 46
  • Models with similar sizes or layer depths align more closely, while cross-family alignment can exceed within-family alignment for selected feature layers.ResNet101 and ResNet50 align more than ResNet101 and ResNet34; block 3 and block 2 representations align closely, and ResNet50 can align more with VGG-16 than with ResNet34.Source: E Compare in Semantic Space · p. 46

F Audit in Semantic Space

  • ImageNet audits compare valid and spurious concept alignment for model components, exposing differences in how ResNet50v2 and VGG-16 support Ox predictions.The audit uses textual embeddings, subtracts alignment to an empty embedding, and filters low-relevance neurons before plotting component-level alignments.Source: F Audit in Semantic Space · p. 46, §F.1.1 · p. 46

E Compare in Semantic Space

  • ResNet50v2 aligns best across training recipes, with alignment correlating with test accuracy and training duration; 600-epoch models outperform shorter-trained variants in alignment.The comparison is shown in Supplementary Fig. E.2, whose top-left panel evaluates ResNet50 models with different training recipes.Source: E Compare in Semantic Space · pp. 45–46, Figure E.2 · p. 47

F.1.1 Ox Example

  • For Ox recognition, ResNet neurons reach 60.8% relevance versus 8.0% for VGG, while ResNet additionally uses copyright-watermark, Indian-person, and palm-tree concepts.Both models rely on cart- and savanna-related concepts, but the ResNet’s extra spurious concepts are consistent with its stronger palm-tree response in generated-data tests.Source: §F.1.1 · p. 47

F.1.3 Aligned Models Are More Interpretable and Performant

  • Across ImageNet models, polysemantic or abstract components reduce alignment, whereas larger, more performant models show more specialized components and greater alignment scores.VGG-16 contains components serving multiple decision-making roles, while ResNet101v2 exhibits more class-specific and monosemantic components; wider models such as ResNet50 and ResNet101 align more strongly.Source: §F.1.3 · pp. 53–54

F.2 Medical Use Case

  • The medical ISIC use case applies CLIP-based semantic embeddings to debug VGG-16 representations using 20 test-set examples and labels largely following the ABCDE-rule.The smaller example set reflects ISIC’s 2,533-sample test set, and the twentieth-most activating samples retain over 40% of neurons’ maximum activation on average.Source: §F.2 · p. 54, §F.2.1 · p. 54

F.2.1 Concept Labels

  • F.2.1 Concept Labels: Concept labels cover ABCDE features such as asymmetry, border irregularity, colour variation, diameter, and evolving-lesion indicators.The labels include descriptors ranging from irregular borders and multicoloured lesions to sizes above 6 mm and changing, crusty, or scaly surfaces.Source: §F.2.1 · pp. 54–55

Other

  • Other: The concept inventory also includes disease-associated structures and potential spurious cues such as redness, markers, bandages, rulers, reflections, and hair.It links structures to conditions including squamous cell carcinoma, basal cell carcinoma, melanoma, benign keratosis, and vascular lesions, while separately listing image artifacts and contextual cues.Source: Other · pp. 56–57

F.2.2 Training

  • F.2.2 Training: Models are trained for 300 epochs with sparsity loss, λ = 1.0, stochastic gradient descent, and learning-rate reductions after epochs 240 and 290.The initial learning rate is 0.01 and is decreased by a factor of 10 at each specified milestone.Source: §F.2.2 · p. 57

F.2.3 Evaluation

The evaluation examines artefact sensitivity after pruning or retraining and introduces semantic measures for human interpretability. User studies then assess whether these measures align with human perception.

F.2.3 Evaluation

  • Only retraining strongly reduces sensitivity to band-aid, red-skin, and ruler artefacts, whereas pruning alone has weaker effects.The comparison uses changes in test accuracy after adding artefacts to clean medical images.Source: Figure F.13 · p. 58
  • Pruning or retraining reduces output-logit magnitudes for red colour and band-aid artefacts, indicating reduced over-fitting to these spurious cues.The comparison covers vanilla, pruned, and retrained models on melanoma and other subsets.Source: Figure F.14 · p. 59

G Human-Interpretability Measures

  • The framework quantifies interpretability through clarity, polysemanticity, similarity, and redundancy after mapping concept examples into foundation-model feature space.Clarity measures common-theme coherence, polysemanticity captures multiple semantics, similarity compares concepts, and redundancy measures repeated concepts.Source: G Human-Interpretability Measures · p. 59, Figure G.1 · p. 60

G.1 User Study

  • The user studies evaluate whether computed similarity, clarity, and polysemanticity align with human judgments of concept examples.The studies use controlled tasks and filtered participant results to compare model-based measures with human perception.Source: §G.1 · p. 59

G.1.1 Similarity Measure

The user studies test semantic measures against human judgments of similarity, clarity, and polysemanticity. Results generally support the measures, while revealing that some human difficulty is better predicted by perceived similarity than by computed similarity alone.

G.1.1 Similarity Measure

  • Using mean concept embeddings produced higher correlation with perceived similarity than comparing individual example embeddings, while cosine and Euclidean distances showed no large difference.This ablation compares distance choices and aggregation strategies for concept-example sets.Source: §G.1.1 · p. 60, Figure G.4 · p. 63
  • Semantic similarity achieved over 97 % accuracy in identifying the odd-one-out, compared with 85,6 % for participants.The foundation-model comparison assumes the correct pair has the highest pairwise similarity, while participant performance may reflect rapid survey completion.Source: §G.1.1 · p. 61, Figure G.3 · p. 62
  • Computed similarity was less indicative than perceived similarity or confidence for detecting participant accuracy below 65 %, partly because low similarity within the correct pair also matters.The proposed similarity score is therefore not sufficient by itself to identify difficult human judgments.Source: §G.1.1 · p. 61, Figure G.5 · p. 64

G.1.2 Clarity Measure

  • The clarity measure is evaluated against perceived clarity and participants’ comparative choices using correlations with confidence intervals.The study presents both direct clarity judgments and differences between concept-set clarity scores.Source: §G.1.2 · pp. 61, 63

G.1.3 Polysemanticity Measure

  • The proposed polysemanticity measure generally performs significantly better than overall clarity or clarity increase after clustering for detecting human-perceived polysemantic units.The comparison uses correlation analyses against perceived polysemanticity.Source: §G.1.3 · p. 64

G.2 Optimizing Interpretability

The supplementary experiments examine how training choices affect clarity, polysemanticity, and redundancy. Effects vary by hyperparameter: sparsity generally improves interpretability, whereas task complexity and training duration produce mixed changes.

Dropout regularization

  • Dropout has little effect on ResNet interpretability, while VGG models show higher clarity and redundancy under the tested setting.The study randomly sets 40% of neurons to zero during training.Source: §G.2 · p. 64
  • VGG-13 neurons generally become more class-specific and clearer with dropout, exemplified by a neuron changing from flamingo feathers plus sunset to flamingo feathers alone.The comparison uses the most semantically similar concepts between VGG-13 models with and without dropout.Source: §G.2 · p. 65

Sparsity Loss

  • Sparsity regularization generally decreases redundancy and polysemanticity while increasing clarity, yielding improved overall interpretability.The reported scores show this pattern in almost all tested cases.Source: §G.2 · p. 65

Task complexity

  • Higher task complexity increases clarity and decreases redundancy, but also increases polysemanticity compared with a binary animal-versus-object task.The comparison uses binary and 26-class training scenarios.Source: §G.2 · p. 66

Data Augmentation

  • Data augmentation produces changes such as increased VGG-13 clarity but no generally clear interpretability trends across models.The tested augmentations include rotation, cropping, flipping, and sharpness variation.Source: §G.2 · p. 66

Number of Epochs

  • After the first 100 epochs, further training causes little large-scale change; longer training generally reduces redundancy but also reduces clarity in the ImageNet subset.The authors contrast this pattern with higher clarity observed in more extensively trained models elsewhere and suggest subset overfitting as a possible explanation.Source: §G.2 · pp. 66–67

H Method

The method embeds component-level concept examples into a foundation model’s semantic space and connects those representations to relevance scores, predictions, and data. This supports concept search, labelling, comparison, interpretability measurement, human-alignment evaluation, and auditing workflows.

H.2 Concept Relevance Scores

  • CRP computes neuron importance scores and heatmaps by propagating relevance from model outputs back through latent components and inputs.The implementation uses CRP with an LRP-based composite and supports relevance analysis for individual neurons or groups.Source: §H.2 · p. 69, §H.4 · pp. 70–71

H.3 Human-Interpretability of Concepts

  • Clarity, polysemanticity, and redundancy provide complementary measures for evaluating concept-based interpretability and training effects.Human studies compare clarity and polysemanticity measures with perceived properties of concept-example sets, while the workflows use these measures to assess whole models.Source: Figure G.9 · p. 68, §H.3 · p. 69, §H.4 · p. 73

H.1 Transformation into Semantic Space

  • For each model component, high-activating concept examples are pooled and embedded by a multimodal foundation model into one semantic vector.The resulting vectors form a searchable representation of the model’s internal knowledge.Source: §H.1 · pp. 67–68, §H.4 · p. 70

H.4 Workflows for SemanticLens

  • SemanticLens searches model components using text or image probing embeddings and assigns each component the most aligned expected concept.Expected concepts can include parent categories and valid-versus-spurious labels.Source: §H.4 · p. 70
  • Relevance scores filter concept-aligned components by prediction class or sample, enabling analysis of which concepts contribute to specific outputs and data examples.Relevance can be summarized globally or used per sample; relevance between components can also produce attribution graphs of hierarchical dependencies.Source: §H.4 · pp. 70–71
  • The workflow audits alignment by comparing expected concepts with relevant model components, including unexpected valid or spurious concepts.It also supports comparing embeddings and concept assignments across models, architectures, and training procedures.Source: §H.4 · pp. 71–73

H Method

  • The method represents model components, training examples, semantic interpretations, and predictions as mutually connected information for scalable analysis without a human in the loop.This representation underlies search, description, comparison, interpretability evaluation, and auditing.Source: §H.4 · p. 70, Figure H.1 · p. 73

I Limitations and Future Work

The method’s scope is constrained by the foundation model, the availability of meaningful concept examples, and the image-focused demonstrations. Future work includes broader interpretability measures and stronger validation of audit metrics against actual misbehaviour.

Foundation-model dependence

  • SemanticLens depends on a foundation model with sufficient domain expertise and can be misled by that model’s own biases or spurious correlations.In niche domains without a suitable foundation model, the investigated model may serve as an alternative but may not provide an optimal semantic space.Source: I Limitations and Future Work · p. 73

Concept-example scope

  • SemanticLens is ineffective when model components cannot be described through meaningful concept examples.The authors suggest sparse autoencoders, activation factorization, or interpretability-aware hyperparameter selection as possible directions.Source: I Limitations and Future Work · p. 73

Interpretability measures

  • The three latent interpretability measures do not cover all desirable explanation properties, including spatial localization and minimizing the number of distinct concepts used.Additional measures remain a potential direction for future work.Source: I Limitations and Future Work · p. 73

Data-domain scope

  • The demonstrations use image data, while application to text, audio, or video depends on suitable foundation models for those domains.The paper presents broader-domain application as future work.Source: I Limitations and Future Work · p. 74

Auditing

  • The auditing demonstrations quantify alignment to expectations, but further measures and tests against actual model misbehaviour remain future work.The authors identify correlation with actual misbehaviour as an open evaluation question.Source: I Limitations and Future Work · p. 74

Related papers

Loading 2501.05398v1…