PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks

Tuomas Oikarinen, Tsui-Wei Weng

arXiv:2204.10965v5cs.CVcs.AIcs.LG

TL;DR

Understanding individual neurons is important for interpreting deep vision networks, but existing labeling methods rely on restricted concepts, dense annotations, or human examples. CLIP-Dissect uses multimodal representations to assign open-ended concepts to neurons, achieving stronger labeling results and substantially greater efficiency while retaining important scope limitations.

  • Problem

    Existing neuron-labeling methods are limited by predefined concepts, densely annotated data, or costly human examples, making flexible automated interpretation difficult.

  • Method

    CLIP-Dissect computes neuron activation patterns on arbitrary probing images and matches them to open-ended concepts represented by a multimodal vision-language model.

  • Results

    CLIP-Dissect matches or outperforms prior automated labeling methods qualitatively and quantitatively, while achieving 10×-200× greater computational efficiency.

  • Takeaways & Limitations

    The framework flexibly labels hidden-layer neurons, can detect concepts absent from the probing dataset, and supports future replacement with stronger multimodal models.

  • Takeaways & Limitations

    The method does not use spatial activation information, may struggle with lower-level concepts, depends on concepts and images that CLIP handles well, and cannot fully capture complicated neuron functions.

Abstract

from arXiv · show

In this paper, we propose CLIP-Dissect, a new technique to automatically describe the function of individual hidden neurons inside vision networks. CLIP-Dissect leverages recent advances in multimodal vision/language models to label internal neurons with open-ended concepts without the need for any labeled data or human examples. We show that CLIP-Dissect provides more accurate descriptions than existing methods for last layer neurons where the ground-truth is available as well as qualitatively good descriptions for hidden layer neurons. In addition, our method is very flexible: it is model agnostic, can easily handle new concepts and can be extended to take advantage of better multimodal models in the future. Finally CLIP-Dissect is computationally efficient and can label all neurons from five layers of ResNet-50 in just 4 minutes, which is more than 10 times faster than existing methods. Our code is available at https://github.com/Trustworthy-ML-Lab/CLIP-dissect. Finally, crowdsourced user study results are available at Appendix B to further support the effectiveness of our method.

1 INTRODUCTION

CLIP-Dissect addresses the difficulty of understanding individual neuron functions by using multimodal models to generate open-ended descriptions without labeled data. It is presented as more accurate than prior automated methods for final-layer neurons and substantially more efficient.

  • Motivation and contribution: Manual neuron inspection can provide high-quality explanations but requires substantial human effort, motivating automated alternatives.Existing automated approaches also impose restrictions through predefined concepts or densely annotated data.
  • Motivation and contribution: CLIP-Dissect automatically labels individual neurons with open-ended concepts without requiring concept-labeled data.It leverages pre-trained multimodal models to identify neuron functionality.
  • Motivation and contribution: CLIP-Dissect provides higher-quality descriptions for hidden neurons and more accurate labels for final-layer neurons with known ground truth.The method is also described as computationally more efficient than existing methods.
  • Illustrative comparison: Figure 1 compares CLIP-Dissect, NetDissect, and MILAN on randomly selected ResNet-50 neurons using their five most highly activating images.Descriptions are colored green for matching, yellow for overly generic matches, and red for mismatches.
  • Motivation and contribution: 10×-200× greater computational efficiency is reported for CLIP-Dissect relative to existing methods.The introduction also reports that highly connected neurons usually represent similar concepts.

2 BACKGROUND AND RELATED WORK

Prior neuron-labeling methods compare activation patterns with predefined concepts or generate descriptions from highly activating images. These approaches remain constrained by annotated data, fixed concept sets, or costly human supervision, whereas CLIP uses language-image representations that support flexible labels.

  • Network Dissection: Network Dissection matches neuron activation masks against pixel-level masks for predefined concepts in the densely annotated Broden dataset.A neuron is assigned a concept when its intersection-over-union score exceeds a threshold.
  • Network Dissection: Network Dissection is constrained by dense annotations and a fixed concept set, even when segmentation models replace human labels.The replacement still requires dense labels to train the segmentation model.
  • MILAN: MILAN generates unrestricted neuron descriptions by training an image-to-text model on human annotations of highly activating images.Its formulation treats neuron labeling as captioning the images that most strongly activate a neuron.
  • CLIP: CLIP learns visual representations from natural-language supervision by jointly training image and text encoders to align paired images and texts.After training, it can perform zero-shot classification for arbitrary supplied label names.

3 METHOD

CLIP-Dissect labels neurons by relating their activation patterns on arbitrary probing images to concept representations from a multimodal model. Its three-step pipeline computes concept activations, records neuron responses, and selects the most similar concept using a tunable similarity function.

  • Inputs and outputs: CLIP-Dissect accepts a probed network, an image dataset, and a concept set, then outputs one concept label for each neuron.The probing dataset need not have concept labels, and the image dataset and concept set can be chosen independently.
  • Inputs and outputs: The method separates probing images from concepts, allowing datasets such as CIFAR-100, ImageNet, combined datasets, or unlabeled internet images.This decoupling increases flexibility and efficiency compared with Network Dissection's Broden-based setup.
  • Three-step algorithm: Step 1 computes a concept-activation matrix from CLIP image and text embeddings, while Step 2 records each target neuron's summarized activation across probing images.The default summary function is the spatial mean of each neuron's activation map.
  • Three-step algorithm: Step 3 assigns each neuron the concept maximizing similarity between its activation vector and the concept-activation matrix.The neuron activation vector contains one summarized response for every probing image.
  • Similarity function: Cosine similarity performs poorly because it can place excessive weight on images that do not strongly activate the target neuron.The method therefore compares multiple similarity designs rather than relying on cosine similarity alone.
  • Similarity function: SoftWPMI relaxes WPMI's binary membership of highly activating images by assigning each probing image a probability between 0 and 1.The authors report that SoftWPMI performed best among the four compared similarity functions and used it in experiments.
  • Extensibility: CLIP-Dissect can directly replace CLIP with improved CLIP-like models without changing the algorithm.This compatibility supports future use of stronger multimodal models with similar structure.

4 EXPERIMENTS

Experiments evaluate CLIP-Dissect qualitatively and quantitatively on ResNet-50 and ResNet-18, including label quality, similarity functions, efficiency, and concepts absent from probe images. Across these tests, it produces strong neuron descriptions, achieves high final-layer accuracy, and is substantially faster than existing methods.

  • Evaluation setup: Experiments evaluate CLIP-Dissect on ResNet-50 trained on ImageNet and ResNet-18 trained on Places365 using qualitative and quantitative analyses.The evaluation includes similarity-function ablations, computational-efficiency comparisons, and tests for concepts absent from probing images.
  • Qualitative results: CLIP-Dissect provides more descriptive hidden-neuron labels than existing methods, while Network Dissection misses concepts absent from its dataset and MILAN sometimes produces generic or incorrect descriptions.The qualitative comparison covers randomly chosen neurons across different layers and uses the union of ImageNet validation images and Broden as Dprobe.
  • Quantitative results: CLIP-Dissect labels are closer to ground truth in sentence-embedding space than Network Dissection or MILAN across probe datasets and concept sets.Larger concept sets improve performance, and on Places365 CLIP-Dissect outperforms Network Dissection despite not using Broden’s ground-truth labels.
  • Quantitative results: 95% accuracy is achieved for final-layer neurons when CLIP-Dissect uses ImageNet labels as its concept set.The accuracy metric counts exact class-label matches, and is reported where the exact correct label is available as a candidate.
  • Similarity function: SoftWPMI performs best among the evaluated similarity functions and is used in the remaining experiments.The comparison uses cosine similarity in embedding space and accuracy to assess description quality.
  • Computational efficiency: 4 minutes are sufficient to label five ResNet-50 layers, making CLIP-Dissect more than 10, 60, and 200+ times faster than MILAN, Network Dissection, and Compositional Explanations.The runtime comparison uses the authors’ Tesla P100 GPU hardware.
  • Concepts absent from probe images: CLIP-Dissect correctly identifies dog breeds and bird species even when CIFAR100 probe images contain neither dogs nor birds.Using CIFAR100 as Dprobe, it correctly labels 46 of 118 dog-breed neurons and 22 of 59 bird-species neurons.

5 USE CASE OF CLIP-DISSECT

A use-case analysis of ResNet-50 shows that neuron connectivity can reveal relationships between encoded concepts. Neurons linked by higher weights tend to represent more similar concepts, and some penultimate-layer neurons already encode complete final-layer classes.

  • Concept similarity and weights: Higher weights between neurons are associated with more similar concepts in the ResNet-50 final layer.The relationship is measured by comparing CLIP-Dissect concepts for layer4 neurons with ground-truth class labels for final-layer neurons.
  • Penultimate-layer representations: The three neurons with the highest outgoing weights can already detect the final-layer class concept they connect to.This suggests that some final-layer class representations are already substantially encoded in individual penultimate-layer neurons.
  • Concepts absent from probe images: CLIP-Dissect can identify little blue heron and great white heron neurons from CIFAR images of dolphins and dinosaurs.CIFAR100 contains no bird images, yet the method recovers the bird concepts.
  • Concept similarity and weights: The average concept similarity decreases exponentially with the rank k of connected weights.Figure 5b quantifies similarity among neurons connected through the highest-weight final-layer connections.

6 LIMITATIONS AND CONCLUSIONS

CLIP-Dissect is a flexible, computationally efficient framework for automatically identifying hidden-neuron concepts, but it has limitations in spatial awareness, domain coverage, and describing complex functions.

  • Limitations: CLIP-Dissect does not use spatial information from neuron activations, which makes detecting lower-level concepts more difficult.The method can nevertheless detect many low-level and localized patterns.
  • Limitations: The method works best on concepts and images that CLIP handles well, limiting out-of-the-box use on highly specialized domains such as astronomical imagery.It remains compatible with future CLIP-like models and models trained for specific target domains.
  • Limitations: Some neurons have complicated or multi-function behavior that cannot be captured adequately by simple word-based descriptions.The authors suggest expanding the concept set or using compositional explanations, while noting that current methods may still miss the full function.
  • Conclusions: The paper develops a flexible and computationally efficient framework for automatically identifying concepts in hidden-layer neurons.It also introduces quantitative comparisons for neuron labeling and reports that CLIP-Dissect can match or outperform prior automated methods.
  • Conclusions: CLIP-Dissect can detect concepts absent from the probing dataset, and highly connected neurons often represent similar concepts.These findings extend the method beyond direct probing-set concept matching.

A.1 SIMILARITY FUNCTION DETAILS AND DERIVATION

The similarity-function derivation converts CLIP concept probabilities into WPMI-based scores and then softens neuron-example membership to handle uncertainty about highly activating images.

  • WPMI construction: WPMI is computed from CLIP products without training additional models, using concept probabilities derived from the concept-activation matrix.The method applies softmax to CLIP image–concept products, with a scalar temperature controlling the probabilities.
  • WPMI construction: For neuron k, B_k contains the images that most highly activate that neuron, and p(t_m|B_k) is modeled as the product of concept probabilities across those images.The background probability p(t_m) is approximated by averaging over the neurons in the probed layer.
  • SoftWPMI: SoftWPMI replaces hard inclusion in B_k with probabilistic membership, allowing uncertainty over which probing images belong to the neuron’s example set.When membership probabilities are 0 or 1, the formulation reduces to standard WPMI.
  • Implementation: The resulting SoftWPMI score is used as the CLIP-Dissect similarity function, with experiments also using fixed hyperparameters and top-28 activating images for final-layer evaluation.The stated settings include a=2 and λ=0.6 for the final-layer experiments, while the ranking-based SoftWPMI setup uses a=10 and λ=1.
  • SoftWPMI: The probability p(x∈B_k) can be based on activations or rankings, and the experiments use a ranking-based probability because it performed best.The implementation uses probabilities decreasing from 0.998 for the highest-activating image to 0.97 for the 100th, and zero thereafter.

A.2 ADDITIONAL QUALITATIVE RESULTS

Additional qualitative examples compare CLIP-Dissect with Network Dissection and MILAN on highly interpretable neurons in ResNet-18 and ResNet-50.

  • Additional qualitative results: The figures show explanations for highly interpretable neurons in the second-to-last layers of ResNet-18 trained on Places365 and ResNet-50 trained on ImageNet.Each example is displayed with the five images that most strongly activate the neuron.
  • Additional qualitative results: On ResNet-18, Network Dissection and CLIP-Dissect perform well, while MILAN struggles with some neurons.MILAN is also shown in a version trained on both ImageNet and Places365.
  • Additional qualitative results: On ResNet-50, CLIP-Dissect and Network Dissection generally perform well, with some failures by Network Dissection and frequent difficulties for MILAN.The comparison uses the five most highly activating images for each neuron.

A.3 LOW LEVEL CONCEPTS

Low-level and compositional examples show that CLIP-Dissect can identify many localized concepts and combine concepts, but it also confuses correlated objects and misses some background or uninterpretable patterns.

  • Low-level concepts: CLIP-Dissect accurately detects many low-level concepts, including colors and a text-specific activation, in early ResNet-152 layers.For one neuron, it identifies the text or label without access to the activation pattern, whereas MILAN fails to do so.
  • Failure modes: CLIP-Dissect can confuse a concept with a correlated object, producing higher-level labels such as underwater, zebra, or missing white-background activity.These examples reflect difficulty focusing on backgrounds and distinguishing stripes or colors from associated objects.
  • Failure modes: Some neurons appear uninterpretable: for example, CLIP-Dissect assigns music to Conv1 neuron 2, while human annotators and MILAN also lack a clear concept.The paper directs further analysis of uninterpretable neurons to Appendix A.8.
  • Low-level concepts: The method shows similar low-level observations in ResNet-50, although no human annotations are available for comparison there.The results are presented for highly interpretable neurons in an early layer.
  • Compositional concepts: The method can search concatenations of two concepts to produce compositional explanations, but computational efficiency and consistency remain challenges.The initial search considers combinations among the 100 most accurate single-word labels for each neuron.
  • Architecture transfer: CLIP-Dissect extends to Vision Transformers and finds highly interpretable neurons that are strongly location-focused, such as kitchens or highways.The demonstration uses a ViT-B/16 model trained on ImageNet.

A.6 PREDICTING INPUT CLASS FROM NEURON DESCRIPTIONS

The experiment tests whether neuron descriptions can support input-class prediction from the most class-contributing hidden neuron. CLIP-Dissect descriptions outperform Network Dissection and MILAN, while broader analyses examine interpretability and similarity-function choices.

  • Class prediction from descriptions: The prediction procedure records hidden-neuron activations, selects the neuron with the highest positive contribution, retrieves its description, and matches it to the most similar class name.Class-name similarity averages cosine similarities from CLIP text embeddings and mpnet sentence embeddings.
  • Class prediction from descriptions: 10.28% of ImageNet images were correctly classified using CLIP-Dissect descriptions, versus 3.36% with Network Dissection and 2.31% with MILAN(base).The experiment used ResNet-50 trained on ImageNet and selected the most positively contributing neuron for each predicted class.
  • Interpretation: The class-prediction experiment provides evidence that CLIP-Dissect descriptions are higher quality than descriptions from Network Dissection and MILAN.The authors caution that overall prediction accuracy is not expected to be high because the most contributing neuron may serve a different role from the target class.
  • Interpretability and similarity functions: 69.7% of ResNet-18 Places-365 neurons and 77.8% of ResNet-50 ImageNet neurons were classified as interpretable using the SoftWPMI threshold τ = 0.16.Interpretable neurons had an average description score of 0.75, compared with 0.655 for all neurons and 0.5257 below the cutoff.
  • Interpretability and similarity functions: Simple cosine similarity adequately described only one of eight displayed neurons, highlighting the need for the more sophisticated SoftWPMI similarity function.Figure 14 compares descriptions produced with cosine similarity and SoftWPMI.

A.10 LARGER SCALE EXPERIMENT ON DESCRIPTION QUALITY

A larger evaluation examined CLIP-Dissect descriptions across randomly selected neurons, models, and layers using human judgments. Descriptions were generally good, but agreement was subjective and varied by layer.

  • Evaluation design: 1,000 evaluations covered 50 randomly selected neurons from each of five layers in two models, with evaluators judging whether descriptions matched 10 highly activating images.The interface offered Yes, Maybe, and No responses.
  • Results: 55–80% of neurons were well described depending on layer, with an average description score of 0.655.Scores were assigned as Yes = 1, Maybe = 0.5, and No = 0.
  • Results: Very early and very late layers were most interpretable, while middle layers were harder to describe.The evaluation used random neurons, some of which may not have a simple functional description.
  • Evaluation limitations: The evaluations were subjective: the two evaluators agreed on 68.4% of neurons, while only 2.4% received both a Yes and a No.Most disagreements were between adjacent categories, Yes/Maybe or Maybe/No.
  • Crowdsourced evaluation: The supplementary evaluation was expanded with a Mechanical Turk study using 200 random neurons per layer or all neurons for smaller layers, evaluated by three raters.Descriptions used the union of ImageNet validation images and Broden as Dprobe, SoftWPMI, and 20,000 common English words.

B.2 RESULTS

Crowdsourced ratings indicate that CLIP-Dissect descriptions generally match highly activating images and outperform randomly chosen concepts. Agreement is strongest for the earliest and latest layers, although the random baseline reveals evaluation noise.

  • Crowdsourced results: 3.57 out of 5 was the average crowdsourced rating across both models and layers, with highest agreement on the earliest and latest layers.The results were similar across the two models.
  • Random baseline: 2.61 was the average rating for randomly selected concepts, compared with 3.57 for CLIP-Dissect explanations.The random descriptions performed worse but still scored higher than expected, indicating noise in the crowdsourced evaluations.
  • Overall conclusion: The study found that CLIP-Dissect descriptions were clearly better than random at scale and supported the earlier evaluation using more neurons and evaluators.The authors report this as a larger crowdsourced confirmation of the prior findings.
Loading 2204.10965v5…