Source-linked AI summary
Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks
Ruth Fong, Andrea Vedaldi
TL;DR
The paper addresses limited understanding of intermediate representations and the limitations of interpreting concepts through extremal responses from individual filters. It introduces Net2Vec, which maps concepts to embeddings over filter activations, and finds that concepts commonly require multiple filters while filters often encode multiple concepts.
Problem
Existing approaches often focus on extremal responses from individual filters, while distributed concept encoding remains harder to quantify and explain.
Method
Net2Vec learns concept embeddings aligned with a CNN layer’s filter space to quantify and visualize how concepts are encoded across filters.
Results
Net2Vec shows that multiple filters are often required for a concept and that filters frequently encode multiple concepts.
Takeaways & Limitations
Filter embeddings characterize representation meaning and relationships among concepts more effectively than single-filter activations.
Takeaways & Limitations
The approach is limited by its linear nature, and visual explanations should be evaluated against the full distribution of filter activations rather than only extremes.
Abstract
from arXiv · showhide
In an effort to understand the meaning of the intermediate representations captured by deep networks, recent papers have tried to associate specific semantic concepts to individual neural network filter responses, where interesting correlations are often found, largely by focusing on extremal filter responses. In this paper, we show that this approach can favor easy-to-interpret cases that are not necessarily representative of the average behavior of a representation. A more realistic but harder-to-study hypothesis is that semantic representations are distributed, and thus filters must be studied in conjunction. In order to investigate this idea while enabling systematic visualization and quantification of multiple filter responses, we introduce the Net2Vec framework, in which semantic concepts are mapped to vectorial embeddings based on corresponding filter responses. By studying such embeddings, we are able to show that 1., in most cases, multiple filters are required to code for a concept, that 2., often filters are not concept specific and help encode multiple concepts, and that 3., compared to single filter activations, filter embeddings are able to better characterize the meaning of a representation and its relationship to other concepts.
1. Introduction
Understanding intermediate representations is difficult because their meaning is unclear, while single-filter interpretations may not represent distributed encoding. Net2Vec addresses this gap by quantifying how concepts overlap with combinations of filters.
- Intermediate-layer meanings remain unclear, limiting the ability to characterize deep-network limitations and improve models systematically.
- Extremal-response studies find concept correlations, but the smaller number of feature channels than concepts suggests that representations must partly use filter combinations.
- Net2Vec aligns semantic concepts with filter activations through learned concept embeddings used for segmentation and classification.
- The framework quantifies whether concepts require multiple filters and whether individual filters are shared across diverse concepts.
- Concept and filter activation sets often partially intersect without either set containing the other, indicating non-exclusive overlap.
- The paper evaluates visual explanations by how well they explain the whole activation distribution rather than only maximally activating inputs.
2. Related Work
Related work explains filter semantics through visualizations and studies distributed encodings, but these approaches provide limited quantitative and explanatory coverage of how concepts are encoded.
- Visualizations: Visualization methods inspect real or generated inputs, patches, or segmentation masks that strongly activate individual filters.
- Distributed Encodings: Distributed-encoding studies show that classification and detection can benefit from multiple hidden units or activation clusters.
3. Net2Vec
Net2Vec learns concept embeddings from CNN filter activations to analyze distributed semantic representations. The framework supports segmentation and classification, including comparisons between single filters and filter combinations.
- Framework: Net2Vec learns concept weights over probe-set filter activations, interpreting the resulting weight vectors as semantic concept embeddings.These embeddings align concepts with filters for semantic tasks and subsequent analysis.
- Data: The framework primarily probes AlexNet trained on ImageNet using BRODEN, which contains 1,189 retained concepts across six categories.BRODEN provides pixel- and image-level annotations for scenes, objects, parts, materials, textures, and colors.
- Concept Segmentation by a Single Filter: For single-filter segmentation, activations are thresholded at each filter’s τ = 0.005 quantile and bilinearly upsampled to produce concept masks.The thresholded mask is compared with ground-truth masks using set IoU, with the best filter selected on training data and evaluated on validation data.
- Concept Segmentation by Filter Combinations: For filter-combination segmentation, learned weights linearly combine thresholded filter activations before a sigmoid produces the predicted mask.Weights are learned on training images, while set IoU and per-image IoU are evaluated on validation images.
- Concept Classification by Filter Combinations: For concept classification, spatially averaged filter activations are combined with learned weights and a bias, then passed through a sigmoid to estimate concept presence.The classifiers use balanced positive and negative sampling during training and balanced validation accuracy for evaluation.
- Restricted Filter Sets: Subset comparisons select the top F filters by absolute learned weight and retrain classifiers for varying F, including F = 1 for single-filter evaluation.The same restricted-filter procedure is applied to the segmentation task with its specified filter-count settings.
4. Quantifying the Filter-Concept Overlap
Net2Vec evaluates whether concepts are encoded by individual filters or combinations, finding that learned multi-filter representations generally outperform single filters and that filters are often shared across concepts.
- Are Filters Sufficient Statistics for Concepts?: Learned combinations of filter activations outperform the best single filter on average for both segmentation and classification.The comparison uses validation data and learned weights over filter activations.
- Are Filters Sufficient Statistics for Concepts?: Up to 4× improvement occurs for some concepts, including simple color concepts, when filter combinations replace individual filters.The improvements begin in conv1 and vary substantially across concepts.
- Are Filters Sufficient Statistics for Concepts?: Performance saturates at different numbers of top conv5 filters across concept categories and tasks.Classification usually saturates by F = 50, while segmentation peaks at F = 8 for materials and parts, F = 16 for objects, and F = 128 for colors.
- Are Filters Sufficient Statistics for Concepts?: Around 10% of concepts do not perform better with learned weights than with the best single filter.Failure cases are associated with few training examples or very small objects, which can cause overfitting or unstable training.
- Are Filters Shared between Concepts?: A single filter can respond to related concepts such as sheep, horse, and cow through a shared latent pattern resembling an animal torso.AlexNet conv5 filter 66 is shown as selective for a pastoral animal’s torso.
- Are Filters Shared between Concepts?: A substantial portion of filters are selected for many concepts, while others are never selected, indicating that filters are not concept-specific.Around 15% of conv1 filters are selected for at least 20 or 30 concepts, and multi-concept filters are not exclusively overlapped by one concept.
- More Architectures, Datasets, and Tasks: Results extend across additional architectures, datasets, and self-supervised networks, with spatial resolution affecting performance and self-supervised gains exceeding supervised gains.The reported improvements are 5–6× for self-supervised networks versus 2–4× for supervised networks.
5. Interpretability
Net2Vec extends interpretability beyond maximally activating filters by evaluating non-extreme examples and representing concepts through learned embeddings in filter space. These embeddings connect single-filter behavior with distributed encoding and expose relationships among concepts and representations.
- Visualizing Non-Maximal Examples: Maximally activating examples can favor visually appealing cases that do not represent the broader distribution of filter activations.The paper argues that explanations should account for how visualizations vary across examples.
- Visualizing Non-Maximal Examples: Single-filter concept segmentation produces IoUind scores of 0 for many examples, often because thresholding leaves no surviving activations.This indicates that a single filter may not fire consistently strongly for a concept.
- Visualizing Non-Maximal Examples: Learned filter combinations produce informative, salient masks for most dog and airplane examples, including those at the lowest sampled decile.The comparison uses examples sampled across deciles of the non-zero individual-IoU distribution.
- Unifying Single- & Multi-Filter Views: By conv3, 81.14% of segmentation concepts are significantly correlated between max(w, 0) and individual-filter set IoU scores.The reported percentages are 47.33% for conv1, 69.12% for conv2, 81.14% for conv3, 79.13% for conv4, and 82.47% for conv5, using p < 0.01.
- Explanatory Power via Concept Embeddings: Learned weights function as concept embeddings whose dimensions correspond to filters, enabling quantitative comparison of concept relationships across representations.The embeddings are normalized to unit length before cosine-distance analysis.
- Explanatory Power via Concept Embeddings: Segmentation embeddings place similar-category objects near one another, whereas classification embeddings capture compositional relationships such as object parts.For example, cat and horse are near dog in segmentation space, while muzzle and paw are near dog in classification space.
- Explanatory Power via Concept Embeddings: Vector arithmetic on segmentation embeddings yields coherent analogies and transformed concepts, including grass−green resembling sky−blue.Other reported results include non-green, ground-like concepts for grass minus green and floral concepts for tree minus wood.
6. Conclusion
The paper presents Net2Vec as a framework for learning concept embeddings aligned with CNN filter spaces and for analyzing distributed concept encodings. It also proposes non-extreme visualization and acknowledges that the approach is limited by its linear nature.
- Conclusion: Net2Vec learns concept embeddings aligned to a CNN layer’s filter space to quantify how concepts overlap with individual and combined filters.The framework extends binary questions about exclusive encoding into scalar questions about the extent of overlap.
- Conclusion: The approach proposes a standard for visualizing non-extreme examples and uses embeddings to explain distributed concept encodings.These components complement analysis of single-filter and multi-filter perspectives.
- Conclusion: The conclusion identifies the approach’s linear nature as a limitation and points to nonlinear concept representations as future work.The stated scope boundary concerns how concepts are modeled in the framework.
- Conclusion: The paper uses quantitative embedding comparisons to characterize relationships among learned representations and concept categories.Figure 10 compares embedding-space distances and mean concept-category distances.