Source-linked AI summary
Interpreting Deep Visual Representations via Network Dissection
Bolei Zhou, David Bau, Aude Oliva, Antonio Torralba
TL;DR
CNNs learn powerful hidden representations, but their internal semantic structure and interpretability remain difficult to characterize. The paper introduces Network Dissection, which labels units by measuring their alignment with visual concepts, and finds that deep representations are significantly more interpretable than random equivalent bases. Across architectures and training conditions, the method compares interpretability and uses interpreted units to explain CNN predictions.
Problem
The paper addresses how disentangled representations and their factors can be detected and quantified, and whether interpretable hidden units reflect a special alignment of feature space.
Method
Network Dissection identifies human-labeled visual concepts, measures hidden-unit responses to them, and quantifies unit–concept alignment in CNN representations.
Results
Deep representations are significantly more interpretable than expected under a random equivalent basis, with interpretability varying across architectures, training datasets, and conditions.
Takeaways & Limitations
Interpretability measurements provide insights into the hierarchical structure of deep neural networks and support explicit explanations of CNN predictions.
Takeaways & Limitations
Interpretations may miss concepts absent from Broden, while units detecting concepts finer-grained than its labels may receive low IoU scores.
Abstract
from arXiv · showhide
The success of recent deep convolutional neural networks (CNNs) depends on learning hidden representations that can summarize the important factors of variation behind the data. However, CNNs often criticized as being black boxes that lack interpretability, since they have millions of unexplained model parameters. In this work, we describe Network Dissection, a method that interprets networks by providing labels for the units of their deep visual representations. The proposed method quantifies the interpretability of CNN representations by evaluating the alignment between individual hidden units and a set of visual semantic concepts. By identifying the best alignments, units are given human interpretable labels across a range of objects, parts, scenes, textures, materials, and colors. The method reveals that deep representations are more transparent and interpretable than expected: we find that representations are significantly more interpretable than they would be under a random equivalently powerful basis. We apply the method to interpret and compare the latent representations of various network architectures trained to solve different supervised and self-supervised training tasks. We then examine factors affecting the network interpretability such as the number of the training iterations, regularizations, different initializations, and the network depth and width. Finally we show that the interpreted units can be used to provide explicit explanations of a prediction given by a CNN for an image. Our results highlight that interpretability is an important property of deep neural networks that provides new insights into their hierarchical structure.
1 INTRODUCTION
The paper investigates how interpretable semantic structure emerges in deep visual representations and whether individual hidden units align meaningfully with visual concepts. It introduces Network Dissection to quantify and compare that alignment across networks, training settings, and visual tasks.
- Motivation: Deep networks can develop individual hidden units corresponding to human-interpretable concepts without being explicitly constrained to do so.Prior observations include object, part, and object-detector units emerging in networks trained for different visual tasks.
- Research questions: The paper asks how disentangled representations and their factors can be detected and quantified, and whether interpretable units reflect a special feature-space alignment.It also examines how architectures, data sources, and training conditions affect representation entanglement.
- Approach: Network Dissection uses the broadly and densely labeled Broden dataset to identify hidden-unit semantics and align units with interpretable concepts in CNNs.The framework is designed to interpret representations across architectures including AlexNet, VGG, GoogLeNet, ResNet, and DenseNet.
- Scope: The study compares visual representations across architectures, supervised and self-supervised tasks, datasets, regularizations, and transfer settings to characterize variation in interpretability.The paper also positions scalable quantitative interpretation as an alternative to human evaluation of unit visualizations in increasingly large networks.
2 FRAMEWORK OF NETWORK DISSECTION
Network Dissection defines interpretability as alignment between individual CNN units and human-interpretable visual concepts, then quantifies that alignment using semantic segmentation. Broden supplies densely labeled concepts, while IoU scores identify unit–concept detectors and aggregate layer interpretability.
- Framework definition: Interpretability is defined as the alignment between hidden variables and human-interpretable concepts, reflecting whether representation factors correspond to meaningful visual structure.The method focuses on correspondence between single latent variables and individual visual concepts.
- Broden dataset: Broden unifies densely labeled datasets covering colors, materials, textures, parts, objects, and scenes, producing 1197 concept classes.Most examples have pixel-level segmentation, while texture and scene labels cover full images.
- Scoring procedure: Each convolutional unit is evaluated as a binary segmentation model for every Broden concept using activation maps compared with concept masks.Low-resolution activations are upsampled to the annotation resolution before thresholding into binary segmentations.
- Scoring procedure: A unit–concept pair is scored by Intersection over Union, and a unit is considered a detector when IoU exceeds 0.04.Because a unit may detect multiple concepts, the layer metric counts distinct concepts aligned with its units.
- Comparability and scope: IoU provides an objective interpretability confidence score comparable across networks, enabling comparisons among representations from varied CNN architectures and training settings.The tested model collection includes AlexNet, GoogLeNet, VGG, ResNet, and DenseNet.
- Comparability and scope: The method’s interpretability scores depend on the concept vocabulary, so units aligned with concepts absent from Broden may receive low scores.The authors note that future Broden versions should expand the visual-concept vocabulary.
3 EXPERIMENTS OF INTERPRETING DEEP VISUAL REPRESENTATIONS
The experiments validate Network Dissection against human judgments and use basis rotations to test whether interpretability is tied to individual units. They further examine how interpretability changes across layers and representations.
- Method: Network Dissection evaluates convolutional units as detectors for visual concepts using segmentation performance, enabling quantitative comparisons across networks.The method uses Broden concepts and can probe any convolutional layer.
- Human evaluation: Human raters generally agree with automatically assigned labels, although algorithmic and human descriptions sometimes identify different aspects of the same unit.Examples include ‘crosswalk’ versus ‘horizontal lines’ and ‘chequered’ texture versus ‘windows’.
- Axis-aligned interpretability: 80% fewer unique detectors appear after randomly rotating AlexNet conv5, indicating that interpretable alignments are unusual rather than typical directions.The rotated representation retains the same discriminative power as the original.
- Axis-aligned interpretability: Interpretability decreases progressively as larger rotations are applied, while networks can still learn the task from randomly rotated representations.Places365 accuracy improved from 50.3% to 51.9% unrotated and 51.7% rotated after retraining the top layers.
- Axis-aligned interpretability: Complete rotation reduces interpretability on both Places365 and ImageNet, with a larger drop for AlexNet trained on Places365.The larger drop is attributed to that network starting with more interpretable units.
3.3 Network Architectures with Supervised Learning
Supervised representations differ across architectures and datasets in their interpretable detectors. Deeper layers and architectures generally yield more complex or numerous semantic detectors, with Places-trained models outperforming ImageNet-trained models in detector counts.
- Architecture and dataset comparisons: ResNet > DenseNet > VGG > GoogLeNet > AlexNet in the number of unique detectors, while Places > ImageNet.The comparison uses unique detectors and also reports their ratio to total units.
- Detector density: For ResNet152 object detectors, average IoU has a weak positive correlation with the number of detectors for a class (r=0.08).The comparison covers models trained on Places and ImageNet.
- Detector examples: DenseNet and ResNet contain strong bus and airplane detectors with IoU > 0.25, while detector appearance varies within and across networks.The figure compares the two highest-IoU matches among convolutional units for each network.
- Layer depth: Higher layers produce more object and scene detectors across architectures, consistent with greater representational ability at increased depth.The paper links deeper layers with concepts of higher semantic complexity.
3.4 Representations from Self-supervised Learning
Self-supervised training produces interpretable representations whose concepts depend strongly on the pretext task. Compared with supervised training on annotated data, these models tend toward texture detectors and fewer object detectors.
- Overall comparison: Places365 produces the largest number of unique detectors, while self-supervised models produce many texture detectors but relatively few object detectors.The comparison uses AlexNet for the models except transinv, which uses VGG.
- Task dependence: The self-supervised task influences which concepts emerge, with colorization producing almost no color-detection units.The colorization model is trained on colorless images.
- Task dependence: Audio and puzzle models develop some part and object detectors that may support their primary tasks.The audio task associates objects with sound sources, whereas the puzzle task aligns object and scene parts.
- Task dependence: Texture recognition may suffice for colorization and tracking, and these models consequently show texture-oriented representations.The paper describes this as unsurprising for their respective primary tasks.
3.5 Representations from Captioning Images
The paper also studies a CNN trained through image captioning, using captioning supervision as another route for examining interpretable internal representations.
- Captioning setup: A CNN was trained from scratch with an LSTM to generate captions from COCO images, using ResNet18 features as the LSTM input.Each image had five captions.
3.6 Training Conditions
Training conditions affect interpretability independently of discriminative power. During training, semantic detectors emerge early, while batch normalization may make representations easier to rotate and thereby reduce interpretability.
- Regularization effects: Batch normalization may reduce interpretability by smoothing activation scaling and enabling intermediate representations to rotate.The paper cautions that discriminative power alone is insufficient for evaluating representations.
- Regularization effects: Preserving interpretability while retaining the benefits of batch normalization remains an open research direction.The paper states that interpretability is neither required for nor detrimental to discriminative power.
- Training dynamics: Object and part detectors begin emerging at about 10,000 training iterations.Each iteration processes a batch of 256 images.
- Training dynamics: Units do not show evidence of transitioning across concept categories before becoming object or part detectors.For example, conv5 units do not first become texture or material detectors.
- Training dynamics: Some units converge to semantic concepts early, while others evolve from one detector to another over training.Examples include a unit detecting snowy mountains early and units shifting from road to car or airplane.
3.7 Transfer Learning between Places and ImageNet
Fine-tuning between Places and ImageNet accelerates convergence while changing many unit interpretations. The changes are structured: units often retain low-level similarities, and detector counts can evolve differently depending on transfer direction.
- Interpretation changes: Fine-tuning changes unit interpretations, with detector counts following different trajectories for the two transfer directions.Unique object detectors first decrease then increase from ImageNet to Places365, but slowly decrease from Places365 to ImageNet.
- Interpretation changes: ImageNet-to-Places units can shift from animal detectors to scene-related concepts, while Places-to-ImageNet units often become dog detectors.Examples include white dogs becoming waterfalls and dog detectors becoming horse or cow detectors.
- Interpretation changes: Before-and-after concepts often share low-level image similarities such as colors and textures.This similarity persists even when the semantic concepts themselves change.
- Interpretation changes: 139 out of 256 units (54.4%) retain the same concepts from ImageNet to Places365, compared with 135 out of 256 (52.7%) from Places365 to ImageNet.These figures refer to conv5 units during fine-tuning.
3.8 Layer Width vs. Interpretability
Increasing layer width produces more interpretable detectors without materially improving classification accuracy in the tested AlexNet variants. Beyond a point, additional units do not yield more unique concepts.
- Width manipulation: Tripling conv5 from 256 to 768 units produces AlexNet-GAP-Wide and increases the emergence of interpretable detectors.AlexNet-GAP-WideAll also triples the width of preceding convolutional layers except conv1.
- Width manipulation: Widened networks have validation accuracy within approximately 0.5% of standard AlexNet while producing many more emergent detectors.The widened variants are trained on Places365.
- Width limits: Increasing conv5 width to 1024 or 2048 units does not significantly increase the number of unique concepts.The result may reflect a capacity limit or a limit on useful disentangled concepts for scene classification.
- Discrimination and interpretability: Across representations and three selected datasets, the number of unique object detectors is positively correlated with classification accuracy.The paper also notes an exception in which ResNet152-ImageNet is more discriminative on some object-centric datasets despite fewer object detectors than ResNet152-Places365.
3.10 Explaining the Predictions for the Deep Features
Network Dissection converts unit activations and semantic detector labels into image-level explanations for deep-feature predictions. Ranking units by class-weighted activation identifies regions supporting both correct predictions and classification errors.
- Explanation procedure: The explanation procedure ranks units by the product of each activation and its SVM weight for the predicted class.For ResNet, the deep feature is the global-average-pooled activation.
- Explanation procedure: The top-ranked unit’s activation map is upsampled and thresholded at the top 0.2 activation to segment explanatory image regions.The threshold is based on the feature map of the individual image.
- Correct predictions: Correct-prediction explanations associate labels with detector units such as person, arm, plate, pottedplant, tree, river, and water.Examples include Gardening and Fishing predictions in the action40 dataset.
- Incorrect predictions: Error explanations reveal which detector contributed to a mistaken prediction.One example attributes a gardening-to-cutting-vegetables error to a unit mistaking ground for table.
4 DISCUSSION
The discussion identifies threshold choices and dataset- and unit-level biases that constrain Network Dissection’s interpretations and comparisons.
- τ = 0.005 is chosen to reveal fine-grained concept selectivity, whereas looser thresholds associate units with general concepts such as color.A smaller threshold can miss objects, while a larger threshold or no threshold favors broader concepts.
- Network Dissection cannot identify concepts absent from Broden, including difficult-to-name concepts such as ‘the corner of a room’.
- Units detecting concepts more specific than Broden’s categories may receive low IoU and therefore not be counted as concept detectors.The paper gives a wooden stool chair leg as an example of a fine-grained concept that may not align well with the ‘chair’ category.
- The method measures alignment between individual units and concepts, so it does not identify groups of units that jointly represent one concept.
- Misaligned receptive-field centers can shift upsampled activation maps by a few pixels, and the number of unique detectors may favor larger networks.
5 CONCLUSION
Network Dissection converts qualitative unit visualizations into quantitative interpretability measurements and finds deep representations more interpretable than random equivalent bases. It compares architectures, supervision, and training conditions while linking interpretability to generalizable visual features and hierarchical insight.
- Network Dissection translates qualitative visualizations of representation units into quantitative interpretations and measurements of interpretability.
- Deep representation units are significantly more interpretable than expected for a basis of the representation space.
- The study investigates representations produced by different architectures, training supervisions, and training conditions.
- Interpretability is relevant to the power of a representation as a generalizable visual feature and provides insights into hierarchical structure.