Source-linked AI summary
Network Dissection: Quantifying Interpretability of Deep Visual Representations
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba
TL;DR
The paper asks how interpretable, disentangled structure in CNN representations can be quantified and whether it is distinct from discriminative power. It introduces network dissection, which aligns individual hidden units with concepts in a broad visual dataset and compares representations across models and training conditions. The analysis finds that interpretability is axis-aligned, can be destroyed by rotation without changing discriminative power, and varies significantly with architecture, supervision, and training technique.
Problem
The paper addresses how to quantify disentangled representations and determine whether interpretable hidden units reflect special feature-space alignment rather than a chimera.
Method
Network dissection evaluates each CNN hidden unit against human-interpretable concepts from Broden and counts unique concepts aligned with units.
Results
Interpretability is not axis-independent: random rotation can reduce unique detectors by 80% while preserving exactly the same discriminative power.
Takeaways & Limitations
Interpretability is a representation quality distinct from discriminative power and is affected by network layers, architectures, supervision, and training techniques.
Takeaways & Limitations
Network dissection works only as well as Broden: a human-understandable concept absent from the dataset will not score well for interpretability.
Abstract
from arXiv · showhide
We propose a general framework called Network Dissection for quantifying the interpretability of latent representations of CNNs by evaluating the alignment between individual hidden units and a set of semantic concepts. Given any CNN model, the proposed method draws on a broad data set of visual concepts to score the semantics of hidden units at each intermediate convolutional layer. The units with semantics are given labels across a range of objects, parts, scenes, textures, materials, and colors. We use the proposed method to test the hypothesis that interpretability of units is equivalent to random linear combinations of units, then we apply our method to compare the latent representations of various networks when trained to solve different supervised and self-supervised training tasks. We further analyze the effect of training iterations, compare networks trained with different initializations, examine the impact of network depth and width, and measure the effect of dropout and batch normalization on the interpretability of deep visual representations. We demonstrate that the proposed method can shed light on characteristics of CNN models and training methods that go beyond measurements of their discriminative power.
1. Introduction
Interpretable concepts can emerge as individual units in CNNs despite no explicit interpretability constraint, motivating a framework to quantify this structure and its dependence on training.
- Emergent interpretable structure: Individual units can detect objects, parts, or people in networks trained for places, objects, or video generation.Examples include table-lamp, bicycle-wheel, and people detectors.
- Emergent interpretable structure: The emergence of interpretable structure suggests that deep networks may spontaneously learn partly disentangled representations.A disentangled representation aligns variables with meaningful factors of the underlying problem structure.
- Research questions: The paper asks how disentanglement can be quantified, whether interpretable units reflect special feature-space alignment, and which training conditions affect entanglement.These questions frame interpretability as an empirical property of learned representations.
- Proposed framework: Network dissection identifies hidden-unit semantics using Broden concepts and evaluates CNNs across architectures, tasks, datasets, and training techniques.The study includes AlexNet, VGG, GoogLeNet, and ResNet trained for object and scene recognition.
- Core finding: Random rotations can destroy interpretability without affecting discriminative power, showing that interpretability is an axis-aligned property rather than an inevitable consequence of discrimination.The framework is intended to reveal representation characteristics beyond standard discriminative measurements.
2. Network Dissection
Network dissection measures interpretability by aligning individual CNN units with densely labeled visual concepts, then counts the concepts detected by units in each layer.
- 2. Network Dissection: Network dissection defines interpretability as alignment between individual latent variables and human-interpretable visual concepts.The method focuses on correspondence between a single hidden unit and a single concept to assess disentanglement.
- 2. Network Dissection: The procedure identifies labeled concepts, collects hidden-variable responses to them, and quantifies hidden-variable–concept alignment.These are the three operational steps of network dissection.
- 2.1. Broden: Broden unifies densely labeled datasets covering objects, scenes, parts, textures, materials, and colors, with most examples annotated at pixel level.The dataset merges ADE, OpenSurfaces, Pascal-Context, Pascal-Part, and the Describable Textures Dataset.
- 2.2. Scoring Unit Interpretability: Each convolutional unit is evaluated as a binary segmentation model for every Broden concept using activation maps and concept annotation masks.Activation maps are collected, thresholded, resized to annotation resolution, and compared with concept masks.
- 2.2. Scoring Unit Interpretability: Unit–concept quality is scored with intersection over union, and a unit is counted as a detector when IoU exceeds 0.04.Layer interpretability is the number of unique concepts aligned with units; relative network orderings remain stable across thresholds.
- 2.2. Scoring Unit Interpretability: The score is comparable across networks but depends on Broden coverage, so concepts absent from the dataset may be judged as uninterpretable.The authors identify dataset expansion as a future direction.
3. Experiments
The experiments validate Network Dissection and use it to show that interpretability is axis-aligned, varies with architecture, supervision, training conditions, and width, and is distinct from discriminative power.
- 3.1. Human Evaluation of Interpretations: Human raters found the method’s semantic labels comparable to human descriptions at the highest layer, while low-level concepts were insufficient for many lower-layer units.Human agreement was also highest at conv5, where object and part concepts are easier to recognize than lower-level shapes and textures.
- 3.2. Measurement of Axis-Aligned Interpretability: 80% fewer unique detectors after random basis rotation shows that interpretability is not axis-independent, despite unchanged discriminative power.The rotated representation preserves the original network’s discriminative behavior, separating interpretability from discriminative power.
- 3.4. Network Architectures and Supervisions: Interpretability follows the architecture ordering ResNet > VGG > GoogLeNet > AlexNet and the dataset ordering Places > ImageNet.Deeper architectures appear to allow greater interpretability, while scene recognition may encourage more object detectors than object-centric training.
- 3.4. Network Architectures and Supervisions: Places365 produces the most unique detectors, whereas self-supervised models produce many texture detectors but relatively few object detectors.The supervision form also shapes semantics: colorization produces almost no color-detection units, consistent with training on colorless images.
- 3.5. Training Conditions vs. Interpretability: Object and part detectors begin emerging at about 10,000 iterations, without evidence that conv5 units transition from texture or material detectors beforehand.Across training conditions, random initializations converge to similar interpretability, dropout shifts the balance toward texture detectors, and batch normalization appears to reduce interpretability.
- 3.7. Layer Width vs. Interpretability: 0.5% top1 accuracy lower than standard AlexNet, AlexNet-GAP-Wide yields many more emergent concept detectors after Places365 training.Increasing conv5 width beyond 768 units to 1024 or 2048 does not significantly increase the number of unique concepts.
4. Conclusion
Network dissection quantifies CNN interpretability and shows that interpretability is not axis-independent, supporting the view that interpretable units reflect partially disentangled representations. Across layers and training methods, the framework reveals differences in which meanings are disentangled and how interpretable learned representations become.
- Network dissection quantifies the interpretability of CNN representations and tests whether it is independent of the choice of feature-space axes.
- Interpretability is not axis-independent, consistent with interpretable units indicating partially disentangled representations.
- Different layers disentangle different categories of meaning.
- Different training techniques can significantly affect the interpretability of representations learned by hidden units.