Source-linked AI summary
Understanding the Role of Individual Units in a Deep Neural Network
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, Antonio Torralba
TL;DR
The paper addresses how to understand learned representations in deep networks by introducing network dissection, which analyzes individual hidden units in classifiers and generators. It finds interpretable visual concepts and contextual object relationships, and applies the framework to adversarial attacks and semantic image editing.
Problem
Classification accuracy alone does not reveal how networks solve tasks or their sensitivities to shapes, textures, and perturbations.
Method
Network dissection systematically analyzes individual units in image classifiers and generators using their activation patterns and effects when units are manipulated.
Results
The analysis identifies interpretable object concepts in a scene classifier and shows that generator units reveal contextual relationships between object classes in scenes.
Takeaways & Limitations
Observing and manipulating individual units can expose learned knowledge structure and support human interaction with deep networks.
Takeaways & Limitations
Network dissection relies on disentangled, human-interpretable units, and learning better disentangled models remains an open problem.
Abstract
from arXiv · showhide
Deep neural networks excel at finding hierarchical representations that solve complex tasks over large data sets. How can we humans understand these learned representations? In this work, we present network dissection, an analytic framework to systematically identify the semantics of individual hidden units within image classification and image generation networks. First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts. We find evidence that the network has learned many object classes that play crucial roles in classifying scene classes. Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes. By analyzing changes made when small sets of units are activated or deactivated, we find that objects can be added and removed from the output scenes while adapting to the context. Finally, we apply our analytic framework to understanding adversarial attacks and to semantic image editing.
Results
Network dissection identifies interpretable units in classifiers and generators, then tests their roles through unit interventions. These units support scene classification, generate context-sensitive objects, and reveal limitations of unit-based interpretation.
- Emergence of Object Detectors in a Scene Classifier.: Unit 150 detected airplanes absent from explicit training labels, reaching 85.6% balanced classification accuracy on an ImageNet airplane-versus-other-object task.Its agreement with reference airplane segmentations in scene images was IoU = 9.0%.
- Role of Units in a Scene Classifier.: Removing the 20 most important units reduced ski resort accuracy from 81.4% to 53.5%, near chance, while changing all-class accuracy only from 53.3% to 52.6%.Removing 492 least-important units while retaining the 20 most important reduced ski resort accuracy by only 3.7%, but reduced all-class accuracy to 2.1%.
- Role of Units in a Scene Classifier.: Across 365 scene classes, removing each class’s 20 most important units reduced average single-class accuracy to 53.0%, whereas removing 492 least-important units reduced it by 3.6%.The results support a decomposition of individual scene classification into visual-concept subproblems.
- Role of Units in a GAN.: In a Progressive GAN, layer5 contained 19 object-specific units and 41 object-part units, with diverse appearances rather than rigid pixel templates.Object parts outnumbered whole objects, and concepts such as ovens and chairs matched multiple units.
- Role of Units in a GAN.: Window unit 314 predicted large windows with 78.2% accuracy, but some generated large-window images lacked its activation, indicating that other units may synthesize windows.The threshold used was peak activation > 8.03.
- Role of Units in a GAN.: Correlations between units and generated objects do not by themselves establish that those units cause the generator to render the objects.The paper therefore evaluates unit interventions in the GAN.
Applications
The paper applies unit-level analysis to interpret adversarial attacks and enable direct semantic editing of generated images.
- Adversarial attacks can be analyzed through changes in the units important to the target and original scene classes.For a ski-resort-to-bedroom attack, the analysis tracks four important units for each class.
- The attack reduced snow, mountain, house, and tree detections while increasing bed, person-head, and sofa activations in nonexistent locations.
- Across many images and classes, attacks changed the few units most important to each class.
- Unit intervention enables a GAN interface in which users paint with high-level object concepts instead of a palette of colors.Each concept is associated with 20 units maximizing IoU for that concept.
Discussion
Analyzing individual units reveals how classifiers decompose scene recognition into visual concepts and how generators enforce contextual object relationships. The approach supports insight into deep-network internals but depends on interpretable units emerging during training.
- Classification accuracy alone does not reveal whether a network relies on shapes, textures, or perturbation-sensitive representations.
- Individual units reveal the visual concepts that contribute to recognizing specific scene classes.
- Generator units reveal contextual relationships that the model enforces between object classes in a scene.
- Network dissection relies on disentangled, human-interpretable units emerging during training, and training better disentangled models remains open.
- Observing and manipulating units can expose learned knowledge structure and support systems for human interaction with deep models.
Materials and Methods
The study analyzes a Places365-trained VGG-16 scene classifier and uses segmentation-based concept matching to interpret its units. It evaluates 335 object classes, 1452 object parts, 25 materials, and 11 colors, with public code, weights, and datasets for reproduction.
- Places365 contains 1.80 million images labeled across 365 scene classes, with 36,500 held-out validation images.
- The analyzed VGG-16 classifies Places365 scenes using 13 convolutional layers and achieves 53.3% accuracy on the held-out validation set.
- A Unified Perceptual Parsing network and RGB color-name assignments provide human-interpretable visual concepts for unit analysis.
- The reference segmentation system detects 335 object classes, 1452 object parts, 25 materials, and 11 colors.
- The code, trained model weights, and datasets required to reproduce the results are publicly available through GitHub and the project website.