Source-linked AI summary
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, Jeff Clune
TL;DR
The paper asks how to interpret what individual DNN neurons have learned while avoiding the limitations of image retrieval and unconstrained activation maximization. It introduces DGN-AM, which optimizes a learned generator’s code to produce highly activating images, yielding realistic and interpretable visualizations, with some limits across architectures and non-canonical image classes.
Problem
Existing probing may require training-like image distributions, miss informative stimuli, or produce unrealistic and uninterpretable images through unconstrained activation maximization.
Method
DGN-AM optimizes the input code of a deep generator network so its generated image highly activates a target neuron, while using the generator as a learned natural-image prior.
Results
DGN-AM produces realistic-looking, interpretable preferred images that reflect neuron-learned features, including properties learned from modified training images rather than only the prior.
Takeaways & Limitations
The visualizations improve understanding of which features neurons detect and can support comparative analysis of trained DNNs.
Takeaways & Limitations
The learned prior generalizes only imperfectly across different network architectures, and non-canonical visualizations can reflect non-canonical training-image distributions.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) have demonstrated state-of-the-art results on many pattern recognition tasks, especially vision classification problems. Understanding the inner workings of such computational brains is both fascinating basic science that is interesting in its own right - similar to why we study the human brain - and will enable researchers to further improve DNNs. One path to understanding how a neural network functions internally is to study what each of its neurons has learned to detect. One such method is called activation maximization (AM), which synthesizes an input (e.g. an image) that highly activates a neuron. Here we dramatically improve the qualitative state of the art of activation maximization by harnessing a powerful, learned prior: a deep generator network (DGN). The algorithm (1) generates qualitatively state-of-the-art synthetic images that look almost real, (2) reveals the features learned by each neuron in an interpretable way, (3) generalizes well to new datasets and somewhat well to different network architectures without requiring the prior to be relearned, and (4) can be considered as a high-quality generative method (in this case, by generating novel, creative, interesting, recognizable images).
1 Introduction and Related Work
The paper studies how to reveal what individual DNN neurons detect by synthesizing preferred inputs. It addresses limitations of image-based probing and unconstrained activation maximization with a learned image prior.
- Motivation: Preferred stimuli reveal the concepts that individual biological or artificial neurons selectively respond to.The paper connects DNN visualization to neuroscience methods that identify neurons through highly activating images.
- Motivation: Image-based probing requires training-like data, may miss informative images, and makes it difficult to identify which image features drive a neuron.These limitations motivate synthesizing preferred inputs rather than selecting them from an existing image distribution.
- Activation maximization: Unconstrained activation maximization can produce unrealistic, uninterpretable fooling images because the space of possible images is vast.The method starts from a random image and uses backpropagation to change pixels toward higher neuron activation.
- Proposed approach: DGN-AM optimizes the input code of a deep generator network so its output image highly activates a target neuron.The generator is trained to produce images resembling natural ImageNet images, restricting the search to its image manifold.
2 Methods
The method visualizes neurons by optimizing codes passed through a fixed generator, while the target networks and generator parameters remain fixed. Code clipping and regularization further constrain the search.
- Networks: The visualization targets include CaffeNet, GoogLeNet, and ResNet, spanning distinct convolutional architectures trained on ImageNet.CaffeNet’s fc6, fc7, and fc8 are its final three fully connected layers, with fc8 producing 1000 class outputs.
- Image generator network: Instead of directly optimizing pixels, the method optimizes the input code y of a generator G so G(y) maximizes target-neuron activation in Φ.The generator acts as a learned prior over images while Φ denotes the DNN being visualized.
- Image generator network: The generator networks are trained to reconstruct images from hidden-layer feature representations within CaffeNet.The setup uses publicly available generator networks trained under GAN-based principles, with CaffeNet truncated at selected encoder layers.
- Optimization: A small L2 regularization term is used, with empirically selected λ = 0.005.The regularization is included in the activation-maximization objective.
- Optimization: Code values are clipped to the activation range [0, 3σ], where σ is one standard deviation around the mean activation.The range is computed from validation-set images passed through the corresponding encoder; clipping substantially improves image quality.
3 Results
DGN-AM produces more coherent visualizations from fully connected codes, generalizes across datasets and some architectures, and reveals features learned by hidden neurons. Its quality reflects both the learned prior and the target neuron, but varies with architecture similarity and feature diversity.
- 3.1 Comparison between priors trained to invert features from different layers: Fully connected codes produce more coherent global structure than convolutional codes, which typically yield repeated fragments.The authors attribute this to fully connected representations incorporating information from all image regions.
- 3.1 Comparison between priors trained to invert features from different layers: fc6 optimization produces the best visualizations, so the authors use the G6 DGN as their default prior.The images also qualitatively appear more realistic than visualizations from previous methods.
- 3.2 Does the learned prior trained on ImageNet generalize to other datasets?: The ImageNet-trained prior generalizes well to MIT Places and other CaffeNet DNNs trained on different datasets.For the MIT Places model, the prior visualizes scene-classification output units despite the dataset difference.
- 3.3 Does the learned prior generalize to visualizing different architectures?: Visualization quality is best when the target DNN matches the CaffeNet encoder and tends to degrade as their architectures become more distant.The authors identify generalization across architectures as an open challenge.
- 3.4 Does the learned prior generalize to visualizing hidden neurons?: DGN-AM often agrees with real-image-based methods about hidden-neuron features while producing more realistic color and texture.On MIT Places hidden neurons, synthetic visualizations also fall within human-provided feature categories and can reveal additional associations.
- 3.5 Do the synthesized images teach us what the neurons prefer or what the prior prefers?: Visualizations reflect modified training features, including cut-up objects, BRG colors, and washed-out details, rather than being produced exclusively by the prior.Canonical structure is less reliable when a neuron responds to diverse, multimodal image sets.
4 Discussion and Conclusion
DGN-AM uses a learned deep generator prior to synthesize realistic, interpretable preferred images for neural-network neurons. The method improves visualization quality, supports cross-dataset and partial cross-architecture generalization, and suggests broader generative applications.
- DGN-AM uses a learned deep generator network prior for activation maximization, restricting synthesis to images the prior can generate.This produces preferred inputs by optimizing generator codes rather than unrestricted pixels.
- The synthesized images are described as the most realistic-looking and interpretable preferred images produced by activation maximization to date.
- DGN-AM visualizations help reveal features learned by individual neurons while also producing visually interesting images.
- The learned prior generalizes to new datasets and somewhat to different architectures without requiring relearning, although architecture transfer is unequal.
- The approach could support image synthesis from text descriptions and more realistic deep-dream images, while motivating more general priors for comparative network analysis.
Supplementary materials for: Synthesizing the preferred inputs for neurons in
The paper presents deep generator networks as the learned-prior mechanism underlying its activation-maximization method.
- The method synthesizes preferred neural inputs through activation maximization using a learned deep generator network prior.
S5 Why do visualizations of some neurons not show canonical images?
Non-canonical visualizations are investigated as either method failures or reflections of the features neurons learned from their training images. The results indicate that DGN-AM reproduces canonical structure when canonical examples dominate, while diverse training images yield non-canonical outputs.
- The experiment tests whether non-canonical visualizations reflect a methodological shortcoming or properties of the neurons.
- Five classes with few canonical images were split into canonical and original-image classes, then added back to ImageNet training.The modified classes were school bus, cup, Irish terrier, tabby cat, and hartebeest.
- DGN-AM produced canonical visualizations for neurons trained on canonical images but non-canonical visualizations for neurons trained on the original image sets.
- Canonical-image proportions were 2% for school bus, 3% for tabby cat, 4% for Irish terrier, 6% for hartebeest, and 18% for cup.
- Classes whose visualizations often appeared canonical had higher canonical-image proportions, including 31% for table lamp, 49% for brambling, 29% for lipstick, 19% for joystick, and 39% for beacon.
- The evidence suggests that non-canonical outputs arise because some neurons detect diverse, multimodal feature sets rather than canonical poses.
S6 Visualizing under-trained, well-trained, and overfit networks
Visualization quality tracks a network’s validation accuracy during early training and is worse for networks trained on less data, supporting its use for studying feature learning and generalization.
- The experiment visualizes preferred stimuli for output and hidden neurons across CaffeNet training snapshots taken every 10,000 iterations.The networks span under-trained, well-trained, and overfit states.
- The visualization quality correlates with DNN accuracy during the first 200,000 iterations, evolving from blurry features to clearer ones as accuracy increases.The “swimming trunks” neuron shifts from associating the concept with people in a blue ocean to the clothing item around 300,000 iterations.
- 100% training accuracy with 20.2% or 31.5% validation accuracy produces recognizable but worse visualizations than a network with 57.4% validation accuracy.The comparison uses networks trained on 10%, 25%, and 100% of ImageNet images.
- Visualization quality correlates with validation accuracy, but low class accuracy can still yield beautiful images when the network confuses similar classes.The authors distinguish validation accuracy from class accuracy as explanations for visualization quality.
S7 Visualizing an activity recognition network
The method is extended to synthesize preferred videos for an activity-recognition network, producing sensible but lower-quality sequences and revealing that the model ignores frame order.
- The synthesized videos are qualitatively sensible but not as strong as the method’s best images, partly because LRCN uses a hybrid architecture rather than CaffeNet.The architecture difference is offered as an explanation for the weaker video quality.
- The model’s classification is largely insensitive to frame order, explaining the non-smooth transitions in its synthetic videos.Shuffling frames in a real video also does not substantially change the DNN’s classification decision.
S8 Synthesizing creative art by activating two neurons instead of one
Activating two neurons jointly requires balancing their activations rather than merely summing objectives. The resulting visualizations blend concepts, create artistic hybrids, and expose previously unseen facets of learned features.
- S8 Synthesizing creative art by activating two neurons instead of one: Adding a second activation objective can let the easier neuron dominate, producing only one concept in the final image.For example, an easier-to-activate bell pepper neuron can suppress the candle concept.
- S8 Synthesizing creative art by activating two neurons instead of one: The method instead adds an L2 penalty on the distance between two neuron activations while optimizing a generator code.The penalty encourages the two activations to be similar; γ controls its weight.
- S8 Synthesizing creative art by activating two neurons instead of one: The resulting visualizations are diverse and depend on the selected neuron pair.Observed outcomes include sensible object blends and unrealistic but artistically interesting hybrids.
- S8 Synthesizing creative art by activating two neurons instead of one: Joint activation can reveal new facets of a neuron, such as people eating lobster or an outdoor prison scene at night.These examples arise from combining the “American lobster” or “prison” neuron with “candles.”
- S8 Synthesizing creative art by activating two neurons instead of one: The authors identify two uses: generating creative art and uncovering unique facets of what neurons detect.They frame this as multifaceted feature visualization for understanding DNNs.
- S8 Synthesizing creative art by activating two neurons instead of one: Canonical training images produce canonical neuron visualizations, while similar classes can remain visually indistinguishable when their training examples are also similar.The canonical-class experiment supports correspondence between synthesized images and learned features; similar-class comparisons include hartebeest versus impala and Indian versus African elephant.
- S8 Synthesizing creative art by activating two neurons instead of one: Priors inverted from fully connected encoder layers produce more coherent global structures than priors inverted from convolutional layers.The comparison is attributed to the larger receptive fields of higher-level representations.
- S8 Synthesizing creative art by activating two neurons instead of one: Across CaffeNet, AlexNet, and scene-classification examples, the method produces recognizable visualizations and reveals object detectors in intermediate layers.The comparisons include prior activation-maximization methods and hidden neurons from all eight layers.