Source-linked AI summary
Visualizing and Understanding Convolutional Networks
Matthew D Zeiler, Rob Fergus
TL;DR
Large convolutional networks achieved strong image-classification performance, but their internal operation and avenues for improvement were poorly understood. The paper introduces visualization and ablation methods, finding improved ImageNet architectures and strong transfer to Caltech-101 and Caltech-256.
Problem
The internal operation of complex convolutional models and how they achieve strong performance remained poorly understood, limiting understanding beyond trial-and-error development.
Method
The paper visualizes feature activations with a multi-layered deconvolutional network and uses architectural ablations to analyze and improve convolutional models.
Results
The improved model beat Krizhevsky et al.’s single-model ImageNet result by 1.7% (test top-5) and achieved 74.2% versus 55.2% on Caltech-256 with 60 training images per class.
Takeaways & Limitations
The findings show that learned convolutional features are structured and increasingly invariant and class-discriminative across layers, while ImageNet features transfer strongly to similar datasets.
Takeaways & Limitations
The reported ImageNet performance was later surpassed in the recent ImageNet 2013 competition.
Abstract
from arXiv · showhide
Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark. However there is no clear understanding of why they perform so well, or how they might be improved. In this paper we address both issues. We introduce a novel visualization technique that gives insight into the function of intermediate feature layers and the operation of the classifier. We also perform an ablation study to discover the performance contribution from different model layers. This enables us to find model architectures that outperform Krizhevsky \etal on the ImageNet classification benchmark. We show our ImageNet model generalizes well to other datasets: when the softmax classifier is retrained, it convincingly beats the current state-of-the-art results on Caltech-101 and Caltech-256 datasets.
1. Introduction
Convolutional Networks achieved strong visual-classification performance, but their internal operation remained poorly understood. This paper introduces a feature-visualization technique, uses it to improve ImageNet architectures, and tests transfer by retraining only the softmax classifier on other datasets.
- Introduction: Convolutional Networks advanced from digit classification and face detection to strong performance on challenging visual-classification tasks.Recent progress was supported by larger labeled datasets, GPU implementations, and regularization strategies such as Dropout.
- Introduction: The paper addresses limited understanding of how complex convolutional models operate and achieve high performance.Without clearer understanding, developing better models risks becoming trial-and-error.
- Introduction: The authors introduce a visualization technique that reveals input stimuli exciting individual feature maps at any model layer.Existing feature visualization was mostly limited to the first layer, while higher-layer interpretation had fewer methods.
- Introduction: Starting from Krizhevsky et al.’s architecture, the authors discover architectures that outperform it on ImageNet.They also evaluate generalization to other datasets by retraining only the softmax classifier, framing this as supervised pre-training.
2. Approach
The paper uses supervised convolutional networks trained for image classification and introduces deconvolutional networks to map intermediate feature activations back to input pixels. The reconstruction reverses convolutional processing through unpooling, rectification, and filtering, revealing image structures contributing to activations.
- Convnet models: The convnet maps a color 2D image through learned-filter layers and rectified linear functions to a probability vector over C classes.Each layer applies convolution followed by relu(x) = max(x, 0).
- Convnet models: Models are trained on N labeled images using cross-entropy loss, with parameters updated by backpropagation and stochastic gradient descent.The parameters include convolutional filters, fully connected weight matrices, and biases.
- Deconvnet visualization: A deconvnet attaches to each convnet layer and maps a selected activation continuously back toward image pixels to reveal the input pattern that caused it.All other activations at the examined layer are set to zero before reconstruction.
- Deconvnet visualization: The deconvnet reconstructs activations by successively unpooling, rectifying, and filtering feature maps.Unpooling uses recorded max-pooling switches, rectification applies relu, and filtering uses transposed versions of the learned filters.
- Deconvnet visualization: Because pooling switches are specific to each input, a single-activation reconstruction resembles a weighted piece of the original image and indicates discriminative input structures.The weighting reflects each structure’s contribution toward the feature activation.
3. Training Details
The model follows a Krizhevsky et al.-style ImageNet architecture but replaces GPU-induced sparse connections in layers 3–5 with dense ones. Training used standardized 256-pixel preprocessing, 224×224 crops and flips, mini-batch SGD, and filter renormalization to stabilize dominant filters.
- Architecture: Layers 3–5 use dense connections instead of the sparse connections in Krizhevsky’s model, which arose from splitting the model across two GPUs.The architecture is otherwise described as similar to Krizhevsky et al. (2012) for ImageNet classification.
- Visualization-guided design: Inspection of visualizations motivated additional changes to layers 1 and 2.The passage refers to Section 4.1 for the specific changes.
- Data preprocessing: Training used ImageNet 2012’s 1.3 million images across 1000 classes, with resizing, center cropping, mean subtraction, and ten 224×224 sub-crops per image.The ten sub-crops comprised corners and center regions, with and without horizontal flips.
- Optimization: Stochastic gradient descent used a mini-batch size of 128 to update the model parameters.The supplied passage begins to specify a learning rate but does not provide its value.
- Filter stabilization: Filters with RMS values above 10^-1 were renormalized to 10^-1, addressing domination revealed by first-layer visualization during training.The procedure was especially important because input images were roughly in the [-128,128] range.
4. Convnet Visualization
Deconvolutional visualizations reveal a hierarchy from edges and textures to class-specific objects, while tracking training and transformations shows that lower layers converge early and upper features become increasingly invariant. Occlusion experiments further indicate object localization and implicit correspondence between recurring parts.
- Feature Visualization: Top-9 feature visualizations expose multiple structures exciting each feature map and reveal invariance to input deformations.Each activation is projected separately into pixel space alongside its corresponding image patch.
- Feature Visualization: Layer 2 detects corners and edge/color conjunctions, layer 3 captures texture invariances, layer 4 becomes class-specific, and layer 5 represents whole objects across poses.Examples include mesh patterns and text in layer 3, dog faces and bird legs in layer 4, and keyboards and dogs in layer 5.
- Feature Evolution during Training: Lower-layer features converge within a few epochs, whereas upper-layer features develop only after roughly 40–50 epochs.The evolution is visualized by projecting each feature map’s strongest training-set activation back into pixel space.
- Feature Invariance: Small transformations strongly affect first-layer features but have less impact on top-layer features, with translation and scaling producing quasilinear changes and stable network outputs.The analysis examines translation, rotation, and scaling through feature-vector distances and true-label probabilities.
- Object Localization: Systematic image occlusion shows that the model localizes objects, because the correct-class probability drops significantly when the object is covered.The experiment masks different portions of each input with a gray square while monitoring classifier outputs.
- Part Correspondence: Lower correspondence scores for eyes and noses than random object parts at layer 5 indicate implicit correspondence, while layer-7 scores are more similar across parts.The experiment compares repeated face-part occlusions across five dog images using the ∆ score.
5. Experiments
The experiments show that modest architectural changes improve ImageNet performance, while ImageNet-pretrained convolutional features transfer strongly to Caltech-101 and Caltech-256. Performance improves with deeper retained feature hierarchies, whereas training comparable models from scratch on the transfer datasets performs poorly.
- ImageNet: The replicated Krizhevsky et al. architecture achieves an ImageNet 2012 validation error rate within 0.1% of their reported value.This establishes a close reproduction of the published baseline.
- ImageNet: The revised model beats Krizhevsky et al.’s single-model ImageNet result by 1.7% test top-5, while multiple models reach 14.8% test error.The revision uses 7×7 filters in layer 1 and stride 2 convolutions in layers 1 and 2; the 14.8% result was the best published performance at the time, though later surpassed in the ImageNet 2013 competition.
- Transfer learning: ImageNet-pretrained features beat the best reported Caltech-101 result by 2.2% for 30 images per class, whereas the convnet trained from scratch achieves only 46.5%.The pretrained model keeps layers 1–7 fixed and trains a new softmax classifier for the target dataset.
- Transfer learning: For Caltech-256 with 60 training images per class, the ImageNet-pretrained model reaches 74.2% accuracy versus 55.2% for Bo et al. (2013), while the scratch-trained model again performs poorly.The experiments also vary the number of training images per class, including a one-shot-learning analysis.
- Feature hierarchy: On Caltech-101 and Caltech-256, classification improves steadily as more pretrained layers are retained, with the best results using all layers.Both a linear SVM and a softmax classifier are evaluated on features from different layers.
6. Discussion
The study finds that convolutional network features become increasingly compositional, invariant, and class-discriminative across layers, while visualization helps diagnose model problems. The ImageNet-trained model generalizes strongly to Caltech-101 and Caltech-256 but less well to PASCAL, possibly because of dataset bias.
- Visualization and model understanding: Visualization shows that learned features are nonrandom and become increasingly compositional, invariant, and class-discriminative in higher layers.The technique also helps debug problems with the model.
- Cross-dataset generalization: The ImageNet-trained model beats the best reported results on Caltech-101 and Caltech-256, significantly on the latter.This result questions the utility of benchmarks with small training sets, defined here as fewer than 10^4 examples.
- Cross-dataset generalization: The model generalizes less well to PASCAL, perhaps because of dataset bias, although it remains within 3.2% of the state of the art.The passage attributes the possible dataset bias to Torralba & Efros (2011).