Source-linked AI summary
How transferable are features in deep neural networks?
Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson
TL;DR
The paper asks where and how features transition from general to specific across a deep network. It quantifies layer-wise transferability and finds that specialization and co-adaptation impede transfer, while transferred initialization can improve generalization after fine-tuning.
Problem
The paper examines where and whether the transition from general to specific features occurs suddenly or across several network layers.
Method
The authors quantify layer generality through cross-task transfer performance using paired convolutional networks trained on separate ImageNet class groups.
Results
Transferability is limited by both feature specificity and optimization difficulties from splitting co-adapted layers, with either issue dominating at different network depths.
Takeaways & Limitations
Transferred features from almost any number of layers can boost generalization after fine-tuning, and even distant-task features can outperform random features.
Takeaways & Limitations
Performance was aggregated over several layers because obtaining each point was computationally expensive, leaving few data points per layer.
Abstract
from arXiv · showhide
Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.
1 Introduction
The paper quantifies how convolutional-network features transition from general first-layer filters to task-specific last-layer representations by measuring their transferability. It identifies distinct causes of transfer degradation, task-distance effects, advantages over random features, and lasting benefits from transferred initialization.
- First-layer features commonly resemble Gabor filters or color blobs across natural-image datasets and training objectives, whereas last-layer features depend strongly on the dataset and task.
- The paper quantifies layer generality or specificity by measuring how well features transfer between tasks, then characterizes the layer-by-layer transition using paired ImageNet convolutional networks.
- Two separate issues degrade performance without fine-tuning: feature specificity and optimization difficulties from splitting co-adapted neighboring-layer neurons; either can dominate at different network depths.
- Transfer benefits decrease as the base and target tasks become more dissimilar.
- Transferred weights outperform random lower-layer weights on ImageNet, whether transferred features are frozen or fine-tuned.The paper reports lower performance for random lower-layer weights than previously reported on smaller datasets.
- Initializing with transferred features from almost any number of layers boosts generalization after fine-tuning, and the effect of seeing the first dataset persists through extensive fine-tuning.
2 Generality vs. Specificity Measured as Transfer Performance
The paper measures feature generality by how well features learned on one classification task transfer to another, with transferability interpreted relative to task similarity. It evaluates frozen and fine-tuned transfers across layers using paired ImageNet classification tasks built from complementary class splits.
- Defining transferability: Generality is defined by the extent to which features learned on task A can be used for another task B.The definition depends on the similarity between tasks A and B.
- Experimental design: The experiments randomly split ImageNet’s 1000 classes into two 500-class tasks, each containing approximately 645,000 examples, and train separate eight-layer convolutional base networks.The networks are called baseA and baseB.
- Experimental design: For each transfer depth n ∈ {1, 2, . . . , 7}, early layers from one base network are copied into a target-task network, while higher layers are randomly initialized and trained on the target dataset.The transfer network A3B illustrates copying the first three layers from A and training layers 4–8 on B; the self-transfer network B3B is the control.
- Transfer conditions: Transferability is tested with transferred layers either frozen or fine-tuned, using A3B versus baseB performance as evidence about the generality of third-layer features.Fine-tuned variants allow all layers to learn, whereas frozen variants keep the transferred layers fixed.
3 Experimental Setup
The experiments prioritize studying transfer results over maximizing absolute performance. They use Caffe’s reference implementation on a well-known architecture to support comparability, extensibility, and broad usefulness.
- Experimental focus: The study evaluates transfer results rather than optimizing absolute model performance.This focus distinguishes the experiments from efforts centered on tuning large convolutional models for peak accuracy.
- Experimental focus: Experiments use a well-known convolutional architecture implemented through Caffe’s reference implementation.The implementation choice follows broad interest in large convolutional models after the ImageNet 2012 competition.
- Experimental focus: The implementation is intended to make results comparable, extensible, and useful to many researchers.
4 Results and Discussion
The results compare transfer learning on similar and dissimilar ImageNet task splits with networks using random weights. They show that baseline difficulty, task similarity, and optimization affect performance across layers.
- Random A/B splits: Random 500-class networks achieve 0.625 top-1 accuracy, or 37.5% error, versus 42.5% top-1 error for the 1000-class network.The lower error is attributed to having half as many classes, and therefore half as many ways to make mistakes.
- Man-made/natural split: Transfer performance is expected to decline as base and target tasks become less similar, tested using random splits versus a man-made/natural ImageNet split.The man-made/natural split assigns man-made classes to one task and natural classes to the other, maximizing dissimilarity within ImageNet.
- Man-made/natural split: Networks targeting natural categories outperform those targeting man-made categories in the dissimilar-task experiment.The difference may reflect 449 versus 551 classes, an easier natural-category task, or both.
- Random weights: Random filters cause performance to fall quickly in layers 1 and 2 and reach near-chance levels for layers 3+.This suggests random weights are not straightforwardly effective in these larger convolutional networks, although comparisons with earlier work are not direct.
- Random weights: Layers 5, 6, and 7 show almost identical performance across random upper-layer initialization draws, suggesting similar results across repeated runs.Performance at each layer uses different random upper-layer initialization weights.
- Random weights: Three random layers produced chance-level validation performance because training loss failed to converge, though different hyperparameters might improve convergence.The result illustrates an optimization limitation in the random-weight experiment.
5 Conclusions
The paper introduces a method for quantifying feature transferability across network layers, revealing feature generality or specificity. It identifies optimization difficulties from splitting co-adapted layers and higher-layer specialization as distinct causes of reduced transferability.
- The method quantifies feature transferability at each network layer to reveal whether features are general or specific.
- Transferability is reduced by optimization difficulties when networks are split within fragilely co-adapted layers.
- Transferability is also reduced when higher-layer features specialize to the original task at the expense of target-task performance.
Supplementary material for: How transferable are
The supplementary material accompanies the paper “How transferable are features in deep neural networks?” by Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.
- The supplementary material is associated with the paper “How transferable are features in deep neural networks?”
- The authors are Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.
- The listed affiliations span Cornell University, the University of Wyoming, and the University of Montreal.
A Training Details
The study used a commonly studied reference convolutional architecture rather than optimizing absolute performance, trained with specified SGD settings, and achieved a 42.5% final top-1 validation error.
- Architecture: The experiments used the reference implementation of a commonly studied convolutional architecture instead of adopting later filter-size and stride improvements.Zeiler and Fergus recommended reducing first-layer filters from 11 × 11 to 7 × 7 and stride from 4 to 2, but this study prioritized a standard architecture.
- Optimization: Training used SGD with momentum, batch size 256, momentum 0.9, weight decay 0.0005, and a learning rate reduced tenfold every 100,000 iterations until 450,000 iterations.Each iteration took about ∼1.7 seconds on an NVidia K…
- Baseline performance: 42.5% final top-1 validation error was comparable to Donahue et al.’s 42.9% and 1.8% worse than Krizhevsky et al.’s result.These checks were intended to show reasonable convergence rather than improve state-of-the-art performance.
- Reproducibility: The authors released the code and parameter files needed to reproduce the experiments.The materials were made available at yosinski.com/transfer.
B How Much Does an AlexNet Architecture Overfit?
An AlexNet trained on the full ImageNet dataset shows only slight overfitting: validation accuracy nearly reaches an asymptote as the number of training examples increases. This helps explain why random filters perform poorly here compared with earlier results on smaller networks and Caltech-101.
- Random-filter comparison: Random filters perform relatively poorly in ImageNet-trained AlexNet, contrasting with reported successes in smaller convolutional networks trained on Caltech-101.The paper hypothesizes that the difference arises because ImageNet is large enough to train AlexNet without excessive overfitting.
- Overfitting assessment: The nearly zero slope between 1000 and 1300 examples per class indicates that adding more data would not significantly improve validation accuracy.Figure S1 describes the rightmost line segment as nearly flat and interprets the result as evidence of only slight overfit.
- Overfitting assessment: Validation accuracy nearly converges at the largest dataset sizes, indicating that AlexNet’s overfitting on ImageNet is slight.The experiment varied the maximum number of examples per class from 1 to 1300, with n = 1300 corresponding to the complete ImageNet dataset.
C Man-made vs. Natural Split · Supplementary References
The study constructs maximally dissimilar ImageNet task splits by using WordNet structure to separate man-made artifacts from natural organisms. The resulting categories contain 522 and 410 classes, leaving 68 unassigned classes.
- C Man-made vs. Natural Split: The split seeks two disjoint ImageNet subsets that are as semantically unrelated as possible.WordNet nodes are annotated by the number of ImageNet classes reachable through parent-to-child traversal.
- C Man-made vs. Natural Split: The WordNet ranking begins with entity, which contains all 1000 ImageNet categories.The listed node counts include entity at 997 reachable classes and physical entity at 958.
- C Man-made vs. Natural Split: 522 classes belong to artifact, the first ranked subset containing approximately half of ImageNet.Artifact is identified as the man-made category.
- C Man-made vs. Natural Split: 410 classes belong to organism, which is mutually exclusive with artifact and forms the natural category.The organism subset is the next approximately half-sized subset encountered in the ranking.
- C Man-made vs. Natural Split: 68 ImageNet classes remain outside the man-made and natural categories.This follows from 1000−522−410 = 68.
- C Man-made vs. Natural Split: The man-made category includes manually placed classes such as hotdog, dough, pizza, traffic light, and toilet tissue.The passage labels these as classes manually placed into the man-made category.
- C Man-made vs. Natural Split: The natural category includes manually placed classes such as strawberry, cabbage, seashore, pineapple, lemon, mushroom, and cucumber.The supplied passage labels these as classes manually placed into the natural category.