Source-linked AI summary
Factors of Transferability for a Generic ConvNet Representation
Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, Stefan Carlsson
TL;DR
The paper asks how generic ConvNet representations can be transferred effectively to target visual-recognition tasks. It systematically studies source-training and feature-extraction factors, then reports improved performance across diverse tasks and a correlation between optimal settings and source-target task distance.
Problem
The paper addresses how to maximize transfer from a generic ConvNet representation when target labeled data, time, and computational resources are limited.
Method
The authors systematically identify and experimentally evaluate source-training and post-learning factors across 17 visual-recognition tasks.
Results
Up to 50% relative error reduction is reported on 16 popular visual-recognition datasets after optimizing transferability factors.
Takeaways & Limitations
Optimal factor settings correlate with target-task distance from the source task, and remain constant within categorical task groups while varying consistently across the ordering.
Abstract
from arXiv · showhide
Evidence is mounting that Convolutional Networks (ConvNets) are the most effective representation learning method for visual recognition tasks. In the common scenario, a ConvNet is trained on a large labeled dataset (source) and the feed-forward units activation of the trained network, at a certain layer of the network, is used as a generic representation of an input image for a task with relatively smaller training set (target). Recent studies have shown this form of representation transfer to be suitable for a wide range of target visual recognition tasks. This paper introduces and investigates several factors affecting the transferability of such representations. It includes parameters for training of the source ConvNet such as its architecture, distribution of the training data, etc. and also the parameters of feature extraction such as layer of the trained ConvNet, dimensionality reduction, etc. Then, by optimizing these factors, we show that significant improvements can be achieved on various (17) visual recognition tasks. We further show that these visual recognition tasks can be categorically ordered based on their distance from the source task such that a correlation between the performance of tasks and their distance from the source task w.r.t. the proposed factors is observed.
I. INTRODUCTION
The paper studies how to maximize transfer from a generic source ConvNet representation to target visual-recognition tasks, especially when target data and resources are limited. It identifies transferability factors, evaluates them across diverse tasks, and reports substantial gains alongside a distance-related pattern in their optimal settings.
- The paper asks how a ConvNet representation should be learned and adjusted to maximize transfer to a particular target task under limited labeled data, time, and computation.
- The authors identify source-training and post-learning factors, including architecture, training-data distribution, fine-tuning, extraction layer, and representation processing.
- The optimal settings of transferability factors are correlated with the target task’s distance from the source task, with occasional exceptions receiving plausible explanations.
- Deep ConvNet representations trained on ImageNet outperform published non-ConvNet methods on 16 of 17 diverse standard computer-vision databases.
- Up to 50% relative error reduction is reported after optimizing transferability factors across 16 popular visual-recognition datasets.The evaluation uses a linear SVM for classification and Euclidean distance for instance retrieval.
- The study evaluates transferability across 17 visual-recognition tasks spanning five subfields, ordered by their similarity to ImageNet object classification.
II. RANGE OF TARGET TASKS EXAMINED
The study evaluates 17 visual recognition tasks spanning five subfields and orders them by their distance from ImageNet object classification. The ordering reflects category structure, instance matching, visual attributes, fine-grained distinctions, and compositional interactions.
- The evaluation covers object and scene classification, visual attribute detection, fine-grained classification, compositional recognition, and instance retrieval.
- Instance retrieval is placed furthest from the source because it matches exemplars without explicit category information or a classification-learning phase.
- Attribute detection is placed before fine-grained recognition because attributes can separate source object classes, whereas fine-grained tasks require distinguishing subtle within-category differences.
- Compositional recognition is challenging because class labels depend on interactions between objects rather than isolated object categories.
- Task closeness also depends on semantic-class and image distributions, including overlap with ImageNet categories and higher-level concepts inferred from object interactions.
III. EXPERIMENTS
The experiments separate factors considered before learning a ConvNet representation from factors applied when using an off-the-shelf network.
- The paper divides transferability factors into learning factors and post-learning factors.
A. Learning Factors
The learning-factor experiments examine source task, network size and structure, and training-data properties. Across tasks, depth is more stable than width, source-task alignment matters, and diversity can be especially important.
- Network Width: Increasing width helps tasks close to the source but can cause overspecialization on distant tasks; Tiny uses 10 times fewer parameters than OverFeat while preserving most performance.
- Network Width: The largest network performs best on Pascal VOC, MIT 67, UIUC attribute, and Oxford pets, whereas OverFeat consistently suffers on retrieval tasks.
- Network Depth: Increasing convolutional depth improves performance on nearly all datasets, with degradation only on UKB and Holidays retrieval.
- Depth versus Width: Increasing depth is a more stable and parameter-efficient form of over-parametrization than increasing width, although additional layers increase sequential processing and computational complexity.
- Source Task: Source-task choice improves performance on target tasks close to the corresponding source, while the Hybrid model remains behind ImageNet on more distant tasks.
- Source Task: Late fusion of ImageNet and Places features outperforms the Hybrid model and achieves the best results on all tasks except subcategory recognition.
- Training Data: With a fixed annotation budget, reducing source-task diversity hurts target performance more than reducing distribution density.
B. Post-learning Factors
Post-learning choices affect transferability, including which layer supplies features, how convolutional outputs are pooled, whether representations are reduced, and whether networks are fine-tuned. Their best settings vary with target-task characteristics and distance from the source task.
- Training data: Training-data diversity is more important than density for transfer, because lower-diversity settings consistently underperform lower-density settings in point-to-point comparisons.Density changes images per class, whereas diversity changes the number of classes.
- Layer selection: The first fully connected layer is most generic when the final task is unknown, while later layers work best for tasks semantically closer to ImageNet.Layer 6 works best for most datasets farther from the source task; Layer 7 is strongest for UIUC attributes and MIT indoor scenes, while the 1000-way output is effective for VOC classification.
- Spatial pooling: Spatial pooling makes last-convolutional-layer representations suitable for retrieval, with higher-resolution grids needed for sculptures and historical buildings.A 1 × 1 grid acts as a soft bag-of-words representation over the whole image.
- Fine-tuning: Fine-tuning is consistently effective, with larger proportional gains for target tasks farther from ImageNet.Fine-tuning initializes the network from ILSVRC12 weights and uses target-task labels with a reduced learning rate to limit forgetting.
- Dimensionality reduction: PCA dimensionality reduction helps all instance-retrieval tasks, usually insignificantly, by addressing distance-based effects associated with high-dimensional representations.Retrieval evaluates Euclidean distances between samples in ConvNet representation space.
- Training data: Increasing source-task data improves transferred representations: Places-trained features reach 69.3% on MIT Indoor Scenes versus 62.6% for SUN397-trained features.Places uses 2.5M images, compared with 130K for SUN397; the same trend appears on other datasets.
IV. OPTIMIZED RESULTS
The optimized-results experiments transfer representations using factor settings learned from preceding analyses and compare them with a standard AlexNet-style representation. Optimizing transferability factors substantially reduces error across the evaluated tasks.
- Final comparison: The final results compare optimized representations with linear SVMs against non-ConvNet state-of-the-art systems across diverse visual-recognition tasks.The factors used for each task are reported in the final-results table.
- Source data: Places-trained representations outperform SUN-trained representations when source-task training data is increased from 100K to 2.5M images.The table compares scene-recognition representations trained on SUN397 and Places.
- Final comparison: Optimizing transferability factors decreases the remaining error of the standard representation by up to 50%.The standard representation is a medium-depth AlexNet trained on 1.3M ImageNet images, with features extracted from FC6.
V. IMPLEMENTATION DETAILS
The implementation uses ConvNets, feature normalization and transformation, linear SVMs for classification, and PCA-whitened representations with spatial search for retrieval. Parameters are selected through cross-validation.
- Model fitting: All parameters are selected using 4-fold cross-validation, with Liblinear training the classification SVMs.Retrieval uses L2 distance between whitened ConvNet representations.
- Classification pipeline: Classification features average ConvNet activations from 12 jittered image samples before normalization, signed square-root transformation, and linear SVM training.The 12 samples include crops, the whole resized image, and their mirrors.
- Retrieval pipeline: Retrieval features are L2-normalized, PCA-reduced to a smaller whitened dimension, and renormalized before matching.Spatial search is used because Oxford, Paris, and sculpture images may contain partial or differently scaled objects.
VI. CLOSING DISCUSSION
The paper presents a systematic study of factors governing ConvNet transferability across diverse recognition tasks and organizes tasks by distance from image classification. Optimized factors improve performance, while the discussion motivates generic representations that combine complementary information and invariances.
- Contributions: The study identifies transferability factors, evaluates them across many visual-recognition tasks, and orders tasks categorically by distance from image classification.The evaluated task groups include image classification, attribute detection, fine-grained recognition, compositional recognition, and instance retrieval.
- Results: Optimizing the identified factors improves state-of-the-art performance across a diverse set of standard computer-vision databases.The final results are summarized in Table VIII.
- Results: Factor effectiveness correlates with target-task distance from the source task, and optimal settings remain constant within groups while varying consistently across the ordering.The paper notes exceptions to this general trend and provides explanations for them.
- Discussion: A generic representation should encode global, local, and relational visual information together with invariances.The paper argues that analyzing task requirements by information level helps identify complementary training tasks.
- Discussion: Optimizing a representation for only one type of invariance or visual information is not expected to encode the others optimally.The discussion therefore proposes investigating multitask learning and loss functions that enforce multiple information levels and invariances.