Source-linked AI summary
Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning
Weifeng Ge, Yizhou Yu
TL;DR
Deep networks need substantial labeled data, but many visual tasks lack enough samples and risk poor performance. The paper selectively retrieves source images with similar low-level characteristics and jointly fine-tunes shared convolutional layers, achieving state-of-the-art results across multiple visual classification tasks.
Problem
Insufficient training samples make deep networks difficult to train effectively for visual tasks such as fine-grained classification, with overfitting a reported concern.
Method
Selective joint fine-tuning jointly trains a data-poor target task and data-rich source task using only source images retrieved for similar low-level characteristics.
Results
The scheme achieves state-of-the-art performance on multiple visual classification tasks with much less training data than recent deep learning architectures require.
Takeaways & Limitations
The approach reuses existing labeled source datasets without requiring additional labeling effort beyond those datasets.
Takeaways & Limitations
The source domain must be large enough to train a deep convolutional network from scratch and ideally provide diversified low-level characteristics.
Abstract
from arXiv · showhide
Deep neural networks require a large amount of labeled training data during supervised learning. However, collecting and labeling so much data might be infeasible in many cases. In this paper, we introduce a source-target selective joint fine-tuning scheme for improving the performance of deep learning tasks with insufficient training data. In this scheme, a target learning task with insufficient training data is carried out simultaneously with another source learning task with abundant training data. However, the source learning task does not use all existing training data. Our core idea is to identify and use a subset of training images from the original source learning task whose low-level characteristics are similar to those from the target learning task, and jointly fine-tune shared convolutional layers for both tasks. Specifically, we compute descriptors from linear or nonlinear filter bank responses on training images from both tasks, and use such descriptors to search for a desired subset of training samples for the source learning task. Experiments demonstrate that our selective joint fine-tuning scheme achieves state-of-the-art performance on multiple visual classification tasks with insufficient training data for deep learning. Such tasks include Caltech 256, MIT Indoor 67, Oxford Flowers 102 and Stanford Dogs 120. In comparison to fine-tuning without a source domain, the proposed method can improve the classification accuracy by 2% - 10% using a single model.
1. Introduction
Deep networks need abundant labeled data, but insufficient-data tasks are vulnerable to overfitting. Selective joint fine-tuning addresses this by jointly fine-tuning shared layers using relevant source samples selected through low-level similarity, achieving strong results across visual classification tasks.
- Insufficient training data can make deep CNNs overfit quickly and perform worse than traditional classifiers with handcrafted features.
- Fine-tuning ImageNet- or Places-pretrained networks remains vulnerable to overfitting because the parameter-to-sample ratio is unchanged.
- Selective joint fine-tuning pairs a data-limited target task with a data-rich source task and jointly fine-tunes shared convolutional layers.
- Source samples are selected by comparing low-level characteristics using descriptors derived from linear or nonlinear filter-bank responses.
- Low-level matching supports shared-layer learning because lower convolutional layers provide the foundation for higher-level features while allowing broader semantic variation.
- State-of-the-art performance was demonstrated on Caltech 256, MIT Indoor 67, Oxford Flowers 102, and Stanford Dogs 120 with substantially less training data.
2. Related Work
The paper relates selective joint fine-tuning to multi-task learning and transfer learning, while distinguishing its low-level nearest-neighbor selection from semantic or noisy-data approaches.
- Multi-task learning shares representations or classifiers across related tasks, including deep models with shared convolutional layers and task-specific losses.
- Transfer learning applies knowledge from one domain to related tasks through instance, feature, or model adaptation.
- Unlike approaches that add noisy web images using target-category keywords, this method searches a labeled dataset for source neighbors using low-level features.
- Low-level feature search can outperform high-level semantic search, while fine-tuning remains prone to rapid overfitting on data-limited tasks.
3. Selective Joint Fine-tuning
Selective joint fine-tuning pairs a data-poor target task with a data-rich source task, but trains the source task using only images selected for low-level similarity. Filter-bank descriptors guide image retrieval, while adaptive neighbor counts give additional source examples to uncertain target samples.
- Source Domain Training Images: Only a selected subset of source-domain images is used, with each target image associated with source images having similar low-level characteristics.The source domain is expected to be large and diverse enough to train a deep CNN, while selection narrows its training contribution.
- Source-Target Joint Fine-tuning: The pipeline shares convolutional layers between source and target tasks while using separate output layers and task-specific cost functions.The entire target domain trains the target task, whereas retrieved source images train the source task only.
- Filter Bank and Image Descriptor: Filter-bank response histograms form image descriptors for searching source-domain nearest neighbors.The experiments use Gabor filters or kernels and nonlinear activations from the first two convolutional layers of ImageNet-pretrained AlexNet.
- Filter Bank and Image Descriptor: Histogram construction scans target-domain responses to set per-filter bounds, divides intervals into bins, and concatenates the histograms into feature vectors.Bin widths are adapted so bins contain roughly equal percentages of pixels, reducing concentration in a single bin.
- Hard Samples in the Target Domain: The method increases source neighbors for target samples with high classification uncertainty through iterative fine-tuning.Uncertainty is measured with information entropy after each iteration, and the evolving source subset changes the learned representation.
- Experimental Setup: The filter-bank comparison uses a 50-layer residual network with half the original architecture's convolutional kernels.This experiment compares Gabor filters with combinations of AlexNet kernels in selective joint fine-tuning.
4. Experiments
Experiments evaluate selective joint fine-tuning across four visual classification tasks, using retrieved source images, shared training, and comparisons against conventional fine-tuning and alternative settings.
- Implementation: Experiments use a 152-layer residual network, conventional fine-tuning without source data as the baseline, and mixed source-target mini-batches with separate classifiers.Source and target samples are split after average pooling and sent to their corresponding softmax layers.
- Source Image Retrieval: Retrieved source images share colors, local patterns, and global structures with target images, although low-level similarity can span unrelated semantic categories.The retrieval procedure uses nearest neighbors from ImageNet or Places based on low-level characteristics.
- Fine-grained Object Recognition: 90.2% mean class accuracy on Stanford Dogs 120 is 9.8% higher than conventional fine-tuning without a source domain.Selective joint fine-tuning also exceeds the result reported for expanding the target set with Google image search by 4.3%.
- Fine-grained Object Recognition: 94.7% mean class accuracy on Oxford Flowers 102 is 3.3% higher than the previous best single-network result, while an ensemble reaches 95.8%.Using the validation set as additional training data, the single-network result also outperforms the cited comparison by 1.7%.
- General Object Recognition: Selective joint fine-tuning improves Caltech 256 classification accuracy by 2.6%–4.1% over conventional fine-tuning, with larger gains when fewer target samples are available.In most experiments, performance is also significantly better than previous state-of-the-art results.
- Scene Classification: On MIT Indoor 67, mean class accuracy reaches 85.8% with Places as source, 4.1% above conventional fine-tuning and 4.8% above the previous best single-network result.ImageNet alone reaches 82.8%, while combining ImageNet and Places does not further improve performance; averaging two networks exceeds the previous ensemble result by 0.9%.
- Ablation Study: Ablations show that using all source images, random source subsets, fixed retrieval counts, Gabor filters, or no source domain reduces performance relative to the proposed pipeline.The reported drops include 4.6% and 1.3% for using all source data, 4.7% and 1.5% for random subsets, and 9.8% and 2.4% without a source domain on the two evaluated datasets.
5. Conclusions
The paper introduces selective joint fine-tuning to improve deep learning tasks with insufficient training data, achieving state-of-the-art performance across multiple visual classification tasks. Selecting the most suitable source domain for a target task remains an open problem.
- Selective joint fine-tuning trains a data-poor target task simultaneously with a data-rich source task.
- The scheme borrows samples from existing labeled source datasets without requiring additional labeling effort.
- Experiments achieve state-of-the-art performance on multiple visual classification tasks with insufficient training data.
- Finding the most suitable source domain for a specific target task remains an open problem.