Source-linked AI summary
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning
Yin Cui, Yang Song, Chen Sun, Andrew Howard, Serge Belongie
TL;DR
The paper studies how to improve large-scale FGVC and how to transfer knowledge to small domain-specific FGVC datasets despite scarce expert-labeled data. It uses higher-resolution inputs, long-tail-aware two-stage training, and Earth Mover’s Distance for source-domain selection. The resulting methods achieve top iNaturalist performance and outperform ImageNet pre-training across commonly used FGVC datasets.
Problem
Expert annotation makes domain-specific FGVC data difficult to scale, while effective training for large-scale non-iconic images and source selection for transfer learning remain open questions.
Method
The paper combines higher-resolution input training, balanced-subset fine-tuning for long-tailed data, and Earth Mover’s Distance to select visually similar source-domain subsets.
Results
The method achieves top performance on iNaturalist, improves under-represented categories, and outperforms ImageNet pre-training with state-of-the-art results on common FGVC datasets.
Takeaways & Limitations
Transfer learning benefits from pre-training on a source domain that is visually more similar to the target, while balanced-subset fine-tuning improves tail-category performance.
Takeaways & Limitations
The domain-similarity analysis ignores domain scale by normalizing source and target image counts, although a scale factor could be added.
Abstract
from arXiv · showhide
Transferring the knowledge learned from large scale datasets (e.g., ImageNet) via fine-tuning offers an effective solution for domain-specific fine-grained visual categorization (FGVC) tasks (e.g., recognizing bird species or car make and model). In such scenarios, data annotation often calls for specialized domain knowledge and thus is difficult to scale. In this work, we first tackle a problem in large scale FGVC. Our method won first place in iNaturalist 2017 large scale species classification challenge. Central to the success of our approach is a training scheme that uses higher image resolution and deals with the long-tailed distribution of training data. Next, we study transfer learning via fine-tuning from large scale datasets to small scale, domain-specific FGVC datasets. We propose a measure to estimate domain similarity via Earth Mover's Distance and demonstrate that transfer learning benefits from pre-training on a source domain that is similar to the target domain by this measure. Our proposed transfer learning outperforms ImageNet pre-training and obtains state-of-the-art results on multiple commonly used FGVC datasets.
1. Introduction
The paper addresses large-scale fine-grained categorization and transfer learning to small domain-specific FGVC datasets, where expert annotation is difficult to scale. It proposes training strategies for long-tailed data and domain-similar source selection, achieving strong results on iNaturalist and established FGVC benchmarks.
- Expert-level annotation makes fine-grained datasets difficult to scale, so commonly used FGVC datasets typically contain around 10k labeled training images.
- The paper investigates both effective modeling for large-scale, non-iconic fine-grained images and transfer learning from large datasets to domain-specific FGVC.
- Higher input resolution and two-stage fine-tuning on a more balanced subset improve iNaturalist performance, especially for under-represented categories.
- Earth Mover’s Distance is used to measure visual similarity between source and target domains for selecting transfer-learning source subsets.
- The proposed transfer-learning method outperforms ImageNet pre-training and achieves state-of-the-art results on commonly used fine-grained datasets.
2. Related Work
Prior FGVC work improves recognition through fine-grained representations, auxiliary information, and data augmentation, while transfer learning commonly relies on ImageNet pre-training. This paper emphasizes systematic study of image resolution and long-tailed distributions as underexplored issues.
- Fine-Grained Visual Categorization (FGVC): FGVC methods capture subtle category differences using bilinear or higher-order feature interactions, visual attention, metric learning, and auxiliary information.
- Fine-Grained Visual Categorization (FGVC): Collecting additional web images is a common response to limited FGVC training data, whereas this work transfers networks trained on existing large-scale datasets.
- High-resolution inputs are increasingly used for FGVC, but prior work had not systematically studied their effect on large-scale fine-grained datasets.
- Long-tailed distributions are important in real-world data but remain relatively unexplored because benchmark datasets are often made nearly evenly distributed.
- Transfer Learning: ImageNet-trained CNNs are widely used for transfer learning through feature extraction or fine-tuning, with prior work loosely connecting transfer performance to domain similarity.
3. Large Scale Fine-Grained Categorization
The large-scale FGVC training scheme focuses on higher-resolution inputs and long-tailed category imbalance in iNaturalist. It first learns features from all data, then fine-tunes on a more balanced subset to improve category balance.
- The iNaturalist training scheme specifically targets higher image resolution and long-tailed distribution in large-scale fine-grained categorization.
- The Effect of Image Resolution: Higher-resolution inputs preserve richer subtle details, and experiments varying sizes from 299 × 299 to 560 × 560 show greatly improved iNaturalist performance.
- Long-Tailed Distribution: About 1,500 iNaturalist categories have fewer than 30 training images, while the largest-to-smallest class-size ratio is about 435.
- The Effect of Image Resolution: Table 1 compares default input image resolutions across network architectures and reflects the trend toward higher-resolution inputs.
- Long-Tailed Distribution: The two-stage scheme trains first on the original imbalanced dataset, then fine-tunes with a small learning rate on a more balanced subset.
4. Transfer Learning
The paper defines transfer learning between fine-grained domains by measuring domain similarity with Earth Mover’s Distance (EMD). It normalizes category sizes, represents categories by mean features, and greedily selects source categories most similar to the target.
- 4.1. Domain Similarity: Image distances are computed as Euclidean distances between feature representations from a generic feature extractor.The experiments use penultimate-layer features from a ResNet-101 trained on the large-scale JFT dataset.
- 4.1. Domain Similarity: The similarity calculation normalizes domain image counts and therefore ignores domain scale, an assumption the authors report works well in practice.The paper notes that a scale factor could be added to generalize the definition to domain size.
- 4.1. Domain Similarity: The method measures transfer-learning domain similarity by treating transfer as moving source images toward a target domain and minimizing total feature-space work with EMD.Image-to-image work is defined by feature distance, while domain distance is the least total work over feasible flows.
- 4.1. Domain Similarity: Source and target domains are represented by categories weighted by their normalized image counts, with category features summarized by mean image features.The category-level representation makes the computation more tractable than operating on every image individually.
- 4.1. Domain Similarity: Figure 3 visualizes source and target categories as red and green circles, with circle size encoding normalized category image counts and blue arrows encoding optimal EMD flows.The arrows show flows from source to target after solving the EMD optimization problem.
- 4.1. Domain Similarity: For source selection, the method greedily ranks individual source categories by their similarity to the target and retains the top k categories.Each source category is evaluated against the target as a singleton source domain.
5. Experiments
Experiments show that higher-resolution inputs and balanced-subset fine-tuning improve large-scale iNat recognition, while domain-similar source subsets improve transfer across FGVC datasets.
- Experimental setup: The transfer-learning evaluation uses ImageNet and iNat as source domains and seven fine-grained datasets spanning natural and man-made categories as targets.The evaluated architectures include ResNet, Inception, and SENet variants.
- Large-scale FGVC: Higher input resolutions achieve better performance on iNat when using Inception-v3.The compared input resolutions are 299, 448, and 560.
- Large-scale FGVC: 1.95% top-1 and 0.92% top-5 improvements occur for head categories, versus 5.74% top-1 and 2.71% top-5 improvements for tail categories after balanced-subset fine-tuning.Tail categories contain fewer than 100 training images; head categories contain at least 100.
- Large-scale FGVC: The winning iNaturalist challenge entry relies on higher image resolution and further fine-tuning on a more balanced subset.The balanced-subset strategy addresses the long-tailed training distribution.
- Domain similarity and transfer learning: Transfer performance is generally better when fine-tuning from a more visually similar source domain, except on Food101, where performance is nearly unchanged.The authors attribute Food101’s exception to its relatively large training set of 750 images per class.
- Domain similarity and transfer learning: Subset B performs well across FGVC datasets and surpasses ImageNet pre-training by a large margin on CUB200 and NABirds.Subset B was constructed from categories selected by the proposed domain similarity measure.
6. Conclusions
The paper concludes that higher-resolution training and balanced-subset fine-tuning support strong large-scale iNat recognition, while Earth Mover’s Distance-based domain similarity improves transfer learning.
- Conclusions: The proposed iNat training scheme combines higher-resolution input images with fine-tuning to address the long-tailed distribution.The paper reports top performance on the large-scale iNaturalist dataset.
- Conclusions: Earth Mover’s Distance is used to capture domain similarity, with better transfer learning achieved from more similar domains.The conclusion frames domain similarity as a factor beyond large-scale pre-training alone.
- Conclusions: The transfer-learning approach achieves state-of-the-art performance on all compared FGVC datasets, especially CUB200 and NABirds.The comparison uses the same 448 × 448 input size by convention.