Source-linked AI summary
The Devil is in the Tails: Fine-grained Classification in the Wild
Grant Van Horn, Pietro Perona
TL;DR
The paper asks whether visual classifiers trained in data-rich settings handle real-world long-tailed category distributions, where most classes have few examples. Using eBird-derived long-tail datasets and a pretrained Inception-v3 model, it finds that performance depends strongly on training data, while class-count increases matter little when data is sufficient and transfer across classes is negligible.
Problem
Real-world recognition involves many categories with few examples, but current algorithms are commonly developed and benchmarked on uniformly distributed datasets.
Method
The study evaluates a pretrained Inception-v3 classifier on uniform, approximate long-tail, and modeled eBird long-tail datasets with controlled class and image distributions.
Results
Performance is excellent with abundant data, declines sharply as training examples decrease, changes minimally with more classes when data is sufficient, and shows negligible transfer between classes.
Takeaways & Limitations
Uniform training datasets do not address the long-tailed challenge, motivating continued research on transfer and low-shot learning.
Abstract
from arXiv · showhide
The world is long-tailed. What does this mean for computer vision and visual recognition? The main two implications are (1) the number of categories we need to consider in applications can be very large, and (2) the number of training examples for most categories can be very small. Current visual recognition algorithms have achieved excellent classification accuracy. However, they require many training examples to reach peak performance, which suggests that long-tailed distributions will not be dealt with well. We analyze this question in the context of eBird, a large fine-grained classification dataset, and a state-of-the-art deep network classification algorithm. We find that (a) peak classification performance on well-represented categories is excellent, (b) given enough data, classification performance suffers only minimally from an increase in the number of classes, (c) classification performance decays precipitously as the number of training examples decreases, (d) surprisingly, transfer learning is virtually absent in current methods. Our findings suggest that our community should come to grips with the question of long tails.
1 Introduction
Real-world visual categories follow long-tailed distributions: a few classes have many examples, while most have few. The paper asks whether current algorithms generalize well under these conditions and finds that they do not when data is scarce.
- Recent progress in visual recognition has been enabled by deep networks, large annotated datasets, and increased computational resources.
- Large datasets provide both training data and shared benchmarks that guide comparisons and architectural progress.
- Real-world datasets of birds, trees, and other species contain a few frequent classes and many classes represented by few specimens.
- The paper tests whether state-of-the-art classifiers trained on uniformly distributed datasets handle real-world long tails effectively.
- When data is abundant, machine vision can rival humans, but performance is not comparable when most classes have scarce data, even if a few are abundant.
- The paper analyzes this question experimentally and organizes the study around datasets, training, experiments, and conclusions.
2 Related Work
Related work spans fine-grained classification, imbalanced datasets, transfer learning, and low-shot learning. This paper focuses specifically on how long-tailed training distributions affect fine-grained classification within a single domain.
- Fine-Grained Visual Classification: Fine-grained datasets cover birds, dogs, airplanes, flowers, leaves, trees, and cars, but are generally uniform or designed with enough data per task.
- Fine-Grained Visual Classification: Prior fine-grained studies often propose specialized models, although newer convolutional networks can sometimes match their performance without custom modifications.
- Imbalanced Datasets: Imbalanced-dataset methods include algorithmic approaches such as cost-sensitive learning and data approaches such as over-sampling or under-sampling.
- Transfer Learning: Transfer learning adapts representations from one domain to another, commonly through pretrained features or fine-tuning on a target dataset.
- Transfer Learning: This study examines whether knowledge transfers from well-represented to sparsely represented classes within one long-tailed domain.
- Low Shot Learning: Low-shot learning studies novel concepts from few examples, including methods that synthesize features or learn regressions between classifiers trained with different data amounts.
3 Experiment Setup
The experiments use eBird images to compare uniform, approximate long-tail, and modeled long-tail datasets with a fine-tuned pretrained Inception-v3 classifier. Dataset sizes and class frequencies are systematically varied while evaluation uses held-out images and top-1 accuracy.
- The study constructs uniform, long-tail, and approximate long-tail datasets from eBird images curated by citizen scientists and regional experts.
- Uniform Datasets: Uniform datasets contain equal images per class—10, 100, 1K, or 10K—and 10, 100, or 1K classes when data permits.
- Approx. Long Tail Datasets: Approximate long-tail datasets use 1K classes divided into head and tail groups with equal within-group image counts.
- Long Tail Datasets: The full eBird distribution is modeled with a two-piece broken power law, varying y to shift image counts and h to shift the number of head classes.
- Long Tail Datasets: The modeled datasets use exponents a1 = −0.3472 and a2 = −1.7135, a minimum of 10 images per class, and y ∈ {1K, 10K}, h ∈ {10, 100}.
- Model and Training: All experiments fine-tune a pretrained Inception-v3 network across layers using a fixed batch size of 32 and a prescribed learning-rate schedule.
- Testing: Testing selects the highest-validation-accuracy model after overfitting-based stopping and evaluates center-cropped images with top-1 image accuracy.
4 Experiments
Experiments show strong performance on uniform datasets when training data is abundant, but long-tailed performance is dominated by scarce tail classes. Changing sampling does little overall, and additional head-class data transfers poorly to the tail.
- 4.1 Uniform Datasets: 96% accuracy is achieved on 10 bird species with 10K training images per class, demonstrating excellent performance with abundant uniform data.Fine-tuning is beneficial except in the extreme 10-class, 10-images-per-class case, where the model overfits quickly.
- 4.1 Uniform Datasets: A 10x increase in classes causes only a small error increase, while each 10x increase in training data cuts error by at least 2x.The network scales well from more classes but remains highly sensitive to the number of examples per class.
- 4.2 Uniform vs. Natural Sampling: Natural sampling is typically as good as or better than uniform sampling overall, with head classes benefiting clearly from natural sampling.For tail classes, average accuracy is 32.1% versus 34.2% and 33.5% versus 38.6% under uniform versus natural sampling in the two visualized datasets.
- 4.3 Transferring Knowledge from the Head to the Tail: 94.7% head-class accuracy is reached while tail accuracy remains near 33.2% when head-class data increases from a 1K-class, 10-images-per-class baseline.With 100 images per class across 1K classes, head accuracy reaches 94% while tail accuracy remains at 71%.
- 4.3 Transferring Knowledge from the Head to the Tail: Tail classes do not benefit from head classes: tail-only models perform as well or better than joint head-and-tail models.The joint model’s overall error is 48.6%, and experiments on eBird-shaped long-tail distributions reproduce the same pattern.
5 Discussion and Conclusions
The paper examines state-of-the-art classification under real-world long-tailed distributions, where most categories have few examples. Performance is excellent with abundant data, but scarce tail data sharply degrades accuracy and current models show negligible transfer between classes.
- Real-world datasets contain a few highly represented categories and many categories observed only rarely, unlike common uniformly distributed benchmarks.
- The study experimentally evaluates a state-of-the-art classification model on approximate and realistic long-tailed datasets.
- About 4% species classification error is achieved on eBird when each species has 10^4 training images.
- With sufficient training data, classification performance suffers only minimally as the number of classes increases, despite potentially tens of millions of categories.
- 48.6% average classification error occurs for a 550,692-image long-tail dataset, versus about 27% when the same images are distributed uniformly across 2215 classes.
- Classification error more than doubles whenever training images are reduced tenfold, while transfer learning between classes is negligible on the tested datasets.
- The findings emphasize continued research on transfer and low-shot learning and on datasets and algorithms that reflect real-world long-tailed statistics.
Supplementary
Supplementary experiments compare head and tail performance across uniform and long-tailed training regimes. Tail data can improve head-class performance, but the benefit declines as head data increasingly outweighs tail data.
- Detailed Results from Experiments: Head accuracy increases when moving from uniform to approximate long-tailed image distributions, whereas tail accuracy benefits little, if at all.
- Detailed Results from Experiments: Tail Isolated ACC measures tail performance after removing head images and ignoring head predictions, probing representation benefits from additional head-class data.
- Detailed Results from Experiments: The whole model is trained with both head and tail classes, with overall, head, and tail performance reported separately.
- Increasing Performance on the Head Classes: Joint training with equally represented tail classes reduces head-class error compared with training exclusively on head classes.
- Increasing Performance on the Head Classes: 6.3%–32.5% error reduction is obtained for head classes by training with the approximate long tail.
- Increasing Performance on the Head Classes: The benefit of tail training typically decreases as the ratio of head images to tail images increases, becoming harmful when that ratio exceeds 10.
- Increasing Performance on the Head Classes: Experiments monitor all classes using a uniform validation set of 10 images per class to select the model iteration for testing.
- Increasing Performance on the Head Classes: Figure 7 compares head-only and head-plus-tail models while ignoring tail predictions at test time; dashed lines represent additional tail categories.