Source-linked AI summary
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition
Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, Li Fei-Fei
TL;DR
Fine-grained recognition traditionally relies on costly expert annotation, limiting data availability and scalability. The paper trains generic recognition models using noisy web data and simple filtering instead. It reports strong results across four datasets without their manually annotated training labels and scales to more than 10,000 categories.
Problem
Manual expert or crowdsourced annotation limits the available data and makes fine-grained recognition difficult to scale.
Method
The paper trains effective fine-grained recognition models from noisy web data using simple, generic recognition methods and filtering.
Results
92.3% top-1 accuracy on CUB-200-2011, 85.4% on Birdsnap, 93.4% on FGVC-Aircraft, and 80.8% on Stanford Dogs were achieved without using their manually annotated training sets.
Takeaways & Limitations
The approach is effective across four fine-grained datasets and scalable to 14,553 categories.
Takeaways & Limitations
Performance can be limited by domain shift, mismatched category names or definitions, and categories lacking publicly available imagery.
Abstract
from arXiv · showhide
Current approaches for fine-grained recognition do the following: First, recruit experts to annotate a dataset of images, optionally also collecting more structured data in the form of part annotations and bounding boxes. Second, train a model utilizing this data. Toward the goal of solving fine-grained recognition, we introduce an alternative approach, leveraging free, noisy data from the web and simple, generic methods of recognition. This approach has benefits in both performance and scalability. We demonstrate its efficacy on four fine-grained datasets, greatly exceeding existing state of the art without the manual collection of even a single label, and furthermore show first results at scaling to more than 10,000 fine-grained categories. Quantitatively, we achieve top-1 accuracies of 92.3% on CUB-200-2011, 85.4% on Birdsnap, 93.4% on FGVC-Aircraft, and 80.8% on Stanford Dogs without using their annotated training sets. We compare our approach to an active learning approach for expanding fine-grained datasets.
1 Introduction
Fine-grained recognition traditionally depends on expert- or crowd-annotated datasets, limiting available data and scalability. The paper instead uses noisy web data and generic recognition methods, achieving strong benchmark results without manually annotated training labels and scaling beyond 10,000 categories.
- Fine-grained recognition distinguishes visually similar categories such as dog breeds, bird species, and car models.
- The standard pipeline collects expert-annotated data, then trains recognition models that may also require parts, attributes, or relationship annotations.
- Manual annotation constrains both performance and scalability because the available data is limited.
- Scaling the traditional approach to 14,000 bird species, 278,000 butterfly and moth species, or 941,000 insect species is prohibitive.
- 92.3% top-1 accuracy on CUB-200-2011, 85.4% on Birdsnap, 93.4% on FGVC-Aircraft, and 80.8% on Stanford Dogs were achieved without manually annotated training labels from those datasets.
- The approach also reports initial recognition results for over 10,000 bird species and 14,000 butterfly and moth species.
2 Related Work
Prior fine-grained recognition research emphasizes better classifiers, dataset curation, and human-in-the-loop collection, while rarely addressing the impact and scalability of training data. This work presents noisy web data as an alternative to manual annotation.
- Most fine-grained recognition research focuses on improving classification models given limited data.
- Some prior methods augment labeled datasets with web images or combine fine-grained supervision with tasks that require no fine-grained labels.
- Dataset curation and interactive recognition methods collect fine-grained images effectively but remain limited by manual annotation requirements.
- Web-data learning methods have generally addressed coarse-grained detection and classification rather than fine-grained recognition.
3 Noisy Fine-Grained Data
The paper analyzes publicly available web imagery for fine-grained categories across birds, aircraft, Lepidoptera, and dogs. Web search supplies substantially more data than curated datasets, but introduces cross-domain and cross-category noise that filtering targets selectively.
- The analysis measures the quantity, distribution, and noise of web imagery across birds, aircraft, Lepidoptera, and dogs.
- The study constructs category lists covering 10,982 bird species, 14,553 Lepidoptera species, 409 aircraft types, and 515 dog categories.
- Images are retrieved through Google image search using scientific names for birds and Lepidoptera and category names for aircraft and dogs.
- Web search provides an order of magnitude more images than existing fine-grained datasets; Birdsnap's 94.8 images per category represent only 13% of the images obtainable by simple search.
- 9.8 million images were collected for 26,458 categories.
- Cross-domain noise consists of images outside the relevant fine-grained domain, whereas cross-category noise assigns the wrong label within that domain.
- Cross-domain noise reaches a maximum of 34.2% for L-Butterfly and tends to decrease as the number of images per category increases.
- Web images show moderately more cross-category noise than the clean CUB test set, while retaining a similar overall confusion pattern.
4 Data via Active Learning
The paper compares noisy web data with an active-learning alternative that iteratively selects images for human annotation. The active-learning system trains a classifier, samples images by class confidence, and uses an annotation interface with multiple safeguards.
- The active-learning system starts with labeled seed images, selects images for annotation, obtains human labels, and retrains iteratively.
- A convolutional neural network serves as the classifier for sample selection and retraining.
- Sample Selection: For each category c, confidence-based sampling selects the b ˆP(c) images with the highest class scores fc(x).Here, ˆP(c) is the desired class prior, b is the annotation budget, and fc(x) is the classifier output.
- Sample Selection: Even highly confident images can be false positives, while uncertain samples are often occluded, unusually viewed, or ambiguously bred.The reported false-positive rate is about 20% in the highest confidence range.
- Human Annotation: The annotation interface supplies positive and negative examples plus hidden gold-standard images, with final decisions made by majority vote.Three annotators determine each final category decision.
5 Experiments
Experiments show that noisy web images, filtered or combined with large-scale pretraining, can match or exceed curated and actively annotated data across fine-grained recognition tasks. The approach also scales to thousands of categories, though domain shift and dataset comparability constrain interpretation.
- Main recognition results: On CUB-200-2011, raw web search results already outperform the annotated training set, while filtering improves results by 1.3%.Training first on L-Bird further improves CUB performance by 2.9%.
- Transfer and scale: Large-scale pretraining on L-Bird improves CUB by 2.9% and Birdsnap by 4.6%, indicating transfer within the fine-grained domain.Additional fine-tuning on annotated ground-truth data adds 0.3% on CUB and 1.1% on Birdsnap.
- Limitations: Web-trained performance is sensitive to domain shift and category-name or definition mismatches, while adding ground-truth training data can correct this shift.The authors also caution that test-set images may inadvertently enter web training data and that benchmark results may not be strictly comparable to single-dataset methods.
- Main recognition results: 92.3% top-1 accuracy on CUB-200-2011, 85.4% on Birdsnap, 93.4% on FGVC-Aircraft, and 80.8% on Stanford Dogs were achieved without the respective ground-truth training data.Using 144 test-time crops, the corresponding with-training-data figures were 92.8%, 85.4%, 95.9%, and 85.9%.
- Active learning comparison: With matched training-set sizes, filtered web data trails cleaner active-learning data by only 0.4%, and adding Stanford Dogs training images yields a 0.5% improvement over active learning.Pure filtered web data is 4.4% better than non-fine-tuned active learning but somewhat behind fine-tuned models.
- Very large-scale recognition: The system reaches 73.1%/75.8% top-1 accuracy on L-Bird, 65.9%/68.1% on L-Butterfly, and 72.7%/77.5% on L-Aircraft using single/144 crops.These evaluations cover 10,982, 14,553, and 409 categories, respectively, with 42,115, 42,046, and 3,131 test images.
6 Discussion
The paper finds that generic classification models combined with filtered web data can solve fine-grained recognition effectively and scale far beyond manually annotated datasets. It also identifies imagery availability and localization as important boundaries.
- Noisy web data combined with a generic classifier and simple filtering effectively discriminates fine-grained categories.The approach compares favorably with active learning while avoiding its annotation requirements.
- The approach scales experimentally to 14,553 fine-grained categories, exceeding the practical scale of manual annotation.Traditional dataset construction is prohibitive across the world’s many bird, butterfly, moth, and insect species.
- Categories lacking publicly available or adequately described imagery may require an alternative such as active learning.
- A classification-only focus may be problematic when applications contain multiple objects or require localization.
A.1 Interface
The active-learning annotation interface gives raters positive and negative examples, class-focused batches, and feedback to support accurate, scalable fine-grained decisions.
- The interface is designed to simplify decisions and provide raters with information needed for generic, scalable annotation.
- Instructional positive images show within-class variation, while negative images clarify boundaries against easily confused categories.The interface also links raters to Google Image Search for broader class research.
- Negative examples from the model’s two most confused categories help raters focus on fine-grained discriminative attributes.For Bernese mountain dogs, contrasting classes can shift attention from fur color to hair length.
- Batching questions by class lets raters learn and focus on one fine-grained category at a time.The authors suggest batching may support a human form of semi-supervised learning, though this is harder to isolate and measure.
- Golden questions provide immediate and longer-term feedback using known correct and incorrect images from the supervised seed dataset.Immediate feedback appears during the task, while longer-term feedback summarizes performance over a day.
A.2 Rater Quality Improvements
A control experiment found that the improved annotation interface made nonexpert raters both more accurate and faster on difficult fine-grained questions.
- The improved interface reduced annotation error by 16.5% relative to a standard crowdsourcing interface.The comparison used difficult binary questions involving images mistaken by a Stanford Dogs test-set classifier.
A.3 Annotation Statistics and Examples
The appendix reports how active learning’s collected positives vary across Stanford Dogs categories and provides qualitative examples across mining iterations.
- The 10 categories with the most positives after two mining rounds include Pug, Golden Retriever, Boston Terrier, and Labrador Retriever.The full list also includes West Highland White Terrier, Boxer, Maltese, German Shepherd, and Sealyham Terrier.
- Figure 14 distributes positive training examples across Stanford Dogs categories after active-learning selection of 1,000 images per class.Categories are ordered by the number of positives collected in the first active-learning iteration.
- Positive-example counts reflect both category prevalence in YFCC100M and the active learner’s ability to find relevant images.
- Figure 15 compares positive training examples obtained for selected categories in active-learning iterations 1 and 2.
B Deduplication Details
The paper removes ground-truth images from web search results using scalable image similarity and illustrates active-learning examples from YFCC100M.
- The deduplication procedure removes ground-truth images from web search results before training.It uses a learned image-similarity metric whose output is binarized for efficient hashing-based exact search.
- Hamming distance in the binarized representation measures image dissimilarity, with identical images assigned distance 0.
- Figure 15 shows positive training examples obtained through active learning from YFCC100M for selected Stanford Dogs categories.
C Remaining Errors: Qualitative
A qualitative error analysis finds that some CUB test images depict species outside the benchmark’s 200 categories, exposing a mismatch between labels and semantic category meaning.
- The image-search model identifies CUB test-set examples whose true species is absent from the 200 benchmark categories.The highlighted example is labeled “Hooded Oriole” but depicts a “Black-Hooded Oriole.”
- Using category names ties web training data more strongly to semantic category meaning than a one-of-K label.
- Benchmark irregularities can be learned during fine-tuning, supporting the paper’s domain-shift hypothesis for higher dataset-specific performance.
D Network Visualization
The network visualization examines which input pixels influence activation and compares gradient patterns for models trained on CUB-200 and the larger L-Bird dataset.
- Gradients of the squared norm of the last convolutional layer are backpropagated into input images to indicate pixel importance for network activation.
- The figure’s comparison is organized around how training data affects the input-gradient visualization.
- Figure 18 compares, row by row, an input image with gradients from a CUB-200-trained model and a larger L-Bird-trained model.Gradients are scaled to the range [0,255].