Source-linked AI summary
Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach
Timnit Gebru, Judy Hoffman, Li Fei-Fei
TL;DR
Fine-grained recognition in the wild is hindered by dataset shift and the infeasible cost of collecting annotations for every new scenario. The paper proposes attribute-based multi-task domain adaptation and reports improved adaptation performance on fine-grained car recognition.
Problem
Fixed datasets provide sparse and biased samples of the real world, while collecting annotations for every object category and scenario is infeasible.
Method
The paper uses a multi-task adaptation objective that jointly learns category and attribute recognition, leveraging cross-category relationships and attribute labels.
Results
Attributes raise baseline accuracy from 9.28% to 14.37%, while domain-confusion improvement increases from approximately 5% without attributes to approximately 10% with attributes.
Takeaways & Limitations
Fine-grained classes sharing attributes can transfer knowledge from training classes to unseen target classes in semi-supervised adaptation.
Takeaways & Limitations
Performance gains are much smaller on the office dataset because WordNet labels may not provide visually distinctive attributes.
Abstract
from arXiv · showhide
While fine-grained object recognition is an important problem in computer vision, current models are unlikely to accurately classify objects in the wild. These fully supervised models need additional annotated images to classify objects in every new scenario, a task that is infeasible. However, sources such as e-commerce websites and field guides provide annotated images for many classes. In this work, we study fine-grained domain adaptation as a step towards overcoming the dataset shift between easily acquired annotated images and the real world. Adaptation has not been studied in the fine-grained setting where annotations such as attributes could be used to increase performance. Our work uses an attribute based multi-task adaptation loss to increase accuracy from a baseline of 4.1% to 19.1% in the semi-supervised adaptation case. Prior do- main adaptation works have been benchmarked on small datasets such as [46] with a total of 795 images for some domains, or simplistic datasets such as [41] consisting of digits. We perform experiments on a subset of a new challenging fine-grained dataset consisting of 1,095,021 images of 2, 657 car categories drawn from e-commerce web- sites and Google Street View.
1. Introduction
Fine-grained recognition models trained on fixed, fully annotated datasets struggle to generalize to real-world appearances. The paper studies adapting models trained on cheaper annotated sources to real-world domains, using attributes and sparse target labels to exploit cross-category relationships.
- Fixed datasets provide biased samples of the world, making additional annotations for every new real-world scenario infeasible.
- Field guides and e-commerce websites provide cheaper image-category annotations, but their statistics differ from real-world images.
- Fine-grained domain adaptation addresses dataset shift between easily acquired annotated images and the real world, a setting not previously studied in this context.
- The proposed multi-task objective jointly learns and adapts recognition at the attribute and category levels, exploiting relationships shared across categories.
- Semi-supervised adaptation uses target labels available for attributes even when category labels are missing for some classes.
- Experiments use 170 of 2,657 car classes and 71,030 images, extending adaptation evaluation to a large-scale fine-grained dataset.
2. Related Work
Prior fine-grained recognition models often rely on single-domain sources and may not generalize to real-world photographs. Related domain-adaptation and multitask-learning work motivates using attributes and structured label relationships to improve adaptation and class-level performance.
- Fine-grained models trained on field-guide or product-shot images may generalize poorly to real-world photos because object appearance varies across domains.
- Domain adaptation improves models trained in one domain when they are applied to another, including unsupervised settings without labeled target images.
- Attributes have been used for object classification and zero-shot learning, motivating their use in unsupervised and semi-supervised domain adaptation.
- The paper uses KL divergence between predicted label distributions to enforce consistency between attribute and class predictions rather than hard semantic constraints.
- Prior multitask-learning results motivate the observation that attribute-based multitask adaptation can improve class-level performance.
- The adaptation architecture includes shared-weight CNNs and separate classifiers for attributes and fine-grained classes, with adaptive losses for source-target alignment.
3. Multi-Task Domain Adaptation for Fine-Grained Recognition
The method adapts fine-grained recognition by jointly learning category and shared attribute predictions, then enforcing consistency between them. Attribute annotations provide additional supervision when target classes lack labels, while adaptation losses operate at both attribute and class levels.
- Multi-task objective: The model jointly optimizes fine-grained classification, attribute classification, attribute consistency, and domain-adaptation losses.Its architecture uses independent softmax classifiers for attributes and categories, with a consistency term linking their predictions.
- Multi-task objective: Attribute annotations can supervise classes lacking target labels when related classes share attributes.The method assumes at least as many labeled source and target images are available for each attribute as for each class.
- Classification loss: The final multi-task softmax loss is a weighted sum of attribute-level and fine-grained classification losses.Attribute classifiers operate on the CNN feature map, with one classifier for each attribute.
- Attribute consistency: The attribute consistency loss penalizes disagreement between attributes inferred from the fine-grained classifier and those predicted by independent attribute classifiers.It uses a symmetric KL divergence because the objective minimizes distance between two predicted distributions rather than matching a reference distribution.
- Augmenting adaptation: Existing adaptation algorithms are augmented with attribute-level objectives alongside fine-grained soft-label and domain-confusion losses.This allows attribute annotations from classes without labeled target examples to contribute to adaptation.
- Dataset structure: The evaluation data are imbalanced across attributes: each fine-grained class has fewer than 500 labeled GSV images, while some body types have nearly 12,000.GSV cars also have smaller bounding boxes on average than web cars: 9,117 versus 146,848 pixels.
4. Evaluation
The evaluation examines domain shift and tests multi-task adaptation across car and office datasets. Results show that attribute-level transfer improves adaptation, especially for semi-supervised car recognition, while gains vary across categories and datasets.
- Datasets and setup: 1,095,021 images across 2,657 car categories provide a large-scale benchmark spanning web and Google Street View domains.Experiments use 170 common classes, with 89 models, 17 makes, and 10 body types represented.
- Domain shift: 73.9% source-domain accuracy drops catastrophically when the source-only model is evaluated on GSV images, confirming substantial domain shift.A jointly trained source-target model even outperforms the fully supervised target-only model.
- Car adaptation: 19.11% semi-supervised accuracy rises from 12.34% when attribute-level transfer is combined with domain confusion and soft-label losses.Using labeled GSV classes can reduce held-out-class generalization through overfitting, which the combined method addresses.
- Per-class analysis: 75% of held-out classes improve in semi-supervised adaptation, while 66% of target classes improve in the unsupervised setting.Accuracy is unchanged for 14% and 16% of classes and drops for 10% and 18% in the two regimes, respectively.
- Per-class analysis: Attribute-label availability helps explain category-level variation: top gains occur among body types with many labeled target examples, while the largest loss is a crew-cab class with only 57 labeled GSV images.Across classes, accuracy gain correlates −0.29 with the number of labeled source examples.
5. Conclusion
The paper presents a multi-task CNN for semi-supervised domain adaptation that uses shared fine-grained attributes to transfer knowledge across classes. It evaluates this approach on a large, multi-domain car dataset and identifies directions for refining attribute-based and hierarchical adaptation.
- The proposed semi-supervised domain adaptation pipeline uses a multi-task CNN architecture.
- The approach leverages shared attributes among fine-grained classes to transfer knowledge from training classes to unseen classes.
- The method is evaluated on a subset of a dataset containing approximately 1M images across 2,657 car categories and multiple domains.
- The paper also evaluates the method on the standard office dataset using additional WordNet labels.
- Future work will refine attribute incorporation and investigate hierarchical adaptation when attribute labels are unavailable.