Source-linked AI summary
Simultaneous Deep Transfer Across Domains and Tasks
Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko
TL;DR
Deep CNN adaptation can require substantial target-domain labeling, despite unlabeled or sparsely labeled data often being available. The paper jointly learns domain-invariant representations and matches soft label distributions to transfer task information, outperforming baselines across supervised and semi-supervised benchmarks. In semi-supervised Office shifts, it reports a 13% average relative improvement over baselines on four challenging shifts.
Problem
Fine-tuning deep CNNs in a new domain may require hundreds or thousands of labeled examples per category, while new environments may provide only sparse labels and unlabeled data.
Method
A new CNN architecture jointly optimizes domain confusion and soft-label matching to align domains and transfer source-learned category relationships.
Results
13% average relative improvement over the baselines is reported on the four most challenging semi-supervised shifts in the Office dataset.
Takeaways & Limitations
The method provides an alternative fine-tuning strategy when limited or no labeled data is available per target category.
Takeaways & Limitations
The approach assumes source and target domains become sufficiently similar under the learned representation for the source classifier to perform well on the target.
Abstract
from arXiv · showhide
Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias. Fine-tuning deep models in a new domain can require a significant amount of labeled data, which for many applications is simply not available. We propose a new CNN architecture to exploit unlabeled and sparsely labeled target domain data. Our approach simultaneously optimizes for domain invariance to facilitate domain transfer and uses a soft label distribution matching loss to transfer information between tasks. Our proposed adaptation method offers empirical performance which exceeds previously published results on two standard benchmark visual domain adaptation tasks, evaluated across supervised and semi-supervised adaptation settings.
UC Berkeley, EECS & ICSI
The paper lists an affiliation with UMass Lowell, CS, and contact information for the Berkeley authors.
- The paper lists Kate Saenko with UMass Lowell, CS.
- The paper provides contact information for authors at eecs.berkeley.edu.
1. Introduction
The paper addresses domain shifts that make deep-model adaptation data-intensive by using unlabeled target data and a few labels to transfer both domain statistics and category relationships.
- Motivation: Fine-tuning deep CNNs for a new environment may require hundreds or thousands of labeled examples per object category.
- Motivation: The proposed algorithm uses unlabeled target data and a few labeled examples from some categories.
- Approach: Domain transfer aligns source and target marginal feature distributions, while task transfer preserves source-learned category relationships.
- Approach: The method jointly combines domain confusion and softmax cross-entropy losses in a CNN architecture for supervised and semi-supervised adaptation.
- Evaluation: The evaluation covers the Office benchmark and a cross-dataset collection, with results reported for classification across visually distinct domains.
2. Related work
Prior visual domain adaptation work primarily aligns source and target representations or regularizes target classifiers, while this paper combines domain-invariant learning with source semantic-structure transfer.
- Domain adaptation: Visual domain adaptation methods commonly address distribution shifts by transforming source and target representations into an aligned feature space.
- Domain adaptation: Supervised adaptation methods also regularize target classifiers against source classifiers, sometimes while learning a feature transformation.
- CNN-based methods: Deep CNN representations reduce the effects of resolution and lighting on domain shifts, but some invariant-representation architectures require labels for every training instance.
- CNN-based methods: Earlier joint source-target CNN approaches were limited to shallow networks, while another approach combined denoising pretraining with an MMD domain confusion loss.
- Task transfer: Classifier output distributions had been used for model distillation, but this paper applies them to transfer class correlations between domains.
- Task transfer: Other domain-invariance methods did not address task transfer in the semi-supervised setting.
- Proposed architecture: The paper’s architecture uses domain confusion over labeled and unlabeled source and target data while matching source semantic activation distributions.
3. Joint CNN architecture for domain and task transfer
The architecture jointly learns a representation that aligns source and target domains while transferring source category relationships to sparsely labeled target data. It combines domain confusion with soft-label training, supporting supervised and semi-supervised adaptation.
- Architecture: The method learns a representation that aligns visual domains and transfers semantic structure from a well-labeled source to a sparsely labeled target.The target labels may cover only a subset of categories.
- Joint optimization: The joint loss combines domain confusion and softmax cross-entropy losses, with hyperparameters controlling their influence during optimization.The architecture is applicable to supervised adaptation with labels from every category and semi-supervised adaptation with labels from only a subset.
- Aligning domains via domain confusion: Domain confusion trains a feature representation in which the best domain classifier performs poorly, making source and target representations domain invariant.The method alternates updates to the domain classifier and representation parameters because their objectives oppose one another.
- Aligning source and target classes via soft labels: Soft-label training matches target activations to source per-category output distributions, transferring inter-class relationships such as visual similarity between bottles and mugs.A soft label is computed by averaging source softmax activations within each category.
- Aligning source and target classes via soft labels: Soft-label loss updates categories without labeled target data to produce non-zero probabilities, improving evaluation on unlabeled categories.This benefit is examined in experiments using target labels from only a subset of categories.
4. Evaluation
The evaluation tests the method on Office and cross-dataset benchmarks under supervised and semi-supervised adaptation settings. Across these settings, soft labels and domain confusion improve adaptation, especially when target labels are scarce or absent for evaluated categories.
- Evaluation settings: The method is evaluated on the Office dataset and a large-scale cross-dataset domain adaptation challenge.The experiments cover supervised and semi-supervised adaptation settings.
- Office dataset: In supervised Office adaptation, soft-label or domain-confusion fine-tuning improves over hard-label training in 5 of 6 domain shifts.Combining soft labels with domain confusion produces marginally higher average performance.
- Office dataset: In semi-supervised Office adaptation, all method variants outperform the source-only and prior domain-adaptation baselines on 16 held-out categories.The evaluated categories have no labeled target data, so both domain confusion and soft labels contribute significantly to performance improvement.
- Office dataset: For Amazon→Webcam, the method correctly classifies held-out-category examples that the baseline misclassifies.Examples include notebooks confused with letter trays and black mugs confused with black computer mice.
- Cross-dataset adaptation: On ImageNet→Caltech adaptation, the method using soft-label loss, with or without domain confusion, outperforms source-only and standard fine-tuning baselines.The full architecture performs best overall, with the largest benefit when few labeled target examples are available and operation possible with zero target labels.
- Cross-dataset adaptation: Target-only fine-tuning yields accuracies of 36.6 ± 0.6, 60.9 ± 0.5, and 67.7 ± 0.5 for 1, 3, and 5 labeled examples per category, respectively.All three values underperform the source-only model, according to the reported evaluation.
5. Analysis
The analysis confirms that domain confusion produces domain-invariant representations and that soft labels transfer information between categories with sparse target supervision.
- Domain invariance: Domain invariance is evaluated by testing whether an optimal classifier can predict an image’s originating domain.The method expects a trained domain classifier to perform poorly on representations learned with domain confusion.
- Domain invariance: 56% domain-classifier accuracy shows that the domain-confusion representation makes Amazon and Webcam images much harder to distinguish.The baseline CaffeNet representation reaches 99% test accuracy, whereas the domain-confusion representation performs only slightly above random.
- Task transfer: Soft labels transfer category information from source to target when labeled target examples exist for only half of the categories.This enables information from related labeled categories to support held-out target categories.
- Task transfer: For a held-out monitor example, the soft-label model predicts monitor correctly, while the baseline predicts ring binder.The source soft labels for laptop computer assign relatively high weight to monitor, providing related-category information during training.
6. Conclusion
The proposed CNN adapts to new domains with limited or no labeled data per target category by jointly supporting domain transfer and task-information transfer. It improves adaptation performance in supervised and semi-supervised benchmark settings, including a 13% average relative improvement on four challenging Office shifts.
- 6. Conclusion: The CNN architecture adapts to a new domain with limited or no labeled data per target category.It is designed for target settings where category-level supervision is sparse or unavailable.
- 6. Conclusion: The method jointly optimizes domain invariance for domain transfer and cross-entropy soft-label loss for transferring task information between domains.These objectives are combined within the proposed CNN architecture.
- 6. Conclusion: 13% average relative improvement over the baselines was achieved on the four most challenging shifts in the Office dataset under semi-supervised adaptation.The architecture also improves adaptation performance in supervised and semi-supervised settings across two standard benchmark datasets.