Source-linked AI summary
Label Efficient Learning of Transferable Representations across Domains and Tasks
Zelun Luo, Yuliang Zou, Judy Hoffman, Li Fei-Fei
TL;DR
The paper addresses transferring representations across domains and tasks when target data are unlabeled or sparsely labeled. It jointly aligns domains with a multilayer adversarial loss and transfers semantic information with a similarity objective, achieving superior performance to fine-tuning in evaluated settings.
Problem
Existing approaches often address domain adaptation, few-shot learning, or task transfer separately, while the paper targets their joint setting with new target tasks and few annotations.
Method
The framework jointly learns from labeled source data and unlabeled or sparsely labeled target data using multilayer domain adversarial alignment and a semantic similarity transfer objective.
Results
The method achieves superior performance over fine-tuning and demonstrates transfer from SVHN 0-4 to MNIST 5-9 and from ImageNet images to UCF-101 videos.
Takeaways & Limitations
The framework supports transferable representations across domains and tasks while reducing the need for annotated target-domain and target-task data.
Takeaways & Limitations
The target domain is assumed to be only sparsely labeled, with mt much smaller than nt and ns, and one evaluation uses non-overlapping digit classes across domains.
Abstract
from arXiv · showhide
We propose a framework that learns a representation transferable across different domains and tasks in a label efficient manner. Our approach battles domain shift with a domain adversarial loss, and generalizes the embedding to novel task using a metric learning-based approach. Our model is simultaneously optimized on labeled source data and unlabeled or sparsely labeled data in the target domain. Our method shows compelling results on novel classes within a new domain even when only a few labeled examples per class are available, outperforming the prevalent fine-tuning approach. In addition, we demonstrate the effectiveness of our framework on the transfer learning task from image object recognition to video action recognition.
1 Introduction
The paper targets joint transfer from a labeled source domain and task set to a sparsely labeled target domain with new tasks. It combines domain adaptation and similarity-based semantic transfer, and evaluates this setting across digit recognition and image-to-video action recognition.
- Existing deep models commonly assume fixed domains and tasks during supervised training, making generalization an observed byproduct.
- The paper addresses adapting a representation to a new domain with new tasks and few annotations, using labeled source data plus sparsely labeled target data.
- The proposed approach jointly uses multilayer unsupervised domain adversarial alignment and cross-domain and within-domain class similarity objectives.
- The similarity objective can transfer information even when source and target domains have non-overlapping classes.
- Experiments cover SVHN digits 0-4 to MNIST digits 5-9 and ImageNet object-centric images to UCF-101 videos for action recognition.
2 Related work
Related work spans domain adaptation, transfer learning, few-shot learning, and unsupervised representation learning. These areas use distribution alignment, parameter reuse, similarity or meta-learning, and reconstruction or pretext objectives.
- Domain adaptation: Domain adaptation learns from related source domains for a target distribution, commonly assuming the same task and sparse or absent target labels.
- Transfer learning: Transfer learning reuses knowledge from related tasks or domains, including transferring image classification knowledge to object detection and semantic segmentation.
- Few-shot learning: Few-shot learning addresses new concepts with few annotations through similarity-ranking networks, matching networks, or meta-learning.
- Unsupervised learning: Unsupervised learning includes reconstruction-based models, probabilistic models, generative models, autoregressive models, and self-supervised pretext tasks.
3 Method
The method jointly transfers representations across domains and semantic task spaces using multilayer domain adversarial alignment and metric-based semantic transfer. It combines supervised target learning with domain and semantic objectives to support sparsely labeled targets, including non-overlapping label spaces.
- Problem setting: The model transfers knowledge from a large labeled source domain to a sparsely labeled target domain with potentially different task and label spaces.The target contains labeled and unlabeled images, with substantially fewer labeled than unlabeled examples.
- Joint optimization: The total objective jointly optimizes target supervision, domain transfer, and semantic transfer losses.The domain and semantic terms are weighted by hyperparameters α and β.
- Domain alignment: A multilayer domain discriminator aligns source and target representations by aggregating activations across corresponding network layers.Unlike single-layer alignment, the discriminator simultaneously uses information from multiple layers and back-propagates its losses through the relevant lower layers.
- Semantic transfer: The semantic transfer objective minimizes entropy over temperature-scaled pairwise similarities between unlabeled target examples and labeled source or target examples.The objective supports transfer from labeled source-to-unlabeled target data and labeled target-to-unlabeled target data.
- Semantic transfer: The softmax temperature controls how broadly target examples may match source examples, allowing semantic transfer when source and target label spaces do not overlap.Small temperatures encourage similarity to one source class when label sets overlap, whereas larger temperatures allow similarity to multiple source classes.
4 Experiment
The experiments evaluate joint transfer across domains and tasks using sparse target labels and unlabeled target data, including digit and image-to-video settings. Results show gains over fine-tuning and competing domain-adversarial methods, while the multilayer objective improves optimization stability.
- Experimental settings: The evaluation covers SVHN-to-MNIST digit transfer and ImageNet-to-UCF-101 transfer from object recognition to video action recognition.The digit experiment uses SVHN digits 0-4 and MNIST digits 5-9; the video experiment uses sparse labeled clips and remaining unlabeled videos.
- Experimental settings: The framework is trained with labeled source data, few labeled target examples, and unlabeled target data.For the digit setting, k = 2, 3, 4, 5 determines the number of labeled examples per class used in the target subset.
- SVHN 0-4 → MNIST 5-9: Fine-tuning is only marginally better than training from scratch, while the proposed method achieves significant gains, especially when k = 2.Fine-tuning on the full target dataset reaches 99.65% accuracy for reference.
- ImageNet → UCF-101: The method improves average-across-frame UCF-101 performance over standard fine-tuning for every tested k.The reported video analysis distinguishes per-frame accuracy from average-across-frame video accuracy.
- Ablation: unsupervised domain adaptation: The multilayer domain-adversarial objective yields a 6.5% performance gain over the best competing domain-adversarial approach.The experiments also associate the multilayer approach with improved optimization stability, evidenced by a small standard error.
5 Conclusion
The paper proposes a data-efficient method for learning representations transferable across domains and tasks. The framework jointly addresses domain shift, transfer to new tasks, and learning from unlabeled data, and reports superior performance over fine-tuning.
- The proposed representation-learning method targets transfer across different domains and tasks in a data-efficient manner.
- The framework is jointly trained to minimize domain shift, transfer knowledge to new tasks, and use large amounts of unlabeled data.
- The method shows superior performance over the popular fine-tuning approach.
Network Architecture
The paper reports embedding and discriminator network structures, with the embedding network identified as ResNet-18.
- Table 4 reports the embedding network structure.
- Tables 5 and 6 report discriminator structures.
- The embedding network is a ResNet-18.
- Tables 7 and 8 also report embedding and discriminator network structures.