PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-training for Few-shot Transfer Across Extreme Task Differences

Cheng Perng Phoo, Bharath Hariharan

arXiv:2010.07734v2cs.CVcs.AIcs.LG

TL;DR

Few-shot transfer is challenging when labeled base data from the target domain is unavailable and the available source domain is extremely different. STARTUP adapts source representations using unlabeled target data and source-classifier-induced groupings. On BSCD-FSL, it improves 1-shot performance by an average of 2.9 points over Transfer.

  • Problem

    Few-shot learners and self-supervised methods struggle when the labeled source and target domains differ extremely, leaving target domains without large labeled base datasets with limited options.

    Source: §1 · p. 1

  • Method

    STARTUP trains target-domain representations on unlabeled data to replicate the groupings induced by a pre-trained source classifier.

    Source: §1 · p. 2

  • Results

    On BSCD-FSL, STARTUP improves 1-shot performance by an average of 2.9 points over Transfer.

    Source: §5.1.1 · p. 6

  • Takeaways & Limitations

    Even when source classes are irrelevant, the source classifier’s target-domain groupings can still provide informative similarity structure for few-shot adaptation.

    Source: §1 · p. 2

    The setup assumes a labeled base dataset from a very different, disjoint image space and class set, plus unlabeled data from the target domain.

    Source: §2 · p. 3

Abstract

from arXiv · show

Most few-shot learning techniques are pre-trained on a large, labeled "base dataset". In problem domains where such large labeled datasets are not available for pre-training (e.g., X-ray, satellite images), one must resort to pre-training in a different "source" problem domain (e.g., ImageNet), which can be very different from the desired target task. Traditional few-shot and transfer learning techniques fail in the presence of such extreme differences between the source and target tasks. In this paper, we present a simple and effective solution to tackle this extreme domain gap: self-training a source domain representation on unlabeled data from the target domain. We show that this improves one-shot performance on the target domain by 2.9 points on average on the challenging BSCD-FSL benchmark consisting of datasets from multiple domains. Our code is available at https://github.com/cpphoo/STARTUP.

1 INTRODUCTION

Extreme source–target gaps leave few-shot learning without suitable labeled base data, while training from unlabeled target data alone can also struggle. STARTUP adapts source representations using target-domain groupings and reports gains on BSCD-FSL.

1 INTRODUCTION

  • Few-shot learners rely on a large labeled base dataset, often unavailable in domains such as medical or satellite imagery, where collecting annotations is costly.The introduction gives crop-type recognition in satellite images as an example of expensive annotation.Source: §1 · p. 1
  • Across extreme differences such as internet photos versus X-rays, few-shot methods can underperform even naive ImageNet transfer.Self-supervised learning also struggles to beat naive ImageNet transfer with more practical amounts of unlabeled data.Source: §1 · p. 1
  • STARTUP trains target-domain representations to replicate groupings induced by a pre-trained source classifier, retaining source knowledge insofar as it remains relevant.The source classifier’s classes may be irrelevant to the target, while its induced notions of similarity and dissimilarity may still be informative.Source: §1 · p. 2
  • STARTUP uses source-induced target similarities rather than treating each target image as its own class, as many self-supervised approaches do.The approach aims to adapt representations while maintaining source-task knowledge to the extent that it is relevant.Source: §1 · p. 2
  • STARTUP gains up to 2.9 points on average over few-shot learning, transfer learning, and self-supervision state-of-the-art on BSCD-FSL.The benchmark comprises datasets from extremely different domains.Source: §1 · p. 2

2 PROBLEM SETUP

The setup considers few-shot classification in a target domain when the labeled base dataset comes from a very different source. It adds unlabeled target data for representation learning before few-shot evaluation.

2 PROBLEM SETUP

  • The target-domain few-shot task presents a small labeled support set and a small query test set; the learner must quickly learn the classes and classify queries.Each task selects classes from the target domain’s unknown label space.Source: §2 · p. 2
  • The labeled base dataset is assumed to come from a very different distribution, with an image space and class set disjoint from the target domain.This differs from conventional few-shot learning setups described in the paper.Source: §2 · p. 3
  • The learner also receives an unlabeled dataset from the target domain.This additional data is part of the setup because the base dataset is so different from the target domain.Source: §2 · p. 3
  • Representation learning uses both the labeled base dataset and unlabeled target data before evaluation on target-domain few-shot tasks.Source: §2 · p. 3

3 RELATED WORK

The related work spans few-shot learning, cross-domain transfer, and methods that use unlabeled data. The paper distinguishes its setting by addressing extreme domain gaps and a change in label space.

Few-shot Learning (FSL)

  • Few-shot methods commonly assume base and novel classes share discriminative features, variation patterns, or useful initializations.Approaches include metric learning, transfer learning, meta-learning, augmentation, and class-agnostic denoising.Source: §3 · p. 3

Cross-domain Few-shot Classification (CD-FSL)

  • Prior cross-domain work either simulates transfer with diverse training domains and smaller test-time gaps, studies similar internet-image domains, or uses more test annotations.The paper targets a more extreme gap between base and novel domains.Source: §3 · p. 3

Few-shot learning with unlabeled data

  • Semi-supervised and transductive few-shot methods use unlabeled data during evaluation, whereas this approach uses it during representation learning; the innovations can be combined.Source: §3 · p. 3

Self-Training

  • Self-training and knowledge distillation train a student to mimic a teacher, but prior work addresses a single task without a change in label space.The paper frames transfer to novel domains with a wholly different label space as an unexplored scenario.Source: §3 · pp. 3–4

Domain Adaptation

  • Domain adaptation commonly assumes source and target share a label space, unlike few-shot learning in the paper’s setting.Source: §3 · p. 4

Self-supervised Learning

  • Contrastive self-supervision treats images and their augmentations as separate classes, but its utility across large base-to-novel domain gaps had not been evaluated.The paper focuses on that challenging scenario.Source: §3 · p. 4

4 APPROACH

STARTUP trains a student on base data and teacher-generated soft labels for unlabeled target data, adding a self-supervised loss to adapt its representation. Evaluation freezes that representation for few-shot linear classification, and the default student uses the teacher’s embedding with a randomly initialized classifier.

4 APPROACH

  • STARTUP trains a base-data teacher, assigns probability-distribution pseudo-labels to unlabeled target images, then trains a student on base and pseudo-labeled data.Source: §4 · p. 4
  • SimCLR supplies the unlabeled-data loss, bringing two augmentations of the same image closer in feature space than other images in the batch.Source: §4 · p. 4
  • The KL term is intended to make features emphasize target-domain groupings induced by the teacher’s pseudo-labels.Source: §4 · p. 4

4.1 EVALUATION

  • At evaluation, STARTUP freezes the adapted representation and trains a linear classifier on the support set to predict query examples.Source: §4.1 · p. 5

4.2 INITIALIZATION STRATEGIES

  • With SimCLR omitted, no initialization strategy wins consistently across datasets; the default uses the teacher’s embedding with a randomly initialized classifier.Source: §4.2 · p. 5

5 EXPERIMENTS

Across drastically different domains, STARTUP improves one-shot transfer over the Transfer baseline; experiments also examine how it works and whether it helps when domains are similar.

5.1.1 RESULTS

  • 2.9 points is STARTUP’s average improvement over Transfer in 1-shot classification, and it significantly outperforms few-shot techniques on most datasets except ChestX, where methods are similar.The improvement is almost 6 points on CropDisease.Source: §5.1.1 · pp. 5–6, Table 1 · p. 6
  • STARTUP performs significantly better than SimCLR on ISIC and EuroSAT, but SimCLR is strong on CropDisease; using SimCLR as an auxiliary loss benefits STARTUP.The authors conjecture that the base embedding is not a good starting point for CropDisease.Source: §5.1.1 · p. 6
  • STARTUP outperforms Transfer + SimCLR in most cases, while results on EuroSAT show its advantage is not solely from SimCLR: STARTUP (no SS) also beats both baselines.The stated exceptions are 5-shot ChestX and 1-shot ISIC.Source: §5.1.1 · p. 6
  • With a ResNet-18 teacher pretrained on ILSVRC 2012, STARTUP’s absolute gains over Transfer remain similar on most datasets and are larger on EuroSAT and CropDisease.The experiments also find that larger, richer embeddings do not always transfer better.Source: Table 2 · p. 7, §5.1.1 · p. 7

5.2.1 HYPOTHESIS 1: STARTUP ADDS NOISE WHICH INCREASES ROBUSTNESS.

  • 70.40 ± 0.86 (5-way 1-shot) and 88.78 ± 0.54 (5-way 5-shot) on CropDisease underperformed naive Transfer when unlabeled training data also came from ImageNet.This test suggests that unlabeled data are not merely adding noise; STARTUP learns structure in the target domain useful for downstream classification.Source: §5.2.1 · p. 7

5.2.2 HYPOTHESIS 2: STARTUP ENHANCES TEACHER-INDUCED GROUPINGS

  • Teacher-induced groupings agree with ground-truth labels on EuroSAT and CropDisease, the two datasets with the best transfer performance and greatest STARTUP improvement, suggesting a correlation.AMI measures agreement between the teacher-induced grouping and ground-truth labels.Source: §5.2.2 · pp. 7–8, Table 3 · p. 8
  • t-SNE plots show more separation after STARTUP on EuroSAT and CropDisease, consistent with STARTUP strengthening the discriminability of representations around teacher-induced groupings.The authors describe STARTUP as inducing a potentially meaningful grouping and training a representation that emphasizes it.Source: Figure 2 · p. 8, §5.2.2 · p. 8

5.3 FEW-SHOT TRANSFER ACROSS SIMILAR DOMAINS

  • On miniImageNet, STARTUP variants mostly neither help nor hurt relative to Transfer; on tieredImageNet, they can outperform Transfer with initialization matched to unlabeled-data amount.Teacher initialization is beneficial in the less-data setup, while training from scratch is superior in the more-data setup.Source: §5.3 · p. 8, Table 4 · p. 9

6 CONCLUSION

STARTUP adapts feature representations to novel domains using unlabeled target-domain data while retaining class groupings induced by the base classifier. It outperforms prior art on extreme cross-domain few-shot transfer.

6 CONCLUSION

  • STARTUP adapts feature representations to novel domains using unlabeled data while retaining class groupings induced by the base classifier.The approach targets performance degradation caused by large domain and task differences.Source: §6 · p. 9

A.1 IMPLEMENTATION DETAILS

The authors implemented STARTUP by modifying the publicly available BSCD-FSL implementation.

A.1 IMPLEMENTATION DETAILS

  • STARTUP was implemented by modifying the publicly available BSCD-FSL implementation.The benchmark implementation is by Guo et al. (2020).Source: §A.1 · p. 14

A.1.1 TRAINING THE TEACHER

Teacher training follows the BSCD-FSL code with a larger batch size for miniImageNet and fewer epochs for tieredImageNet.

A.1.1 TRAINING THE TEACHER

  • The miniImageNet teacher uses the BSCD-FSL code with batch size increased from 16 to 256.All other settings are kept the same.Source: §A.1.1 · p. 14
  • The tieredImageNet teacher uses the miniImageNet setup but trains for 90 epochs without image augmentation.The passage specifies both changes relative to the miniImageNet setup.Source: §A.1.1 · p. 14

A.1.2 TRAINING THE STUDENT

The appendix reports implementation choices and additional experiments on BSCD-FSL, including the effects of self-supervision, initialization, unlabeled-data amount, and base-data use. Across higher-shot settings, the conclusions of the main experiments generally hold.

A.1.2 TRAINING THE STUDENT

  • Student optimization runs for 1,000 epochs with batch size 256 and SGD with momentum 0.9 and weight decay 1e-4; validation loss selects the starting learning rate and final model.The starting rate is chosen from seven candidates using internal validation data; the rate is halved when training loss has not decreased for 20 epochs.Source: §A.1.2 · p. 14

A.1.3 TRAINING LINEAR CLASSIFIER.

  • SimCLR uses NT-Xent temperature 1 and dataset-specific stochastic augmentations; the linear classifier uses cross-entropy and SGD for 100 epochs at learning rate 0.01, momentum 0.9, and weight decay 1e-4.Source: §A.1.2 · p. 14, §A.1.3 · p. 14

A.2 FULL RESULTS ON BSCD-FSL

  • Higher-shot miniImageNet→BSCD-FSL results generally preserve the conclusions from the main experiments.Tables 5–8 report results for 1-, 5-, 20-, and 50-shot settings across miniImageNet and ImageNet transfer.Source: §A.2 · p. 15, Table 5 · p. 16, Table 6 · p. 17

A.3 USING ROTATION FOR SELF-SUPERVISION.

  • STARTUP with rotation self-supervision outperforms Transfer on CropDisease and EuroSAT but generally underperforms its SimCLR counterparts.The rotation variant is reported for miniImageNet→BSCD-FSL.Source: §A.3 · p. 15

A.4 INITIALIZATION STRATEGIES FOR THE STUDENT

  • Teacher-embedding initialization with a randomly initialized classifier is best or second best across scenarios, despite no initialization strategy winning everywhere.Random initialization is best on CropDisease but worst on ISIC; the authors use the teacher-embedding option as the default.Source: §A.4 · p. 15, Table 9 · p. 18

A.5 IMPACT OF DIFFERENT AMOUNT OF UNLABELED EXAMPLES

  • STARTUP benefits from more unlabeled examples in miniImageNet→ISIC experiments.The experiment sets aside 20% of the dataset, or 2,003 examples, for evaluation; Figure 3 plots accuracy with means and 95% confidence intervals over 600 tasks.Source: §A.5 · p. 15, Figure 3 · p. 19

A.6 TRAINING THE STUDENT WITHOUT THE BASE DATASET

  • Removing the base-dataset cross-entropy loss makes student training equivalent to fine-tuning; results show no discernible difference except significant 5-shot degradation on ISIC.This comparison is for miniImageNet→BSCD-FSL.Source: §A.6 · p. 15, Table 10 · p. 19

A.7 STARTUP ON DIFFERENT UNLABELED DATA

  • For ImageNet→BSCD-FSL, unlabeled data for STARTUP should come from the target domain of interest.The appendix compares STARTUP trained on different unlabeled datasets.Source: §A.7 · p. 15, Table 11 · p. 19
Loading 2010.07734v2…