PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Representation Learning from Multiple Domains for Few-shot Classification

Wei-Hong Li, Xialei Liu, Hakan Bilen

arXiv:2103.13841v1cs.CV

TL;DR

Few-shot classification seeks to adapt to unseen classes and domains using few labeled examples. The paper distills multiple domain-specific networks into a single universal representation and refines it for new tasks; on Meta-Dataset, it reports improved performance over prior methods with greater efficiency.

  • Problem

    Few-shot classification must generalize to previously unseen classes and domains from only a few labeled examples.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    The method distills domain-specific networks into a shared representation using adapter-aligned features and a CKA-inspired loss, then learns a task-specific transformation from support examples.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    The method outperforms state-of-the-art methods on seen and unseen domain generalization while using fewer parameters and less computation than other multi-domain techniques.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    A single universal network can provide multi-domain representations with fixed inference cost regardless of the number of domains.

    Source: §1 · p. 2

    The setup assumes the training collection Db comprises K subdatasets, each sampled from a different domain.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

In this paper, we look at the problem of few-shot classification that aims to learn a classifier for previously unseen classes and domains from few labeled samples. Recent methods use adaptation networks for aligning their features to new domains or select the relevant features from multiple domain-specific feature extractors. In this work, we propose to learn a single set of universal deep representations by distilling knowledge of multiple separately trained networks after co-aligning their features with the help of adapters and centered kernel alignment. We show that the universal representations can be further refined for previously unseen domains by an efficient adaptation step in a similar spirit to distance learning methods. We rigorously evaluate our model in the recent Meta-Dataset benchmark and demonstrate that it significantly outperforms the previous methods while being more efficient. Our code will be available at https://github.com/VICO-UoE/URL.

1. Introduction

Few-shot classification adapts to unseen classes from few labeled examples, and the paper focuses on generalizing across domains as well as classes. It proposes distilling multiple domain-specific networks into one shared representation, then refining it for each test task.

1. Introduction

  • Few-shot learning adapts a classifier to classes not seen during training using only a few labeled samples.The paper considers generalization to unseen data distributions as well as unseen classes.Source: §1 · p. 1
  • Meta-Dataset evaluates few-shot generalization across ten datasets from different domains, including domains held out for meta-testing.The benchmark later added three unseen domains to test generalization.Source: §1 · p. 1
  • Earlier methods often used separate domain-specific feature extractors and retrieved or combined them for each target task, but these approaches require multiple inference passes.The paper contrasts its single shared feature extractor with this retrieve-or-combine strategy.Source: Figure 1 · p. 1, §1 · p. 2
  • The method distills knowledge from multiple domains into one network, aligns features using task-specific adapters and a CKA-inspired loss, and adapts the resulting representations to new tasks.The task-specific mapping is learned discriminatively, rather than using the approach in Simple CNAPS.Source: §1 · p. 2
  • The proposed method outperforms state-of-the-art few-shot methods on seen and unseen domain generalization while using fewer parameters and less computation than other multi-domain techniques.It uses a single universal network at inference, unlike approaches that pass queries through multiple domain-specific networks.Source: §1 · p. 2

2. Related Work

Related work spans meta-learning and transfer-learning approaches to few-shot classification, cross-domain methods, and knowledge distillation or universal representations. The paper distinguishes its single-network, feature-alignment approach from methods that adapt models or select among domain-specific networks.

Meta-learning based few-shot classification

  • Meta-learning methods either learn discriminative feature spaces for distance-based classification or learn model parameters that can adapt quickly to new tasks.Examples include metric-based methods such as Siamese networks and optimization-based methods such as MAML and Reptile.Source: §2 · p. 2

Transfer learning based few-shot classification

  • Transfer-learning methods train on available data and then adapt at test time, ranging from updating only a classifier to fine-tuning the full network.Some methods also assume the query set is available during fine-tuning.Source: §2 · p. 2

Cross-domain few-shot classification

  • Cross-domain methods adapt feature encoders or select among domain-specific features; SUR and URT use multiple networks, whereas this paper learns one network across domains.The paper also maps features to a task-specific space using each support set, a step it relates to Simple CNAPS [2].Source: §2 · p. 2

Knowledge distillation and universal representation

  • The paper combines multi-task distillation with universal representations, using task-specific adapters and a CKA-inspired feature-matching loss for the larger domain gaps of cross-domain learning.Unlike SUR and URT, its approach learns universal representations without domain-specific weights.Source: §2 · p. 3

3. Method

The method first learns a single multi-domain feature extractor by distilling knowledge from domain-specific networks, then adapts its features to each few-shot task. It frames few-shot classification around support and query sets, with meta-training and meta-test classes kept mutually exclusive.

3.1. Few-shot Task Formulation

  • Few-shot tasks contain labeled support examples that define the task and query examples whose labels must be predicted.Meta-training produces a general feature extractor from one dataset, while meta-test tasks use mutually exclusive classes from another dataset.Source: §3.1 · p. 3

3.2. Learning multiple domain representations

  • The method replaces selecting among domain-specific feature extractors with one multi-domain network distilled from pretrained domain-specific networks.The single extractor has the same capacity as each domain-specific extractor and has fixed inference cost regardless of the number of domains.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • Feature distillation maps each domain-specific representation into a common space with adapters and uses RBF-kernel CKA to match features across diverse domains.CKA kernel matrices are computed per minibatch rather than over entire datasets.Source: §3.2 · pp. 4–5
  • The multi-domain model is trained to match domain-specific features and predictions while predicting ground-truth classes.The method freezes the pretrained domain-specific feature extractors during this distillation stage.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Feature adaptation in meta-test

  • A task-specific linear transformation is learned from support-set examples, then query images are classified by their nearest adapted class centroid.The transformation is optimized on the support set to increase intra-class and reduce inter-class similarity.Source: §3.3 · p. 5, Figure 3 · p. 5

4. Experiments

The experiments compare the proposed method with baselines and state-of-the-art methods on Meta-Dataset, then examine its components, qualitative predictions, and retrieval representations. The evaluation covers seen and unseen domains and multiple few-shot settings.

4.1. Experimental setup

  • Meta-Dataset evaluation uses eight datasets for training and five unseen datasets for testing, with 600 sampled tasks per dataset and mean accuracy with 95% confidence intervals.The experiments use ResNet-18 backbones; the multi-domain network shares layers across domains except for the classifier.Source: §4.1 · pp. 5–6

4.2. Results

  • The method outperforms other methods on seven of eight seen datasets and four of five unseen datasets, with average rank 1.3 versus 5.0 for SUR and 4.4 for URT.Reported gains over the second-best approach include Aircraft (+2.8), Birds (+2.1), Texture (+4.2), VGG Flower (+1.5), Traffic Sign (+6.1), and MSCOCO (+3.8).Source: Table 1 · p. 6, §4.2 · p. 6
  • The method beats Best SDL and MDL on all datasets except QuickDraw, while MDL can transfer representations across domains but may also suffer negative transfer.At inference, the proposed method uses one universal network, requiring fewer parameters and computations than SUR and URT, which pass queries through multiple single-domain networks.Source: §4.2 · p. 7

4.3. Further results

  • In varying-way five-shot and five-way one-shot settings, the method retains gains over other methods on most datasets, including the one-shot setting with only one support image per class.The five-shot setting is more challenging because it uses fewer support images; the top-two methods remain the same, while Simple CNAPS ranks above SUR in five-way one-shot, reversing their order in prior settings.Source: Table 2 · p. 6, §4.3 · p. 7

4.4. Analyses

  • Combining CKA and KL divergence gives the best performance among the individual distillation-loss alternatives; CKA outperforms KL in most domains, while L2 and cosine are insufficient for very diverse features.For classifiers, the proposed combination outperforms SVM, LR, and NCC+MD in most seen datasets, but is worse in some unseen domains, including Traffic Sign and MNIST.Source: §4.4 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 7
  • In qualitative comparisons across four datasets, the method retrieves more correct neighbors than URT and favors semantic similarity over similarity in color, shape, and background.Figure 4 marks correct predictions in green and false predictions in red.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5. Global retrieval

  • For global retrieval, the method achieves the best performance in ten of thirteen domains, with significant gains in Aircraft, Birds, Textures, and Fungi.URT and SUR require support-set adaptation, so the retrieval comparison instead uses concatenated or summed features from domain-specific networks as their baselines.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusion

The paper learns universal representations by aligning a multi-domain network with single-domain networks, then refines features for unseen tasks. On Meta-Dataset, the approach achieves state-of-the-art performance while using fewer parameters and less computation than other multi-domain techniques.

5. Conclusion

  • The universal representations with feature refinement achieve state-of-the-art performance on Meta-Dataset, with fewer parameters and greater computational efficiency than other multi-domain techniques.The conclusion reports this overall comparison without giving numerical results.Source: §5 · p. 8
  • The method aligns a multi-domain network’s features with multiple single-domain networks using linear adapters and a CKA-inspired loss.The multi-domain network is optimized simultaneously over multiple domains.Source: §5 · p. 8
  • Universal features can be refined for unseen tasks from few examples by learning a transformation similar to distance learning.Source: §5 · p. 8

A. Implementation details

ResNet-18 serves as the backbone for both single-domain and multi-domain networks.

A. Implementation details

A.1. Training details of single-domain models

Single-domain models use ResNet-18 and follow the training protocol described for the experiments, with hyperparameters listed in Table 6.

A.1. Training details of single-domain models

  • Each training dataset has its own ResNet-18 model, trained with SGD, cosine annealing, momentum 0.9, and weight decay 7 × 10−4.The learning rate, batch size, annealing frequency, and maximum iterations are listed in Table 6.Source: §A.1 · p. 11, Table 6 · p. 11
  • Single-domain training uses the same data augmentations as [13], including random crops and random color augmentations.Source: §A.1 · p. 11
  • Table 6 presents the training hyperparameters for single-domain learning.Source: Table 6 · p. 11

A.2. Training details of our method

The method’s multi-domain training shares network layers across domains, while meta-test adaptation optimizes a feature adaptor from an identity initialization. The further-results section covers single-domain evaluations, adaptor effects, and additional classification settings.

A.2. Training details of our method

  • The multi-domain network shares all layers except the final classifier across domains and trains for 240,000 iterations.Each batch contains 50% ImageNet data; the loss weights are 4 for ImageNet and 1 for other datasets, and are linearly annealed.Source: §A.2 · p. 11
  • Meta-test feature adaptation initializes ϑ as the identity matrix and optimizes it for 40 iterations with Adadelta.The learning rate is 0.1 for the first eight datasets and 1 for the last five.Source: §A.2 · p. 11

B. More results

  • The further-results section evaluates single-domain models, adaptor effects, varying-way five-shot and five-way one-shot classification, and qualitative and retrieval results.Source: B. More results · p. 11

B.1. Complete results of single domain learning

Single-domain features generalize unevenly across Meta-Dataset, with ImageNet often transferring well and domain-specific models excelling on their own datasets. The adaptor ablation reports accuracy gains on several datasets, consistent with adaptors helping align features across networks.

B.1. Complete results of single domain learning

  • ImageNet features lead on four of eight seen datasets and four of five unseen datasets, while several domain-trained models perform best on their own datasets.The Omniglot model also generalizes well to MNIST; the strongest single-domain model is used as a competitive baseline for universal representation learning.Source: §B.1 · p. 11, Table 7 · p. 12

B.2. Effect of adaptors in knowledge distillation

  • Adaptors improve accuracy by 1.7 on Birds, 3.6 on VGG Flower, and 1.3 on MSCOCO when used with CKA distillation.The authors interpret these gains as evidence that adaptors help align features from multi-domain and single-domain networks trained on very different domains.Source: §B.2 · p. 11, Table 9 · p. 12

B.3. Complete results of varying-way five-shot and five-way one-shot

In varying-way five-shot evaluation, performance declines for all methods relative to the conventional setting, but the proposed method remains significantly better on ten datasets. In five-way one-shot evaluation, it maintains a consistent gain.

B.3. Complete results of varying-way five-shot and five-way one-shot

  • The proposed method performs significantly better than the compared methods on ten datasets in varying-way five-shot evaluation, although scores generally fall versus the conventional setting.The authors attribute the lower overall performance to the five-shot setting providing fewer support images.Source: §B.3 · p. 12
  • In five-way one-shot evaluation, the proposed method maintains a consistent gain, which the authors connect to the value of universal representations with limited labeled samples.Simple CNAPS ranks above SUR in this setting, reversing their relative ranking in previous settings.Source: §B.3 · p. 13

B.4. Qualitatively results

Across qualitative nearest-neighbor comparisons on the test datasets, the proposed method retrieves more correct neighbors and more often finds semantic rather than superficial visual matches.

B.4. Qualitatively results

  • The proposed method produces more correct nearest neighbors than MDL, Simple CNAPS, SUR, and URT, including cases where other methods match colors, shapes, or backgrounds instead of semantics.The paper cites Figures 13 and 14 as examples and interprets the comparisons as indicating more useful, general representations.Source: §B.4 · p. 13

B.5. Complete global retrieval results

The paper evaluates universal representations in non-episodic retrieval using Recall@k. The method achieves the best performance in ten of thirteen domains, with significant gains in four named domains.

B.5. Complete global retrieval results

  • The method achieves the best retrieval performance in ten of thirteen domains, with significant gains in Aircraft, Birds, Textures, and Fungi.Evaluation uses Recall@1, Recall@2, Recall@4, and Recall@8; Simple CNAPS is compared directly, while concatenated and summed domain-specific features replace SUR and URT because those methods require support-set adaptation.Source: §B.5 · p. 13

Related papers

Loading 2103.13841v1…