PaperlayerView exact revision on arXiv

Source-linked AI summary

A Closer Look at Few-shot Classification

Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, Jia-Bin Huang

arXiv:1904.04232v2cs.CV

TL;DR

Few-shot classification needs fairer comparisons and more realistic tests because implementation differences and same-domain evaluation obscure progress. The paper establishes a consistent comparison, analyzes baseline and meta-learning methods, and introduces cross-domain evaluation; deeper backbones narrow method gaps, while a baseline remains competitive and performs favorably under domain shift.

  • Problem

    Implementation differences make relative performance difficult to assess, while same-domain base and novel classes make standard few-shot evaluation unrealistic.

  • Method

    The paper conducts consistent comparative experiments, evaluates baseline and meta-learning methods, and introduces an evaluation setting with domain shift between base and novel classes.

  • Results

    Deeper backbones narrow method gaps with limited domain differences, a distance-based baseline is competitive on mini-ImageNet and CUB, and the baseline performs favorably under domain shift.

  • Takeaways & Limitations

    Reducing intra-class variation matters more with shallow backbones, while adapting to domain differences remains important for realistic few-shot classification.

  • Takeaways & Limitations

    For real-world applications, learning adaptation during meta-training remains an important direction for future few-shot research.

Abstract

from arXiv · show

Few-shot classification aims to learn a classifier to recognize unseen classes during training with limited labeled examples. While significant progress has been made, the growing complexity of network designs, meta-learning algorithms, and differences in implementation details make a fair comparison difficult. In this paper, we present 1) a consistent comparative analysis of several representative few-shot classification algorithms, with results showing that deeper backbones significantly reduce the performance differences among methods on datasets with limited domain differences, 2) a modified baseline method that surprisingly achieves competitive performance when compared with the state-of-the-art on both the \miniI and the CUB datasets, and 3) a new experimental setting for evaluating the cross-domain generalization ability for few-shot classification algorithms. Our results reveal that reducing intra-class variation is an important factor when the feature backbone is shallow, but not as critical when using deeper backbones. In a realistic cross-domain evaluation setting, we show that a baseline method with a standard fine-tuning practice compares favorably against other state-of-the-art few-shot learning algorithms.

1 INTRODUCTION

The paper addresses unfair comparisons in few-shot classification and unrealistic same-domain evaluation. It provides a consistent empirical study showing that deeper backbones narrow method gaps, a distance-based baseline is competitive, and domain-shifted evaluation changes the comparison.

  • Few-shot classification seeks to recognize unseen classes using very few labeled examples, addressing annotation costs and data scarcity.Current vision systems typically require abundant labeled instances with diverse visual variations.
  • Implementation discrepancies obscure relative performance gains and can substantially underestimate baseline methods.The paper identifies training without data augmentation as one example of an implementation difference.
  • Same-dataset sampling of base and novel classes omits domain shift, making standard evaluation scenarios unrealistic.
  • Deeper feature backbones reduce performance gaps among methods when domain differences between base and novel classes are limited.
  • A baseline using a distance-based classifier achieves competitive performance with state-of-the-art meta-learning methods on mini-ImageNet and CUB.
  • Under domain shift between base and novel classes, sophisticated few-shot algorithms do not improve performance over the baseline.

2 RELATED WORK

Related work organizes few-shot classification into initialization-based, distance-metric-based, and hallucination-based approaches. It also connects the paper’s cross-domain analysis to domain adaptation while noting the unusually limited target-domain data in few-shot learning.

  • Initialization-based methods: Initialization-based methods learn model parameters or optimizers so novel-class classifiers can be adapted from few labeled examples with few updates.
  • Distance-metric methods: Distance-metric methods classify unseen inputs by comparing them with labeled examples through learned similarity functions.Examples include cosine, Euclidean, relation-module, ridge-regression, and graph-neural-network metrics.
  • Distance-metric methods: The paper compares three distance-metric methods and finds that a simple distance-based baseline can perform competitively.
  • Hallucination-based methods: Hallucination-based methods learn generators from base classes to synthesize novel-class data or transfer appearance variation for augmentation.
  • Domain adaptation: The paper studies domain differences in few-shot classification in relation to domain adaptation, but with only a small amount of target-domain labeled data.

3 OVERVIEW OF FEW-SHOT CLASSIFICATION ALGORITHMS

The paper defines baseline transfer-learning and meta-learning procedures for adapting to novel classes from limited labeled data. Baseline++ changes the classifier to use cosine similarity, while meta-learning conditions prediction or adaptation on support sets.

  • Few-shot algorithms train novel-class classifiers from abundant base-class data and a small amount of novel-class labeled data.
  • Baseline: The baseline pre-trains a feature extractor and classifier on base classes, then fixes the extractor and trains a new classifier on novel-class examples.
  • Baseline++: Baseline++ keeps the baseline training procedure but replaces its linear classifier with cosine-similarity scores between input features and class weight vectors.The resulting class scores are normalized with softmax, explicitly reducing intra-class variation during training.
  • Baseline++: The paper clarifies that Baseline++’s distance-based classifier is not presented as a novel network design.Distance-based classification had already been studied and revisited in few-shot classification.
  • Meta-learning: Meta-learning trains a classifier on sampled support and query sets from N base classes by minimizing N-way query prediction loss.At meta-testing, the classifier adapts to novel classes using their support set.
  • Meta-learning: MatchingNet and ProtoNet compare query features with support representations, RelationNet learns a relation module, and MAML adapts initial parameters through support-set gradient updates.

4 EXPERIMENTAL RESULTS

The experiments compare few-shot methods across standard, deeper-backbone, and cross-domain settings. They find that backbone depth narrows method gaps on CUB, while domain shift exposes adaptation limitations and favors a fine-tuned baseline.

  • Experimental settings: The evaluation covers generic object recognition, fine-grained classification, and cross-domain adaptation, using mini-ImageNet, CUB, and mini-ImageNet →CUB.The cross-domain setting uses mini-ImageNet base classes and CUB validation and novel classes.
  • Standard setting: Reimplemented meta-learning methods fall no more than 2% behind reported performance after implementation details are standardized.The authors attribute remaining differences to random seeds and minor implementation changes, including a shared optimizer.
  • Increasing network depth: On CUB, deeper backbones drastically reduce performance gaps among methods, and ProtoNet improves rapidly as backbone depth increases.The authors attribute the reduced gap to lowering intra-class variation with a deeper feature backbone.
  • Increasing network depth: On mini-ImageNet 5-shot, deeper backbones improve Baseline and Baseline++, while some meta-learning methods become worse relative to them.The authors relate this more complicated pattern to dataset differences and larger base-to-novel domain divergence in mini-ImageNet.
  • Domain differences and further adaptation: In mini-ImageNet →CUB, Baseline outperforms all meta-learning methods because retraining a classifier on novel data adapts more directly to domain shift.Baseline also exceeds Baseline++, possibly because reducing intra-class variation compromises adaptability; further adaptation improves MatchingNet and MAML but can harm ProtoNet under small domain differences.

5 CONCLUSIONS

The paper finds that Baseline and Baseline++ can match state-of-the-art methods under standard and cross-domain few-shot settings, especially with deeper backbones.

  • Baseline++ is competitive with state-of-the-art methods under standard evaluation conditions.
  • Baseline achieves competitive performance with recent meta-learning algorithms on CUB and mini-ImageNet when using a deeper feature backbone.
  • Baseline compares favorably against all evaluated meta-learning algorithms when base and novel classes differ in domain.

A1 RELATIONSHIP BETWEEN DOMAIN ADAPTATION AND FEW-SHOT CLASSIFICATION

Few-shot classification and domain adaptation differ in their class relationships, but can overlap when novel classes are evaluated across domains. The paper introduces mini-ImageNet →CUB to expose this domain shift.

  • Domain adaptation generally adapts source knowledge to the same class in a target dataset, whereas few-shot classification transfers from base classes to novel classes.
  • Some prior work lies at the intersection of domain adaptation and few-shot classification, but may still evaluate the same target classes.
  • The mini-ImageNet →CUB setting highlights domain shift by sampling base and novel classes from different datasets.
  • Table A1 summarizes the relationship between domain adaptation and few-shot classification, including minor shifts between base and novel classes.

A2 TERMINOLOGY DIFFERENCE

The paper notes that few-shot learning studies use inconsistent terminology and documents these differences to clarify comparisons.

  • Different meta-learning studies use different terminology, creating inconsistency that the appendix documents.
  • Table A2 records terminology differences across related works and marks terms that match this paper's usage.

A3 ADDITIONAL RESULTS ON OMNIGLOT AND OMNIGLOT→EMNIST

Additional Omniglot and Omniglot→EMNIST experiments compare few-shot methods within-domain and across domains. Meta-learning methods lead in 1-shot, while methods become comparable with 5 shots, and classifier choice affects results.

  • The appendix adds character-recognition and cross-domain character-recognition evaluations.
  • Omniglot is expanded to 6,492 rotated classes and split into 4,112 base, 688 validation, and 1,692 novel classes.
  • Omniglot→EMNIST uses 1,597 Omniglot base classes and 62 EMNIST classes, split into 31 validation and 31 novel classes.
  • Both settings use a Conv-4 backbone with 28x28 inputs and no data augmentation.
  • Meta-learning methods outperform baseline and baseline++ in 1-shot, but all methods reach comparable performance in 5-shot classification.The authors attribute the 1-shot disadvantage of baseline methods to over-fitting without data augmentation; more novel examples reduce its impact.
  • Table A3 reports 5-way results with Conv-4 and no data augmentation, while Table A4 compares softmax and 1-NN classifiers.
  • A 1-NN classifier outperforms softmax in 1-shot, whereas softmax performs better in 5-shot evaluation.The reported values are not directly comparable with Vinyals et al. because the paper uses a different mini-ImageNet dataset.
  • Baseline and baseline++ use five training epochs in the Omniglot and Omniglot→EMNIST experiments.

A5 MAML AND MAML WITH FIRST-ORDER APPROXIMATION

First-order MAML reaches the same final validation accuracy as full MAML on Omniglot 5-shot with a Conv-4 backbone, although full MAML converges faster. This supports using the first-order approximation for improved memory efficiency without affecting final accuracy.

  • Full MAML converges faster, but first-order MAML reaches similar accuracy by the end of training.
  • First-order MAML is used throughout the experiments to improve memory efficiency without affecting final accuracy.
  • Both MAML versions converge to the same validation accuracy on Omniglot with 5-shot and a Conv-4 backbone.

A6 INTRA-CLASS VARIATION AND BACKBONE DEPTH

Deeper backbones reduce intra-class variation and narrow accuracy gaps among few-shot methods, while Baseline++ performs favorably in experiments that increase the number of classes at meta-testing. Larger-way meta-training, however, creates substantial memory demands for deeper networks.

  • Intra-class variation and backbone depth: Deeper backbones decrease intra-class variation in both base- and novel-class features on CUB.The paper measures cluster tightness with the Davies-Bouldin index.
  • Intra-class variation and backbone depth: Accuracy gaps among methods diminish as backbone depth increases on CUB, and deeper-backbone Baseline can beat some meta-learning methods on mini-ImageNet 5-shot.
  • More-way in meta-testing stage: The experiment compares 5-way meta-training with 5-, 10-, and 20-way meta-testing to evaluate mismatched training and testing scenarios.
  • More-way in meta-testing stage: Baseline++ compares favorably with Baseline, MatchingNet, ProtoNet, and RelationNet as the meta-testing task increases to larger N-way classification.
  • More-way in meta-testing stage: Baseline++ is attributed better performance than Baseline to further intra-class variation reduction, while meta-learning methods may degrade when testing uses more ways than training.
  • More-way in meta-testing stage: A 20-way meta-training batch with 5 support and 15 query images per class requires 400 images, potentially preventing deeper-backbone training without specialized hardware.
Loading 1904.04232v2…