PaperlayerView exact revision on arXiv

Source-linked AI summary

A Baseline for Few-Shot Image Classification

Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, Stefano Soatto

arXiv:1909.02729v5cs.LGcs.CVstat.ML

TL;DR

Few-shot learning needs methods that work with scarce labels while addressing noisy comparisons across episodes and protocols. The paper evaluates a simple cross-entropy pre-training and transductive fine-tuning baseline, finding it outperforms state-of-the-art methods on standard benchmarks and motivates a hardness metric for systematic evaluation.

  • Problem

    Few-shot evaluation is hindered by high accuracy variation across sampled episodes and by methods using different models and hyper-parameters across protocols.

  • Method

    The paper pre-trains a network with standard cross-entropy, fine-tunes it on support data, and uses unlabeled query predictions through transductive entropy penalization.

  • Results

    Transductive fine-tuning improves over state-of-the-art methods by 2-7% for 1-shot 5-way episodes across the reported benchmark datasets.

  • Takeaways & Limitations

    The results motivate a baseline and a hardness metric for more systematic few-shot evaluation, while indicating that existing benchmark results warrant cautious interpretation.

  • Takeaways & Limitations

    Transductive fine-tuning is about 300× slower than prototypical networks for a 1-shot 5-way episode with 15 query shots.

Abstract

from arXiv · show

Fine-tuning a deep network trained with the standard cross-entropy loss is a strong baseline for few-shot learning. When fine-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100 with the same hyper-parameters. The simplicity of this approach enables us to demonstrate the first few-shot learning results on the ImageNet-21k dataset. We find that using a large number of meta-training classes results in high few-shot accuracies even for a large number of few-shot classes. We do not advocate our approach as the solution for few-shot learning, but simply use the results to highlight limitations of current benchmarks and few-shot protocols. We perform extensive studies on benchmark datasets to propose a metric that quantifies the "hardness" of a few-shot episode. This metric can be used to report the performance of few-shot algorithms in a more systematic way.

1 INTRODUCTION

Few-shot learning addresses the growing annotation burden of massive and rare image categories, but high episode variability and inconsistent protocols complicate claims of progress. The paper presents transductive fine-tuning as a simple baseline that outperforms existing methods across standard benchmarks while motivating more systematic evaluation.

  • Few-shot learning uses only a few labeled samples per class as annotation costs and rare-category image procurement become more difficult.
  • High accuracy variance across episodes means progress can appear more substantial when judged only by mean accuracies.The paper notes that variance of the mean estimate differs from variance of accuracies, which is extremely large in the benchmark comparison.
  • The baseline pre-trains with standard cross-entropy and fine-tunes on few-shot data, with transductive fine-tuning outperforming state-of-the-art methods across standard protocols.The approach does not require specialized training for different ways or shots.
  • The baseline outperforms state-of-the-art methods on Mini-ImageNet, Tiered-ImageNet, CIFAR-FS, and FC-100 using the same hyper-parameters.It also works with a single labeled example and a single test datum per class.
  • The authors present the baseline as evidence that existing few-shot results and benchmark protocols should be interpreted cautiously, not as the definitive solution.They propose a metric for quantifying episode hardness and systematic reporting across protocols.

2 PROBLEM DEFINITION AND RELATED WORK

The paper formalizes few-shot classification around support and query datasets, then situates its approach among meta-learning, metric-based, transductive, and semi-supervised methods. Its method uses the support set to initialize a classifier and adapts parameters using the test datum and query information.

  • Problem definition: A few-shot episode consists of support and query datasets drawn from disjoint class sets, with ways denoting classes and shots denoting labeled support examples per class.
  • Problem definition: Standard supervised classification compresses the support set into parameters, whereas the paper adopts a more general classifier that can revisit the support data during inference.The conventional form is convenient because it separates training and inference, but may lose generality when few labeled examples are available.
  • Related work: Meta-learning partitions learning into a base level performing supervised learning and a meta level that uses information from the base level to make few-shot training efficient.
  • Related work: Metric-based methods learn embeddings for comparing or clustering query samples, while gradient-based methods learn mappings for base-level updates but face challenging bilevel optimization.
  • Related work: The transductive approach trains on the meta-training set, initializes with the support set, and fine-tunes parameters using the test datum.
  • Related work: The method penalizes Shannon entropy on query predictions at test time, distinguishing its use of unlabeled queries for transductive learning from semi-supervised training with extra unlabeled data.

3 APPROACH

The approach pre-trains a backbone with cross-entropy, appends a support-initialized classifier for new classes, and fine-tunes all parameters. A transductive variant additionally regularizes predictions on unlabeled queries using entropy.

  • Pre-training: Cross-entropy pre-training provides the simplest form of meta-training and yields a pre-trained backbone for few-shot adaptation.The loss includes a regularizer and is typically minimized with stochastic gradient descent-based algorithms.
  • Fine-tuning: Few-shot training amounts to fine-tuning the pre-trained model after modifying it to predict the new classes.Careful initialization can make this process efficient.
  • Classifier adaptation: The model appends a fully connected classifier after the backbone logits, using ReLU-transformed logits as input and trainable weights and biases for the target classes.The backbone logits are z(x; θ), while the classifier parameters are w and b.
  • Classifier adaptation: Support-based initialization averages each target class’s transformed features and normalizes the result, maximizing cosine similarity between class weights and features.The construction is related to weight imprinting, which treats each classifier weight vector as a class template.
  • Classifier adaptation: Using backbone logits rather than intermediate features is motivated by their peaked responses on the correct class, whereas outputs of all layers are entangled.This design avoids re-initializing the backbone’s final classifier layer.
  • Transductive fine-tuning: Transductive fine-tuning fits labeled support samples while encouraging peaked, low-entropy predictions on multiple unlabeled query samples.Tuning the entropy coefficient or query-softmax temperature per dataset and protocol improved results by 1-2%.

4 EXPERIMENTAL RESULTS

Experiments show that transductive fine-tuning is a strong baseline across benchmark datasets and backbone architectures, while large-scale pre-training and episode-hardness analysis expose broader evaluation patterns. The method improves standard benchmark results, scales to ImageNet-21k, and motivates systematic performance reporting.

  • Benchmark datasets: Transductive fine-tuning uniformly outperforms state-of-the-art algorithms on standard few-shot benchmark protocols.The comparison covers benchmark datasets including Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100.
  • Benchmark datasets: 2-7% improvement over state-of-the-art is obtained for 1-shot 5-way episodes across all benchmark datasets.For 5-shot 5-way episodes, improvement is 1.5-4% except on Mini-ImageNet, where performance is matched.
  • Benchmark datasets: 2-8% improvements result from using validation data to pre-train the backbone on Mini-ImageNet, CIFAR-FS and FC-100.The improvement is smaller for Tiered-ImageNet, suggesting benefits from additional pre-training classes.
  • Large-scale few-shot learning: 89 ± 1.86% accuracy is achieved for 1-shot 5-way ImageNet-21k episodes, compared with 72.87 ± 0.71% on Tiered-ImageNet.ImageNet-21k results remain extremely high even for large way, and were evaluated over 80 episodes.
  • Analysis: Accuracy changes logarithmically with way and support shot, providing thumb rules for constructing few-shot systems.Accuracy improves with query shot when support shot is low, while the effect is minor for Tiered-ImageNet.
  • A proposal for reporting few-shot classification performance: The hardness metric predicts accuracy across protocols: accuracy degrades linearly with hardness, and area under fitted curves supports systematic algorithm comparison.Hardness varies within fixed protocols and is a good indicator of accuracy; its efficacy against other metrics remains future work.

5 DISCUSSION

The discussion frames transductive fine-tuning as a simple baseline and hardness metric intended to ground few-shot learning evaluation. It emphasizes benchmarking across datasets with disjoint class splits and standardized experimental settings.

  • 5 DISCUSSION: Transductive fine-tuning provides a simple baseline for re-evaluating few-shot learning results and developing new algorithms.The paper stresses that the baseline is not novel but performs better than existing algorithms on standard benchmarks.
  • 5 DISCUSSION: The paper proposes systematic evaluation through a hardness metric alongside its baseline.The stated aim is to provide grounding for few-shot learning practice and guidelines for developing new methods.
  • 5 DISCUSSION: The benchmark suite includes Mini-ImageNet, Tiered-ImageNet, CIFAR-FS, and FC-100, covering datasets with different class counts and semantic overlaps.Mini-ImageNet and CIFAR-FS use random class splits, whereas Tiered-ImageNet and FC-100 reduce semantic overlap between splits.
  • 5 DISCUSSION: Each dataset separates training, validation, and test classes, with few-shot episodes constructed from uniformly sampled test classes.Experiments compare meta-training on training classes alone against training plus validation classes.

A.2 PRE-TRAINING

Pre-training uses standard image-classification practice: a WRN-28-10 backbone, cross-entropy-based training, and regularization methods including mixup and label smoothing. Fine-tuning uses a fixed Adam schedule without regularization.

  • A.2 PRE-TRAINING: Pre-training uses WRN-28-10, while conv (64)×4, ResNet-12, and WRN-16-4 are reserved for additional analysis.All networks are trained with SGD and batch-size 256, with Nesterov momentum 0.9.
  • A.2 PRE-TRAINING: Mixup augments training examples by linearly interpolating input images and their one-hot labels.The interpolation coefficient λ is sampled from Beta(α, α).
  • A.2 PRE-TRAINING: Label smoothing assigns probability 1−ϵ to the target class and ϵ/(K−1) to every other class.The paper uses label smoothing to constrain output-neuron ratios and support large-scale training.
  • A.2 PRE-TRAINING: The experiments set ϵ=0.1 for label smoothing and α=0.25 for mixup across all experiments.Batch-normalization parameters are excluded from weight decay.
  • A.2 PRE-TRAINING: Fine-tuning runs for 25 epochs with Adam at a fixed learning rate of 5 × 10^-5 and uses no regularization.Adam is selected for robustness to loss and gradient magnitude changes across episode ways.

A.4 DATA AUGMENTATION

The experiments use fixed image normalization and augmentation, standardized episode sampling, and 1,000-episode evaluation. ImageNet-21k uses the same general setup but fewer evaluation episodes.

  • A.4 DATA AUGMENTATION: Images use ImageNet-1k normalization, random horizontal flips, 4px padding, and brightness and contrast changes of ±40%.The same augmentation is applied during pre-training and fine-tuning.
  • A.4 DATA AUGMENTATION: Affine-transform augmentation has a minor effect with no consistent trend in numerical results.
  • A.4 DATA AUGMENTATION: Episodes uniformly sample classes, support samples, and query samples; query shot is fixed at 15 unless noted otherwise.Networks are evaluated over 1,000 episodes, reporting mean accuracy and its 95% confidence interval.
  • A.4 DATA AUGMENTATION: ImageNet-21k contains 14.2M images across 21,814 classes, with episodes built from 13,007 classes containing at least 10 images.The study meta-trains on 7,491 classes with more than 1,000 images and evaluates 80 episodes.

C ADDITIONAL ANALYSIS

Additional analyses examine dataset imbalance, transductive embedding changes, backbone size, and latency. They show query embeddings moving toward supports after fine-tuning, while smaller backbones reduce accuracy and remain slower than non-updating inference.

  • C ADDITIONAL ANALYSIS: ImageNet-21k is highly imbalanced, ranging from about 3K images in its most frequent class to one image in its rarest.
  • C ADDITIONAL ANALYSIS: In a 1-shot 5-way Mini-ImageNet episode, query logits move toward their supports during transductive fine-tuning while support logits remain relatively stable.The cited figure uses colors for labels, crosses for supports, circles for queries, and transparency to distinguish before and after tuning.
  • C ADDITIONAL ANALYSIS: 64% initial accuracy rises to 73.3% after transductive fine-tuning in the illustrated episode.
  • C ADDITIONAL ANALYSIS: WRN-16-4 obtains 63.28 ± 0.68% and 77.39 ± 0.5% on Mini-ImageNet for 1-shot 5-way and 5-shot 5-way, respectively.The 2.7M-parameter WRN-16-4 performs worse than the 36M-parameter WRN-28-10.
  • C ADDITIONAL ANALYSIS: WRN-16-4 transductive fine-tuning is 20–70× slower than the same-backbone prototypical-network implementation in the 1-shot 5-way scenario.Reported times are 0.87 versus 0.04 seconds for one query shot and 2.85 versus 0.04 seconds for 15 query shots.
  • C ADDITIONAL ANALYSIS: On Mini-ImageNet, transductive fine-tuning with WRN-16-4 reaches 63.28 ± 0.68% versus 57.29 ± 0.40% for prototypical networks in 1-shot 5-way classification.

C.5 USING MORE META-TRAINING CLASSES

The experiments compare classifier construction and backbone updating choices, showing that retaining backbone logits and fine-tuning the network outperform reinitialization or freezing in key few-shot settings.

  • Pre-training classes: WRN-28-10 has 0.03% more parameters when pretrained on Mini-ImageNet training plus validation data than on training data alone.The passage notes that this parameter difference is small relative to the comparison involving more pre-training classes.
  • Classifier initialization: 64.20 ± 0.65% and 81.26 ± 0.45% are obtained on Mini-ImageNet with reinitialized classifiers for 1-shot 5-way and 5-shot 5-way.The corresponding Tiered-ImageNet accuracies are 67.14 ± 0.74% and 86.67 ± 0.46%.
  • Classifier initialization: The authors append a classifier to the pretrained backbone rather than reinitializing its final fully connected layer.They attribute the stronger few-shot behavior to well-clustered backbone logits.
  • Backbone updating: 58.38 ± 0.66% and 75.46 ± 0.52% are obtained on Mini-ImageNet when the backbone is frozen for 1-shot 5-way and 5-shot 5-way.Tiered-ImageNet accuracies are 67.06 ± 0.69% and 83.20 ± 0.51%, with the 1-shot results much lower than Table 1 counterparts.

C.8 USING MIXUP DURING PRE-TRAINING

The mixup experiment reports a roughly one-percent few-shot-accuracy improvement, while evaluation over 10,000 episodes produces results consistent with 1,000-episode evaluation and smaller confidence intervals.

  • Using mixup: About 1% accuracy improvement is attributed to mixup during pre-training.Without mixup, WRN-28-10 reaches 67.06 ± 0.71% and 79.29 ± 0.51% on Mini-ImageNet for 1-shot 5-way and 5-shot 5-way.
  • Episode evaluation: 67.77 ± 0.21% and 80.24 ± 0.16% are reported on Mini-ImageNet over 10,000 episodes for 1-shot 5-way and 5-shot 5-way.Tiered-ImageNet results are 72.36 ± 0.23% and 85.70 ± 0.16%.
  • Episode evaluation: The 10,000-episode accuracies are consistent with results from 1,000 episodes, while their confidence intervals are smaller.The experiment addresses the high standard deviation of accuracies across few-shot episodes.

C.10 EVALUATION ON META-DATASET

On Meta-Dataset, the baseline is compared with prior best results, support-based initialization, and fine-tuning variants across tasks using consistent evaluation details.

  • Evaluation protocol: The comparison uses 600 few-shot episodes, with best results in each row shown in bold and † marking support-based results that are better than or comparable to existing algorithms.These conventions are specified in the Table 4 caption.
  • Meta-Dataset results: Transductive fine-tuning is better than state-of-the-art on 8 out of 10 Meta-Dataset tasks.The passage says these improvements are most times significant.
  • Meta-Dataset results: Support-based initialization is better than or comparable to state-of-the-art on 8 out of 10 tasks.The comparison uses meta-training on ImageNet-1k (ILSVRC).
  • Evaluation protocol: For ImageNet-1k and Omniglot, few-shot classes were sampled uniformly rather than with the hierarchical sampling used by Triantafillou et al.The hierarchical strategy samples classes far apart in the hierarchy and therefore easier to distinguish.

D FREQUENTLY ASKED QUESTIONS

The frequently asked questions position transductive fine-tuning as a systematic accuracy baseline rather than a claimed novel solution, while identifying benchmark, latency, and online-setting caveats.

  • Method rationale: The baseline combines support-based classifier initialization with transduction, using unlabeled query samples alongside scarce labeled support samples.The paper presents both components as important in the few-shot regime.
  • Benchmark comparisons: The authors compare methods on matched backbone architectures because increasingly sophisticated approaches often use different architectures for modest reported gains.They establish the baseline to support fairer comparisons.
  • Benchmark comparisons: Published tables retain missing entries because reimplementing methods without original code is impractical and can produce inferior results judged unfair.The paper attributes current benchmark difficulty to many performance-critical design choices.
  • Novelty and interpretation: The paper claims no novelty for fine-tuning itself and presents transductive fine-tuning as a baseline assembled from non-novel techniques.Its reported performance motivates reinterpreting benchmark results and reevaluating the literature’s status quo.
  • Limitations: WRN-16-4 transductive fine-tuning is about 20–70x slower than same-backbone metric-based approaches, despite significantly better accuracy.The paper’s stated goal is an accuracy baseline, not an analysis centered on test-time latency.
  • Transductive setting: Transductive fine-tuning depends on the query set, making it unsuitable for online sequences and motivating public test episodes to deter query-set hacking.The paper notes that transduction can operate even with a single test datum.
Loading 1909.02729v5…