PaperlayerView exact revision on arXiv

Source-linked AI summary

Free Lunch for Few-shot Learning: Distribution Calibration

Shuo Yang, Lu Liu, Min Xu

arXiv:2101.06395v3cs.LGcs.CV

TL;DR

Few-shot classifiers can overfit because limited samples form a biased estimate of the evaluation distribution. The paper calibrates Gaussian feature distributions by transferring statistics from similar well-sampled classes, then trains classifiers on generated features. The strategy reaches state-of-the-art results, including an approximately 5% improvement on miniImageNet over the next-best method, while requiring no extra parameters.

  • Problem

    Few-shot training samples can form a biased distribution that causes models to overfit and generalize poorly to the ground-truth evaluation distribution.

  • Method

    The method transfers Gaussian mean and variance statistics from similar many-shot classes to calibrate few-shot feature distributions and sample additional training features.

  • Results

    Approximately 5% improvement over the next-best method is reported on miniImageNet, with state-of-the-art accuracy across the evaluated datasets.

  • Takeaways & Limitations

    Distribution calibration can be paired with classifiers and feature extractors without extra learnable parameters while providing an accurate approximation of the feature distribution.

  • Takeaways & Limitations

    The approach assumes Gaussian feature distributions, and broader applicability to settings such as multi-domain few-shot classification remains future work.

Abstract

from arXiv · show

Learning from a limited number of samples is challenging since the learned model can easily become overfitted based on the biased distribution formed by only a few training examples. In this paper, we calibrate the distribution of these few-sample classes by transferring statistics from the classes with sufficient examples, then an adequate number of examples can be sampled from the calibrated distribution to expand the inputs to the classifier. We assume every dimension in the feature representation follows a Gaussian distribution so that the mean and the variance of the distribution can borrow from that of similar classes whose statistics are better estimated with an adequate number of samples. Our method can be built on top of off-the-shelf pretrained feature extractors and classification models without extra parameters. We show that a simple logistic regression classifier trained using the features sampled from our calibrated distribution can outperform the state-of-the-art accuracy on two datasets (~5% improvement on miniImageNet compared to the next best). The visualization of these generated features demonstrates that our calibrated distribution is an accurate estimation.

1 INTRODUCTION

Few-shot training data can bias the estimated distribution and cause overfitting, so the paper calibrates feature-space distributions using statistics transferred from similar many-shot classes. Sampling from the calibrated distributions improves classifier generalization and achieves strong benchmark performance without extra learnable parameters.

  • 1 INTRODUCTION: Few-shot models can overfit their limited examples because those samples poorly mirror the ground-truth distribution used for evaluation.This distributional bias can damage generalization.
  • 1 INTRODUCTION: The method calibrates distributions in the lower-dimensional feature space rather than the original data space.Feature-space calibration is described as easier to perform.
  • 1 INTRODUCTION: Mean and variance statistics are transferred from similar many-shot classes to better estimate few-shot class distributions, then additional features are sampled for classifier training.The approach assumes Gaussian feature dimensions and uses class similarity to guide statistic transfer.
  • 1 INTRODUCTION: 12% accuracy gain is reported for a 5way1shot task versus a baseline trained only on the provided few samples.The strategy uses a simple logistic regression classifier and requires no extra learnable parameters.
  • 1 INTRODUCTION: The generated features provide broader coverage of the test cases, and their visualization supports the calibrated distribution as an accurate ground-truth approximation.The paper reports state-of-the-art accuracy on three datasets.

2 RELATED WORKS

Related few-shot methods improve adaptation or augment scarce data through meta-learning, metric learning, generative models, or feature transformations. Distribution calibration instead estimates class-level feature distributions and generates samples without learnable parameters.

  • 2 RELATED WORKS: Optimization-based and metric-based methods learn adaptation procedures or classify by distances to class representatives.Examples include learned optimization and representative-based comparison.
  • 2 RELATED WORKS: Generation-based methods use GANs, autoencoders, or related models to synthesize samples or features for training augmentation.These approaches introduce models conditioned on tasks or other feature information.
  • 2 RELATED WORKS: Many generative approaches require complex models and loss functions to learn how to generate additional samples or features.The paper contrasts this requirement with its parameter-free strategy.
  • 2 RELATED WORKS: Traditional and learned augmentation methods create varied images or features from individual samples, feature representations, or intra-class variance.The cited methods differ in their choices of augmentation inputs and learned transformations.
  • 2 RELATED WORKS: Distribution calibration estimates class-level distributions to reduce single-sample inductive bias and produce more diverse generations.Its feature sampling procedure and classifier training use no learnable parameters.

3 MAIN APPROACH

The approach calibrates few-shot class distributions in feature space by transferring statistics from similar base classes, then samples additional features to train task-specific classifiers.

  • 3.1 PROBLEM DEFINITION: Few-shot tasks use N-way-K-shot episodes with N novel classes and K labeled support examples per class, evaluated on query examples.Base and novel classes are disjoint, and performance is averaged across tasks sampled from novel classes.
  • 3.2 DISTRIBUTION CALIBRATION: Distribution calibration transfers statistics from data-rich base classes to better estimate Gaussian feature distributions for data-scarce novel classes.Base-class statistics are more accurately estimated, and class similarity guides the transfer.
  • 3.2 DISTRIBUTION CALIBRATION: The method operates at feature level and is agnostic to the feature extractor, allowing pretrained extractors without further costly finetuning.The experiments use a pretrained WideResNet feature extractor.
  • 3.2.2 CALIBRATING STATISTICS OF THE NOVEL CLASSES: For each support feature, the method selects k nearest base classes using Euclidean distance between the feature and base-class means.The selected classes provide the statistics used to calibrate the novel-class distribution.
  • 3.2.2 CALIBRATING STATISTICS OF THE NOVEL CLASSES: The evaluation reports 5way1shot and 5way5shot accuracy on miniImageNet and CUB with 95% confidence intervals.Bold values indicate intervals intersecting with the most accurate method.
  • 3.2.2 CALIBRATING STATISTICS OF THE NOVEL CLASSES: For multi-shot tasks, calibration is repeated for each support feature to reduce single-sample bias and produce more diverse distribution estimates.The resulting calibrated statistics are used to generate labeled feature vectors from Gaussian distributions, which join the support features as classifier training data.

4 EXPERIMENTS

The experiments evaluate distribution calibration across datasets, few-shot settings, visualization, and design choices including Tukey’s transformation and generated-feature counts.

  • Research questions: The evaluation asks whether the strategy matches state-of-the-art performance, accurately approximates class distributions, and benefits from Tukey’s transformation and feature generation.These questions structure the comparative, visualization, and ablation experiments.
  • Evaluation setup: The study compares distribution calibration on miniImageNet, tieredImageNet, and CUB across 5way1shot and 5way5shot settings.Top-1 accuracy is averaged over 10,000 tasks.
  • Datasets: The datasets span diverse classes, hierarchical categories, and fine-grained bird species, enabling evaluation across different feature-space distribution granularities.miniImageNet has 100 classes, tieredImageNet 608 classes, and CUB 200 bird classes.
  • Implementation details: The feature extractor is a WideResNet trained on base classes, with representations taken from its penultimate ReLU layer.The ReLU activation makes feature values non-negative for Tukey’s transformation.
  • Visualization: Figure 2 uses t-SNE visualizations to compare support-set, generated, and query-set features across classes.Colors identify classes; stars mark support features, crosses mark query features, and triangles mark generated features.
  • Ablations: Figure 3 varies Tukey’s transformation power and the number of generated features, comparing training with and without transformed or generated features.The left panel varies transformation power, while the right panel varies generated-feature count.

4.2 COMPARISION TO STATE-OF-THE-ART

The comparison experiments test distribution calibration against state-of-the-art methods and examine its portability across visualizations, backbones, and baseline classifiers. The results report strong performance with simple classifiers and no extra learnable parameters.

  • State-of-the-art comparison: Simple SVM and logistic-regression classifiers equipped with distribution calibration achieve the best performance on 1-shot and 5-shot settings across miniImageNet, tieredImageNet, and CUB.The method is compared with optimization-based, metric-based, and generation-based approaches.
  • State-of-the-art comparison: 10% accuracy separates distribution calibration from the state-of-the-art generation-based method in the 5way1shot setting.This comparison is reported as evidence that the strategy handles extremely low-shot classification tasks.
  • Generated-feature visualization: The calibrated distribution overlaps areas covered by query features that are missed by the one-example support set.The visualization indicates that generated features can reduce mismatch between few-shot support estimates and the ground-truth distribution.
  • Backbone applicability: Around 10% accuracy improvement is observed when distribution calibration is applied across conv4, conv6, resnet18, WRN28, and rotation-trained WRN28 backbones.The reported gains support applicability across different feature extractors.
  • Baseline applicability: Over 10% accuracy improvement is reported when distribution calibration is applied to both Baseline and Baseline++ few-shot classifiers.This extends the evaluation beyond the paper’s own simple classifier configurations.

4.5 EFFECTS OF FEATURE TRANSFORMATION AND TRAINING WITH GENERATED FEATURES

The experiments examine how Tukey’s feature transformation and generated-feature training affect few-shot classification, including sensitivity to generated-feature count and calibration hyperparameters.

  • Ablation: Over 10% performance declines when both Tukey’s transformation and generated features are removed in the 5way1shot setting.Removing either component individually causes an approximately 5% performance drop.
  • Feature transformation: λ = 0.5 is the optimum Tukey transformation power with and without generated features.The transformation makes query features more aligned with the calibrated Gaussian distribution.
  • Generated features: Below 500 generated features, increasing the sample count benefits both transformed and untransformed settings.With more samples, performance begins to decline for classifiers tested on untransformed features.
  • Generated features: Training with generated samples yields a 12% relative performance improvement in a 1-shot classification setting.This result is reported for the simple logistic regression classifier.
  • Calibration hyperparameters: The calibration uses k = 2 base-class statistics, while α controls the dispersion of sampled features and can affect the classifier’s decision boundary.Hyperparameters are selected using validation-set performance.

5 CONCLUSION AND FUTURE WORKS

The paper concludes that distribution calibration is a simple, effective strategy for few-shot classification and identifies broader settings and methods for future study.

  • Conclusion: Distribution calibration enables a simple logistic regression classifier to outperform current state-of-the-art methods by approximately 5% on miniImageNet.The strategy uses generated features without complex generative models, specialized loss functions, or extra learnable parameters.
  • Conclusion: Visualization indicates that the calibrated distribution accurately estimates the feature distribution.
  • Future work: Future work will examine distribution calibration in multi-domain few-shot classification and with additional methods, including metric-based meta-learning algorithms.

A AUGMENTATION WITH NEAREST CLASS FEATURES

This appendix compares calibrated-distribution sampling with nearest-class feature retrieval as alternatives for augmenting the support set.

  • Augmentation comparison: Nearest-class feature retrieval improves performance over using only the support set but can damage performance as more retrieved features are added.The comparison also includes samples drawn from the calibrated distribution.

B DISTRIBUTION CALIBRATION WITHOUT NOVEL FEATURE

This appendix studies distribution calibration without averaging the novel feature into the calibrated mean.

  • Mean calibration: The calibrated novel-class mean is normally formed by averaging the novel-class mean with retrieved base-class means.The appendix evaluates calibration without this averaging step.

C THE EFFECTS OF TUKEY’S TRANSFORMATION

The analysis examines Tukey’s transformation by comparing class distributions before and after transformation, alongside calibration comparisons and similarity-based performance analysis.

  • Distribution visualization: Base-class features fit the Gaussian assumption better than skewed novel-class features before transformation.After Tukey’s transformation, the novel-class distribution becomes more aligned with the Gaussian-like base-class distribution.
  • Calibration comparison: Distribution calibration is compared with and without the novel feature ˜x.
  • Similarity analysis: Performance improvement is analyzed across similarity levels between a query novel class and its most similar base classes.

D THE SIMILARITY LEVEL ANALYSIS

The similarity analysis finds that calibration helps more when retrieved base-class distributions are more similar to the novel-class ground-truth distribution.

  • Similarity-performance relationship: Under the 5-way-1-shot setting, higher distribution similarity corresponds to greater performance improvement from the method.
  • Similarity definition: The comparison uses similarity between the retrieved base-class distribution and the novel-class ground-truth distribution.
  • Experimental setting: Table 9 reports performance improvement according to similarity between each query novel class and retrieved base classes.
Loading 2101.06395v3…