PaperlayerView exact revision on arXiv

Source-linked AI summary

Embedding Propagation: Smoother Manifold for Few-Shot Classification

Pau Rodríguez, Issam Laradji, Alexandre Drouin, Alexandre Lacoste

arXiv:2003.04151v2cs.CVcs.LG

TL;DR

Few-shot classification must generalize across disjoint training and test classes despite distribution shift. The paper introduces embedding propagation as a non-parametric manifold-smoothing regularizer, reporting state-of-the-art few-shot results and improvements of up to 16% accuracy points in semi-supervised settings.

  • Problem

    Few-shot classification needs representations that generalize from training classes to disjoint test classes under distribution shift.

  • Method

    EPNet applies embedding propagation to interpolate neural features using a similarity graph and regularize the embedding manifold.

  • Results

    EPNet achieves state-of-the-art results across standard and semi-supervised few-shot scenarios, improving Imagenet-FS accuracy by more than 2%.

  • Takeaways & Limitations

    Embedding propagation smooths decision boundaries and improves transductive and semi-supervised few-shot classification within the evaluated settings.

  • Takeaways & Limitations

    The semi-supervised formulation assumes a specific support-labeling setup, including 5-shot 5-way episodes with only 20% to 60% of support labels available.

Abstract

from arXiv · show

Few-shot classification is challenging because the data distribution of the training set can be widely different to the test set as their classes are disjoint. This distribution shift often results in poor generalization. Manifold smoothing has been shown to address the distribution shift problem by extending the decision boundaries and reducing the noise of the class representations. Moreover, manifold smoothness is a key factor for semi-supervised learning and transductive learning algorithms. In this work, we propose to use embedding propagation as an unsupervised non-parametric regularizer for manifold smoothing in few-shot classification. Embedding propagation leverages interpolations between the extracted features of a neural network based on a similarity graph. We empirically show that embedding propagation yields a smoother embedding manifold. We also show that applying embedding propagation to a transductive classifier achieves new state-of-the-art results in mini-Imagenet, tiered-Imagenet, Imagenet-FS, and CUB. Furthermore, we show that embedding propagation consistently improves the accuracy of the models in multiple semi-supervised learning scenarios by up to 16\% points. The proposed embedding propagation operation can be easily integrated as a non-parametric layer into a neural network. We provide the training code and usage examples at https://github.com/ElementAI/embedding-propagation.

1 Related Work

The paper situates EPNet between metric learning, transfer learning, transductive learning, and manifold regularization. It addresses underexplored regularization in few-shot classification by smoothing feature manifolds during training.

  • Few-shot classification: Episodic learning simulates few-shot train and test scenarios by constructing episodes with support and query sets.
  • Few-shot classification: Metric learning learns a shared feature space in which labeled support examples predict query classes, while transfer learning learns general features before task-specific adaptation.
  • Regularization for Generalization: EPNet applies embedding propagation during training and differs from concurrent manifold-regularization work in its training phase, inference classifier, self-supervision, and semi-supervised evaluation.
  • Transductive learning: Transductive learning predicts only test points and has outperformed inductive learning with few labeled examples, motivating its use in few-shot classification.
  • Semi-Supervised learning: For semi-supervised few-shot classification, prior work uses unlabeled data through soft k-means, label propagation, or selected examples, while EPNet further uses pseudo-labels.

2 Proposed Method

EPNet extracts features, propagates them over a similarity graph to form interpolated embeddings, and then classifies with label propagation. It is trained through pretraining and episodic fine-tuning, with an unlabeled-data extension for semi-supervised prediction.

  • Pipeline: EPNet maps feature-extractor outputs to interpolated embeddings through embedding propagation, then uses those embeddings for classification.
  • Embedding propagation: Embedding propagation computes pairwise feature distances, constructs a similarity-based graph, and uses propagation to obtain embeddings that are weighted sums of neighboring features.
  • Embedding propagation: Embedding propagation removes undesired noise, is compatible with varied feature extractors and classifiers, and has negligible complexity for small few-shot episodes.
  • Few-shot classification setup: EPNet assumes base, novel, and validation datasets, with novel classes disjoint from base classes, and evaluates episodes using n-way k-shot support and query sets.
  • Few-shot classification setup: At inference, EPNet jointly propagates support and query features, then applies label propagation using one-hot support labels to predict query logits.
  • Training procedure: Training first learns general representations on the base dataset, then fine-tunes episodically for novel-class generalization using classification, rotation, and label-propagation objectives.
  • Semi-supervised learning: In semi-supervised learning, EPNet pseudo-labels unlabeled images, augments the support set with them, and reruns inference to classify queries.
  • Semi-supervised learning: Entropy regularization is effective when decision boundaries lie in low-density regions, and embedding propagation seeks a similar boundary through manifold smoothing.

3 Experiments

The experiments evaluate EPNet across standard, semi-supervised, ablation, and manifold-smoothness settings. Results compare embedding propagation with related methods and an otherwise identical model without EP.

  • Experimental setup: EPNet is evaluated on miniImagenet, tieredImagenet, CUB, and Imagenet-FS using standard few-shot and semi-supervised scenarios.Accuracies are averaged over 1,000 episodes, with common feature extractors and dataset-specific image resolutions.
  • Main results: EPNet obtains state-of-the-art accuracy on miniImagenet, tieredImagenet, and CUB for 1-shot and 5-shot benchmarks.The comparison includes models with more parameters or higher-resolution images.
  • Main results: EP improves all Imagenet-FS benchmarks by approximately 2% accuracy and remains scalable to the larger dataset.The results also indicate orthogonality with other embedding transformations such as denoising autoencoders.
  • Semi-supervised learning: Up to 16% accuracy points are gained over previous state of the art in the 1-shot WRN-28-10 SSL setting with 100 unlabeled samples.EPNet also improves over previous state of the art by up to 2.7% when 40% of the support set is labeled.
  • Ablation studies: The improvement is larger when label propagation is combined with embedding propagation, supporting their role in smoothing decision boundaries.The ablation identifies the EP–LP combination as the main source of improvement rather than rotation loss.
  • Embedding propagation on manifold smoothness: Embedding propagation projects and interpolates embeddings so that inter-class boundaries become denser and smoother.The experiments use cross-class interpolation and a two-moons visualization to examine the resulting manifold.

4 Conclusion

The conclusion presents EPNet as a simple embedding-propagation regularizer for distribution-shifted few-shot learning. Empirical results link smoother decision boundaries with improved transductive and semi-supervised performance across several benchmarks.

  • 4 Conclusion: EPNet regularizes feature representations through embedding propagation to address distribution shift in few-shot learning.The method is described as a simple embedding propagation step.
  • 4 Conclusion: Embedding propagation smooths decision boundaries, a property associated with improved generalization.The conclusion reports this relationship empirically.
  • 4 Conclusion: EPNet achieves state-of-the-art results on miniImagenet, tieredImagenet, and CUB in standard and semi-supervised scenarios.The conclusion also reports significant improvements in transductive and semi-supervised settings.
  • 4 Conclusion: EP improves state-of-the-art Imagenet-FS accuracy by more than 2% in all reported setups.The comparison is made against wDAE-GNN.
  • 4 Conclusion: Smoothing alone improves 1-shot miniImagenet accuracy by 4.8% with Wide Residual Networks.This comparison uses EPNet and a non-smooth version of the same model.

Appendix

The appendix extends evaluation to higher-shot and higher-way settings and reports additional implementation-related analyses. It also includes carbon-emissions reporting for the research.

  • Appendix: Additional appendix analyses ablate parts of the propagator matrix and report the CO2 emissions associated with producing the research.These analyses supplement the main experimental results.

10-shot test accuracy.

The 10-shot evaluation compares EPNet and EPNetSSL on miniImagenet, tieredImagenet, and CUB. The reported methods improve accuracy over TADAM and Discriminative by 5%.

  • 10-shot test accuracy: EPNet and EPNetSSL improve accuracy over TADAM and Discriminative by 5% on the reported 10-shot benchmark.The results are reported for miniImagenet, tieredImagenet, and CUB.
  • 10-shot test accuracy: EPNetSSL does not improve much over EPNet in this benchmark, suggesting a larger embedding-propagation impact with fewer labeled data.The authors note that few methods have been evaluated on this benchmark, limiting direct assessment of embedding propagation.

Higher-way results

EPNet attains higher test accuracies than previous state-of-the-art methods across all evaluated higher-way miniImagenet settings, including 10-way, 15-way, and 20-way scenarios.

  • 38.6% accuracy versus 36.5% marks EPNet’s improvement in the 1-shot 20-way miniImagenet scenario.

Additional ablation experiments

The ablations show that EP performs best when using the full propagator matrix, indicating that neighboring embeddings contribute important information beyond simple rescaling.

  • Neighbor information is important for EP performance, as shown by comparing full, off-diagonal-only, and diagonal-only propagator matrices.The diagonal-only version merely rescales original embeddings, whereas the off-diagonal version uses neighboring embeddings.
  • The full propagator matrix achieves the best performance among the tested EP variants.

CO2 Emission Related to Experiments

The experiments used substantial computational resources and produced an estimated 146.88 kgCO2eq in emissions, with reported offsets through Gold Standard.

  • 146.88 kgCO2eq of total emissions were estimated from 24480 hours of computation on Tesla V100 hardware.The infrastructure used a carbon emission factor of 0.02 kg/kWh and hardware with a 250 W TDP.
  • 1000 kgCO2eq, or 685%, were offset through Gold Standard.
Loading 2003.04151v2…