PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian Prototypical Networks for Few-Shot Learning on Omniglot

Stanislav Fort

arXiv:1708.02735v1cs.LGcs.CVcs.NEstat.ML

TL;DR

Few-shot learning asks whether machines can recognize new categories from only a few examples. Gaussian prototypical networks predict image-dependent covariance estimates alongside embeddings and use them to construct a direction- and class-dependent distance metric; on Omniglot, the authors report state-of-the-art results across 1-shot and 5-shot, 5-way and 20-way classification, with 5-shot 5-way comparable to prior state of the art.

  • Problem

    Few-shot learning seeks machine recognition of new categories from a single or small number of examples, inspired by humans’ ability to learn from few examples.

    Source: §1.1 · p. 1

  • Method

    Gaussian prototypical networks predict image-dependent covariance estimates alongside embeddings and use them to construct a direction- and class-dependent distance metric for classification.

    Source: §1.2 · p. 2, §3 · p. 3

  • Results

    The authors report state-of-the-art Omniglot results in 1-shot and 5-shot, 5-way and 20-way classification, with 5-shot 5-way comparable to prior state of the art.

    Source: §1.2 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    On Omniglot, estimating a single real number atop the embedding outperformed diagonal or full covariance estimates; the authors hypothesize the method may show more strength on lower-quality, less homogeneous data.

    Source: §4 · p. 7, §6 · p. 12

    The authors say Omniglot is too simple to fully use the network’s covariance-estimation ability, and hypothesize greater strength on inhomogeneous data with varying point quality.

    Source: §4 · p. 7

Abstract

from arXiv · show

We propose a novel architecture for $k$-shot classification on the Omniglot dataset. Building on prototypical networks, we extend their architecture to what we call Gaussian prototypical networks. Prototypical networks learn a map between images and embedding vectors, and use their clustering for classification. In our model, a part of the encoder output is interpreted as a confidence region estimate about the embedding point, and expressed as a Gaussian covariance matrix. Our network then constructs a direction and class dependent distance metric on the embedding space, using uncertainties of individual data points as weights. We show that Gaussian prototypical networks are a preferred architecture over vanilla prototypical networks with an equivalent number of parameters. We report state-of-the-art performance in 1-shot and 5-shot classification both in 5-way and 20-way regime (for 5-shot 5-way, we are comparable to previous state-of-the-art) on the Omniglot dataset. We explore artificially down-sampling a fraction of images in the training set, which improves our performance even further. We therefore hypothesize that Gaussian prototypical networks might perform better in less homogeneous, noisier datasets, which are commonplace in real world applications.

1 Introduction

Few-shot learning asks models to adapt quickly to unseen classes from only a few labeled examples. The paper extends prototypical networks with image-quality estimates and a confidence-dependent distance metric, reporting strong Omniglot results and hypothesizing greater benefit on less homogeneous data.

1.1 Few-shot learning

  • Few-shot learning requires rapid adaptation to unseen classes from k labeled examples, whereas standard deep learning relies on supervised training with abundant data.The paper describes the influence of individual training examples as small in that setting, preventing rapid introduction of new classes after training.Source: §1.1 · p. 1

1.2 Gaussian prototypical networks

  • Gaussian prototypical networks predict an image-dependent confidence region as a Gaussian covariance matrix and use it to construct a direction- and class-dependent embedding-space distance metric.The authors present this as a preferred use of additional trainable parameters compared with vanilla prototypical networks.Source: §1.2 · p. 2
  • The authors intentionally corrupt parts of the dataset to explore whether weighting individual examples could matter on noisy, inhomogeneous real-world data.They frame this as an exploration of the method's extendability, not as evidence from real-world datasets.Source: §1.2 · p. 2
  • The authors report state-of-the-art Omniglot performance in 1-shot and 5-shot, 5-way and 20-way classification, except that 5-way 5-shot is comparable to prior state of the art.They hypothesize that the advantage may be greater on lower-quality, inhomogeneous datasets.Source: §1.2 · p. 2

2 Related work

Prior few-shot approaches include learned metric embeddings, episodic matching, and meta-learning. This paper instead adapts prototypical networks with learned confidence estimates and a direction- and class-dependent metric.

2 Related work

  • Direct input-space distances perform poorly because image classes relate nonlinearly to pixels, and kNN is sensitive to the chosen distance metric.The passage cites kNN's metric sensitivity as a limitation.Source: §2 · p. 2
  • Learned metric embeddings have supported kNN classification, while matching networks learn pairwise image distances and use episodes that mimic data-poor test conditions.The cited work reports improved few-shot performance from this episodic training scheme.Source: §2 · p. 2
  • Other cited approaches train recurrent or temporal-convolutional meta-learners, and combinations of parametric and non-parametric methods have recently been successful in few-shot learning.The passage cites high Omniglot accuracies for meta-learning approaches.Source: §2 · p. 2
  • The proposed image-classification approach does not use meta-learning; it extends prototypical networks with learned, image-dependent covariance estimates and a direction- and class-dependent distance metric.The resulting clustering is used for classification.Source: §2 · p. 2

3 Methods

Gaussian prototypical networks extend prototypical networks by interpreting part of the encoder output as covariance estimates, enabling class- and direction-dependent distances. Episodic training uses support examples to define class prototypes and classify query examples; the section also specifies encoder variants and evaluation procedures.

3 Methods

  • The Gaussian network retains the vanilla encoder architecture but interprets part of its output as covariance estimates around embeddings, reflecting individual data-point quality.The models use a multi-layer CNN to map images into Euclidean embedding vectors.Source: §3 · pp. 2–3, §3.1 · p. 3

3.1 Encoder

  • The encoder outputs an embedding and covariance components, with radius and diagonal estimates explored; full covariance was judged needlessly complex for these tasks.The radius estimate uses one value per image and is not direction-sensitive; the diagonal estimate allows direction-dependent confidence but remains axis-aligned.Source: §3.1 · p. 3

3.2 Episodic training

  • Episodic training samples classes, support examples, and query examples; support embeddings define prototypes, while query-to-prototype distances drive classification and loss.For Gaussian networks, support-point covariances also contribute to class covariance, which modifies the distance metric.Source: §3.2 · p. 4

3.3 Defining a class

  • Support embeddings are combined using inverse-covariance weights to form class prototypes, allowing less important examples to be down-weighted.For one-shot classification, the sole support embedding is the prototype and its covariance is inherited by the class.Source: §3.3 · pp. 5–6

3.4 Evaluating models

  • Evaluation measures k-shot accuracy for k from 1 to 19 in 5-way and 20-way test classification, averaging results from the five highest training-accuracy stages.For each k, the query count is 20 − k because Omniglot provides 20 examples per class.Source: §3.4 · p. 6

4 Datasets

The experiments use Omniglot with a recommended alphabet-based train/test split and augment classes through rotations. The authors also down-sample some training data to probe covariance estimation and its potential value on less homogeneous datasets.

4 Datasets

  • Omniglot contains 1,623 character classes from 50 alphabets, with 20 handwritten grayscale 105 × 105 examples per class, down-sampled to 28 × 28 × 1, mean-subtracted, and inverted.Source: §4 · p. 6
  • The recommended split assigns 30 alphabets to training and 20 to testing, with 964 training classes and 659 test classes and no class overlap; no separate validation set was used.The best-performing model was chosen using training accuracies alone.Source: §4 · pp. 6–7
  • Rotating each character by 90°, 180°, and 270° and treating each rotation as a new class quadrupled the class count, but symmetric characters made 100% accuracy unreachable.The figure illustrates the original character and its three rotated classes.Source: §4 · p. 7, Figure 3 · p. 7
  • The authors down-sampled part of the training set in some experiments because Omniglot may be too simple to fully use covariance estimates, and hypothesize greater strength on inhomogeneous, variable-quality data.Source: §4 · p. 7

5 Experiments

Experiments compare Gaussian and vanilla prototypical networks across model sizes, covariance estimates, embedding dimensions, and training conditions. The best models achieve state-of-the-art results on most Omniglot test regimes, while radius estimates and partial training-set down-sampling provide key performance advantages.

5 Experiments

  • The experiments vary encoder capacity, embedding dimensionality, covariance construction, and distance metric, and also compare Gaussian networks with vanilla prototypical networks.The tested metrics include cosine, √L2, and L2.Source: §5 · p. 7
  • Training used 60-way episodes with one support point per class, while evaluation included 20-way tests and final 5-way tests for the best models.The authors report that limiting training support points led to better classification accuracy.Source: §5 · pp. 7, 9
  • With matched parameter counts, the radius covariance estimate outperforms both the diagonal estimate and the vanilla prototypical network.The radius estimate adds one real number per embedding vector; the diagonal estimate does not seem to improve performance over the matched-parameter alternative.Source: §5 · pp. 7–8, Figure 4 · p. 8
  • Purposefully down-sampling part of the training set improves k-shot test accuracy for k > 1, which the authors attribute to better use of covariance estimates.The down-sampling choices were arbitrary and not optimized.Source: §5 · pp. 7–8, Figure 5 · p. 9
  • The models achieve state-of-the-art 1-shot and 5-shot results in 20-way classification and 1-shot 5-way classification, while 5-shot 5-way results are comparable to prior state of the art.The best 5-way 5-shot model reaches 99.73 ± 0.02%.Source: §5 · p. 9, Table 3 · p. 11

5.1 Usage of covariance estimate

  • For undamaged data, most covariance estimates take the same value; after down-sampling, the aligned distributions show a heavier tail toward higher predicted s.The authors interpret higher s for damaged examples as down-weighting them; histogram alignment accounts for batch-normalization shifts, and differences from the dominant value influence classification.Source: §5.1 · pp. 10–11, Figure 9 · p. 11

6 Conclusion

The conclusion presents Gaussian prototypical networks as a parameter-efficient improvement over vanilla prototypical networks and reports strong Omniglot results. It identifies more complex, lower-quality datasets as a priority for further evaluation.

6 Conclusion

  • Gaussian prototypical networks outperform vanilla prototypical networks with comparable parameter counts, and predicting one real number per embedding works better than diagonal or full covariance estimates.The authors suspect less homogeneous, lower-quality datasets might benefit from more complex covariance estimates.Source: §6 · p. 12
  • The authors report state-of-the-art 1-shot and 5-shot results in 5-way and 20-way Omniglot tests, except that 5-way 5-shot performance is comparable to prior state of the art.They report results close to perfect in 5-way classification and conclude that further few-shot development should focus on more complex datasets than Omniglot.Source: §6 · p. 12
  • Training on partially down-sampled Omniglot improves accuracy, particularly for k > 1, and the authors hypothesize uncertainty learning may be more beneficial on poorer-quality datasets.They connect this hypothesis to the potential value of down-weighting some data points for faithful classification.Source: §6 · p. 12

A Embedding space snapshots

The figures show two-dimensional PCA projections of the embedding space during training. They depict clustering relevant to classification, including query embeddings around class prototypes.

A Embedding space snapshots

  • A PCA projection shows similar characters clustering in the embedding space, which is used to classify unknown query images.Source: Figure 10 · p. 14
  • Together, the projections illustrate clustering in the learned embedding space, including query points around prototypes.Source: Figure 10 · p. 14, Figure 11 · p. 14
  • A separate PCA projection shows query-point embeddings clustering around class prototypes.Source: Figure 11 · p. 14
Loading 1708.02735v1…