PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning Probabilistic Inference For Prediction

Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, Richard E. Turner

arXiv:1805.09921v4stat.MLcs.LG

TL;DR

Few-shot learning needs flexible, data-efficient prediction across related small datasets, while existing probabilistic meta-learning views cover limited method families. The paper introduces ML-PIP and VERSA, which amortizes posterior inference through flexible forward-pass networks; VERSA achieves state-of-the-art benchmark results and demonstrates one-shot ShapeNet view reconstruction.

  • Problem

    Existing probabilistic interpretations of meta-learning are limited to specific method families, motivating a broader framework for flexible, data-efficient learning.

  • Method

    ML-PIP frames meta-learning as amortized approximate probabilistic inference, while VERSA maps few-shot datasets to distributions over task-specific parameters in forward passes.

  • Results

    VERSA sets new state-of-the-art results on miniImageNet and Omniglot benchmarks and demonstrates challenging one-shot view reconstruction.

  • Takeaways & Limitations

    VERSA supports arbitrary shot counts and, for classification, arbitrary class counts at training and test time while avoiding gradient-based test-time optimization.

  • Takeaways & Limitations

    The context-independent inference assumption is an approximation, and predictive-posterior recovery is discussed under identifiability conditions for task-specific parameters.

Abstract

from arXiv · show

This paper introduces a new framework for data efficient and versatile learning. Specifically: 1) We develop ML-PIP, a general framework for Meta-Learning approximate Probabilistic Inference for Prediction. ML-PIP extends existing probabilistic interpretations of meta-learning to cover a broad class of methods. 2) We introduce VERSA, an instance of the framework employing a flexible and versatile amortization network that takes few-shot learning datasets as inputs, with arbitrary numbers of shots, and outputs a distribution over task-specific parameters in a single forward pass. VERSA substitutes optimization at test time with forward passes through inference networks, amortizing the cost of inference and relieving the need for second derivatives during training. 3) We evaluate VERSA on benchmark datasets where the method sets new state-of-the-art results, handles arbitrary numbers of shots, and for classification, arbitrary numbers of classes at train and test time. The power of the approach is then demonstrated through a challenging few-shot ShapeNet view reconstruction task.

1 INTRODUCTION

The paper addresses the need for flexible, data-efficient learning across many related small datasets. It introduces ML-PIP and VERSA to unify probabilistic meta-learning and amortize fast, versatile inference.

  • Few-shot learning seeks rapid adaptation to many small, related datasets by sharing information across tasks.
  • ML-PIP provides a unifying probabilistic framework that reframes and extends existing meta-learning interpretations to cover broader method families.It includes gradient-based, metric-based, amortized MAP, and conditional probability approaches.
  • The framework combines hierarchical task structure, meta-learned inference, and amortization to support uncertainty-aware and rapid learning.
  • VERSA replaces test-time optimization with inference-network forward passes that output distributions over task-specific parameters in one pass.Its amortization network supports arbitrary shot counts and, for classification, arbitrary class counts at training and test time.

2 META-LEARNING PROBABILISTIC INFERENCE FOR PREDICTION

ML-PIP models few-shot prediction as amortized approximate probabilistic inference over task-specific parameters. It learns inference networks by optimizing predictive accuracy across episodically sampled tasks while using shared global parameters.

  • The framework uses a multi-task probabilistic model with shared parameters and task-specific parameters for few-shot learning.
  • ML-PIP approximates each task’s posterior predictive distribution with an amortized inference network conditioned on the training dataset and test input.The network first approximates the posterior over task-specific parameters, then forms the predictive distribution.
  • The inference network is trained by minimizing expected KL divergence between true and approximate posterior predictive distributions across tasks.Training therefore targets approximate Bayesian held-out predictive performance.
  • Under sufficient posterior expressiveness and identifiability, global optimization recovers the true task-specific posterior while supporting accurate prediction.The paper notes that predictive inference remains valid regardless of task-parameter posterior identifiability.
  • The end-to-end objective optimizes shared parameters and inference-network parameters using episodic train/test splits and Monte Carlo approximation.The procedure can learn the prior over task-specific parameters implicitly through the amortization network.
  • ML-PIP is presented as a framework that unifies a broad class of recent meta-learning methods and motivates versatile learning systems.

3 VERSATILE AMORTIZED INFERENCE

VERSA makes amortized inference both fast and flexible by processing variable-sized few-shot datasets and producing task-specific parameters for classification and view reconstruction. Its context-independent classifier inference avoids output dimensionality tied to the number of classes.

  • VERSA defines rapid inference as a feed-forward computation and flexible inference as handling variable shots or classes without retraining.
  • Permutation-invariant instance pooling lets the amortization network process few-shot datasets with arbitrary numbers of training observations.
  • For classification, a shared feature extractor feeds task-specific linear softmax classifiers whose weights and biases are inferred probabilistically.
  • Context-independent inference predicts each class’s weight vector from that class’s examples instead of amortizing the full classifier matrix.This reduces output dimensionality and removes the need to specify the number of classes in advance.
  • VERSA constructs the classification parameters with C forward passes and can train across different class counts, though context independence is an approximation.The paper reports theoretical and empirical justification for this approximation.
  • For view reconstruction, VERSA maps a small set of oriented training images to a task-specific latent input, then combines it with a test angle to generate a new image.The generator parameters are global, while the latent input is task-specific.

4 ML-PIP UNIFIES DISPARATE RELATED WORK

ML-PIP provides a unified approximate-inference perspective spanning gradient-based, metric-based, amortized MAP, and conditional-modeling approaches. Within this view, VERSA is distributional, more flexible than activation averaging, and supports end-to-end multi-task learning.

  • ML-PIP unifies gradient-based, metric-based, amortized MAP, and conditional-modelling approaches as approximate inference in hierarchical models.
  • Gradient-Based Meta-Learning: One-step gradient updates recover MAML as semi-amortized ML-PIP, while VERSA removes gradient computation at test time and back-propagation through updates during training.
  • Metric-Based Few-Shot Learning: Metric-based averaging of class activations recovers prototypical networks, whereas VERSA uses a distributional and more flexible amortization function.
  • Amortized MAP inference: Amortized MAP inference predicts class weights from activations, while VERSA supports end-to-end full multi-task learning and distributions beyond point estimates.
  • Comparison to Variational Inference (VI): Standard amortized VI differs from ML-PIP by using KL regularization and omitting meta train/test splits.

5 EXPERIMENTS AND RESULTS

VERSA is evaluated through toy posterior-inference experiments, few-shot classification on Omniglot and miniImageNet, and ShapeNet view reconstruction. Across these settings, it demonstrates accurate approximate inference, flexible test-time conditions, competitive classification, and improved reconstruction quality.

  • Few-shot classification: VERSA achieves state-of-the-art performance despite adapting only the weights of the top-level classifier, unlike methods adapting all learned parameters.The comparison includes approaches with comparable training procedures and convolutional feature extraction architectures; pre-trained and residual-network methods are excluded.
  • Inference comparisons: VERSA outperforms amortized variational inference and non-amortized variational inference, while the latter forms posteriors significantly more slowly through many forward and backward passes.The comparison uses the same model and evaluates log-likelihood and accuracy across few-shot settings.
  • Few-shot classification: On Omniglot, Figure 5 evaluates test accuracy while varying the number of classes and shots, including models trained with different way-shot combinations.The shot-variation panel evaluates all models on 5-way classification.
  • ShapeNet view reconstruction: In ShapeNet view reconstruction, VERSA generates sharper, more detailed images than a matched conditional variational autoencoder, with metrics improving as shots increase to 5.Both methods capture the correct object orientation, while VERSA often imputes occluded information from learned object statistics.

Model MSE SSIM

Table 2 reports view-reconstruction performance using MSE and SSIM measured against ground-truth images.

  • Model MSE SSIM: MSE is lower-is-better, whereas SSIM is higher-is-better for comparing generated and ground-truth views.Error bars are not shown because they are insignificant.

6 CONCLUSIONS

ML-PIP provides a probabilistic framework that unifies a broad class of meta-learning methods, while VERSA amortizes posterior inference for fast few-shot prediction. The paper reports state-of-the-art performance and compelling visual results on few-shot tasks, including 1-shot view reconstruction.

  • ML-PIP unifies a broad class of recently proposed meta-learning methods and suggests alternative approaches.
  • VERSA avoids gradient-based optimization at test time by amortizing posterior inference over task-specific parameters.
  • VERSA achieved state-of-the-art performance across several few-shot learning tasks and compelling visual results on a challenging 1-shot view reconstruction task.

B.2 EMPIRICAL JUSTIFICATION

The empirical study examines whether class-specific weights can be inferred independently of the other classes in a task. The results generally support this approximation, while exposing overlap and capacity-related deviations for similar classes.

  • The experiment evaluates context-independent inference using fifty randomly generated tasks and free-form variational inference on task-specific weights.
  • The study uses 5-way MNIST classification with fixed weight dimensions dθ = 16 and dθ = 2, trained episodically to convergence.
  • The trained model achieves 99% accuracy on held-out test examples for the tasks.
  • Weights generally cluster by class and show similar means across tasks, supporting approximate task independence for each class.
  • Some overlap occurs between similar classes, and class ‘2’ weights move away from their cluster when tasks contain both classes ‘1’ and ‘2’.
  • The deviations are attributed to limited model capacity for assigning class weights to distinct regions of weight space.

C VARIATIONAL INFERENCE DERIVATIONS FOR THE MODEL

The variational-inference derivation defines amortized inference through a neural network with shared parameters, while retaining the same objective derivation as non-amortized inference. Optimization uses task sampling and Monte Carlo integration, with differences from the paper’s final objective.

  • Amortized variational inference parameterizes qφ(ψ|D(t), θ) with a neural network having fixed-sized shared parameters φ.
  • Non-amortized inference instead optimizes local parameters independently for each new task at test time.
  • The ELBO derivation is unchanged between amortized and non-amortized variational inference.
  • The stochastic estimator samples tasks and uses Monte Carlo integration over ψ with samples from qφ(ψ|D(t), θ).
  • The derivation differs from the paper’s objective because the ELBO includes a KL term and does not distinguish training from test data within a task.

D.1 OMNIGLOT FEW-SHOT CLASSIFICATION TRAINING PROCEDURE

The Omniglot procedure uses episodic few-shot classification with randomly sampled classes and held-out examples within each task. Evaluation is performed on randomly selected tasks from the test set.

  • Omniglot contains 1623 handwritten characters from 50 alphabets, with 20 instances per character.
  • Images are resized to 28 × 28 pixels, and character classes are augmented with 90-degree rotations.
  • The augmented data are split into 4400 training, 400 validation, and 1292 test classes.
  • Each episodic task samples C classes, uses kc character instances for training, and reserves 15 instances for testing.
  • Final evaluation uses 600 randomly selected tasks from the test set.

D.2 miniIMAGENET FEW-SHOT CLASSIFICATION TRAINING PROCEDURE

The miniImageNet experiments use predefined class splits and episodic few-shot training, with separate schedules for 5-way 1-shot and 5-way 5-shot models. The architectures include feature extraction, amortization, and linear-classifier components.

  • miniImageNet contains 60,000 84 × 84 color images across 100 classes, with 600 instances per class.
  • The experiments use 64 training, 16 validation, and 20 test classes from established splits.
  • The models are trained episodically, following the same procedure used for Omniglot.
  • The 5-way 5-shot model uses 4 tasks per batch for 100,000 iterations with a constant learning rate of 0.0001.
  • The 5-way 1-shot model uses 8 tasks per batch for 50,000 iterations with a constant learning rate of 0.00025.
  • The architecture comprises a feature extractor, amortization network, and linear classifier, with the amortization network producing mean-field Gaussian parameters for classifier weight distributions.

E.1 VIEW RECONSTRUCTION TRAINING PROCEDURE AND NETWORK ARCHITECTURES

The ShapeNet view-reconstruction experiments use a 12-category subset of ShapeNetCore v2, partitioned into training, validation, and test objects. Each object is represented by 36 grayscale 32 × 32 views, with dedicated encoder, amortization, generator, and category-list specifications.

  • ShapeNetCore v2 contains 55 object categories and approximately 51,300 unique objects; the experiments use 12 of the largest categories.
  • The selected categories are concatenated into 37,108 objects and randomly split into 25,975 training, 3,710 validation, and 7,423 testing objects.
  • Each object has V = 36 rendered 128 × 128 views spaced every 10 degrees in azimuth.
  • The rendered views are converted to grayscale and reduced to 32 × 32 pixels before training.
  • The ShapeNet setup specifies separate encoder, amortization, and generator networks for few-shot learning, with no dropout or batch normalization in the encoder and generator.
Loading 1805.09921v4…