PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning with Latent Embedding Optimization

Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell

arXiv:1807.05960v3cs.LGcs.CVstat.ML

TL;DR

High-dimensional gradient-based meta-learning can struggle when only a few examples are available. LEO learns a data-dependent latent generative representation of model parameters and performs adaptation in its low-dimensional latent space, achieving state-of-the-art few-shot classification results on miniImageNet and tieredImageNet.

  • Problem

    Using only a few samples to compute gradients in a high-dimensional parameter space can make optimization-based meta-learning generalization difficult.

  • Method

    LEO learns a stochastic, data-dependent low-dimensional latent space that generates model parameters and supports gradient-based adaptation through a differentiable decoder.

  • Results

    LEO achieves state-of-the-art performance on both miniImageNet and tieredImageNet few-shot classification tasks.

  • Takeaways & Limitations

    LEO provides task-specific parameter initialization and enables more effective adaptation by optimizing in latent space.

Abstract

from arXiv · show

Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this low-dimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.

1 INTRODUCTION

Few-shot learning exposes the difficulty of adapting high-dimensional models from very few examples. LEO addresses this by conditioning parameter initialization on task data and performing meta-learning in a low-dimensional latent space, achieving state-of-the-art results on miniImageNet and tieredImageNet.

  • Few-shot learning requires models to learn new concepts or behaviors from very few examples or limited experience.
  • Optimization-based meta-learning adapts a shared parameter initialization with a few gradient steps, but high-dimensional gradients from typically 1 or 5 samples can hinder generalization.
  • LEO learns a low-dimensional latent embedding of model parameters and performs optimization-based meta-learning in that space.
  • LEO conditions task initialization on training data and uses a relation network to account for joint relationships among inputs, while stochasticity represents few-shot ambiguities.
  • LEO achieves state-of-the-art results on both miniImageNet and tieredImageNet few-shot classification tasks.

2 MODEL

LEO reframes optimization-based meta-learning by generating task-conditioned model parameters from a learned latent representation and adapting in that lower-dimensional space. It retains the episodic, inner-loop/outer-loop structure of methods such as MAML while extending the approach beyond the specific classification setting.

  • Problem definition: Few-shot meta-learning adapts a shared parameter initialization to new tasks using a few gradient-descent steps, with validation performance guiding meta-training.Tasks contain training and validation sets, while separate meta-training, validation, and test sets use disjoint target classes.
  • Latent Embedding Optimization: LEO decouples optimization-based meta-learning from high-dimensional parameter space by learning a data-conditioned stochastic latent space from which model parameters are generated.The latent representation uses an information bottleneck and replaces a single explicit parameter vector with a conditional distribution over parameters.
  • Latent Embedding Optimization: Given a task’s examples, LEO encodes them into a latent code, decodes that code into initial parameters, and performs adaptation by differentiating the task loss with respect to the latent code.The optimized latent code is decoded again to produce final task-adapted parameters.
  • Initialization: LEO’s encoder uses a relation network to model pairwise relationships among examples and produce context-dependent, task-specific parameter initialization.For K-shot classification, pairwise relation outputs are grouped by class and averaged before forming the latent representation.
  • Beyond classification and linear output layers: LEO can generate entire parameter vectors for nonlinear models and optimize any differentiable loss computed on task training and validation sets, beyond few-shot classification and single linear output layers.The general formulation uses one latent code to generate all model parameters.

3 RELATED WORK

Prior few-shot meta-learning work includes metric-based, memory-based, and optimization-based methods, alongside parameter-generating and probabilistic approaches. Neural Processes are related through latent representations but differ in their lack of inner-loop adaptation.

  • Few-shot meta-learning methods include metric-based approaches for learning class similarity, memory-based approaches using stored examples or adaptation algorithms, and optimization-based approaches seeking parameters suited to fast gradient adaptation.
  • Parameter-generating approaches produce some or all parameters of another network, including methods that modify biases, generate class-conditional output weights, or predict top-layer parameters.
  • MAML differentiates through inner-loop adaptation, whereas REPTILE uses an L2 loss toward instance-specific adapted models; both operate on full high-dimensional model parameters.
  • Probabilistic meta-learning methods learn Gaussian posteriors or variational extensions to MAML, but extending them to more complex distributions and diverse task sets remains unclear.
  • Neural Processes map to and from latent spaces for few-shot function estimation, but do not perform inner-loop adaptation and instead optimize a variational objective.

4 EVALUATION

LEO is evaluated on uncertain, multimodal few-shot regression and on 1-shot and 5-shot ImageNet classification. The results show uncertainty modeling, latent adaptation, and data-conditional parameter generation supporting strong performance across these settings.

  • Few-shot regression: LEO models uncertainty in noisy 5-shot regression, especially where sine and line solutions are ambiguous.It samples different models from both families while fitting targets well near observed data.
  • Few-shot regression: In ambiguous regression instances, LEO samples distinct solutions from both sine and line families.The generated distribution is multimodal when both explanations fit the training data.
  • Few-shot classification: LEO achieves state-of-the-art performance on both miniImageNet and tieredImageNet classification tasks.With multi-view miniImageNet features, it reports 63.97 ± 0.20% for 1-shot and 79.49 ± 0.70% for 5-shot accuracy.
  • Ablation study: Ablations indicate that the low-dimensional bottleneck, data-conditional encoding, and latent-space adaptation are critical to LEO’s performance.Removing latent adaptation or replacing data-dependent encoding with a random prior decreases performance.
  • Latent-space analysis: Adapted latent codes develop more class-conditional separation than initial encoder outputs, while LEO’s latent-space curvature is 2 orders of magnitude higher than parameter-space curvature.The decoder expands space by at least one order of magnitude, so fixed latent steps can change the underlying function more strongly.

5 CONCLUSIONS AND FUTURE WORK

LEO uses a low-dimensional, data-dependent latent embedding and gradient-based adaptation to achieve state-of-the-art results on challenging few-shot image-classification tasks. The authors identify joint feature-extractor training and applications to reinforcement learning or sequential data as future directions.

  • LEO captures task-relevant parameter distributions with a low-dimensional, data-dependent latent embedding and adapts through gradient updates in that space.This provides task-specific parameter initialization and is intended to make adaptation more effective.
  • LEO achieves state-of-the-art results on 5-way 1-shot and 5-shot miniImageNet and tieredImageNet classification.
  • Future work: Future work could jointly learn the feature extractor through meta-learning instead of using a pre-trained one.
  • Future work: Future work could apply LEO to reinforcement-learning tasks or sequential data.

A.1 REGRESSION TASK DESCRIPTION

The regression experiments use noisy 1D 5-shot tasks drawn from a multimodal distribution combining sinusoids and lines.

  • The experiments follow Finn et al. (2018) and evaluate 1D 5-shot regression with inputs sampled uniformly from [−5, 5].
  • The task distribution is multimodal, with half sinusoidal instances and half linear instances.Sinusoid amplitudes and phases are sampled from [0.1, 5] and [0, π], while line slopes and intercepts are sampled from [−3, 3].
  • Gaussian noise with standard deviation 0.3 is added to the regression targets.

A.2 LEO NETWORK ARCHITECTURE

The LEO architecture generates model parameters from latent codes conditioned on few-shot data, using a 3-layer MLP model and a 16-dimensional bottleneck. The architecture details are reported for 5-way 1-shot image classification with 12 task instances processed in parallel.

  • LEO generates the complete parameter tensor θ of a 3-layer MLP from a latent code conditioned on concatenated training inputs and regression targets.
  • The encoder uses a 3-layer, 32-unit-per-layer MLP, and the latent bottleneck has size nz = 16.
  • The relation network and decoder are 3-layer MLPs with 32 units per layer and no biases.
  • Table 3 reports architecture shapes for 5-way 1-shot miniImageNet and tieredImageNet during meta-training, using a meta-batch of 12 task instances in parallel.

B EXPERIMENTAL SETUP - CLASSIFICATION

The classification experiments use standard episodic 5-way 1-shot and 5-shot tasks, with separate training, validation, and testing examples. Meta-learning uses the corresponding meta-sets without data augmentation or feature averaging, except for a separately reported multi-view condition.

  • Each classification task contains 5 randomly sampled categories and either 1 or 5 training examples per class.Validation examples are used during meta-training, while remaining examples are used for validation and testing.
  • The standard meta-learning setup does not use data augmentation, feature averaging, or additional data beyond the corresponding training and validation meta-sets.
  • The multi-view embedding condition averages features from four corner crops, a central crop, and their horizontal mirrors for comparison with Qiao et al. (2017).

B.2 FEATURE PRE-TRAINING

Feature embeddings were pretrained on meta-training classes before LEO meta-learning, using a Wide Residual Network and dimensionality reduction. The resulting 640-dimensional embeddings were cached to simplify meta-learning.

  • A WRN-28-10 pretrained on the meta-training set classified 80 × 80 pixel images into training classes before meta-learning.The setup used 64 classes for miniImageNet and 351 for tieredImageNet.
  • The feature extractor used three steps of dimensionality reduction and dropout inside residual blocks during training.Dropout used pkeep = 0.5 and was disabled during evaluation and dataset export.
  • The feature extractors were trained with learning-rate annealing schedules specified separately for miniImageNet and tieredImageNet.
  • 640-dimensional feature embeddings were precomputed from layer 21 activations using spatial average pooling.Caching these embeddings substantially simplified the subsequent meta-learning process.

B.3 LEO NETWORK ARCHITECTURE

LEO uses a shared low-dimensional parameter-generator architecture with encoder, relation, and decoder networks, while jointly learning adaptation-related parameters and regularizing the meta-optimization.

  • The encoder and decoder are linear networks with a bottleneck embedding space of size nz = 64.The relation network has three fully connected layers with 128 units per layer and rectifier nonlinearities.
  • The Meta-SGD (our features) baseline uses the same one-layer softmax classifier as the base model.
  • The inner loop runs in parallel across batches of 12 problem instances, and relation-network use adds negligible computational cost when k2 is small.
  • The relation network is applied to pairs of training examples and averaged into class-specific means and variances.For the described setup, it is applied 25 times and produces five class-specific means and variances.
  • Dropout is applied independently to feature embeddings at every step, while meta-gradients are clipped at an absolute value and norm of 0.1.The clipping excludes inner-loop latent-space adaptation gradients.
  • LEO jointly optimizes encoder, relation, and decoder parameters together with per-parameter learning rates in latent and parameter spaces.Optimization uses Adam over the meta-learning objective with early stopping based on validation accuracy.
  • Hyperparameters are selected by random search using meta-validation accuracy, and each LEO baseline receives a separate random search.Reported performance averages five independent runs with fixed selected hyperparameters.

B.6 TRAINING TIME

LEO training separates feature pretraining, hyperparameter selection, and final meta-training, then adapts each evaluation task in latent space before inference. Cached embeddings make LEO training substantially cheaper than image-extractor training.

  • B.6 TRAINING TIME: LEO training takes 1–2 hours on miniImageNet and around 5 hours on tieredImageNet per independent run using a multi-core CPU.Feature embeddings can be cached before LEO training, yielding an efficient meta-learning process.
  • B.6 TRAINING TIME: Training the image extractor takes 5 hours for miniImageNet and around a day for tieredImageNet using 32 GPUs.
  • B.6 TRAINING TIME: The three-stage procedure pretrains embeddings, trains LEO with meta-validation early stopping and random-search selection, then retrains LEO five times.
  • B.6 TRAINING TIME: Final-stage meta-learning uses embeddings from both meta-training and meta-validation sets while retaining early stopping on meta-validation.This early stopping is used as a proxy for generalization to classes excluded during feature pretraining.
  • B.6 TRAINING TIME: During evaluation, stochasticity and dropout are disabled, and adapted parameters are used only for inference on the sampled task.
  • B.6 TRAINING TIME: Evaluation encodes few-shot samples into latent space, decodes their means to initialize parameters, and performs latent-space adaptation followed optionally by parameter-space adaptation.
  • B.6 TRAINING TIME: The resulting parameters form the final adapted model for that particular problem instance.
Loading 1807.05960v3…