PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Transfer Learning for Few-Shot Learning

Qianru Sun, Yaoyao Liu, Tat-Seng Chua, Bernt Schiele

arXiv:1812.02391v3cs.CV

TL;DR

Few-shot learning methods must adapt effectively from very limited labeled data, while conventional meta-learning often uses shallow learners to avoid overfitting. This paper introduces meta-transfer learning, which adapts pretrained deep networks through Scaling and Shifting and trains with a hard-task curriculum. Across miniImageNet and Fewshot-CIFAR100, the approach achieves top few-shot performance, with the proposed components also supporting convergence and accuracy.

  • Problem

    Few-shot learning must learn new concepts from few labeled examples, while conventional meta-learning often uses shallow base learners because deep networks overfit in this setting.

  • Method

    MTL adapts pretrained DNNs by meta-learning lightweight Scaling and Shifting operations on their weights, combined with hard-task meta-batch training.

  • Results

    MTL with HT meta-batch achieves top performance on miniImageNet and Fewshot-CIFAR100 few-shot benchmarks.

  • Takeaways & Limitations

    The paper reports that MTL's operations efficiently adapt pretrained DNN learning experience to unseen tasks, especially in extreme 1-shot settings.

Abstract

from arXiv · show

Meta-learning has been proposed as a framework to address the challenging few-shot learning setting. The key idea is to leverage a large number of similar few-shot tasks in order to learn how to adapt a base-learner to a new task for which only a few labeled samples are available. As deep neural networks (DNNs) tend to overfit using a few samples only, meta-learning typically uses shallow neural networks (SNNs), thus limiting its effectiveness. In this paper we propose a novel few-shot learning method called meta-transfer learning (MTL) which learns to adapt a deep NN for few shot learning tasks. Specifically, "meta" refers to training multiple tasks, and "transfer" is achieved by learning scaling and shifting functions of DNN weights for each task. In addition, we introduce the hard task (HT) meta-batch scheme as an effective learning curriculum for MTL. We conduct experiments using (5-class, 1-shot) and (5-class, 5-shot) recognition tasks on two challenging few-shot learning benchmarks: miniImageNet and Fewshot-CIFAR100. Extensive comparisons to related works validate that our meta-transfer learning approach trained with the proposed HT meta-batch scheme achieves top performance. An ablation study also shows that both components contribute to fast convergence and high accuracy.

1. Introduction

Few-shot learning seeks to learn new concepts from very few labeled examples, but existing meta-learning methods often rely on costly task collections and shallow base learners. The paper proposes meta-transfer learning with Scaling and Shifting operations and a hard-task curriculum to adapt pretrained deep networks.

  • Motivation: Few-shot learning aims to learn new concepts from few labeled examples, but machine learning systems struggle to match human learning efficiency.On CIFAR-100, a state-of-the-art method achieves 40.1% accuracy for 1-shot learning versus 75.7% in the all-class fully supervised case.
  • Limitations of existing methods: Existing task-based meta-learning requires many similar training tasks and typically uses shallow base learners to reduce overfitting.For miniImageNet, MAML uses a 4-layer convolutional network and achieves optimal performance after learning on 240k tasks.
  • Proposed method: MTL transfers large-scale pretrained DNN weights to few-shot tasks through lightweight Scaling and Shifting operations, αX + β.The operations are meta-learned as task-level hyperparameters while preserving the pretrained weights.
  • Proposed method: MTL uses pretrained weights to provide good initialization, requiring only 8k miniImageNet tasks, 30 times fewer than MAML.The lightweight operations also use fewer learnable parameters and reduce the chance of overfitting.
  • Training strategy: The hard-task meta-batch strategy re-samples harder tasks based on past failures with the lowest validation accuracy.This curriculum is designed to make meta-transfer learning improve through more challenging training examples.

2. Related work

The related work considers supervised meta-learning methods and positions the paper's hard-task strategy alongside metric, memory-network, gradient-based, and hard-sample-mining approaches.

  • Few-shot learning methods: Supervised few-shot meta-learning methods include metric learning, memory networks, and gradient-descent-based approaches.These categories learn efficient similarity spaces, store experience from seen tasks, or adapt models through gradient-based procedures.
  • Paper positioning: The paper's method is presented as a meta-learning paradigm with a hard-task meta-batch training strategy, contrasted with transfer learning and conventional meta-learning.Figure 2 separately depicts the method's three-phase pipeline, including pretrained DNN training, MTL, and meta-test.
  • Hard-task curriculum: Hard-task sampling is inspired by hard negative mining, where training on confusing examples is associated with greater robustness and better performance.The paper applies online hard-task sampling to MTL and reports that it can also enhance MAML.

3. Preliminary

Meta-learning alternates between meta-training on sampled classification episodes and meta-testing on unseen tasks, where prior experience guides base-learner adaptation and evaluation.

  • Meta-training: A meta-training example is a classification episode sampled from a task distribution, with training and test splits serving the base-learner and meta-learner.Meta-training learns from multiple episodes sampled from p(T).
  • Meta-test: During meta-test, an unseen task is used to adapt the base-learner through the trained meta-learner.The adaptation can use mechanisms such as initialization, after which the task's test split evaluates performance.
  • Meta-test: When multiple unseen tasks are evaluated, their test results are averaged to obtain the final evaluation.The averaging is performed across the unseen tasks' test splits.

4. Methodology

MTL first pre-trains a DNN, freezes its feature extractor, and adapts it to few-shot tasks through neuron-level Scaling and Shifting parameters. Its HT meta-batch curriculum repeatedly resamples difficult tasks identified from prior failures.

  • DNN training on large-scale data: MTL pre-trains a DNN on large-scale benchmark data, freezes the feature extractor, and discards the large-scale classifier before meta-training.The subsequent few-shot tasks can use different classification objectives, such as 5-class rather than 64-class classification.
  • Meta-transfer learning: During meta-training, MTL adapts frozen feature-extractor neurons by learning Scaling and Shifting parameters while optimizing a task-specific classifier.The classifier is updated using training loss, while the SS parameters are optimized using test loss.
  • Meta-transfer learning: Compared with parameter-level Fine-Tuning, SS updates fewer parameters and is intended to reduce overfitting and catastrophic forgetting in few-shot adaptation.The figure example states that SS reduces the number of learning parameters to below 2^9.
  • Meta-transfer learning: SS applies per-neuron scaling and shifting to weights and biases, reducing trainable parameters while leaving the original DNN weights unchanged.For a neuron, the operation scales the input transformation and adds a shift to the original bias; element-wise multiplication is used for scaling.
  • Hard task (HT) meta-batch: The HT meta-batch curriculum identifies failure classes using test-class accuracies and resamples harder tasks from those classes for continued training.The lowest-accuracy class is returned for hard-task sampling, and indirectly sampling new examples from that class generally works better than reusing current-task samples.

5. Experiments

The experiments evaluate MTL and HT meta-batch learning on miniImageNet and FC100 using few-shot recognition accuracy and convergence speed. MTL with pre-trained ResNet-12, Scaling and Shifting, and HT meta-batches achieves strong results, while ablations clarify the roles of its components.

  • Experimental setup: Experiments evaluate 5-class few-shot recognition on miniImageNet and FC100, including 1-shot, 5-shot, and 10-shot settings.miniImageNet uses 1-shot and 5-shot tasks, while FC100 also includes 10-shot tasks.
  • Results: 61.2% accuracy is achieved by MTL with Scaling and Shifting, HT meta-batch, and pre-trained ResNet-12 on miniImageNet 5-class 1-shot tasks.The same configuration reaches 75.5% on 5-class 5-shot miniImageNet tasks, compared with TADAM’s reported 76.7%.
  • Results: MTL consistently outperforms MAML on FC100, by around 7% according to the reported comparison.The experiments also compare MTL against ablative methods and state-of-the-art approaches.
  • Ablation study: Updating only the classifier θ performs better than updating both the feature extractor Θ and classifier θ, with an approximately 5% reduction for the latter on miniImageNet 1-shot.The authors attribute this to having too many parameters to optimize from few-shot data.
  • Ablation study: SS* consistently outperforms FT*, while HT meta-batches and pre-trained ResNet-12 improve other meta-learning variants.MAML with HT meta-batch gains an average of 1% across two datasets, and replacing 4CONV with pre-trained ResNet-12 yields 10% and 9% improvements on miniImageNet.
  • Convergence: MTL reaches best performance using 8k tasks on miniImageNet and at most 2k tasks on FC100, compared with MAML’s 240k tasks on miniImageNet.The authors associate faster convergence with pre-trained ResNet-12 and learning fewer parameters through Scaling and Shifting.
  • Convergence: HT meta-batches consistently improve recognition accuracy over conventional meta-batches and reach top FC100 performance after about 2k, 1k, and 1k iterations for 1-, 5-, and 10-shot tasks.Figure 4 compares accuracy and convergence speed across miniImageNet and FC100 settings.

6. Conclusions

The paper concludes that MTL trained with the HT meta-batch curriculum achieves top few-shot performance on miniImageNet and FC100. It highlights efficient adaptation through pre-trained DNN neurons and especially strong benefits in extreme 1-shot settings and FC100 convergence.

  • Conclusion: MTL trained with the HT meta-batch curriculum achieves top performance on few-shot learning problems, particularly in extreme 1-shot cases on miniImageNet and FC100.The conclusion identifies both the method and curriculum as central to this outcome.
  • Conclusion: HT meta-batches show consistently good performance across baselines and ablations, with particular usefulness for accelerating convergence on FC100.The design is described as generalizable when task hardness can be evaluated online.

Supplementary materials

The supplementary materials provide additional methodological, implementation, dataset, variance, ablation, and interpretation details, along with open-source code.

  • Supplementary materials: Supplementary materials cover network architecture, implementation, FC100 dataset splits, variance analysis, additional ablations, and interpretations of the meta-learned model.The authors also provide open-source code on GitHub.

A. Network architectures

The paper documents 4CONV and ResNet-12 feature-extractor architectures used in the experiments. ResNet-12 consists of four residual blocks and produces a 512-dimensional feature vector from an RGB image.

  • Architectures: The supplement presents 4CONV and ResNet-12 as the two feature-extractor architectures used in the paper.ResNet-12 is shown through both an individual residual block and the complete four-block network.
  • ResNet-12: ResNet-12 maps a 3-channel RGB image to a 512-dimensional feature vector and uses leakyReLU activations with a = 0.1.The network contains four residual blocks followed by mean pooling.

B. Implementation details

The experiments specify pre-training, episodic meta-training and meta-testing procedures across miniImageNet and FC100. They also define FC100 splits to prevent fine-grained overlap between meta-training and meta-testing tasks.

  • Pre-training uses Adam with a 0.001 initial learning rate, halving every 5k iterations until below 0.0001, dropout keep probability 0.9, batch size 64, and a 10k-iteration limit.
  • Meta-training samples 5-class episodes with 1, 5, or 10 support examples per class and 15 test samples, following MAML’s setting.The base learner uses batch gradient descent with learning rate 0.01.
  • Hard tasks are sampled after every 10 meta-batches from failure classes observed across 20 tasks, with validation-selected counts varying by dataset and shot setting.MiniImageNet uses 10 and 4 hard tasks for 1-shot and 5-shot; FC100 uses 20, 10, and 4 for 1-, 5-, and 10-shot settings.
  • Meta-testing samples 600 episodes per dataset, with each episode containing equal train and test shots of 1, 5, or 10 examples per class.These settings are explicitly matched to MAML.
  • FC100 episodes are sampled from dataset splits defined by super-class labels, so meta-training and meta-testing tasks have no fine-grained information overlap.

D. Standard variance analysis

The variance analysis reports mean accuracies and standard variances over 600 meta-test tasks, noting that episode-test sample counts affect variance. Supplementary evaluations use 100 episode-test samples to compare confidence intervals with TADAM.

  • Final accuracy results are means and standard variances computed from 600 meta-test tasks.
  • Episode-test sample count affects standard variance, with the MAML-matched setting using fewer test samples and therefore higher result variance than evaluations using more samples.
  • Using 100 episode-test samples, MTL without HT has confidence intervals of 0.71% and 0.54% for miniImageNet 1-shot and 5-shot, versus TADAM’s 0.3% values.
  • On FC100 with 100 episode-test samples, MTL without HT reports 0.70%, 0.63%, and 0.58% confidence intervals for 1-, 5-, and 10-shot, compared with TADAM’s 0.4%, 0.4%, and 0.5%.

E. Additional ablation study

The additional ablations examine architecture, classifier depth, parameter updates, and learned scaling-and-shifting parameters. They report inferior results for several larger or alternative configurations and characterize the learned parameter distributions.

  • Updating both feature-extractor parameters Θ and classifier parameters θ causes a performance drop because Θ has too many parameters for the available few-shot data.
  • Changing the single fully connected classifier layer θ to two or three fully connected layers reduces performance.
  • Training 4CONV on large-scale data produces inferior results compared with the reported MTL setting.
  • In miniImageNet 1-shot, learned scaling parameters have mean 0.9962 and variance 0.0084, while shifting parameters have mean 0.0003 and variance 0.0002.The distributions become approximately Gaussian after meta-training, despite uniform initialization.
  • The interpretation is that most learned patterns remain close to large-scale pre-training, while a small tail of scale-and-shift values adapts the model and supports fast convergence.
Loading 1812.02391v3…