PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Meta-Learning: Learning to Learn in the Concept Space

Fengwei Zhou, Bin Wu, Zhenguo Li

arXiv:1802.03596v1cs.LG

TL;DR

Few-shot learning remains challenging for meta-learning because its instance-space representation may not capture high-level concepts from only a few examples. Deep meta-learning jointly trains a concept generator, meta-learner, and discriminator so that few-shot learning occurs in concept space. The authors report that it substantially improves vanilla meta-learning across few-shot image-recognition problems.

  • Problem

    Few-shot learning remains challenging because a few instances may not provide a good representation of high-level concepts for meta-learning.

    Source: §1 · p. 1

  • Method

    Deep meta-learning jointly trains a concept generator and meta-learner on related tasks while a concept discriminator enhances the generator using external image-recognition tasks.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    The authors report that deep meta-learning substantially improves vanilla meta-learning on few-shot image recognition.

    Source: §1 · p. 2, §4.4 · p. 7

  • Takeaways & Limitations

    The framework learns to perform few-shot learning in concept space, where the concept generator provides representations that ease meta-learning.

    Source: §5 · p. 8

    Experiments train the concept generator and discriminator on one 200-class dataset; using multiple large-scale datasets would require more careful design and computing resources.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

Few-shot learning remains challenging for meta-learning that learns a learning algorithm (meta-learner) from many related tasks. In this work, we argue that this is due to the lack of a good representation for meta-learning, and propose deep meta-learning to integrate the representation power of deep learning into meta-learning. The framework is composed of three modules, a concept generator, a meta-learner, and a concept discriminator, which are learned jointly. The concept generator, e.g. a deep residual net, extracts a representation for each instance that captures its high-level concept, on which the meta-learner performs few-shot learning, and the concept discriminator recognizes the concepts. By learning to learn in the concept space rather than in the complicated instance space, deep meta-learning can substantially improve vanilla meta-learning, which is demonstrated on various few-shot image recognition problems. For example, on 5-way-1-shot image recognition on CIFAR-100 and CUB-200, it improves Matching Nets from 50.53% and 56.53% to 58.18% and 63.47%, improves MAML from 49.28% and 50.45% to 56.65% and 64.63%, and improves Meta-SGD from 53.83% and 53.34% to 61.62% and 66.95%, respectively.

1. Introduction

Few-shot learning challenges meta-learning because a few instances may not represent high-level concepts, and current meta-learning still operates in a complicated instance space. Deep meta-learning proposes learning in concept space by jointly training a concept generator, meta-learner, and concept discriminator.

1. Introduction

  • Few-shot learning remains difficult because the complicated instance space makes a few examples insufficient to describe high-level concepts, and meta-learning does not solve this problem.The paper identifies the lack of a good data representation as a reason meta-learning's potential remains unrealized.Source: §1 · p. 1
  • Deep meta-learning combines a concept generator, meta-learner, and concept discriminator, jointly learning representations and few-shot learning in concept space.The generator is trained with related meta-learning tasks and external large-scale image-recognition tasks.Source: §1 · p. 2
  • The framework integrates deep learning's representation power into meta-learning and improves vanilla meta-learning on few-shot image recognition.The authors believe the framework may also apply to reinforcement learning and regression.Source: §1 · p. 2
  • The authors report extensive experiments instantiating the framework with Matching Nets, MAML, and Meta-SGD, and say it uses data more efficiently than existing methods.The framework is described as able to evolve continuously as fresh labeled samples arrive.Source: §1 · p. 2

2. Related Work

Related work spans meta-learning, memory-augmented approaches, transfer and multi-task learning, and other few-shot recognition methods. The paper positions its method as meta-learning in concept space rather than instance space.

2. Related Work

  • Meta-learning methods learn task-agnostic learning algorithms from tasks, while this paper places few-shot learning in concept space rather than instance space.Memory-augmented approaches add retrieval costs or task-specific memory constraints; the concept generator is described as a memory that avoids exhaustive retrieval.Source: §2 · p. 2
  • Transfer learning moves knowledge from a labeled source domain to a low-label target domain, whereas meta-learning adapts automatically to new tasks.Fine-tuning requires manually tailored frozen layers and learning rates for each new task.Source: §2 · p. 3
  • Multi-task learning jointly learns related tasks through a shared representation, while meta-learning can apply to new tasks after learning from related tasks.The passage notes that multi-task methods may struggle with the many parameters induced by very large task sets.Source: §2 · p. 3
  • Prior few-shot recognition approaches include Bayesian and generative models, feed-forward learners, example generation, memory networks, and metric learning.The paper characterizes its own approach as meta-learning that learns to learn in concept space.Source: §2 · p. 3

3. Deep Meta-Learning

Deep meta-learning integrates deep-learning representations into meta-learning so that few-shot learning takes place in a high-level concept space. Its jointly learned concept generator, meta-learner, and concept discriminator support that framework.

3. Deep Meta-Learning

  • DEML moves few-shot learning from the complicated instance space into a concept space, integrating deep-learning representations into meta-learning.The paper says this makes the few-shot learning problem easier.Source: §3 · p. 3, §3.5 · p. 5

3.1. Framework

  • The framework jointly learns a concept generator and meta-learner for task representations, while a concept discriminator trains the generator on external labeled data.The generator extracts task-agnostic representations; the meta-learner uses them for task-specific few-shot learning.Source: §3.1 · p. 3

3.2. Modules

  • The generator can be a deep neural network, the discriminator predicts labels for generated concepts, and the meta-learner can use existing methods such as Matching Nets, MAML, or Meta-SGD.The three modules are abstract and may be implemented with suitable models.Source: §3.2 · p. 4

3.3. Criterion

  • The joint criterion combines concept-discrimination loss with generalization loss on meta-learning tasks, where the meta-learner adapts a learner for each task.The concept pipeline trains representations to capture concepts while the meta-learning pipeline operates in the high-level concept space.Source: §3.3 · p. 4

3.4. Algorithm

  • The algorithm instantiates the framework with Meta-SGD, balances the two learning objectives with λ, and uses Adam to optimize the objective.The paper describes Adam as a variant of stochastic gradient descent.Source: §3.4 · p. 5

3.5. Comparison with Related Work

  • Unlike vanilla meta-learning in instance space, DEML uses concept-level representations; the paper also proposes principled generator training and systematic few-shot guidance rather than relying only on pretraining and fine-tuning.A generator trained on different tasks may help little, and fine-tuning may forget previously learned concepts.Source: §3.5 · p. 5
  • With more labeled concept-discrimination data, the generator could retain learned concepts and provide more effective representations, gradually improving performance on new tasks over time.The paper describes this as a possible extension to lifelong learning.Source: §3.5 · p. 5

4. Experiments

The experiments evaluate DEML on few-shot image recognition datasets, comparing it with vanilla and deep meta-learners, transfer-learning variants, and different values of λ. DEML outperforms vanilla versions, while comparisons and the λ study examine the roles of joint learning and balancing external and meta-level knowledge.

4.1.1. DATASETS FOR CONCEPT DISCRIMINATION

  • Experiments use ImageNet-200 for concept discrimination and MiniImagenet, Caltech-256, CIFAR-100, and CUB-200 for meta-learning.ImageNet-200 uses 200 classes sampled from 900, excluding MiniImagenet’s classes.Source: §4.1 · p. 5, §4.1.1 · p. 5

4.1.2. DATASETS FOR META-LEARNING

  • The meta-learning datasets use separate training, validation, and test classes; CUB-200 tests on 40 bird-species classes, while CIFAR-100 images are rescaled to 32 × 32.MiniImagenet uses 64/16/20 classes, Caltech-256 uses 150/56/50, and CUB-200 uses 140/20/40 for training/validation/testing.Source: §4.1.2 · p. 5

4.2. Baselines

  • The baselines are Matching Nets, MAML, and Meta-SGD, evaluated both in their original designs and with DEML’s architecture and inputs, excluding the concept discriminator.The latter implementations are called Deep Matching Nets, Deep MAML, and Deep Meta-SGD.Source: §4.2 · p. 6

4.3. Implementation

  • DEML uses a ResNet-50 concept generator and a one-layer concept discriminator; its few-shot learner architecture depends on the chosen meta-learner.Meta-training combines ImageNet-200 image-recognition batches with 5-way tasks from a meta-learning dataset, using 1 or 5 training examples per class and 5 test examples per class.Source: §4.3 · p. 6, Figure 2 · p. 6

4.4. Results and Discussion

  • DEML versions outperform vanilla Matching Nets, MAML, and Meta-SGD by a wide margin across the reported comparisons.The comparison is summarized in Table 1.Source: Table 1 · p. 7, §4.4 · p. 7
  • Decaf+kNN performs best on MiniImagenet and Caltech-256, but its performance drops on CIFAR-100 and CUB-200, which differ more from ImageNet-200.The authors describe joint DEML as combining task-agnostic meta-level concepts with external concepts.Source: §4.4 · pp. 7–8, Table 3 · p. 8
  • DEML+Meta-SGD performs better than Decaf+Meta-SGD and Decaf+Fine-Tune+Meta-SGD on CIFAR-100 and CUB-200, while deeper networks and larger training data alone do not raise accuracy.The authors attribute DEML’s performance to joint learning that balances learning from few-shot tasks and the external dataset.Source: §4.4 · pp. 7–8, Table 2 · p. 7, Figure 3 · p. 8
  • As λ increases, concept-recognition accuracy rises, while 5-way-5-shot accuracy on CIFAR-100 rises initially and then falls.The authors conclude that balancing external knowledge and internal meta-level knowledge is useful.Source: Figure 4 · p. 8, §4.4 · p. 8

5. Conclusion and Future Work

Deep meta-learning jointly trains a concept generator on external concept discrimination and few-shot tasks, enabling learning in concept space. Experiments show improved vanilla meta-learning, while the conclusion identifies broader training and lifelong learning as future work.

5. Conclusion and Future Work

  • Joint training lets the concept generator capture external and meta-level concepts, while it and the meta-learner evolve synergistically to improve generalization.The framework is trained on large-scale concept discrimination and few-shot learning tasks.Source: §5 · p. 8
  • Experiments train the concept generator with the concept discriminator on one dataset containing 200 classes.Source: §5 · p. 8
  • Future work would train the generator on multiple large-scale datasets, requiring more careful design and computing resources.The authors also identify lifelong learning challenges: balancing external and few-shot concept learning and avoiding forgetting previously learned concepts.Source: §5 · p. 9

Related papers

Loading 1802.03596v1…