Source-linked AI summary
Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, Yongxin Yang
TL;DR
Learning effectively from few examples remains an open machine-learning challenge. The paper proposes a shared, task- and actor-conditioned meta-critic for reinforcement and supervised learning. It reports promising results across a variety of problems and describes support for unlabelled data.
Problem
Learning effectively from few examples is a capability that machine learning has yet to replicate, motivating methods that transfer knowledge to new tasks.
Source: §1 · p. 1Method
A shared meta-critic conditioned on task and actor information supervises task-specific actors, serving as a trainable task-parameterised loss generator in supervised learning.
Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 6Results
The authors report promising results on a variety of reinforcement and supervised learning problems.
Source: §5 · p. 10Takeaways & Limitations
The framework is presented as flexible across reinforcement and supervised learning and able to exploit unlabelled data.
Source: §1 · p. 2, §5 · p. 10
Abstract
from arXiv · showhide
We propose a novel and flexible approach to meta-learning for learning-to-learn from only a few examples. Our framework is motivated by actor-critic reinforcement learning, but can be applied to both reinforcement and supervised learning. The key idea is to learn a meta-critic: an action-value function neural network that learns to criticise any actor trying to solve any specified task. For supervised learning, this corresponds to the novel idea of a trainable task-parametrised loss generator. This meta-critic approach provides a route to knowledge transfer that can flexibly deal with few-shot and semi-supervised conditions for both reinforcement and supervised learning. Promising results are shown on both reinforcement and supervised learning problems.
1 Introduction
The paper targets learning from few examples and proposes a shared meta-critic that trains task-specific actors across reinforcement and supervised learning. Its task- and actor-conditioned supervision is intended to transfer across tasks and support learning with unlabelled data.
1 Introduction
- Learning effectively from few examples remains a machine-learning capability that the authors say has not yet matched human performance.They motivate learning-to-learn as extracting problem-independent information from multiple learning problems to help with new tasks.Source: §1 · p. 1
- The paper proposes a shared meta-critic that learns to supervise actors across tasks, rather than transferring knowledge through a shared initialization or generated task-model weights.The critic is conditioned on both the actor and the task; in supervised learning it acts as a trainable, task-parameterised loss generator.Source: §1 · p. 2
- The meta-critic is designed to support few-shot learning in both reinforcement and supervised learning, including varied output types and learner architectures.The introduction also describes it as applicable to continuous and discrete outputs and as not requiring a fixed number of shots.Source: §1 · p. 2
- The authors describe meta-critic supervision as usable on unlabelled examples or unrewarded interactions after meta-training.At meta-testing, the meta-critic is fixed and the new task's actor is trained using its supervision.Source: §1 · p. 2
2 Related Work
The related-work section contrasts meta-critic learning with approaches that transfer initializations, synthesize task-specific weights, or use single-task teachers. Its central distinction is learning to supervise multiple actors across different tasks.
Few-shot learning
- Unlike shared-initialization methods such as, the meta-critic is described as more robust to diverse tasks, where an average-task prior can be poor for a multimodal task distribution.The related-work section frames as learning a shared initialization for fine-tuning individual tasks.Source: §2 · pp. 2–3
Task-parametrised Networks
- Weight-synthesis methods generate task-specific network weights, whereas this approach uses task encodings to drive supervision without requiring matching meta- and target-network architectures.The authors also identify application to RL and training on unlabelled data as advantages of learning to supervise.Source: §2 · p. 3
Distillation and Multi-Actor Methods
- Unlike distillation's single-task teacher-student setup or multi-actor methods addressing the same task, one shared meta-critic teaches actors solving distinct tasks.The section connects the approach to unlabelled-data training while distinguishing its cross-task teacher role.Source: §2 · p. 3
3 Methodology
The methodology extends actor-critic learning by conditioning a shared critic on task and actor information, then applies that supervision to discrete-action RL, supervised learning, and semi-supervised settings.
3 Methodology
- In continuous-action RL, an actor maps states to actions and maximizes critic-estimated discounted return, while the critic learns expected return from temporal-difference error.The actor and value network are alternatively updated.Source: §3 · p. 4
A Meta-Critic and Task-Actor Encoder for Multiple Tasks
- A task-actor encoder maps learning traces into an embedding that lets the shared meta-critic condition its supervision on both the task and the actor.The encoder uses state-action-reward triplets; state-action pairs reflect the actor and rewards reflect the task. The actor update treats the embedding as constant.Source: §3 · p. 4, A Meta-Critic and Task-Actor Encoder… · pp. 4–5
3.1 Discrete-action RL
- For discrete-action RL, the policy outputs a categorical action distribution and uses actor-critic policy gradients with the multi-task update framework.The policy update is modified for the discrete-action setting.Source: §3.1 · p. 5
3.2 Supervised Learning
- For supervised learning, the actor predicts a target and the critic estimates negative loss, making the critic a task-conditioned loss generator that can transfer across tasks.The task encoder uses few-shot examples; the authors report that [x, y, 0] works slightly better than the mirrored [x, ŷ, −ℓ(y, ŷ)] representation.Source: §3.2 · pp. 5–6
3.2.1 Seamless Integration with Semi-Supervised Learning
- At meta-testing, the meta-critic is fixed while the new task's actor can receive its supervision on unlabelled examples or unrewarded interactions.Few-shot examples describe the task to the encoder; actor training may also include unlabelled testing-task data.Source: §3.2.1 · p. 6
4 Experiments
Experiments test the meta-critic in supervised regression and reinforcement-learning tasks, using few-shot adaptation and out-of-sample evaluation. Results include stronger performance in mixed regression tasks and cartpole control, alongside task embeddings that reflect hidden task properties.
4.1 Implementation and Details
- Experiments use meta-training, few-shot meta-testing, and out-of-sample final testing, with MLP actors and value nets and an LSTM task-action encoder.The TAEN uses 30 LSTM cell units and outputs a 3-dimensional embedding; the actor and meta-value network each use two ReLU hidden layers.Source: §4.1 · pp. 6–7
4.2 Supervised Learning: Regression
- Meta-Critic performed comparably to MAML on sine-only regression tasks and clearly better on mixtures of sine and linear functions.Meta-testing used K ∈ {4, 6, 8} training pairs per task; Table 1 reports final-testing MSE.Source: Table 1 · p. 8, Figure 2 · p. 8, §4.2 · p. 8
4.3 Reinforcement Learning: Dependant Multi-arm Bandit
- Meta-Critic achieved higher average reward than competitors at every tested trial count in dependent 2-, 4-, and 6-arm bandits.The bandits were meta-trained on 1,000 tasks and tested on 100 new bandits with only a few trials per task.Source: Table 2 · p. 9, §4.3 · p. 9
4.4 Reinforcement Learning: Cartpole Control
- 26% of cartpole meta-tests succeeded with Meta-Critic, whose reward improved faster per training episode than alternatives despite a slightly lower starting point.A successful meta-test required success on all 10 final episodes; a successful episode meant balancing the pole for more than 195 iterations.Source: §4.4 · p. 9, Figure 3 · p. 10
- The learned cartpole task embedding follows pole length, although the agent receives no direct pole-length input and learns the task manifold from learning traces.Figure 3’s middle panel plots task embeddings, with points representing tasks and colors indicating pole length.Source: §4.4 · p. 10, Figure 3 · p. 10
5 Conclusion
The paper presents a flexible few-shot meta-learning method for reinforcement and supervised learning that can exploit unlabelled data, with promising results across varied problems. It identifies more challenging tasks and continual learning as future directions.
5 Conclusion
- The method applies to few-shot reinforcement and supervised learning and can seamlessly exploit unlabelled data.The authors report promising results on a variety of problems.Source: §5 · p. 10
- The authors propose evaluating the method on more challenging supervised-learning and reinforcement-learning control problems with greater difficulty and input/output dimensionality.These are identified as future evaluation directions.Source: §5 · p. 10
- Extending the method to continual learning is identified as future work.Source: §5 · p. 10