Source-linked AI summary
Meta-Q-Learning
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, Alexander J. Smola
TL;DR
Meta-RL seeks to adapt policies to new environments from few data, but existing benchmarks may not require sophisticated meta-training. MQL uses context-conditioned Q-learning, average-reward multi-task training, and propensity-weighted replay-buffer reuse, and obtains comparable or better validation returns across most evaluated environments, with faster convergence in many cases.
Problem
Meta-RL aims to adapt to new environments with few data, but real robots limit data collection and may differ from training simulations.
Method
MQL combines context-conditioned off-policy Q-learning, average-reward multi-task training, and propensity-weighted reuse of meta-training replay data for adaptation.
Results
MQL obtains comparable or better validation returns in all evaluated environments except Walker-2D-Params and Ant-Goal-2D, often converging faster than existing algorithms.
Takeaways & Limitations
The results suggest that context-conditioned Q-learning and simple multi-task training can be competitive on current meta-RL benchmarks, while replay-buffer data can support adaptation.
Takeaways & Limitations
Current meta-RL benchmarks may be insufficient to evaluate meta-RL because training and validation tasks are quite similar.
Abstract
from arXiv · showhide
This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-Reinforcement Learning (meta-RL). MQL builds upon three simple ideas. First, we show that Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory. Second, a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies. Third, past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates. MQL draws upon ideas in propensity estimation to do so and thereby amplifies the amount of available data for adaptation. Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with the state of the art in meta-RL.
1 INTRODUCTION
Meta-RL targets adaptation to new environments with few data, but a context-conditioned off-policy baseline is already competitive on standard benchmarks. MQL extends this finding with simple multi-task training and propensity-corrected replay-buffer reuse for adaptation.
- Meta-RL trains across many environments to adapt to a new environment with few data, addressing limited real-robot data and simulation-to-reality differences.
- TD3-context is competitive with prototypical meta-RL algorithms even without meta-training or adaptation on a new task.
- MQL maximizes average rewards across meta-training tasks using off-policy updates as a simple meta-training procedure.
- For new-task adaptation, MQL samples similar transitions from the meta-training replay buffer and uses propensity estimation to address the resulting bias.
- MQL combines new-task updates, propensity-weighted updates on old data, and an automatically adapting proximal term that prevents policy degradation.
- MQL policies obtain higher average returns on new tasks while being meta-trained for fewer time-steps than state-of-the-art algorithms.
2 BACKGROUND
The background formalizes task-specific MDPs, off-policy value learning, and meta-RL as learning an initialization or inductive bias for rapid adaptation. It also introduces importance ratios and effective sample size for comparing training and target data distributions.
- The paper models tasks as MDPs sharing state and action spaces while potentially differing in dynamics and reward functions.
- Deterministic Policy Gradient learns an approximate value function by minimizing Bellman error and optimizes a policy against that approximation.
- Off-policy algorithms can optimize a policy using expectations computed from data generated by another policy.
- Meta-RL trains across many tasks to learn an inductive bias that accelerates learning on a new task.
- Gradient-based meta-RL optimizes performance after task-specific updates, using the resulting parameters as an initialization for adaptation.
- The propensity score is an importance ratio measuring the odds that a sample comes from one distribution rather than another, estimated from finite samples with logistic regression.
- Normalized Effective Sample Size estimates the relative target-sample count needed to match an importance-sampling estimator’s performance and approaches one when distributions are close.
3 MQL
MQL uses multi-task off-policy meta-training with trajectory context, then adapts to new tasks by reweighting replay-buffer transitions with propensity estimates while controlling policy drift.
- 3.1 META-TRAINING: MQL meta-trains an off-policy policy by maximizing average returns across the meta-training tasks.TD3 is used as the underlying algorithm, with double-Q-learning targets to reduce over-estimation bias.
- 3.1 META-TRAINING: MQL’s multi-task objective differs from gradient-based MAML because MAML’s adaptation speed can come at the cost of under-fitting individual tasks.The under-fitting effect increases with the number of gradient steps and is controlled by the adaptation parameters.
- 3.1.1 DESIGNING CONTEXT: The policy and value functions use a deterministic GRU context derived from the trajectory of states, actions, and rewards.The context is not permutation invariant, allowing temporal direction to convey task information.
- 3.2 ADAPTATION TO A NEW TASK: For a new task, MQL first performs vanilla off-policy adaptation while penalizing distance from the meta-trained parameters.The quadratic penalty reduces variance when adapting from few new-task samples.
- 3.2 ADAPTATION TO A NEW TASK: MQL then reweights meta-training replay-buffer transitions using a logistic classifier that estimates their propensity of belonging to the new task.The context variable is the classifier feature, enabling reuse of past data despite task mismatch and extrapolation error.
- 3.2 ADAPTATION TO A NEW TASK: The adaptation objective combines new-task updates, propensity-weighted old-data updates, and an automatically adapting proximal term.The effective sample size controls the penalty coefficient: similarity to meta-training tasks relaxes the penalty, while dissimilarity strengthens it.
- 3.2 ADAPTATION TO A NEW TASK: MQL requires access to the meta-training replay buffer during adaptation, although the reported experiments used buffers of at most 3 GB.The paper notes that clustering techniques could select important transitions when storage is limited.
4 EXPERIMENTS
Experiments evaluate MQL on standard continuous-control meta-RL benchmarks, using fixed disjoint training and validation tasks and comparisons with established algorithms. MQL generally achieves comparable or better returns, often with faster convergence, while ablations examine adaptation, replay-buffer reuse, and ESS-based regularization.
- Experimental setup: The evaluation uses MuJoCo continuous-control meta-RL benchmarks with disjoint meta-training and validation tasks, following Rakelly et al.'s evaluation protocol.The protocol addresses inconsistent task splits, rewards, episode lengths, and adaptation-data budgets across prior studies.
- Results: Vanilla TD3 with a trajectory context variable, trained using the multi-task objective without adaptation, is competitive with state-of-the-art meta-RL algorithms.The result suggests that context enables policies to generalize to validation tasks without parameter updates during adaptation.
- Results: MQL obtains comparable or better validation returns than existing algorithms in all environments except Walker-2D-Params and Ant-Goal-2D.Figure 3 compares MQL with MAML, RL2, PROMP, and PEARL on average undiscounted return.
- Results: MQL converges faster than existing algorithms in most environments, while MAML and ProMP require about 100M time-steps to reach worse returns.The faster convergence is particularly evident on Humanoid-Direc-2D.
- Ablation experiments: Ablations show that adaptation improves performance, old replay-buffer data is useful, and λ = 1 − d_ESS can adjust regularization across tasks without per-task hyper-parameter search.The experiments compare MQL with and without adaptation, replay-buffer updates, and ESS-based modulation.
- Methodological context: MQL uses off-policy updates and propensity scores to exploit past replay-buffer data for adaptation despite the potential bias from transitions collected on other tasks.The method can use substantially more old-data updates than new-task data during adaptation.
5 DISCUSSION
The discussion argues that MQL's simple components challenge assumptions about meta-RL and highlight weaknesses in current benchmarks. It also identifies replay-buffer reuse as a broader data-efficiency idea.
- Discussion: MQL combines contextual Q-learning, average-reward multi-task training, and replay-buffer recycling through propensity estimation.These are presented as three simple ideas underlying the algorithm.
- Broader relevance: Replay-buffer data is described as essentially free and potentially reusable beyond reinforcement learning, including few-shot and zero-shot image classification.The proposed reuse extends the paper's data-efficiency perspective beyond its evaluated RL setting.
- Benchmark implications: The competitiveness of vanilla contextual Q-learning suggests that current benchmarks may have similar training and validation tasks and may insufficiently evaluate meta-RL.The discussion calls for better benchmark problems that drive new algorithmic innovation.
A PSEUDO-CODE
The pseudo-code separates MQL into off-policy meta-training and test-task adaptation. Meta-training builds a contextual policy and replay buffer, while adaptation updates the policy with new data and estimates transition propensity against stored data.
- MQL - Meta-training: Meta-training initializes an off-policy learner and replay buffer, samples tasks from Dmeta, gathers contextual trajectories, and stores them for updates.The procedure uses an off-policy method such as TD3 and feeds transitions through a context GRU.
- MQL - Adaptation: During adaptation, MQL initializes from the meta-trained policy and first updates it using data gathered from the test task.The test task is denoted D, and the gathered transitions are stored in a temporary buffer.
- MQL - Adaptation: MQL then fits a logistic classifier using test-task and meta-training transitions to estimate propensity and effective sample size.These estimates support the subsequent adaptation objective involving stored replay-buffer data.
B OUT-OF-DISTRIBUTION TASKS
MQL performs strongly on out-of-distribution tasks, including settings where training and test target velocities are disjoint or far apart. Its propensity-weighted adaptation automatically limits reliance on replay-buffer samples unlike the new task.
- MQL significantly outperforms PEARL when training and test target velocities come from disjoint sets.
- β(z) ≈ 0.2 and λ ≈0.8 on the medium-hard task, indicating adaptation uses few replay-buffer samples when test transitions differ from training data.
- MQL exceeds TD3-context by a large margin on the hard task with either 200 or 1200 adaptation time-steps.The 1200-step condition uses six episodes; high standard deviation reflects the environment’s difficulty.
- On the hard task, λ ≈1 and β(z) ≈0 for much of training, so MQL discards samples unrelated to the new test task.
- The ablation studies report that MQL adaptation improves performance and that additional replay-buffer data remains useful even when it comes from different tasks.
C MORE ABLATION STUDIES
Additional ablations compare Half-Cheetah-Vel and Walker-2D-Params to analyze MQL components and adaptation behavior. Adaptation is more useful when test and training tasks are less similar.
- Adaptation is more useful for Half-Cheetah-Vel than Walker-2D-Params because Walker-2D-Params has more similar test and training tasks.
D HYPER-PARAMETERS AND MORE DETAILS OF THE EMPIRICAL RESULTS
The benchmark experiments use specified hyper-parameters for MQL and TD3 across continuous-control meta-RL tasks. The configurations use identical network depth and Adam batch size, with hyper-parameters selected by grid search.
- MQL and TD3 use two fully connected layers, while Adam uses a batch size of 256 across all environments.
- The hyper-parameters for the continuous-control benchmark tasks were tuned by grid search.