Source-linked AI summary
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, Sergey Levine
TL;DR
Meta-RL research has often used narrow task distributions that provide limited evidence about generalization to entirely new behaviors. This paper introduces Meta-World, an open-source benchmark of 50 diverse robotic manipulation tasks, and evaluates existing methods. The experiments show that current methods struggle with diverse multi-task learning and generalization to new tasks.
Problem
Narrow task distributions limit evaluation of whether meta-RL methods generalize to entirely new tasks.
Method
The paper introduces an open-source benchmark of 50 diverse simulated manipulation tasks with shared robot, environment, and control structure.
Results
Current methods generally struggle to learn diverse task sets and generalize effectively to entirely new tasks.
Takeaways & Limitations
Meta-World provides a challenging evaluation setting for developing and testing meaningful generalization in multi-task and meta-RL.
Takeaways & Limitations
The benchmark remains a simulated setting and future work should consider image observations and sparse rewards for more realistic robotics tasks.
Abstract
from arXiv · showhide
Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 7 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the full potential of these methods.
1 Introduction
Robots need methods that reuse shared structure across many skills, but existing reinforcement-learning research has not demonstrated generalization to diverse tasks. Meta-World addresses this gap with 50 diverse manipulation tasks and evaluates prior multi-task and meta-RL methods.
- Current reinforcement-learning methods require substantially more experience than humans to acquire narrowly defined skills.
- Existing methods have not yet demonstrated generalization to diverse tasks in reinforcement-learning settings.
- Established benchmarks and evaluation protocols have not adequately reflected realistic multi-task and meta-learning use cases.
- Meta-World contributes 50 diverse simulated manipulation tasks sharing a tabletop environment, Sawyer arm, and common environment and control structure.
- The benchmark is designed to evaluate whether methods can efficiently learn many tasks and quickly generalize to new tasks.
- Current methods generally struggle to learn diverse task sets and generalize to entirely new tasks.
2 Related Work
Prior reinforcement-learning benchmarks often emphasize single tasks or highly dissimilar tasks, limiting conclusions about positive transfer. Meta-World instead provides related but diverse manipulation tasks with shared robot, action-space, and workspace structure.
- Many prior reinforcement-learning benchmarks focus on single-task learning settings.
- Atari games differ substantially in visual appearance, controls, and objectives, and prior methods have observed substantial negative transfer between them.
- Meta-World studies related yet diverse tasks that share the same robot, action space, and workspace.
- Prior meta-RL evaluations include maze navigation, parametric continuous control, bandits, arcade-game levels, and locomotion with varying dynamics.
- Meta-World is intended as a robotic-manipulation testbed covering both parametric and non-parametric task variation.
- The benchmark continues a broader robotics tradition of simulation benchmarks spanning navigation, driving, grasping, and manipulation.
3 The Multi-Task and Meta-RL Problem Statements
Multi-task RL learns one task-conditioned policy across training tasks, whereas meta-RL learns from training tasks to adapt quickly to unseen tasks. Meta-RL generalization depends on a sufficiently broad task distribution with shared structure.
- Meta-World frames each task as a finite-horizon Markov decision process with states, actions, transitions, rewards, horizon, and discount factor.
- Multi-task RL learns a single task-conditioned policy that maximizes average expected return across a task distribution.
- Multi-task RL typically evaluates average performance on training tasks and has no separate test set of tasks.
- Meta-RL trains on multiple tasks and adapts its policy to unseen test tasks using a small number of samples.
- Meta-RL assumes training and test tasks come from the same distribution and that this distribution contains shared structure for efficient adaptation.
- Broad meta-training distributions are necessary when the goal is generalization to entirely new tasks.
4 Meta-World
Meta-World is a benchmark of 50 qualitatively distinct manipulation tasks designed to test multi-task learning and adaptation to new tasks across broad distributions with shared structure. Its protocols vary from within-task goal adaptation to multi-task learning and few-shot generalization, using shared robot interfaces, structured rewards, and interpretable success metrics.
- The Space of Manipulation Tasks: Meta-World defines 50 qualitatively distinct manipulation tasks because narrow task distributions cannot support generalization to entirely new control tasks.Tasks are defined by reward function, initial object position, and target position.
- The Space of Manipulation Tasks: Parametric variation in object and goal positions complements non-parametric task differences by discouraging memorization and increasing overlap across manipulation tasks.The benchmark varies positions within each task while retaining qualitatively distinct behaviors across tasks.
- Task and Interface Design: The benchmark reuses one Sawyer arm and combinations of reaching, pushing, and grasping across objects with different shapes, joints, and connectivity.This shared structure supports a broad range of manipulation tasks while keeping the interaction platform common.
- Task and Interface Design: All tasks use a shared 39-dimensional observation space and a two-part action space consisting of end-effector motion and normalized gripper torque.Actions range from −1 to 1, and absent objects or goals are represented by zeroed quantities.
- Task and Interface Design: Structured, well-shaped rewards make tasks individually solvable while preserving shared reward structure across tasks.The design addresses both single-task learnability and the risk that differing reward scales obscure common structure.
- Evaluation Protocol: The evaluation spans ML1, MT1, MT10, MT50, ML10, and ML45, progressing from goal variation within one task to multi-task training and adaptation to held-out tasks.MT10 and MT50 cover 10 and 50 training environments, while ML10 and ML45 hold out five tasks and omit task IDs.
- Evaluation Protocol: Success is measured with task-specific interpretable criteria, typically whether the relevant object reaches its goal within a small distance threshold.The benchmark uses distance-based success rather than treating raw reward values as directly indicative of policy success.
5 Experimental Results and Analysis
The experiments evaluate multi-task and meta-reinforcement learning methods across increasingly broad task suites. Performance is reasonable on simpler settings but declines or remains limited as task diversity and generalization demands increase.
- Around 68% success on MT10 is achieved by multi-task SAC, while multi-task PPO and TRPO reach only around 30%.MT-SAC learns the ten-task suite substantially better than the compared on-policy methods.
- Around 35-38% success on MT50 is achieved by MT-SAC and MT-PPO, indicating substantial room for improvement on the broader suite.Scaling from ten to fifty distinct tasks reduces the reported success rates for these methods.
- 35% and 31% success on ML10 test tasks is reached by MAML and RL2, respectively, while PEARL achieves 13%.These results measure generalization to meta-test tasks after learning the meta-training tasks.
- Around 39.9% and 33.3% of ML45 meta-test tasks are solved by MAML and RL2, respectively.The ML45 results show meaningful but limited generalization on the more challenging benchmark.
- RL2 reaches 86.9% on training tasks, but its generalization is only 35.8%, compared with 31.6% for MAML.The training-task advantage is much larger than the reported generalization difference.
- MT-SAC performs best on MT10, but its performance does not scale to MT50, where MT-PPO performs better.The benchmark comparison spans ten-task and fifty-task multi-task settings.
6 Conclusion and Directions for Future Work
Meta-World provides a broad benchmark for testing generalization to entirely new robotic manipulation tasks. Experiments show that current meta-RL methods struggle with diverse meta-training tasks, motivating broader algorithmic development and benchmark extensions.
- Conclusion: The benchmark contains many simulated robotic manipulation tasks and emphasizes generalization to new objects and interaction scenarios.This differs from evaluations focused only on parametric changes in goals.
- Conclusion: Current meta-RL methods cannot yet generalize effectively to entirely new tasks or learn meta-training tasks effectively across multiple distinct tasks.The conclusion identifies both held-out-task generalization and multi-task training as unresolved challenges.
- Directions for Future Work: Highly diverse meta-training tasks remain difficult for current meta-RL algorithms, paralleling challenges observed in multi-task reinforcement learning.The authors identify developing methods for broader task distributions as a promising direction.
- Directions for Future Work: Future benchmark extensions include image observations, sparse rewards, and compositional long-horizon tasks for more realistic and broader robotics evaluation.Image-based meta-learning must prevent direct task inference from observations to avoid memorization.
- Conclusion: The task suite is presented as a step toward evaluating multi-task and meta-learning algorithms on diverse robotic manipulation problems.The authors position it as a gauge of progress toward broad task generalization.
A Task Descriptions
This appendix documents the Meta-World task suite and provides a list and description of its tasks.
- Task Descriptions: Table 2 lists all 50 Meta-World tasks and provides a description of each task.The table serves as the appendix’s task-description reference.
B Benchmark Verification with Single-Task Learning
The benchmark tasks are individually solvable with sufficient data, as verified using SAC and PPO. SAC solves all tasks to some degree, while PPO solves most.
- Benchmark Verification: The verification evaluates independent single-task policies using SAC and PPO.This experiment validates task solvability rather than defining the benchmark’s official evaluation protocol.
- Benchmark Verification: SAC can learn all 50 tasks to some degree, while PPO can solve a large majority.The evaluation uses three random seeds.
C Learning curves
The learning-curve evaluations compare efficiency and performance across meta-learning and multi-task settings. Off-policy SAC-based multi-task learning is more data-efficient, while RL2 performs strongly on several meta-RL evaluations and PEARL underperforms on ML-45.
- Learning Curves: Meta-learning adaptation requires only a few trajectories, but meta-training can remain highly data-inefficient.The learning curves evaluate this distinction across ML1, MT10, MT50, ML10, and ML45.
- Learning Curves: Off-policy algorithms such as SAC learn with substantially less data than on-policy algorithms in multi-task settings.Sample efficiency is treated as a critical metric because it motivates multi-task learning.
- MT-10: MT-SAC vastly outperforms on-policy counterparts in performance and sample efficiency on MT-10.The comparison is shown in Figure 15.
- MT-50: On MT-50, MT-SAC has higher sample efficiency, but its performance tapers off and MT-PPO surpasses it with more training.The comparison is shown in Figure 16.
- ML-10: On ML-10, RL2 significantly outperforms other methods in sample efficiency and test-task performance, although MAML performs better early in training.RL2 overtakes MAML with more training.
- ML-45: On ML-45, PEARL underperforms MAML and RL2, while RL2 and MAML show similar test-task performance.RL2 significantly outperforms other methods in sample efficiency and train-task performance.
D.1 Single Task SAC
This section presents the hyperparameters used for Garage experiments with Single Task SAC.
- Table 3 lists the hyperparameters used for Garage experiments with Single Task SAC.
D.2 Single Task PPO
This section reports hyperparameter tables for single-task, multi-task, task-embedding, and meta-reinforcement-learning experiments, with experiment seeds chosen individually at random.
- Table 4 lists hyperparameters for Garage experiments with Single Task PPO.
- Seed values were individually chosen at random for each experiment.
- Tables 5–8 list hyperparameters for Multi-Task PPO, Multi-Task TRPO, Multi-Task SAC, and Task Embeddings PPO.
- Tables 9–11 list hyperparameters for MAML, RL2, and PEARL experiments in Garage.
E Reward Functions and Single-Task Results
The reward functions combine tolerance terms, grasp-related guidance, and caging components to provide smooth, dense task rewards. Task-specific designs also add geometric guidance, object-orientation penalties, and staged movement incentives where needed.
- Caging rewards: The caging reward describes gripper behavior along the actuated-finger axis and the other two axes, then combines these values with a Hamacher product.
- Caging rewards: The caging reward has medium-density and high-density modes, with c1, c2, and c3 passed as arguments to C.
- Reward design: A post-grasp guidance term activates when the Sawyer successfully grasps the object.The variable A is non-zero if and only if the object is successfully grasped.
- Reward design: Reward functions commonly sum tolerance terms, while harder tasks add Hamacher Products combining tolerances, grip effort, and caging rewards.These combinations are intended to produce smooth, dense rewards.
- Geometric guidance: Funnel-shaped surfaces guide gripper motion around table holes or bins to prevent collisions during object manipulation.The surface height is represented by alt.
- Implementation caveat: One implementation uses constant rather than dynamic caging-reward margin parameters, while reporting mostly the same behavior.
- Task-specific rewards: Assembly, disassembly, and hammer rewards penalize tilted object orientations in addition to their other reward components.