Source-linked AI summary
Meta-World+: An Improved, Standardized, RL Benchmark
Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, K. R. Zentner, Ryan Julian, J K Terry, Isaac Woungang, Nariman Farsad, Pablo Samuel Castro
TL;DR
Meta-World’s undocumented version changes create a fairness gap for comparing multi-task and meta-RL algorithms. This work empirically analyzes past reward functions and releases a reproducible, more ergonomic benchmark with customizable task sets. The results show that inconsistent rewards can produce incorrect performance conclusions and that tested multi-task methods perform better under V2 rewards.
Problem
Undocumented Meta-World changes, especially inconsistent reward functions, prevent fair comparison of multi-task and meta-RL algorithm results.
Method
The paper empirically compares algorithms across past reward functions and re-engineers Meta-World with new task sets, customization, reproducibility, and updated software compatibility.
Results
All tested algorithms perform better on V2 than V1 rewards, while inconsistent reward functions produce incorrect conclusions about algorithm performance.
Takeaways & Limitations
Benchmark results should use explicit versioning, consistent rewards, documented design decisions, and independently run baselines rather than copied numbers.
Takeaways & Limitations
The benchmark assumes all tasks share fixed state and action spaces, although other task components may differ.
Abstract
from arXiv · showhide
Meta-World is widely used for evaluating multi-task and meta-reinforcement learning agents, which are challenged to master diverse skills simultaneously. Since its introduction however, there have been numerous undocumented changes which inhibit a fair comparison of algorithms. This work strives to disambiguate these results from the literature, while also leveraging the past versions of Meta-World to provide insights into multi-task and meta-reinforcement learning benchmark design. Through this process we release a new open-source version of Meta-World (https://github.com/Farama-Foundation/Metaworld/) that has full reproducibility of past results, is more technically ergonomic, and gives users more control over the tasks that are included in a task set.
1 Introduction
RL benchmarks must assess both diverse-skill mastery and generalization, but existing benchmarks often specialize in one training and evaluation mode. Meta-World supports both multi-task and meta-RL research, while this work addresses version inconsistencies and improves customization and reproducibility.
- RL agents trained for only one task remain limited when the goal is to master diverse skills simultaneously.The motivation spans real-world applications including plasma control, balloon positioning, and power-grid management.
- Meta-World was proposed to support both simultaneous multi-task learning and meta-learning for broader task generalization.These approaches overlap substantially, motivating a shared benchmark.
- Undocumented benchmark version changes have obscured fair comparisons between multi-task and meta-RL algorithms.The paper empirically demonstrates these inconsistencies and re-engineers the benchmark to improve benchmarking and reproducibility.
- The release adds two task sets, supports custom task-set creation, and updates compatibility with Gymnasium and current MuJoCo bindings.It removes dependencies on unsupported OpenAI Gym and MuJoCo-Py packages.
2 Related Works
Related work addresses multi-task RL through gradient-conflict optimization or task-specific architectures, while meta-RL targets rapid adaptation to unseen tasks. These approaches motivate benchmarks that evaluate both shared-skill learning and generalization.
- Existing robotics benchmarks often use differing state and action spaces, limiting their suitability for multi-task and meta-RL evaluation.The paper contrasts these benchmarks with suites designed for manipulation, causal reasoning, and generalization.
- Multi-task RL methods either address conflicting task gradients or build architectures specialized for sharing and separating task information.PCGrad projects one gradient onto another when their cosine-based conflict criterion is met; other methods use multi-head, modular, compositional, or orthogonal-expert designs.
- Generalization to unseen tasks is a central motivation for extending RL beyond agents trained only on multiple known tasks.The related methods therefore concern both simultaneous task optimization and transfer to novel tasks.
- Meta-RL evaluates whether agents can quickly adapt to tasks within a distribution after gathering experience.MAML and RL2 are identified as two well-known meta-RL algorithms.
3 RL Problem Statements
The paper formalizes RL as policy optimization over trajectories and extends it to task distributions. Multi-task agents condition on task identity and train across observed tasks, whereas meta-RL trains on one task distribution and adapts to held-out tasks.
- RL models sequential decision making as an MDP in which a policy maps states to actions while maximizing expected discounted returns over trajectories.The MDP includes states, actions, transitions, initial-state distribution, rewards, and a discount factor.
- Multi-task RL samples tasks from a distribution and learns a task-conditioned policy using a task descriptor, represented here with one-hot encoding.The objective averages expected returns across tasks, and the agent can observe both the goal location and task descriptor.
- The multi-task and meta-RL formulations assume a task distribution whose tasks may differ in MDP components while sharing state and action spaces.This shared-space assumption defines the common interface across tasks.
- Meta-RL separates training and testing task distributions so agents learn to gather experience and adapt quickly to the current task.Training uses p(B_train), while evaluation uses held-out tasks from p(B_test).
4 Meta-World
Meta-World provides 50 Sawyer-arm manipulation tasks and evaluates multi-task learning on tasks seen during training versus meta-RL adaptation to held-out tasks. Its historical V1 and V2 reward functions differ in design and scale, making cross-version results incomparable.
- Meta-World contains 50 robotic manipulation tasks for a single Sawyer robot arm, spanning force application, grasping, and combined manipulation skills.Examples include door-close, drawer-close, coffee-push, pick-place, and assembly.
- MT10 and MT50 train and evaluate agents on the same 10 or 50 tasks, including parametric goal or object-location variations.The task sets vary both task types and within-task parameters.
- ML10 and ML45 train on 10 or 45 tasks and evaluate on 5 held-out tasks requiring skill acquisition and novel composition.The evaluation tests adaptation to unseen tasks rather than performance only on training tasks.
- Both multi-task and meta-RL settings use mean success rate across all evaluation tasks as their evaluation metric.
- V1 rewards were structured by modifying pick-place components, whereas V2 used fuzzy constraints with per-timestep rewards in (0, 10) and narrower cross-task return distributions.The differing reward philosophies and scales make results reported across the two versions not comparable.
5 Empirical Results
The experiments show that Meta-World reward versions materially affect multi-task results, while meta-RL results are less sensitive overall; task-set size also affects algorithms differently.
- 5.1 Multi-task RL Results: On MT10, algorithms generally perform worse with V1 rewards, while PCGrad and SM remain the top-performing methods under V2.The results indicate that V2 rewards are somewhat easier to optimize in this setting.
- 5.1 Multi-task RL Results: On MT50, V1 performance is much lower than V2 performance, with reward-related optimization difficulties becoming more pronounced as task count increases.SM and PCGrad continue to outperform most baselines, while MOORE is on par with SM under V2 rewards.
- 5.1 Multi-task RL Results: V2 rewards produce higher success rates than V1 across tested multi-task algorithms, likely because their more consistent scales improve Q-function modeling.The V2-trained MTMHSAC has lower Q-function loss and higher success, and all tested algorithms perform better with V2 rewards.
- 5.2 Meta-RL Results: MAML-V1, MAML-V2, and RL2-V2 show no statistically significant performance difference, whereas RL2-V1 suffers a large drop likely linked to unnormalized raw rewards in observations.The authors attribute the contrast to differences between value-based multi-task methods and policy-gradient meta-RL methods.
- 5.2 Meta-RL Results: Meta-RL performance remains modest on Meta-World, whose disjoint and compositionally complex task spaces challenge generalization across ML10 and ML45.The authors identify a gap between easier parametric variation in ML1 and the more difficult non-parametric distributions in ML10 and ML45.
6 Overall Benchmark Improvements
The improved benchmark preserves both historical reward functions while adopting Gymnasium integration, supporting reproducibility and broader tooling.
- Reward compatibility: Meta-World retains selectable V1 and V2 reward functions instead of introducing a new V3 system, preserving comparability with historical results.The implementation uses a preservation-based approach to address reward-version inconsistencies.
- Gymnasium integration: Gymnasium integration replaces the custom environment implementation and enables access to Gymnasium’s ecosystem of tools, infrastructure, and environment creation.The upgrade also aligns the library with the Gymnasium standard.
7 Conclusion
The paper shows that Meta-World versioning and reward inconsistencies distorted algorithm comparisons, then releases an updated benchmark supporting reproducibility and more rigorous future evaluation.
- Inconsistent reward functions caused incorrect algorithm-performance conclusions, with multi-task and meta-RL methods sensitive to reward scaling differences.
- Users are encouraged to run their own baselines rather than copy reported numbers from previous works.
- The authors argue that benchmark development should prioritize transparent versioning, reward consistency, and documentation of performance-affecting design decisions.
- Future benchmarks should examine greater task diversity, compositional complexity, and cross-embodiment transfer to better distinguish algorithmic advances.
NeurIPS Paper Checklist
The checklist reports that the paper’s claims match its experiments and that code, implementation details, statistical reporting, and compute information support reproducibility. It also notes that benchmark-focused work does not address the safety, interpretability, or generalizability of real-world RL deployment.
- The paper states that its abstract claims match the experimental results and that its contributions are clearly outlined in the introduction.
- The paper reports no theoretical results and marks human-subject, LLM-use, and high-risk-model safeguards sections as not applicable.
- 5. Open access to data and code: The authors report open-sourcing the updated Meta-World version and scripts used to generate experimental results.
- 6. Experimental setting/details: Hyperparameter choices are described in the main text and made available in the repository, while computational resources are discussed in the paper.
- 7. Experiment statistical significance: Experiments use 10 random seeds, interquartile means, and 95% confidence intervals in each plot.
- The benchmark is intended to reproduce past results, but its scope excludes whether deployed RL agents are generalizable, safe, or interpretable.
A.1.2 Full Reproducibility of Past Results
The updated Meta-World maintains backward compatibility while adding capabilities, enabling researchers to reproduce, verify, and compare results from earlier benchmark versions.
- Preserving core functionality and task definitions from earlier versions enables researchers to verify implementations against established benchmarks and compare results directly.
A.2 Tasks
Meta-World provides shared-space robotic manipulation tasks for multi-task and meta-RL, with configurable task sets and two reward-function versions for comparison and reproducibility.
- Each task has V1 and V2 dense reward functions, allowing comparisons across reward designs and previously published results.
- Meta-World exposes 50 robotic tasks with shared observation and action spaces for multi-task or meta-RL policies.
- Observation & Action Spaces: The 39-dimensional observation combines robot, object, temporal, and goal information, while actions specify end-effector displacement and gripper position.
- Tasks & Task-Sets: Meta-World supports multi-task evaluation on training environments and meta-learning with separate training and testing environments.
- Tasks & Task-Sets: MT25 selects 12 tasks solved and 13 unsolved in MT50 to create a 25-task training set.
- Replication: Replications approximately match published results for SM, PaCo, and MOORE, while PCGrad does not replicate the original result and instead resembles earlier work.
D Raw Results
The original Meta-World reward design adapted a pick-place reward across tasks, but its prescribed action sequence could prevent high success on some tasks.
- Raw Rewards: The original reward function was adapted from pick-place and guided agents through reaching, grasping, lifting, moving, and placing sequences.
- Raw Rewards: For some tasks, optimizing the adapted reward could never produce a policy with a high success rate.
E.2 Meta-World v2 Reward Functions
Meta-World v2 replaces reused task rewards with mostly unique functions designed for trainability, reduced prescriptiveness, comparable scale, and Markovian behavior.
- Objectives and Outcomes: The v2 objectives require high PPO success, minimally opinionated rewards, limited scale differences, and Markovian dependence.
- Objectives and Outcomes: The authors could not perfectly satisfy the first three objectives simultaneously, although the resulting functions performed fairly well on all three.
- Objectives and Outcomes: PPO reaches roughly 90% success after 20 million timesteps on at least 45/50 tasks.
- Objectives and Outcomes: The rewards encode bottleneck states without explicit temporal information and span approximately two orders of magnitude, from 0.1 to 10.
- Fuzzy-Logic Reward Function Paradigm: The fuzzy-logic paradigm combines geometric constraints through Hamacher-product conjunction or weighted-sum disjunction, then rescales rewards by 10.
- Fuzzy-Logic Reward Function Paradigm: Fuzzy constraints support task-specific training improvements, less opinionated keep-out regions, and consistent reward structure and scale across tasks.