Source-linked AI summary
Benchmarking Model-Based Reinforcement Learning
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba
TL;DR
MBRL promised lower sample complexity, but inconsistent environments and implementations left relative algorithm performance unclear. The paper benchmarks diverse MBRL and MFRL methods across 18 environments, including noisy settings, and identifies three recurring challenges. Results show no consistently best MBRL algorithm, while dynamics bottlenecks, planning horizons, and early termination remain key research problems.
Problem
Inconsistent environments, implementations, and reproducibility make it unclear how existing MBRL algorithms compare with one another and standard baselines.
Method
The paper benchmarks 11 MBRL and 4 MFRL algorithms across 18 OpenAI Gym environments, including stochastic observations and actions.
Results
The benchmark finds no clear consistently best MBRL algorithm and characterizes dynamics bottlenecks, planning horizon dilemmas, and early-termination dilemmas.
Takeaways & Limitations
The results suggest opportunities to combine strengths across MBRL approaches and pursue the three identified challenges.
Takeaways & Limitations
Learned dynamics can become unstable and inaccurate over time, while coupling policy learning with dynamics learning makes performance local minima more likely.
Abstract
from arXiv · showhide
Model-based reinforcement learning (MBRL) is widely seen as having the potential to be significantly more sample efficient than model-free RL. However, research in model-based RL has not been very standardized. It is fairly common for authors to experiment with self-designed environments, and there are several separate lines of research, which are sometimes closed-sourced or not reproducible. Accordingly, it is an open question how these various existing MBRL algorithms perform relative to each other. To facilitate research in MBRL, in this paper we gather a wide collection of MBRL algorithms and propose over 18 benchmarking environments specially designed for MBRL. We benchmark these algorithms with unified problem settings, including noisy environments. Beyond cataloguing performance, we explore and unify the underlying algorithmic differences across MBRL algorithms. We characterize three key research challenges for future MBRL research: the dynamics bottleneck, the planning horizon dilemma, and the early-termination dilemma. Finally, to maximally facilitate future research on MBRL, we open-source our benchmark in http://www.cs.toronto.edu/~tingwuwang/mbrl.html.
1 Introduction
MBRL research lacked standardized environments and implementations, making algorithm comparisons and scientific progress difficult. This paper addresses that gap with a broad benchmark and identifies three challenges affecting MBRL performance.
- MBRL can reduce sample complexity by learning an environment model, but modeling errors can produce policies that exploit model deficiencies.Probabilistic models and ensembles have been used to characterize uncertainty and alleviate model bias.
- Comparisons among MBRL methods remain unclear because implementations, observation preprocessing, rewards, and episode horizons vary across studies.Reproducibility and limited open-source code further complicate comparisons with prior work.
- Benchmarking can reveal algorithm strengths and weaknesses while suggesting future research directions, as demonstrated in other areas and model-free RL.Prior platforms such as OpenAI Gym and DeepMind Control Suite helped standardize evaluation.
- The benchmark evaluates 11 MBRL and 4 MFRL algorithms across 18 OpenAI Gym environments, including settings with stochastic observations and actions.The environments span simple 2D tasks and complex domains such as Humanoid.
- The paper identifies dynamics bottlenecks, planning horizon dilemmas, and early-termination dilemmas as causes of stagnating MBRL performance.These challenges concern learned dynamics, search depth, and the integration of early termination into model-based algorithms.
2 Preliminaries
The paper formulates its tasks as finite-horizon MDPs and describes MBRL as learning transition dynamics from collected experience. Learned dynamics are then used to support model-based decision making.
- Each task is a discrete-time finite-horizon MDP defined by (S, A, p, r, ρ0, γ, H).The tuple specifies state and action spaces, transition dynamics, rewards, initial-state distribution, discount factor, and horizon.
- The objective is to learn a policy π that maximizes expected total reward J(π).
- MBRL stores experienced transitions D = {(s_t, a_t, s_t+1)} and uses them to learn a dynamics function ˜f_φ.For stochastic environments, dynamics can be represented with a Gaussian distribution characterized by mean and covariance.
3 Algorithms
The benchmarked algorithms span Dyna-style learning, policy search through model derivatives, and shooting-based planning. Their differences concern how learned dynamics generate experience, gradients, or candidate action trajectories.
- Dyna-style Algorithms: Dyna-style methods alternate between collecting real transitions, learning dynamics, and improving policies with imagined data.They use model-free policy learning with synthetic experience generated without additional real-environment interaction.
- Dyna-style Algorithms: ME-TRPO models dynamics with neural-network ensembles and updates policies with TRPO using experience generated by those models.
- Dyna-style Algorithms: SLBO extends ME-TRPO with theoretical guarantees of monotonic improvement and uses a multi-step L2-norm dynamics loss.
- Dyna-style Algorithms: MB-MPO meta-trains a policy across ensemble dynamics so it can quickly adapt to model variations and reduce model-bias.
- Policy Search with Backpropagation through Time: Policy-search methods backpropagate through model derivatives to compute analytic policy gradients, while PILCO uses Gaussian processes for probabilistic dynamics modeling.PILCO's Gaussian-process inference does not scale to high-dimensional environments.
- Policy Search with Backpropagation through Time: iLQG assumes known dynamics, approximates rewards quadratically and dynamics linearly, and solves the resulting problem with dynamic programming.
- Policy Search with Backpropagation through Time: GPS distills iLQG controllers into a neural-network policy through behavioral cloning while regularizing deviation from recent real-world behavior.
- Shooting Algorithms: Shooting methods approximately solve receding-horizon MPC for nonlinear dynamics and non-convex rewards by evaluating candidate action sequences.
4 Experiments
The benchmark evaluates MBRL and MFRL across 18 continuous-control environments, including modified rewards, early-termination variants, and noisy settings. Results expose algorithm-specific strengths and three recurring challenges: dynamics accuracy, planning horizon, and early termination.
- Benchmarking Performance: Shooting methods are effective across environments: uncertainty-aware PETS variants surpass RS as task difficulty increases, while PETS-CEM usually outperforms PETS-RS.PETS-CEM is less effective than PETS-RS in Ant, Walker2D, and SlimHumanoid, motivating more expressive planning for complex tasks.
- Benchmarking Performance: Dyna-style methods perform well on selected tasks but are inefficient on long-horizon complex domains because of compounding model error.MB-MPO surpasses ME-TRPO in most environments, while SLBO performs particularly well in some complex environments except SlimHumanoid.
- Benchmarking Performance: At 200k time-steps, SAC and TD3 provide stable baselines, while model-free and model-based methods are almost evenly matched overall.Performance is averaged over 4 random seeds, with separately tuned hyperparameters for each algorithm.
- Dynamics Bottleneck: Learned dynamics create a bottleneck: MBRL plateaus below model-free and ground-truth-dynamics performance even after 1M time-steps.Prediction error accumulates on unseen states, and coupling policy and dynamics learning can produce unstable predictions and performance local minima.
- Planning Horizon Dilemma / Early Termination Dilemma: Planning horizons of 20 to 40 work best, because longer horizons often reduce performance; early termination also decreases MBRL performance despite being nearly necessary in complex environments.The authors associate long-horizon degradation with exponentially expanding search spaces and identify early termination as an unresolved integration challenge.
5 Conclusions
The paper benchmarks a wide collection of MBRL algorithms by evaluating sample efficiency, asymptotic performance, and robustness, and reports rankings across 18 environments. No single MBRL algorithm is consistently best, leaving opportunities to combine strengths across approaches.
- The benchmark evaluates MBRL algorithms on sample efficiency, asymptotic performance, and robustness.
- No clear consistently best MBRL algorithm emerges across the benchmark, suggesting opportunities to combine strengths from different approaches.
- The study reports rankings of MBRL algorithms across 18 benchmarking environments.
A Environment Overview
The benchmark uses a broad set of environments with documented observation, action, horizon, and reward specifications. These include locomotion, pendulum, cart-pole, car, and reaching tasks with environment-specific dynamics and goals.
- Classic control: Acrobot and Pendulum evaluate control of pendulums toward height or upright-position objectives.Acrobot uses a two-link pendulum and six-dimensional trigonometric-angle and velocity observations; Pendulum uses a single link and three-dimensional observations.
- Classic control: InvertedPendulum and CartPole use cart-and-pole systems whose rewards favor keeping the pole upright.CartPole discretizes the actuator to −1 and 1, while InvertedPendulum uses a real-valued cart force.
- Classic control: MountainCar requires back-and-forth driving to build momentum before reaching the right mountain’s top.The observation consists of one-dimensional position and velocity.
- Manipulation: Reacher2D evaluates moving a two-link arm’s tip to randomly placed targets quickly while minimizing control input.Its reward combines tip-target Euclidean distance with a control-input penalty.
B.1 iLQG
The iLQG section summarizes its hyper-parameter search and notes that recommended settings generally perform best but may require more computation.
- iLQG hyper-parameters are summarized in Table 8.
- Recommended iLQG hyper-parameters usually achieve the best performance.
- Recommended iLQG settings can require more computation resources.
B.2 Ground-truth CEM and Ground-truth RS
This section documents benchmark-specific search settings and implementation choices across planning-based, model-based, and model-free algorithms. It also records computational trade-offs, altered initial-state conditions, and stability considerations.
- Ground-truth CEM and Ground-truth RS: Ground-truth CEM and RS searches vary planning horizon and search population size across 10 to 100.The reported best planning horizon is usually 20 to 30.
- Ground-truth CEM and Ground-truth RS: RS, CEM, and PETS use distinct search schemes, with planning horizon 30 often better in simpler environments and 100 best for Walker2D and Hopper.PETS-RS omits the elite-size hyper-parameter.
- Ground-truth CEM and Ground-truth RS: PE-E is selected for comparable performance and computational efficiency, taking about 5 hours versus 68 hours for PE-DS on HalfCheetah under the stated setting.The best HalfCheetah models use planning horizon 100 and take about 15 hours.
- MBMF is retuned for 200,000 time-steps although it was originally designed for 1 million time-steps.
- GPS uses Gaussian-distributed initial states in the benchmark, making its environments harder to solve than the original code-base conditions.
- PILCO is reported as unstable across random seeds; additional human-prior reward penalties are removed, and a dataset is used to improve training efficiency within 200,000 time-steps.The section also lists grid-search options for PILCO and other algorithms, including METRPO, SLBO, SVG, MB-MPO, and model-free methods.
C Detailed Bench-marking Performance Results
The appendix provides complete performance curves for the benchmarked algorithms across Figures 4 and 5. Some GPS and PILCO curves are omitted because their reward scales can differ substantially from those of other algorithms.
- Figures 4 and 5 contain the performance curves for the benchmarked MBRL algorithms.Figure 5 continues the performance curves from Figure 4.
- Some GPS and PILCO curves are not shown because their reward scales are sometimes very different from other algorithms.
D Noisy Environments
The appendix evaluates MBRL algorithms under observation and action noise, visualizing performance changes and examining how planning choices affect robustness. It also finds that the planning horizon dilemma persists across population sizes and that observation preprocessing can substantially alter performance.
- Noise Evaluation: Noise experiments report performance curves and end-of-training results for algorithms across HalfCheetah, Pendulum, and Cart-Pole environments.Observation and action noise are represented separately by their standard deviations, while colors indicate performance increases, decreases, or negligible changes.
- Planning Choices: The planning horizon dilemma persists across different population sizes when using learnt PETS-CEM.The experiments include HalfCheetah benchmark environments and environments with further observation preprocessing.
- Planning Choices: Observation preprocessing can affect performance by a large margin.
- Planning Choices: A performance grid examines the combined effects of planning horizon and depth.
F Planning Horizon Dilemma in Dyna Algorithms
This section studies how actual environment length and imaginary environment length, or planning horizon, affect SLBO performance in HalfCheetah and Ant. The effects differ by environment: longer horizons are not clearly beneficial for HalfCheetah, while 100-step horizons usually perform best for Ant.
- Experimental Setup: Experiments vary environment lengths from 100 to 1000 and planning horizons from 100 to 1000 in HalfCheetah and Ant.The evaluation includes horizons matching the environment length as well as longer horizons for reference.
- Findings: For HalfCheetah, increasing the planning horizon does not have obvious effects on performance.
- Findings: For Ant across different environment lengths, a planning horizon of 100 usually produces the best performance rather than longer horizons.
G Early Termination
The appendix examines early termination across MBRL algorithms and compares alternative ways to represent termination during planning. It evaluates these choices with both ground-truth and learned dynamics, including penalties, reward padding, and extra interaction near termination.
- Evaluation: The evaluation reports PETS performance with and without early termination and PETS-CEM performance using learned dynamics at 200k time-steps.
- Scope: The early termination dilemma is universal across the tested Dyna, shooting, and policy-search MBRL algorithms.The appendix begins its detailed study with shooting algorithms, including RS, PETS-RS, and PETS-CEM, where termination occurs during planning.
- Evaluation: Additional comparisons use ground-truth dynamics for CEM and RS.
- Termination Schemes: PETS experiments compare five termination schemes: ignoring termination, penalizing it, padding zero rewards, terminating without consideration during planning, and adding extra interaction time.
- Penalty Design: The appendix also evaluates different alive bonuses or depth penalties during planning.Results for the termination schemes are summarized in Table 24, while alive-bonus and death-penalty results appear in Table 25.