Source-linked AI summary
A Survey on Model-based Reinforcement Learning
Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, Yang Yu
TL;DR
Deep RL’s costly trial-and-error motivates MBRL, which learns environment models to improve sample efficiency and reduce real-world errors. The survey synthesizes model learning, model usage, related RL settings, and real-world applications, while emphasizing model generalization and future directions such as abstract and foundation models. It concludes that MBRL remains a promising route toward more efficient and applicable RL.
Problem
Deep RL requires many costly environment interactions, while model inaccuracies create discrepancies between learning in a model and in the real environment.
Method
The survey reviews MBRL model learning and usage, including tabular and deep models, simulation-based and gradient-based methods, related RL settings, and applications.
Results
The survey identifies sample efficiency as MBRL’s central target and highlights generalizable models, abstract models, and foundation environment models as future directions.
Takeaways & Limitations
MBRL has substantial potential for real-world RL because models provide a training playground where trial-and-error can occur without equivalent real-world costs.
Takeaways & Limitations
Whitebox model-based methods currently suffer from gradient bias caused by model error, leaving bias and variance reduction as future work.
Abstract
from arXiv · showhide
Reinforcement learning (RL) solves sequential decision-making problems via a trial-and-error process interacting with the environment. While RL achieves outstanding success in playing complex video games that allow huge trial-and-error, making errors is always undesired in the real world. To improve the sample efficiency and thus reduce the errors, model-based reinforcement learning (MBRL) is believed to be a promising direction, which builds environment models in which the trial-and-errors can take place without real costs. In this survey, we take a review of MBRL with a focus on the recent progress in deep RL. For non-tabular environments, there is always a generalization error between the learned environment model and the real environment. As such, it is of great importance to analyze the discrepancy between policy training in the environment model and that in the real environment, which in turn guides the algorithm design for better model learning, model usage, and policy training. Besides, we also discuss the recent advances of model-based techniques in other forms of RL, including offline RL, goal-conditioned RL, multi-agent RL, and meta-RL. Moreover, we discuss the applicability and advantages of MBRL in real-world tasks. Finally, we end this survey by discussing the promising prospects for the future development of MBRL. We think that MBRL has great potential and advantages in real-world applications that were overlooked, and we hope this survey could attract more research on MBRL.
1 Overview of Model-based RL
Model-based reinforcement learning addresses deep RL’s high sample complexity by learning environment models that support low-cost interaction and simulation. The survey reviews model learning, model usage, and challenges caused by model inaccuracies.
- Motivation: Deep RL requires extensive interaction data because policy updates change the data distribution, making real-world trial-and-error costly.The relevant sample complexity is the amount of data needed to learn an approximately optimal policy.
- Motivation: MBRL improves sample efficiency by using an environment model as an imagined world for additional interaction and policy learning.The model abstracts environment dynamics and generates simulation data alongside experienced real-environment data.
- Challenges: With a sufficiently accurate model, MBRL can achieve higher sample efficiency than model-free RL, but learning such models is difficult in complex environments.The survey therefore emphasizes careful choices in model learning and model usage.
- Survey scope: The survey organizes MBRL around model learning and model usage, including tabular and approximation models and blackbox, whitebox, value-aware, and policy-aware approaches.It also reviews combinations with other reinforcement-learning settings.
2 Model Learning
MBRL learns the unknown transition dynamics and reward function from historical trajectory data. Because trajectories record the relevant inputs and outputs, model learning can borrow supervised-learning ideas.
- Model-learning setup: The environment model retains the MDP’s transition dynamics and reward function as the components that must be learned.State space, action space, and discount factor are treated as given in this formulation.
- Data: Historical data is represented as trajectories of state-action-reward sequences.These trajectories provide the observations used to learn the environment model.
- Learning objective: Recorded transitions provide inputs and outputs for the transition dynamics, while recorded rewards provide inputs and outputs for the reward function.This correspondence makes supervised-learning methods applicable to model learning.
2.1 Model Learning in Tabular Setting
In tabular MDPs, model learning estimates transitions by counts and rewards by accumulated returns. R-MAX combines these estimates with optimistic exploration, while the estimates converge to the true model with sufficient data.
- Tabular estimation: Tabular model learning records state-action-next-state counts and reward sums to estimate transition probabilities and rewards.The reward estimate uses a preset minimum value when a state-action pair lacks observations.
- Tabular estimation: These tabular estimates are maximum-likelihood estimators that converge to the true transition and reward functions as samples approach infinity.The survey characterizes both estimates as unbiased.
- Exploration: R-MAX assigns optimistic self-transitions and maximum rewards to insufficiently visited state-action pairs, replacing them with empirical averages after more than K visits.This optimistic fictitious model encourages exploration.
- Exploration: R-MAX repeatedly solves an approximately optimal policy in the fictitious model, executes it in the environment, and updates the model with new samples.The exploration incentive comes from the optimistic reward assigned to insufficiently visited pairs.
2.2 Model Learning via Prediction Loss
For large or continuous MDPs, MBRL replaces tabular estimates with learned prediction models trained from transition data. The survey analyzes how model error affects policy evaluation, including compounding horizon effects and practical limitations of worst-case guarantees.
- Approximation models: Large or continuous MDPs require approximation functions such as linear models, neural networks, or decision trees instead of tabular representations.The survey focuses on neural-network models.
- Prediction losses: One-step deterministic models can be trained by minimizing mean squared next-state prediction error, while probabilistic models capture aleatoric uncertainty.Probabilistic models may minimize KL divergence and are often instantiated as Gaussian distributions.
- Prediction losses: Prediction-model losses transform model learning into supervised learning, allowing supervised-learning techniques to train the environment model.This applies to both deterministic and probabilistic prediction objectives.
- Model error: Value loss grows quadratically with the effective horizon under the simulation lemma, motivating short rollouts in learned models.The survey notes that some methods use rollouts shorter than 10 steps.
- Practical limitations: The maximum model-error bound is difficult to achieve or assess in practice because it requires measuring the largest discrepancy over all state-action pairs.Distribution-based analyses instead connect model error more directly to the training loss.
- Model error: Model-based policy evaluation depends on both learned-model bias and divergence between the evaluated policy and the data-collecting policy.Simulation Lemma II makes this dependence explicit through an error measured over the data distribution.
2.3 Model Learning with Reduced Error
Prediction-loss model learning can produce horizon-squared compounding error. The survey discusses constraining models and matching trajectory distributions to reduce this error, while noting trade-offs and remaining policy-divergence effects.
- Motivation: Prediction-loss learning can cause horizon-squared compounding error because it does not capture long-term transition effects.This motivates constraining the model or matching real and model trajectory distributions.
- Model constraints: Wasserstein distance measures similarity between transition distributions and remains applicable when their supports are disjoint.The distance is defined through the minimum expected metric distance over couplings of the two distributions.
- Model constraints: Lipschitz continuity constraints can bound n-step model error under assumptions on the real and learned transition models.The bound uses the minimum of their Lipschitz constants, K = min{K∗, K}.
- Model constraints: When K is small, the evaluation error and compounding error can be controlled relative to the standard simulation-lemma bounds.A smaller learned-model Lipschitz constant may nevertheless increase Wasserstein approximation error when the real model has a larger Lipschitz constant.
- Distribution matching: Distribution matching trains the transition model to match the joint state-action-next-state distribution induced by the real environment.The optimal model minimizes Jensen-Shannon divergence, while related work also considers Wasserstein distance and joint optimization with the policy.
- Distribution matching: Distribution matching yields an improved policy-evaluation bound, but the remaining policy-divergence term can still be large.Using a distribution of data-collecting policies is proposed to reduce divergence from the target policy.
- Distribution matching: The improved bound has a coefficient linear in the effective horizon 1/(1−γ), meeting a lower bound and eliminating the compounding-error issue in general.The matching loss using JS divergence may require more samples than prediction loss to achieve the same value.
2.4 Model Learning for Complex Environments Dynamics
Complex environment dynamics require model-learning strategies beyond standard state-action transition modeling. The survey highlights belief-state estimation for partial observability and representation learning for high-dimensional observations.
- Dynamics models: Ensembles of Gaussian-process dynamics models use neural networks to predict the next-state distribution from the current state-action pair.This architecture has been shown to work well on MuJoCo robot-locomotion environments when state observations are sufficient statistics.
- Partial Observability: In partially observable environments, observations may not provide sufficient statistics for future dynamics, yielding a partially observable MDP.A classic solution learns an observation model and latent transition model, then infers the posterior belief state.
- Representation Learning: Representation learning can improve environment-model building in high-dimensional state spaces such as images.It supports value prediction and model rollout by learning informative latent state or action representations.
3 Model Usage and Integration with Model Learning
The survey organizes model usage around planning, data augmentation, and gradient generation, while also examining how model learning can be aligned with policy improvement. It emphasizes that model errors create important limits and open problems for reliable model-based reinforcement learning.
- Planning with Model Simulation: Planning uses learned models to produce or improve policies through methods such as MPC, MCTS, and background planning.MPC optimizes action sequences, MCTS incrementally extends a search tree, and background planning uses simulated data to improve policy or value learning.
- Planning with Model Simulation: Black-box MPC samples action sequences in a learned model, but random shooting can have high variance and miss high-reward sequences in large action spaces.Recent approaches alter the sampling strategy or sampling space to improve planning efficiency.
- Planning with Model Simulation: Policy imitation of MPC results can provide nearly optimal initial action sequences, simplifying optimization and improving planning efficiency.POPLIN-A uses the policy's prediction to initialize the planning problem at states the policy has encountered.
- Data Augmentation with Model Simulation: Model-based value expansion forms H-step targets by unrolling learned dynamics; under stated conditions, its value-estimation MSE approximately contracts by γ2H.STEVE adapts across rollout horizons by weighting targets according to uncertainty from the value function and transition dynamics.
- Data Augmentation with Model Simulation: Theory links model-based policy improvement to model-rollout quality: improving modeled returns beyond C(ϵm, ϵπ, ϵm′, k) guarantees improvement in true returns.Bidirectional model rollouts can yield a tighter return-discrepancy bound than MBPO with the same total rollout length.
- Gradient Generation with White Box Model Simulation: White-box methods propagate gradients through neural-network models, but model error currently causes gradient bias, making bias and variance reduction an open direction.Related methods integrate model structure with policy or value optimization, and model learning can incorporate value or policy information to better support current updates.
4 Model-based Methods in Other Forms of RL
Model-based techniques extend reinforcement learning beyond standard interaction settings by using learned models for offline, goal-conditioned, multi-agent, meta-, and automated MBRL. Across these settings, the survey emphasizes balancing model generalization and uncertainty against the benefits of simulated data, adaptation, and planning.
- 4.1 Offline RL: Offline RL learns policies from fixed datasets, while model-based offline RL adds a learned environment model to generate training data and support limited exploration.Offline RL avoids interaction before learning a satisfactory policy, but model learning remains constrained by limited data and out-of-distribution states.
- 4.1 Offline RL: Offline RL faces extrapolation error because the behavior policy generating the dataset differs from the current learning policy.Conservative methods avoid out-of-distribution regions, whereas MAPLE uses meta-RL to adapt directly in such regions.
- 4.2 Goal-conditioned RL: Goal-conditioned RL augments observations with goals, and model-based goal planning can generate diverse subgoals for complex tasks.Visual prediction models have enabled subgoal generation and planning for robot manipulation, with substantial gains over model-free and planning baselines.
- 4.3 Multi-agent RL: Model-based MARL analyzes multi-agent environments through opponent and environment-dynamics modeling under game non-stationarity.Theoretical work shows that close game approximations yield close Markov perfect equilibria, while environment-dynamics learning remains relatively rare.
- 4.3 Multi-agent RL: For two-agent zero-sum Markov games, model-based MARL has lower derived sample complexity than prior model-free MARL methods.A stated sufficient sample bound for an α-MPE is ˜O(|S||A|(1 −γ)^−2α^−2).
- 4.4 Meta RL: Model-based meta-RL combines adaptation with MPC to address training-test dynamics gaps and can introduce additional safety constraints during action generation.Reported applications include adaptive control for a Rubik’s cube and robust quadrupedal locomotion, while the generalization of latent adaptation states remains underexplored.
- 4.5 Automated Methods on Model Learning and Usage: MBRL procedures require more human effort because model learning, policy optimization, planning steps, simulated-to-real data ratios, and hyperparameters interact.AutoMBPO automatically schedules key MBPO hyperparameters, and empirical results report higher sample efficiency for automatically optimized MBRL methods.
5 Applications of Model-based RL
MBRL is suited to real-world tasks because learned or hand-crafted models provide a lower-cost setting for training and evaluating policies where errors are costly. The survey highlights simulation-based exploration, pre-deployment assessment, and decision explanation as practical advantages.
- Real-world motivation: Real-world RL requires a high-fidelity, error-tolerant playground because real environments cannot freely support trial-and-error learning.This motivates simulators and learned environment models for cost-sensitive applications.
- Simulation approaches: Hand-crafted simulators support cost-sensitive tasks including autonomous driving, industrial control, traffic control, electricity allocation, financial trading, and tokamak control.They can resemble overall task functionality but require substantial expert time to build.
- Simulation approaches: Learned environment models can reduce simulator-construction costs, and policies trained in Virtual-Taobao maintained similar performance after real-world deployment in A/B tests.MAIL modeled detailed customer behaviors, while later work incorporated hidden factors to improve model learning.
- Practical advantages: Simulators or learned models allow broader exploration and can support reinforcement-learning training even when the model is imperfect by constraining exploration.This can release more of RL’s exploration capacity while limiting the impact of unrealistic model behavior.
- Practical advantages: Model-based evaluation offers a direct way to assess improved policies before deployment when off-policy evaluation has not shown reasonable benchmark effectiveness.Improved policies can induce state-action distributions different from those in collected data, complicating historical-data evaluation.
- Practical advantages: Running a policy in a simulator or model exposes its state-specific decisions, helping decision-makers evaluate confidence and rationality.Decision visibility can support trust when the policy’s actions appear reasonable or reveal better ideas.
6 Conclusions and Future Directions
The survey identifies generalization, abstraction, and broader model-based settings as central directions for improving MBRL. It highlights multi-agent planning, foundation environment models, and adaptive methods as promising areas for future development.
- MBRL has re-emerged in deep RL and plays an important role in achieving state-of-the-art sample efficiency.
- Learning generalizable models: Generalizable models are a key goal, with causal structure and causal-effect modeling identified as routes toward stronger model generalization.
- Learning abstract models: State and temporal abstractions can produce compact, low-dimensional MDPs that simplify model learning, while hierarchical reinforcement learning remains largely unexplored.
- Model-based multi-agent RL: Model-based multi-agent RL remains in its infancy, with open opportunities for improving coordination, sample efficiency, environment learning, planning, and communication.
- Foundation models: Foundation environment models, beyond foundation policy models, represent a large unexplored research area.
- Future work includes tighter value discrepancy bounds, automatic scheduling, adaptive model usage, and lifelong model learning.