Source-linked AI summary
Model-Based Offline Planning
Arthur Argenson, Gabriel Dulac-Arnold
TL;DR
Offline RL seeks useful policies from logged system data without environment access, but model-free policies can be difficult to control externally. MBOP learns a model, behavior prior, and value function for direct MPC planning, achieving competitive offline performance, near-optimal behavior in some low-data tasks, and adaptable goal- and constraint-conditioned control.
Problem
Offline RL must learn policies from logged system data without environment access, while model-free policies can be opaque and difficult to command or integrate into larger systems.
Method
MBOP learns a world model, behavior-cloning policy, and fixed-horizon value function, then uses MPC and trajectory optimization for direct planning from offline logs.
Results
MBOP performs competitively across data regimes, finds near-optimal policies on several simulated tasks with as little as 5000 steps, and supports adaptable goals and constraints.
Takeaways & Limitations
Offline planning can provide data-efficient policies that remain adaptable to changing rewards, goals, and environmental constraints during operation.
Takeaways & Limitations
In higher-data regimes, variants without the behavior-cloning prior can be used for goal and constraint control, while more data-efficient goal and constraint control requires future goal-conditioned priors and value functions.
Abstract
from arXiv · showhide
Offline learning is a key part of making reinforcement learning (RL) useable in real systems. Offline RL looks at scenarios where there is data from a system's operation, but no direct access to the system when learning a policy. Recent work on training RL policies from offline data has shown results both with model-free policies learned directly from the data, or with planning on top of learnt models of the data. Model-free policies tend to be more performant, but are more opaque, harder to command externally, and less easy to integrate into larger systems. We propose an offline learner that generates a model that can be used to control the system directly through planning. This allows us to have easily controllable policies directly from data, without ever interacting with the system. We show the performance of our algorithm, Model-Based Offline Planning (MBOP) on a series of robotics-inspired tasks, and demonstrate its ability leverage planning to respect environmental constraints. We are able to find near-optimal polices for certain simulated systems from as little as 50 seconds of real-time system interaction, and create zero-shot goal-conditioned policies on a series of environments. An accompanying video can be found here: https://youtu.be/nxGGHdZOFts
1 INTRODUCTION
MBOP learns a directly plannable controller from offline logs, combining a learned model, behavior-cloning policy, and value function. It is designed to improve over logged behavior while remaining adaptable to rewards, goals, and constraints.
- Motivation: Offline reinforcement learning learns controllers from system logs without interacting with the corresponding environment.The paper motivates this setting when interaction is costly, unsafe, or unavailable.
- Approach: MBOP combines a learned world model, behavior-cloning policy, and fixed-horizon value function with MPC-based real-time planning.Its trajectory optimizer extends MPPI to provide goal- or reward-conditioned control.
- Capabilities: MBOP can outperform the policy that generated the logs and subsequently accept reward, goal-state, and state-constraint conditioning.These conditions may be non-stationary, supporting operator or hierarchical-system control.
- Evaluation: The paper evaluates MBOP on standard offline-RL benchmarks, zero-shot goal and constraint adaptation, and component ablations.The experiments cover RLU and D4RL tasks and examine the combined contribution of MBOP’s elements.
2 RELATED WORKS
Prior model-based offline methods generally use learned models to train task-specific model-free policies rather than planning directly at deployment. MBOP combines a behavior-cloning prior and value initialization for direct offline planning.
- Prior offline methods: MOPO and MoREL use learned models to train model-free controllers through surrogate offline environments, making the resulting policies task-specific.They penalize model uncertainty but do not use the models for direct planning on the deployment problem.
- Prior offline methods: MOPO can alter rewards by retraining, whereas it cannot dynamically adapt one final policy to arbitrary goals or constrained objectives.This distinguishes retraining-based reward changes from task-time planning adaptation.
- Positioning: MBOP shares behavior-cloning-guided trajectory sampling with related work but performs policy generation through direct planning in a fully offline setting.The comparison places MBOP between model-based planning and offline policy learning.
- Motivation: Planning-based controllers are motivated by flexibility for external interaction, including changing tasks, constraints, and explainability requirements in industrial systems.These properties may matter even at some expense of raw performance.
- Novelty: Previous approaches considered individual MBOP components but not their full combination of a behavior-cloning trajectory prior and value-function initialization for offline learning.The paper also identifies consistent ensemble sampling and ensemble return averaging as important implementation choices for stability.
3 MODEL-BASED OFFLINE PLANNING
MBOP learns an approximate environment model, behavior-cloned action prior, and fixed-horizon value function from offline data, then uses MPC and iterative trajectory optimization to select actions. The planner samples model rollouts, refines trajectories with the prior and stored plan, and weights them by predicted return.
- 3 MODEL-BASED OFFLINE PLANNING: MBOP produces policies entirely from logs of a less-performant policy without interacting with the actual environment.It uses a particle-based trajectory optimizer and MPC to choose actions conditioned on the current state.
- 3.3 OFFLINE LEARNING: The offline setting supplies a fixed dataset of state, action, reward, and next-state tuples while withholding the environment during learning.The learned policy is later benchmarked in the real environment, although offline evaluation may also be needed.
- Learned components: MBOP trains three function approximators: a dynamics-and-reward model, a behavior-cloned action prior, and a truncated value function.The dynamics model rolls out candidate trajectories, the prior guides action sampling, and the value function estimates return over a fixed horizon.
- 3.6 MBOP-TRAJOPT: Trajectory returns combine predicted rewards with a fixed-horizon value estimate, after which trajectories are averaged using return-based weights.The planner predicts next states and rewards with the model and appends the value estimate at the horizon.
- Learned components: Each learned component is implemented with bootstrap ensembles, whose members are trained by supervised learning on the offline dataset.The ensemble construction is described as a way to stabilize planning.
- MPC policy loop: At every timestep, MPC runs fixed-horizon planning, returns a trajectory, and executes its first action while maintaining a trajectory buffer.This receding-horizon loop updates the plan from the current state.
- 3.6 MBOP-TRAJOPT: MBOP-Trajopt rolls out N trajectories of length H with the learned model, uses consistent ensemble members, and combines behavior-cloned actions with the previous trajectory.The stored trajectory amortizes optimization across MPC steps, while β controls how quickly the current trajectory changes.
- Scope and extension: MBOP’s combined elements improve trajectories over behavior data especially in low-data regimes, while future work targets goal-conditioned priors and value functions for more efficient goal and constraint control.In higher-data regimes, variants without the behavior-cloning prior can also support goal- and constraint-based control.
4 EXPERIMENTAL RESULTS
MBOP is evaluated in standard offline settings and on novel objectives and constraints using RLU and D4RL datasets. It achieves strong performance with limited data, adapts to changed goals and constraints, and remains computationally viable, although performance declines on highly variable data.
- Standard offline performance: MBOP finds near-optimal policies on most RLU Cartpole and Quadruped dataset sizes with 5000 steps, while Walker requires approximately 10 minutes of data.The 5000-step datasets correspond to 5 episodes and approximately 50 seconds on Cartpole or 100 seconds on Quadruped; Walker reaches a score of 900 with sufficient data.
- Standard offline performance: MBOP generally outperforms the behavior data and BC prior, and surpasses MOPO on medium and medium-expert D4RL regimes.On random and mixed higher-variance datasets, MBOP is less performant, likely because of its reliance on policy-conditioned priors.
- Zero-shot task adaptation: MBOP integrates constraints absent from the dataset and performs well on objectives different from the behavior policy’s objective.The evaluated tasks include position-constrained Cartpole, heading-conditioned Quadruped, and height-constrained Walker.
- Execution and robustness: MBOP operates from 106 Hz at planning horizon h = 4 to 40 Hz at h = 40, compared with 362 Hz for BC.These measurements use one Intel Xeon W-2135 CPU core and an Nvidia 1080TI.
- Execution and robustness: MBOP maintains consistent performance across broad ranges of trajectory-reweighting κ and planning-horizon H values, degrading mainly near extreme settings.The sensitivity analysis covers the three RLU environments.
5 CONCLUSION
The conclusion presents MBOP as a data-efficient and flexible offline planning method for controllable policy generation. Its performance is competitive across data regimes and supports adaptation to changing goals, costs, and constraints, but degrades on multimodal or random data.
- Ablations: MBOP’s full combination of BC prior, value function, and environment model is consistently more performant than its ablations on the Walker dataset.The ablations remove the policy prior, estimated returns, or both.
- Conclusion: Planning-based offline control allows external systems to interact with learned controllers through changing goals, costs, and environmental constraints.The conclusion attributes MBOP’s flexibility to online planning over learned components.
- Conclusion: MBOP is competitive across data regimes and provides adaptable policies for goal-conditioned and constrained tasks without prior experience in the original data.The stated scope includes policies generated from offline data for more complex task modifications.
- Conclusion: MBOP’s performance degrades when offline data is multimodal or random, limiting its suitability in those data regimes.The conclusion identifies consistent but sub-optimal operating policies as a more favorable use case.
APPENDIX
MBOP is presented as a general offline model-based approach with controllable policies, including constraints and arbitrary cost-based goals. The appendix reports benchmark, ablation, sensitivity, filtering, and execution-speed analyses, while noting limits on control frequency and real-robot validation.
- Scope and limitations: MBOP is evaluated across six physics-bound environments, but the paper does not implement it on actual robots.The authors describe real-robot deployment as upcoming work and motivate applicability from results across under-actuated control, locomotion, and manipulation.
- Contributions: MBOP can learn entirely offline without a simulator, constrain policy operation, and rephrase its goal according to an arbitrary cost function.These capabilities are identified as key contributions for robotics applications.
- Ablations and benchmarks: Figures 4 and 5 visualize the full results for MBOP’s ablations on RLU and D4RL datasets.The appendix also reports mean evaluation performance and associated hyperparameters in task-specific tables.
- Execution speed: On RLU Walker, MBOP can achieve control frequencies below 10Hz but cannot currently attain 100Hz with longer horizons.The paper suggests distilling the controller into a task-specific policy when high-frequency low-level control is important.
- Constrained tasks: 80% of the episode satisfies the height constraint on Walker while maintaining reasonable performance.Across ablations, MBOP better maintains base-task performance at similar constraint-satisfaction rates.
- Sensitivity analyses: Figures 7 and 8 evaluate sensitivity to Beta, Horizon, and Sigma, while Figure 6b examines Horizon and Kappa together.These analyses concern MBOP and associated ablations on RLU datasets or environments.