Source-linked AI summary
Objective Mismatch in Model-based Reinforcement Learning
Nathan Lambert, Brandon Amos, Omry Yadan, Roberto Calandra
TL;DR
MBRL trains dynamics models for prediction likelihood but uses them to optimize downstream control reward, objectives that may not align. The paper analyzes this mismatch and proposes re-weighting model training as an initial mitigation. Its experiments report that likelihood and control reward are not reliably correlated, motivating further study.
Problem
MBRL trains forward dynamics models for prediction likelihood while seeking downstream control performance, but these objectives may not correlate.
Source: §1 · pp. 1–2Method
Results
Experiments show substantial disagreement between model accuracy and reward, with no clear accuracy-reward trend on full-state-space data and no guarantee that higher likelihood improves control.
Source: §4.1 · p. 4Takeaways & Limitations
The authors conclude that objective mismatch warrants deeper study toward a new generation of MBRL algorithms.
Source: §7 · p. 9The experiments are an initial exploration; the cartpole analysis is limited by computational challenges, controller comparisons could be quantified, and other MBRL algorithms remain to be assessed.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks. Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, with little development of the general framework. In this paper, we identify a fundamental issue of the standard MBRL framework -- what we call the objective mismatch issue. Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized. In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t.~the likelihood of the one-step ahead prediction, and the overall goal of improving performance on a downstream control task. For example, this issue can emerge with the realization that dynamics models effective for a specific task do not necessarily need to be globally accurate, and vice versa globally accurate models might not be sufficiently accurate locally to obtain good control performance on a specific task. In our experiments, we study this objective mismatch issue and demonstrate that the likelihood of one-step ahead predictions is not always correlated with control performance. This observation highlights a critical limitation in the MBRL framework which will require further research to be fully understood and addressed. We propose an initial method to mitigate the mismatch issue by re-weighting dynamics model training. Building on it, we conclude with a discussion about other potential directions of research for addressing this issue.
1. Introduction
The paper identifies an objective mismatch in MBRL: dynamics models are trained for prediction accuracy but used to optimize downstream task performance. It formalizes and studies this disconnect, proposing an initial mitigation.
1. Introduction
- MBRL learns a forward dynamics model from trajectories, classically fitting it by maximum likelihood before using it in a control algorithm.The model predicts system evolution under control signals and may be used in model-predictive control.Source: §1 · p. 1
- Figure 1 contrasts training a model to maximize likelihood with using it in control to maximize a reward signal absent from training.This is the paper’s central mismatch between the model-training objective and the downstream control objective.Source: Figure 1 · p. 2
- MBRL optimizes model prediction accuracy separately from policy task performance, and likelihood need not correlate with controller performance.The paper identifies and formalizes this objective mismatch, examines its effects, and proposes an initial mitigation.Source: §1 · pp. 1–2
2. Model-based Reinforcement Learning
The formulation represents the environment with a state-transition distribution and learns a neural-network forward model from collected dynamics data. A controller uses that model to select action sequences with high expected reward.
2. Model-based Reinforcement Learning
- The agent learns an approximate neural-network forward dynamics model from collected dynamics data, then uses it for control.The model approximates the environment’s state transitions.Source: §2 · p. 2
- The controller takes the current state and returns an action sequence that maximizes expected reward over predictive horizon T.The model induces the state transitions used in evaluating the planned sequence.Source: §2 · p. 2
- The paper primarily compares probabilistic neural networks trained to maximize log-likelihood with deterministic networks trained to minimize mean squared error.Unless stated otherwise, it uses PETS models with an expectation-based trajectory planner and cross-entropy-method optimizer.Source: §2 · p. 2
3. Objective Mismatch and its Consequences
The paper argues that MBRL’s data collection and task-specific learning differ from system identification’s assumptions of broad coverage and abundant data. Consequently, model-training objectives may not track control reward; the paper’s analysis excludes effects from changing data distributions.
3. Objective Mismatch and its Consequences
- System identification assumes full state-action coverage, virtually infinite data, and a global, generalizable model that can perform well on desired tasks.These assumptions support collecting broad data to build a sufficiently accurate global model.Source: §3 · p. 3
- Unlike system identification’s offline trajectories designed to cover the state-action space, MBRL data often concentrates on trajectories toward the goal, leaving other regions unexplored.The paper notes that MBRL’s on-policy sampling aims to reduce interactions, so its data generally does not cover the entire state-action space.Source: Figure 2 · p. 3, §3 · p. 3
- In MBRL, task-optimized sampling and non-parametric models can yield models biased toward locally accurate, task-specific data, while generalization is rarely tested.The paper contrasts this setting with system identification’s global-model assumptions.Source: §3 · p. 3
- Model-training cost does not strongly correlate with episode reward, so improving model accuracy does not guarantee better control performance.The paper frames training and control as decoupled optimization problems that may come at the cost of final reward.Source: §3 · p. 3
- The analysis excludes interactions caused by changes in the dynamics-data distribution during learning and does not claim which potential solution is best.The paper leaves identifying the best solution for future work.Source: §3 · p. 4
4. Identifying Objective Mismatch
Across cartpole and half cheetah experiments, model likelihood and episode reward can diverge, challenging the assumed link between model accuracy and policy performance.
4. Identifying Objective Mismatch
- Experiments compared expert, PETS on-policy, and state-space-representative datasets, using retrained models and random seeds to account for stochasticity.The benchmarks were cartpole and half cheetah.Source: §4 · p. 4
4.1. Exploration of Model Loss vs Episode Reward Space
- Across 1,000 cartpole and 2,400 half cheetah PETS models, likelihood and reward substantially disagreed, with no clear accuracy-reward trend on full-state datasets.The results show decreased correlation between validation loss and episode reward, so increasing model accuracy does not guarantee better control performance.Source: §4.1 · p. 4
4.2. Model Loss vs Episode Reward During Training
- The high-reward, increased-likelihood trend weakened as datasets covered more state space than expert trajectories.Each reward point was averaged over 10 trials.Source: §4.2 · p. 5
- On grid cartpole, P/PE reward peaked at a different epoch from minimum validation loss; policy-only training missed full dynamics, while full-dataset on-policy error preceded reward decline.Validation loss was still improving slowly at 500 epochs in both dataset-choice scenarios.Source: Figure 4 · p. 5, §4.2 · p. 6, Figure 5 · p. 6
4.3. Decoupling Model Loss from Controller Performance
- Re-weighting improved performance on smaller training sets but increased variance on larger sets; training too near the optimal trajectory reduced PETS robustness under stochastic MPC.The study varied training-set size from 10 to 2,500 points and optimal-distance bounds from 0.28 to 15.66.Source: §4.3 · p. 7
- CMA-ES reduced cartpole on-policy reward from 176 to 98 over 5 trials while slightly improving likelihood, showing low-loss parameters span a broader space than high-reward ones.The attack fine-tuned the model’s last layer while penalizing drops in validation likelihood.Source: §4.3 · pp. 6–7, Figure 7 · p. 6, Figure 6 · p. 7
5. Addressing Objective Mismatch During Model Training
The paper proposes re-weighting dynamics-model training so that trajectories relevant to the task receive greater emphasis. The approach improves Cartpole sample efficiency, but assumes the optimal trajectory is known in advance.
5. Addressing Objective Mismatch During Model Training
- Re-weighting the dynamics-model loss demonstrated improved sample efficiency on Cartpole.The loss gives optimal-trajectory points weight 1 and points at the edge of the dataset grid weight 0.Source: §5 · p. 8
- The method lowers the loss weight for points farther from the optimal policy, emphasizing accurate modeling along task-relevant trajectories.It retains knowledge of nearby transitions for robustness under a stochastic controller.Source: §5 · pp. 7–8
- The re-weighting uses distance from an expert trajectory as a baseline for constructing training datasets.The authors uniformly sampled 25e6 Cartpole state-action points and measured each point’s minimum orthogonal distance to 200 expert-trajectory elements.Source: §5 · p. 8
- The approach is impractical when the optimal trajectory is unknown; the authors suggest iteratively estimating and re-weighting samples online.Figure 9 depicts loss re-weighting with respect to distance from the optimal trajectory.Source: §5 · p. 8, Figure 9 · p. 8
6. Discussion, Related Work, and Future Work
The discussion considers approaches that connect reward to control or shape dynamics-model training to ease control. The authors describe their experiments as preliminary and identify settings that need further evaluation.
6. Discussion, Related Work, and Future Work
- The discussion frames objective mismatch as a performance fragility and considers research directions beyond likelihood re-weighting.The paper reviews other community approaches to the issue.Source: §6 · p. 8
- The authors discuss methods that optimize task performance by connecting reward to the controller, including differentiable planning and zero-order optimization.They note that such approaches have proven difficult to scale to complex systems in practice.Source: §6 · p. 8
- Alternative heuristics shape model structure or training, including locally linear latent spaces, long-horizon prediction, and stabilizability constraints.The cited works do not formalize or explore the underlying mismatch issue in detail.Source: §6 · pp. 8–9
- The experiments are an initial exploration; Section 4.2 is limited to Cartpole by computational challenges, while Section 4.3 could add quantitative controller comparisons.The authors also call for evaluating objective mismatch in MBPO and POPLIN.Source: §6 · p. 9
7. Conclusion
The conclusion identifies and analyzes objective mismatch in MBRL, reporting experimental evidence of negative effects on a current state-of-the-art algorithm. It presents this work as an initial insight and calls for further research.
7. Conclusion
- Experiments highlight negative effects of objective mismatch on a current state-of-the-art MBRL algorithm.The paper characterizes the mismatch as a disconnect between dynamics-model likelihood and overall task reward.Source: §7 · p. 9
- The paper identifies, formalizes, and analyzes objective mismatch in MBRL.Source: §7 · p. 9
- The authors present their work as an initial insight and call for future research to address objective mismatch.Source: §7 · p. 9
Appendix A. Effect of Dataset Distribution when Learning
The appendix reports that adding task-irrelevant data can suppress learning efficiency in Cartpole motor-babbling experiments. This illustrates the importance of accounting for off-task data in model training.
Appendix A. Effect of Dataset Distribution when Learning
- Adding transitions from a specific state-space area increases the model’s effort on those transitions and can slow learning as more relevant data arrives.The passage says likelihood is biased down as the model learns the additional data.Source: Appendix A · p. 12
- 20, 200, 400, and 2,000 short-horizon babbling rollouts showed some regression in learning speed when more useless data was included.The Cartpole runs used a 10-step horizon and forced the initial babbling state to 0.Source: Appendix A · p. 12
- Additional data not relevant to the task suppressed learning efficiency in the dynamics model.The authors relate this effect to the need for model training to account for potential off-task data.Source: Appendix A · p. 12
Appendix B. Task Generalization in Simple Environments
Cartpole performance saturates before model likelihood reaches its minimum, and transfer to goals away from zero weakens where training data provide less coverage.
Appendix B. Task Generalization in Simple Environments
- Models trained for the zero-position goal generalize to nearby positive and negative goals, but performance drops in less-covered regions.The comparison uses the same dynamics models with different reward functions.Source: Appendix B · p. 12, Figure 12 · p. 13
- The median reward reaches its maximum of 180 well before the dynamics model’s NLL reaches its minimum.The plotted reward is the median across 10 trials.Source: Appendix B · p. 12
- Training positions concentrate around zero, making MPC difficult to control outside that region.The training-data snapshots show how coverage relates to cartpole reward.Source: Figure 13 · p. 13, Appendix B · p. 13
Appendix C. Validating models with trajectories rather than random tuples
The appendix evaluates dynamics models on whole-trajectory validation batches rather than randomly shuffled tuples, finding a tighter likelihood–reward relationship in the reported experiments.
Appendix C. Validating models with trajectories rather than random tuples
- Trajectory batches align validation with MPC’s choice of the best planned trajectory, rather than random one-step predictions.Randomly shuffled batches can have test loss dominated by an outlier.Source: Appendix C · p. 13
- Validating on trajectories improves the likelihood–reward trend for cartpole and half-cheetah compared with random batches and large-batch validation.For cartpole, the relationship has less variance and increased ρ; for half-cheetah, trajectory validation improves the trend over large batches as well.Source: Appendix C · p. 14, Figure 14 · p. 14
Appendix D. Using simple reward as re-weight
The appendices discuss a reward-based re-weighting alternative, ways model mismatch can impair control, experimental settings, and controller design choices.
Appendix D. Using simple reward as re-weight
- Re-weighting by simple reward can emphasize states near the target while neglecting the dynamics along the route, potentially reducing initial-trajectory performance.For distance-to-target rewards, reward isocurves may be orthogonal to the optimal trajectory, and further study of re-weighting alternatives is needed.Source: Appendix D · p. 14, Figure 16 · p. 14
Appendix E. Ways model mismatch can harm the performance of a controller
- Long-horizon roll-outs can diverge from true predictions and enter implausible states, while missed stationarity properties can undermine control.Source: Appendix E · p. 15
- Approximate dynamics can make control optimization harder even when the true optimal action sequence is optimal under the approximate model.Neural-network nonlinearities and non-smoothness can make classical control approaches difficult.Source: Appendix E · p. 15
- Models with similar likelihoods can yield different rewards and occupy different state-space regions, but the reason for the reward differences remains unclear.The paper identifies interpretability and quantifying the impact of model differences as open issues.Source: Appendix E · pp. 15–16
Appendix F. Hyper-paramters and Simulation Environment
- Cartpole and half-cheetah experiments used MuJoCo 1.50.1; datasets include expert, solved-task on-policy, and sampled-space data, with grid slicing unsuitable for half-cheetah.Source: Appendix F · p. 16
Appendix G. Additional Related Works
- Controller inductive biases such as horizon length and slew-rate penalties can improve performance when tuned correctly, but may be difficult to tune.Source: Appendix G · p. 16