PaperlayerView exact revision on arXiv

Source-linked AI summary

The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithms

Anirudh Vemula, Yuda Song, Aarti Singh, J. Andrew Bagnell, Sanjiban Choudhury

arXiv:2303.00694v1cs.LGcs.ROcs.SYeess.SY

TL;DR

MBRL must reduce real-world interactions without incurring repeated exponential planning or fitting models with objectives misaligned to policy performance. The paper introduces PDAM, two no-regret algorithms, and reports statistical and computational gains on simulated benchmarks, while noting exploration and scalability limitations.

  • Problem

    MBRL faces computationally expensive repeated planning and a mismatch between prediction-error model fitting and real-world policy performance.

    Source: §1 · p. 1

  • Method

    The paper introduces PDAM, using expected policy advantage for policy computation and value moment matching for model fitting, with two no-regret algorithms.

    Source: §1 · pp. 1–2, §6 · p. 6

  • Results

    The proposed algorithms demonstrate statistical and computational gains compared with existing MBRL methods on simulated benchmarks.

    Source: §1 · p. 2, §8 · pp. 8–9

  • Takeaways & Limitations

    Lazy policy computation and value-aware model fitting address both computational expense and objective mismatch within one MBRL objective.

    Source: §8 · pp. 8–9

    The algorithms are sensitive to exploration-distribution quality, and LAMPS-MM may be difficult to apply when exact policy evaluation is unavailable in large MDPs.

    Source: §6 · p. 6, §8 · p. 9

Abstract

from arXiv · show

We propose a novel approach to addressing two fundamental challenges in Model-based Reinforcement Learning (MBRL): the computational expense of repeatedly finding a good policy in the learned model, and the objective mismatch between model fitting and policy computation. Our "lazy" method leverages a novel unified objective, Performance Difference via Advantage in Model, to capture the performance difference between the learned policy and expert policy under the true dynamics. This objective demonstrates that optimizing the expected policy advantage in the learned model under an exploration distribution is sufficient for policy computation, resulting in a significant boost in computational efficiency compared to traditional planning methods. Additionally, the unified objective uses a value moment matching term for model fitting, which is aligned with the model's usage during policy computation. We present two no-regret algorithms to optimize the proposed objective, and demonstrate their statistical and computational gains compared to existing MBRL methods through simulated benchmarks.

1. Introduction

MBRL reduces real-world interactions but repeatedly planning in learned models is computationally expensive and can exploit model inaccuracies, while prediction-error fitting may misalign with policy performance. The paper proposes a unified lazy objective, two no-regret algorithms, and simulated-benchmark gains in statistical and computational efficiency.

Motivation

  • The method addresses computationally expensive planning and objective mismatch between model fitting and real-world policy performance.Planning may require exponentially many model interactions, while prediction-error minimization need not target transitions important for policy quality.Source: §1 · p. 1

Approach

  • Performance Difference via Advantage in Model unifies policy computation and model fitting around performance-relevant quantities.It replaces full planning with expected policy advantage under an exploration distribution and uses value-aware model fitting.Source: §1 · pp. 1–2
  • The unified objective enables policy computation with polynomial rather than exponential dependence on the task horizon.The policy step optimizes expected advantage in the learned model under an exploration distribution.Source: §1 · p. 1
  • Value moment matching focuses model accuracy on states critical for policy computation while allowing inaccuracy elsewhere.This aligns model fitting with how the learned model is used during policy computation.Source: §1 · p. 2

Contributions

  • Two no-regret algorithms achieve tighter performance bounds than Ross & Bagnell (2012).Source: §1 · p. 2
  • Simulated benchmarks show statistical and computational gains over existing MBRL methods.Source: §1 · p. 2

2. Related Work

Related work establishes MBRL’s theoretical and practical progress, while highlighting reliance on optimal planning, exploration distributions, and value-aware objectives. This paper instead proposes a unified value-aware objective that is feasible to optimize with no-regret algorithms.

Model-based RL

  • Prior MBRL methods commonly compute an optimal policy from the learned model at every iteration, despite computational costs that can be exponential in the horizon.Source: §2 · p. 2

RL with exploration distribution

  • The framework assumes an exploration distribution that covers the expert distribution, rather than requiring an expert-collected offline dataset.Source: §2 · p. 2

Objective Mismatch in MBRL

  • Prior work identifies a mismatch between maximum-likelihood model fitting and finding the optimal real-world policy.Source: §2 · p. 2
  • Existing value-aware approaches target successor values or model advantage, whereas this paper introduces a unified value-aware objective feasible for no-regret optimization.Source: §2 · p. 2

3. Preliminaries

The MBRL framework alternates data collection, model fitting, and policy computation using an exploration distribution. Its central challenges are exponential planning cost and prediction-error objectives that inadequately reflect policy-relevant transition errors.

3. Preliminaries

  • The exploration distribution supports guarantees against policies whose state-action distributions are close to it, with near-optimality when it is close to D_ω,π⋆.Source: §3 · p. 3

3.1. MBRL Framework

  • The framework alternates learned-policy or exploration-distribution rollouts, model fitting, and policy computation across iterations.Source: §3.1 · p. 3, §3 · p. 3
  • Performance difference decomposes into learned-model policy performance and value differences caused by model transitions along learned-policy and expert trajectories.Source: §3.1 · p. 3

3.2. Challenges in MBRL

  • Optimal planning is computationally expensive: in the illustrative MDP, solving for the optimal policy requires O(2^H) operations.Source: §3.2 · p. 4, Figure 1 · p. 4
  • Prediction-error fitting can mis-rank models because it treats transitions broadly rather than according to their value for policy computation.In Figure 1, M_bad has lower prediction error but makes a crucial mistake at state A, whereas M_good is preferred by the performance-based objective.Source: §3.2 · p. 4, Figure 1 · p. 4

4. Performance Difference via Advantage in Model

PDAM decomposes real-world performance difference into model advantage and value discrepancies, yielding a unified objective for joint policy and model learning. It replaces exponential optimal planning with polynomial disadvantage minimization and makes model fitting value-aware.

4. Performance Difference via Advantage in Model

  • PDAM provides a unified objective for joint model and policy learning by decomposing performance difference through advantage in the learned model.Source: §4 · p. 4
  • Policy computation needs only a policy with small disadvantage over the learned-model optimum on expert-trajectory states, rather than optimal planning.With an exploration distribution covering D_ω,π⋆, this computation is polynomial in the effective horizon.Source: §4 · pp. 4–5
  • PDAM requires only the learned policy’s value function, avoiding the unknown expert value function that forced loose prediction-error bounds.Source: §4 · p. 5
  • Value-aware fitting focuses model updates on transitions critical for policy computation instead of requiring uniformly accurate dynamics.Source: §4 · p. 5

5. LAMPS: Lazy Model-based Policy Search

LAMPS replaces optimal planning with policy optimization over an exploration distribution, reducing policy-computation cost while retaining a regret analysis tied to coverage quality.

Policy optimization

  • LAMPS uses one-step cost-sensitive classification on exploration-distribution states instead of optimal planning over all learned-policy states.This makes policy optimization computationally efficient, whereas optimal planning can require exponentially many horizon-dependent interactions.Source: §5 · p. 5

Algorithm

  • LAMPS combines MLE-based model fitting with exploration-distribution policy optimization as a modification of the standard MBRL procedure.The resulting algorithm solves the computational challenge C1 but not the model-fitting objective mismatch C2.Source: §5 · p. 5

Assumptions and guarantees

  • LAMPS requires an exploration distribution ν that covers the expert policy state-action distribution, quantified by coverage coefficient C.Its regret bound therefore makes policy-optimization error sensitive to exploration quality.Source: §5 · pp. 5–6

Analysis

  • The cited analysis defines agnostic model error as the minimum expected KL divergence between candidate and true models, while noting a relevant quantity can reach the effective horizon 1/(1−γ).Source: §5 · p. 5

Statistical implications

  • LAMPS can obtain tighter regret bounds when the learned-policy value range ˆVmax is smaller than Vmax, with such a reduction observed in experiments.The method also exhibits statistical gains in practice, although its primary benefit is computational efficiency.Source: §5 · p. 6

6. LAMPS-MM: Lazy Model-based Policy Search via Value Moment Matching

LAMPS-MM directly optimizes the unified objective through value moment matching, aligning model fitting with policy computation and improving the associated statistical guarantee.

Value moment matching

  • LAMPS-MM uses value moment matching to fit models by matching true and predicted successor values in expectation over transition data.An FTRL procedure with a strongly convex regularizer provides a no-regret guarantee for this loss.Source: §6 · p. 6

Unified objective

  • LAMPS-MM solves both computational planning and objective-mismatch challenges by focusing model accuracy on states where prediction errors create large value differences.This makes the fitted model useful for policy computation rather than merely accurate under generic prediction error.Source: §6 · p. 6

Statistical guarantee

  • The LAMPS-MM regret bound removes dependence on ˆVmax, which can be as large as 1/(1−γ) in the worst case.When the model class contains the true dynamics, the relevant value-moment model error can vanish.Source: §6 · p. 6

Practical limitations

  • LAMPS-MM is practically difficult in large MDPs because estimating its loss requires evaluating the policy in the learned model.The authors identify finite MDPs and settings with closed-form policy values as realizable cases.Source: §6 · p. 6

Implementation

  • Batch optimization requires aggregating data and value functions across iterations, but avoids differentiating through how model changes affect the value estimates.The supplied discussion also reports a finite-sample analysis for LAMPS-MM.Source: §6 · p. 7

7. Experiments

Across five simulated domains, LAMPS and LAMPS-MM improve the statistical and computational efficiency of model-based RL relative to planning- and model-fitting-based baselines.

7. Experiments

  • Experiments compare the proposed algorithms with SYSID, MBPO, and exploration-distribution variants across five varied domains.SYSID uses MLE and optimal planning, while the MBPO variants support fair comparisons involving exploration data.Source: §7 · p. 7

7.1. Helicopter

  • LAMPS learns a better helicopter policy than SYSID with the same real-world data and exploration distribution, while using substantially fewer expensive LQR solver calls.The policy result indicates statistical gains, and the solver-call comparison demonstrates computational gains.Source: §7.1 · p. 7

7.2. WideTree

  • In WideTree, MLE-based SYSID converges to the bad model, whereas value moment matching makes LAMPS-MM select the good model more often.The bad model matches true dynamics except at the root, so value-aware fitting emphasizes the decision-critical discrepancy.Source: §7.2 · p. 7

7.3. Linear Dynamical System

  • In the linear dynamical system, LAMPS-MM quickly finds a value-relevant model and controls that drive the final state to zero, achieving statistical gains over SYSID.SYSID converges slowly because it attempts to match time-varying dynamics at every timestep, despite costs depending on the final state.Source: §7.3 · p. 7

7.4. Mujoco Locomotion Benchmarks

  • LAMPS consistently finds better policies with fewer real-world interactions than MBPO-SYSID across Mujoco environments, while remaining competitive with twice-computation MBPO-SYSID (2X).The advantage is especially pronounced on humanoid, the most difficult evaluated environment.Source: §7.4 · p. 8

7.5. Maze

  • LAMPS significantly outperforms MBPO-SYSID on sparse-reward PointMaze by focusing policy computation along the exploration distribution and avoiding model inaccuracies elsewhere.The comparison uses partial exploration coverage and a small number of expert trajectories.Source: §7.5 · p. 8

8. Discussion

The discussion presents the unified objective and two algorithms as remedies for computational inefficiency and objective mismatch, while noting exploration and scalability constraints.

8. Discussion

  • LAMPS and LAMPS-MM leverage the unified objective to demonstrate statistical and computational gains on simulated benchmarks.The discussion identifies the unified objective as the central contribution and the two algorithms as its practical instantiations.Source: §8 · pp. 8–9
  • LAMPS-MM may be difficult to apply to large MDPs where exact policy evaluation is difficult, and both algorithms depend on exploration distributions covering relevant policies.Small regret is guaranteed only against policies whose state-action distributions are close to the exploration distribution.Source: §8 · p. 9

A.1. Proofs for Section 3.1

  • The simulation lemma relates a policy’s performance under learned and real transition models, supporting the paper’s performance-difference analysis.Its proof uses a recurrence over successive state distributions and extends the relation to infinite horizon.Source: §A.1 · p. 13
  • The appendix introduces PDPM as the performance-difference lemma for planning in the learned model and derives it by combining model-comparison terms with the simulation lemma.The proof adds and subtracts terms, applies the simulation lemma, and completes the bound.Source: §A.1 · p. 13

A.2. Proofs for Section 4

The PDAM proof establishes the exploration-distribution form of the performance-difference bound using the simulation lemma, coverage, and importance sampling.

A.2. Proofs for Section 4

  • The PDAM lemma relates learned-policy performance to expert performance through policy advantage in the learned model and model-transition discrepancies.The proof begins from the performance difference, applies the simulation lemma to one term, and solves a recurrence for another.Source: §A.2 · pp. 14–15
  • The proof completes the infinite-horizon relation by solving the recurrence and combining it with the simulation-lemma result.This closes the derivation of the PDAM guarantee.Source: §A.2 · p. 15
  • The bound extends from expert-distribution expectations to an exploration distribution using the coverage coefficient and importance sampling.The appendix explicitly identifies coverage and importance sampling as the sources of the resulting terms.Source: §A.2 · pp. 15–16

A.3. Proof for Section 5

The appendices derive regret and model-error guarantees for LAMPS and LAMPS-MM, explain their no-regret procedures, and compare squared and signed losses.

A.3. Proof for Section 5

  • The formal results are first stated for expert distributions and then extended to exploration distributions under the paper’s coverage assumptions.The proof appendix repeatedly distinguishes the cleaner expert-distribution presentation from the exploration-distribution extension.Source: §A.3 · pp. 16, 18, §A.4 · p. 18
  • Section 5 states a corollary for any start state distribution ω, policy π⋆, and transition functions ˆM and M⋆, using MLE fitting and extending results to exploration-distribution data.The analysis uses the MLE model loss function and FTL; for T rounds, the cited bound includes an O(log(T)) term.Source: §A.3 · pp. 16–18

A.4. Proof for Section 6

  • The Section 6 proof shows how value-moment model fitting bounds the model-dependent terms in LAMPS-MM’s regret analysis.FTRL is applied to the value-aware losses, with the proof explicitly invoking the loss from the main text.Source: §A.4 · p. 19

B.1. A warm-up argument using Hedge

  • The warm-up Hedge argument illustrates no-regret data aggregation over a finite model class, but computing every model’s loss each round is computationally inefficient.Each model’s loss records whether it makes mistakes on the current and expert trajectory distributions.Source: §B.1 · p. 20

B.2. Comparing with signed loss

  • The alternative signed-loss LAMPS-MM algorithm uses the same order of regret bound as the squared-loss version.The appendix states that both loss functions yield a ˜O(1/√T)-type result.Source: §B.2 · pp. 20–21

B.3. Finite Sample Analysis of LAMPS-MM

The section combines finite-sample analysis with environment specifications and additional experiments for evaluating LAMPS-MM and related methods. It gives an online-to-batch analysis, describes benchmark structure, and records an additional MuJoCo case where exploration quality limits early performance.

B.3. Finite Sample Analysis of LAMPS-MM

  • The resulting bound has a regret term scaling with log(|H|/δ) and factors involving the discount horizon, but its sample complexity does not directly compare with traditional MBRL.The authors state that the bound is instance-dependent through the size of H and implicitly addresses learned-model sampling through the construction of H.Source: §B.3 · pp. 21–22
  • The finite-sample analysis applies online-to-batch conversion to Algorithm 4 using a function class built from the model class, states, actions, and value functions.The analysis defines sampled variables, losses, and martingale differences before applying concentration and conversion results.Source: §B.3 · p. 21, §B.4 · p. 22

B.4. Auxiliary Lemmas

  • The auxiliary lemmas support the proof through Hoeffding-Azuma concentration and online-to-batch conversion for sequential function estimation.The martingale difference sequence is conditionally mean-zero and bounded, enabling the concentration step.Source: §B.3 · p. 22, §B.4 · pp. 22–23

C.1. Environment Details

  • The benchmark suite includes helicopter hover control with a 20-dimensional state and 4-dimensional action space, plus a linear system with time-varying true dynamics and time-invariant models.The linear model class consists of systems of the form {xt+1 = Axt + But} and cannot represent the true dynamics exactly, making it agnostic.Source: §C.1.1 · p. 23, §C.1.3 · pp. 23–24

C.1.1. HELICOPTER

  • In helicopter control, a single backward pass on the desired trajectory replaces multiple iLQR backward passes because policy optimization is restricted to the exploration trajectory.The desired trajectory serves as the exploration distribution, and the policy is optimized only along that trajectory.Source: §C.1.1 · p. 23

C.2. Additional Mujoco Experiment

  • In the additional MuJoCo experiment, LAMPS is initially outperformed by MBPO-SYSID but reaches the same performance with enough data, suggesting sensitivity to exploration-distribution quality.The stated hypothesis is that the exploration distribution is not high quality in this case.Source: §C.2 · p. 25

C.3. Implementation Details and Hyperparameters

The implementation uses shared model-training infrastructure across baselines while assigning LAMPS a distinct policy objective based on disadvantage over the exploration distribution. The appendix also lists the algorithmic loop and benchmark-specific hyperparameter tables.

C.3. Implementation Details and Hyperparameters

  • LAMPS trains models with the same ensemble infrastructure as the baselines but updates the actor by replacing the Q-value with a disadvantage term.The LAMPS actor update is based only on the exploration distribution, whereas MBPO-SYSID uses the MBPO branch update with SAC.Source: §C.3 · p. 25
  • Most baselines use default hyperparameters, while MBPO-SYSID (2X) doubles the hyperparameters marked with an asterisk.The appendix provides separate hyperparameter tables for HalfCheetah, Ant, Hopper, Humanoid, Walker, and PointMaze.Source: §C.3 · pp. 25–26, Table 1 · p. 26, Table 2 · p. 27
  • Deep LAMPS alternates model training, real-environment data collection, model rollouts, soft Q-value updates, and policy updates for a fixed number of iterations.The model is trained using a mixture of exploration and aggregated data, and model rollouts use sampled states and a finite rollout horizon.Source: §C.3 · p. 26

Related papers

Loading 2303.00694v1…