PaperlayerView exact revision on arXiv

Source-linked AI summary

A Game Theoretic Framework for Model Based Reinforcement Learning

Aravind Rajeswaran, Igor Mordatch, Vikash Kumar

arXiv:2004.07804v2cs.LGcs.AIcs.ROstat.ML

TL;DR

MBRL can use off-policy data and priors for sample efficiency, but stable and efficient algorithms with rich function approximators remain challenging. This paper models MBRL as a Stackelberg game between policy and model players, yielding two algorithm families; experiments report better sample efficiency than prior model-based and model-free methods and matching asymptotic performance to model-free counterparts.

  • Problem

    A clear algorithmic framework for understanding MBRL and unifying recent insights is lacking, while stable and efficient algorithm design with rich function approximators remains challenging.

    Source: §1 · p. 1

  • Method

    The framework casts MBRL as a game between a reward-maximizing policy player and a model player that fits policy-collected data, then derives two algorithm families from a Stackelberg game.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    The algorithms are substantially more sample efficient than prior model-based and model-free algorithms and can achieve the same asymptotic performance as model-free counterparts.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The two algorithm families encompass, generalize, and unify a large collection of prior MBRL works.

    Source: §7 · p. 9

    The domain-adaptation term reflects an exploration problem and is unavoidable when seeking globally optimal policies, though locally optimal policies may have better model transfer.

    Source: §3 · p. 3

Abstract

from arXiv · show

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation. Additional details and code can be obtained from the project page at https://sites.google.com/view/mbrl-game

1. Introduction

The paper frames MBRL as a game between policy and model players and uses a Stackelberg formulation to derive two algorithm families. Experiments report sample-efficiency gains, model-free asymptotic performance, and scaling to dexterous manipulation.

1. Introduction

  • Trust-region policy updates and unconstrained updates with data aggregation are presented as dual approaches to the same game.The contrast addresses distribution shift between the data used to learn the model and the policy's visited states.Source: §1 · p. 1
  • The framework models MBRL as a game in which the policy maximizes learned-model rewards and the model minimizes prediction error on policy-collected data.At equilibrium, the policy is near-optimal for the environment.Source: §1 · p. 2
  • A Stackelberg game, solved through approximate bi-level optimization, is used to develop stable and convergent algorithms.The framework uses the ordered decisions of the two players to make algorithm design tractable.Source: §1 · p. 2
  • Choosing either player as leader yields two complementary algorithm families that unify and generalize prior MBRL algorithms.The leader moves first, followed by the follower.Source: §1 · p. 2
  • The proposed algorithms outperform prior model-based and model-free methods in sample efficiency, match model-free policy-gradient asymptotic performance, and scale to dexterous manipulation.The reported results span sample efficiency, asymptotic performance, and high-dimensional tasks.Source: §1 · p. 2

2. Background and Notations

The section sets up MBRL against an infinite-horizon MDP and contrasts direct policy optimization with using an explicit learned world model. It explains why incremental, task-relevant model learning is useful and how model quality depends on the data distribution.

2. Background and Notations

  • The environment is an infinite-horizon MDP with state and action spaces, transition dynamics, reward, discount, and initial-state distribution.Model-free methods estimate policy gradients or use value functions, whereas model-based methods build an explicit world model.Source: §2 · p. 2

2.1. Model-Based Reinforcement Learning

  • The world model parameterizes transition dynamics as a neural network approximating the environment, while assuming reward and initial-state distribution are known.The model can generate experience for policy optimization without additional world interaction.Source: §2.1 · p. 2
  • With a sufficiently expressive model that approximates the environment everywhere, a uniform performance bound supports optimizing the policy in the model.The section introduces this as an idealized global-model setting.Source: §2.1 · p. 2
  • Global model accuracy can be impractical or unnecessary, motivating incremental refinement in task-relevant regions; model approximation loss depends on the sampling distribution.For isotropic Gaussian distributions, the KL loss reduces to the familiar L2 loss.Source: §2.1 · p. 2

3. Model Based RL as a Two Player Game

The MBRL game couples policy optimization in the learned model with model fitting on policy-induced data. Its approximate equilibria connect model accuracy and policy quality, while the bound includes a transfer term tied to exploration.

3. Model Based RL as a Two Player Game

  • The policy player maximizes performance in the learned model, while the model player minimizes prediction error under the policy-induced state-action distribution.Each player's objective depends on both players' parameters.Source: §3 · p. 3
  • At an approximate Nash equilibrium, the model can simulate and predict policy performance accurately, and the policy is near-optimal.The formulation separates planning and model learning while showing they must be considered together.Source: §3 · p. 3
  • The equilibrium bound includes planning and model-learning suboptimality, which can be reduced with more compute and data given sufficient capacity, plus a domain-adaptation term.The domain-adaptation term captures equilibrium quality and reflects an exploration problem; it is unavoidable when seeking globally optimal policies, while locally optimal policies may have better model transfer.Source: §3 · p. 3
  • A wide initial-state distribution can reduce the domain-adaptation term, while learned state-action representations can support transfer when the initial distribution is not controllable.The section connects these options to domain adaptation methods.Source: §3 · p. 3

4. Algorithms

The section contrasts simultaneous independent learning with a Stackelberg approach that makes one player the leader and solves the coupled problem through approximate bi-level optimization. Choosing the policy or model as leader yields complementary algorithms, PAL and MAL, and connects the framework to robustness heuristics.

4.1. Independent simultaneous learners

  • Independent simultaneous learners update both players without accounting for objective drift, and GDA can converge poorly or use data inefficiently.BR instead approximates each player's best response with many gradient steps, but simultaneous aggressive changes can destabilize learning through distribution mismatch.Source: §4.1 · pp. 3–4

4.2. Stackelberg formulation and algorithms

  • Stackelberg games turn the coupled MBRL problem into bi-level optimization, with the follower responding to the leader and the leader accounting for that response.The framework uses gradient-based updates and approximates follower best responses with multiple optimization steps, dropping the implicit Jacobian term.Source: §4.2 · p. 4
  • PAL makes the policy the leader: it first improves the model on the current visitation distribution, then conservatively improves the policy using that updated model.The policy update can use a conservative algorithm such as NPG or TRPO.Source: §4.2 · pp. 4–5
  • MAL makes the model the leader: it first optimizes the policy for the current model, then conservatively updates the model using data collected by that policy.In practice, MAL can aggregate historical data and train for a few epochs.Source: §4.2 · p. 5
  • The Stackelberg formulation gives a principled account of robustness heuristics, including model ensembles in PAL and entropy regularization or disagreement bonuses in MAL.These approaches correspond to adversarial choices among model or policy best responses.Source: §4.2 · p. 5

5. Experiments

Across the evaluated tasks, PAL and MAL learn stably and sample efficiently, with reported advantages over prior model-based and model-free methods. Their relative strengths differ under dynamics changes versus goal-distribution changes.

Task Suite and Algorithm Details

  • The evaluation uses continuous-control robot tasks, including a 24DOF dexterous hand task, and OpenAI Gym benchmarks; task goals are randomized each episode.The robot-task evaluations use success rate, and the algorithms use neural-network policies and dynamics models with model-based natural policy gradient.Source: §5 · pp. 5–6, Figure 2 · p. 6

Comparison of learning algorithms

  • PAL and MAL show near-monotonic learning on the control tasks, while GDA learns considerably more slowly and BR fails to make consistent progress.The experiments also report that PAL learns faster than MAL on the studied tasks; GDA eventually approaches a near-100% success rate, but learning is slower and fluctuates.Source: §5 · pp. 6–7, Figure 1 · p. 6
  • On OpenAI Gym, PAL and MAL are 10 times as efficient as other model-based and model-free methods, while PAL is twice as efficient as MBPO.The paper also reports that PAL and MAL substantially outperform baselines on these tasks.Source: Figure 3 · p. 7, §5 · p. 7
  • PAL and MAL match the asymptotic performance of model-free counterparts and scale to dexterous manipulation and extended-horizon tasks.The reported results also describe them as substantially more sample efficient than prior model-based and model-free algorithms.Source: §5 · p. 7

Choosing between PAL and MAL

  • PAL recovers faster after dynamics perturbations, whereas MAL adapts faster when the goal distribution changes.PAL's aggressive recent-data model learning can discard inconsistent old data; MAL's broadly accurate model supports adaptation to new goals, while the policy requires additional learning rather than zero-shot transfer.Source: §5 · pp. 7–8, Figure 4 · p. 7

6. Related Work

Related work spans model-based control and system identification, trust-region-like approaches, and conservative model updates paired with aggressive policy optimization. The paper positions PAL and MAL as a unifying framework and distinguishes rollout-based policy improvement from other uses of learned models.

Related work

  • Earlier MBRL work used tabular settings or assumed exploratory data collection, while later incremental methods applied rich function approximation to practical tasks.Tabular representations are described as inadequate for modern applications such as robotics.Source: §6 · p. 8
  • Trust-region-inspired approaches build a local model around the current policy's visitation distribution and conservatively improve the policy; some use time-varying linear models and KL constraints.The paper notes that these model representations may restrict application to complex dynamics beyond trajectory-centric reinforcement learning.Source: §6 · p. 8
  • MAL-like approaches conservatively update models through data aggregation while aggressively optimizing policies, with later work adding methods such as MPC and model ensembles.Prior work describes data aggregation as supporting no-regret model learning even with aggressive policy optimization.Source: §6 · p. 8
  • The paper frames PAL and MAL as a unified, generalized account of prior approaches, connecting them to multiple-timescale algorithms previously studied in actor-critic learning and GANs.The authors state that those ideas had not been extended to study model-based reinforcement learning.Source: §6 · p. 8
  • Unlike Dyna, MBPO, and related methods that use synthetic trajectories for learning targets or off-policy evaluation, this work uses model rollouts for policy improvement.The authors report successful rollouts of hundreds of steps and leave hybrid extensions and other rollout improvements for future work.Source: §6 · p. 8

7. Summary and Conclusion

The paper frames MBRL as a game between policy and model players and develops Stackelberg-based algorithm families. Its conclusion reports sample-efficient learning, model-free asymptotic performance, and scaling to challenging tasks.

7. Summary and Conclusion

  • At equilibrium, the model accurately simulates the policy and predicts its performance, while the policy is near-optimal.The paper also derives sub-optimality bounds and connects equilibrium quality to domain adaptation.Source: §7 · pp. 8–9
  • Choosing the Stackelberg leader yields two algorithm families, PAL and MAL, which encompass, generalize, and unify many prior MBRL works.The framework uses approximate bi-level optimization and supplies an objective for tracking progress toward equilibrium.Source: §7 · p. 9
  • PAL and MAL are substantially more sample efficient than prior model-based and model-free algorithms, match model-free asymptotic performance, and scale to dexterous and long-horizon tasks.The reported task scope includes OpenAI Gym horizons of 1,000 timesteps.Source: §7 · p. 9
  • The authors leave alternate Stackelberg solvers, including full implicit gradients and unrolled optimization, and broader applications to surrogate-based optimization for future study.They identify actor-critic methods as one possible broader application.Source: §7 · p. 9

A.1. Performance with Global Models

  • The appendix presents formal theory and proofs; its global-model simulation lemma is included for completeness and to motivate techniques for the main theorem.The passage notes that this lemma, or closely related forms, was proved in prior work.Source: A. Theory · p. 12, §A.1 · pp. 12–13

A.2. Performance with Task-Driven Local Models

The theory bounds performance for a policy-model pair near equilibrium and distinguishes planning error from transfer error. The transfer term captures how well a model trained under one policy applies to another.

A.2. Performance with Task-Driven Local Models

  • The model-error performance lemma bounds the performance difference for the same policy evaluated in two models, unlike a lemma comparing different policies in one model.The comparison is conditioned on assumptions about the models' transition dynamics and error.Source: §A.2 · pp. 14–15
  • Theorem 1 bounds the performance gap to an optimal policy for a policy-model pair satisfying the stated approximate-equilibrium conditions.The bound is assembled from contributions including planning sub-optimality and model error under a different policy's distribution.Source: §A.2 · p. 15
  • The performance decomposition includes planning sub-optimality and a transfer term for a model with low error under the current policy but evaluated under an optimal policy's distribution.The approximate-equilibrium condition bounds the planning term by ϵπ.Source: §A.2 · p. 15
  • Without additional assumptions, the transfer-learning term cannot be bounded more tightly.The authors state that the term nevertheless captures the spirit of the transfer-learning issue.Source: §A.2 · p. 16
  • Among models with similar low error under the current policy, the framework favors those able to simulate a wider class of policies.A broad start-state distribution, potentially constructed by briefly executing an exploratory policy, may improve transfer between policies.Source: §A.2 · p. 16
  • The theorem assumes model error is small at each timestep, a condition described as slightly stronger than requiring small average error.The text notes that time-indexed trajectories can be used to check timestep-specific error in practice.Source: §A.2 · p. 16

B. Algorithm Implementation Details and Experiments

The implementations pair ensemble dynamics learning with model-based natural policy-gradient updates. PAL and MAL differ in whether policy or model improvement leads each iteration.

B. Algorithm Implementation Details and Experiments

  • Dynamics models are trained on state-transition data to minimize one-step prediction error, using ensembles and normalized, mean-centered state differences.The implementation uses Adam for model learning; ensembles capture epistemic uncertainty and provide robustness for policy optimization.Source: B. Algorithm Implementation Details… · pp. 17–18
  • Model-based NPG uses synthetic rollouts, a reward function, and a value baseline with GAE to compute policy updates.Rollouts may start from initial states or intermediate real-rollout states; termination conditions are enforced when applicable.Source: B. Algorithm Implementation Details… · p. 17
  • PAL alternates model learning, conservative policy improvement, and fresh data collection, retaining a small replay buffer focused on current visitation.Its practical algorithm uses roughly one NPG step per iteration; the buffer discards older data when it exceeds its size limit.Source: B. Algorithm Implementation Details… · p. 18
  • PAL hyperparameters were selected through a coarse DClaw search; four NPG updates worked best, while the hand task used 10×horizon samples per iteration.Five times the environment horizon was selected for DClaw, DKitty, and Reacher; the hand setting was chosen for more stable results.Source: B. Algorithm Implementation Details… · p. 18, Table 2 · p. 19
  • MAL optimizes the policy for the current model before collecting data and retraining the model on the aggregated buffer.The practical configuration uses many NPG steps; K=25 was a performance/computation trade-off, with 20×horizon samples for several tasks and 3000 samples for OpenAI gym tasks.Source: B. Algorithm Implementation Details… · p. 19, Table 3 · p. 19

B.1. Task Suite

The evaluated task suite includes four control tasks, with descriptions provided for DClaw-Turn and DKitty-Orient.

B.1. Task Suite

  • The suite includes DClaw-Turn, DKitty-Orient, 7DOF-Reacher, and InHand-Pen.The latter two are named in the task list but not described in these passages.Source: §B.1 · p. 20
  • DClaw-Turn asks a three-fingered robot to rotate a faucet toward a desired orientation, with reward based on configuration closeness.Observations include claw and faucet joint positions and velocities, plus the desired valve orientation.Source: §B.1 · p. 20
  • DKitty-Orient asks a quadruped to face a desired direction, with reward based on the difference between its current and desired pose.Observations include robot joint poses and velocities and the desired orientation.Source: §B.1 · p. 20

B.2. Results on OpenAI gym benchmarks

On OpenAI gym benchmarks, PAL-NPG and MAL-NPG substantially outperform the compared baselines in sample efficiency. They also use fully model-based implementations and report long rollouts without error amplification.

B.2. Results on OpenAI gym benchmarks

  • PAL and MAL are nearly twice as sample efficient as MBPO on InvertedPendulum and Hopper, and nearly 10× as efficient as other baselines.The benchmarks include InvertedPendulum-v2, Hopper-v2, and Ant-v2; baselines include MBPO, PETS, STEVE, SLBO, and SAC.Source: §B.2 · p. 21
  • For Hopper and Ant, center-of-mass velocity is included in observations so rewards can be computed for synthetic rollouts.The passage notes similar approaches in prior work, including SLBO.Source: §B.2 · p. 21
  • The authors report hundreds-step rollouts without error amplification, which they associate with controlling distribution shift through the Stackelberg formulation.They contrast their fully model-based PAL and MAL implementations with hybrid MBPO, which combines model-based and model-free components.Source: §B.2 · p. 21

B.3. Model Error Amplification

The error-amplification analysis compares model predictions with environment outcomes across training stages and rollout modes. Errors generally fall with training, and policy feedback is associated with smaller errors and more predictable long-term outcomes.

B.3. Model Error Amplification

  • Prediction errors decrease with more training, and closed-loop errors are smaller than open-loop errors.Figure 6 tracks valve-orientation prediction error in radians across look-ahead horizons and training stages; the caption notes long-term error of about 0.1 radians.Source: Figure 6 · p. 22, §B.3 · p. 22
  • Open-loop evaluation reuses actions from an environment trajectory, whereas closed-loop evaluation lets the policy act on each model state.The analysis uses look-ahead horizon to examine model error relevant to policy improvement rather than relying only on one-step generalization error.Source: §B.3 · pp. 21–22
  • PAL has lower error early in training, but MAL has lower error later as its model quality improves faster.The authors suggest MAL's larger, more diverse replay buffer may explain the later advantage.Source: §B.3 · p. 22, Figure 6 · p. 22
  • Error need not rise with horizon: it can shrink near the end as the improving policy more often reaches the desired faucet configuration.The authors interpret this pattern as evidence that long-term policy consequences are more predictable than intermediate transients.Source: §B.3 · p. 22, Figure 6 · p. 22

Related papers

Loading 2004.07804v2…