PaperlayerView exact revision on arXiv

Source-linked AI summary

CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning

Pietro Noah Crestaz, Mohamed Yassine Kabouri, Nicolas Mansard, Andrea Del Prete

arXiv:2609.08853v1cs.RO

TL;DR

Planning-based model-based RL can improve on learned-policy action selection, but recent methods combining planners and policies typically train critics to value the policy rather than the stronger planner-guided behavior. CAST uses a state-value critic trained with alternating real behavior and imagined policy transitions. The authors report strong sample efficiency and competitive final performance on continuous-control tasks, as well as deployment on a Unitree Go2.

  • Problem

    Methods combining online planning with learned policies typically train critics to value the learned policy rather than the stronger planner-guided behavior.

    Source: §I · p. 1

  • Method

    CAST replaces the action-value critic with a state-value critic trained on a real planner-behavior transition followed by an imagined transition under the current policy.

    Source: §I · p. 2, C. A Hybrid Value Target: Mixing… · p. 4

  • Results

    CAST shows strong sample efficiency and competitive final performance against established model-free and model-based baselines on high-dimensional continuous-control tasks, and is demonstrated on a Unitree Go2.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The hybrid target lets the critic draw on planner behavior while regularizing its value estimate with the current policy.

    Source: §I · p. 1

Abstract

from arXiv · show

Model-based reinforcement learning (MBRL) is a family of RL methods that learn a model of the environment and use it for action selection, making it well suited to robotics due to its sample efficiency. Combining learned models with online planning can further improve action selection, as the planner can exploit the model to find better actions than the learned policy alone. Recent methods combining learned policies with online planning typically learn the value of the policy rather than the stronger planner-guided behavior. We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy. CAST replaces the action-value critic with a state-value critic, trained using a target that combines a real planner-guided transition and an imagined transition under the current policy. The resulting value function corresponds to an alternating process between planner-guided behavior and the current policy, allowing it to benefit from the stronger planner behavior while being regularised by the policy being learned. We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2 quadruped performing a dynamic handstand.

I. INTRODUCTION

Model-based RL uses learned environment models for planning and policy optimization, but methods combining planning with learned policies typically train the critic to value the policy rather than the stronger planner-guided behavior. CAST instead learns a state-value critic from a hybrid target, retaining TD-MPC2’s world model and planner while adding model-based policy improvement.

I. INTRODUCTION

  • Planning-based MBRL can improve action selection beyond the learned policy alone, while reducing the real-world interactions required relative to model-free RL.Online trajectory optimization evaluates and refines candidate action sequences using a learned model,.Source: §I · p. 1

II. RELATED WORK

  • Prior methods learn the value of πθ and address its mismatch with planner behavior β by aligning the policy or using behavior cloning,.Other approaches combine latent world models, MPPI planning, and an action-value critic,.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • CAST combines a behavior-policy transition with a policy-imagined transition to regularize value learning, and adds a k-step model-based policy-improvement objective.The authors report validation against five representative RL baselines on 14 high-dimensional control tasks and deployment on a Unitree Go2.Source: §I · pp. 1–2

II. RELATED WORK

  • CAST aims to value β directly rather than align πθ with β or imitate the planner’s actions, because β generally outperforms πθ [10], [12].Bootstrapping policy improvement directly from Vβ performs poorly in practice, motivating a regularized hybrid Bellman target [13].Source: §II · p. 2
  • CAST propagates policy gradients through imagined transitions before bootstrapping with its state-value critic, combining imagination-based policy optimization with online planning,,.Unlike the cited imagination-based approaches, online planning is central to the TD-MPC2 interaction setup [9], [10].Source: §II · p. 2
  • A state-value critic is cheaper than learning Qβ, which would require exploring state and action spaces and querying the planner to construct TD targets.MODIP [13] also uses terminal state value to avoid querying a policy during planning, but CAST’s motivation is to learn β’s value [13].Source: §II · p. 2

III. METHOD

TD-MPC2 jointly learns a latent world model and policy, then uses online planning to select actions. Its replay transitions come from the planner-augmented behavior policy, while its critic target bootstraps using the learned policy.

A. Background: TD-MPC2

  • TD-MPC2 encodes observations as latent states and predicts latent dynamics and rewards without decoding back to observation space.It jointly learns a latent world model and policy.Source: A. Background: TD-MPC2 · p. 3
  • The method uses an ensemble of action-value functions and a stochastic maximum-entropy Gaussian policy, with reward and value targets represented as distributions over symlog-spaced bins.The latent state uses SimNorm, and a target ensemble is maintained with Polyak averaging.Source: A. Background: TD-MPC2 · p. 3
  • TD-MPC2 trains its model components from replay trajectories with multi-step losses and a one-step action-value TD target.The target uses a stop-gradient operator, discounts errors along the rollout, and applies cross-entropy against a two-hot target.Source: A. Background: TD-MPC2 · p. 3
  • At each control step, MPPI plans over model-generated action sequences, while πθ supplies a prior for some candidate rollouts and is separately trained with a maximum-entropy objective.The planner scores sequences using a terminal policy action and executes the first action of the selected sequence.Source: A. Background: TD-MPC2 · p. 3

B. From Action-Values to State-Values

CAST replaces TD-MPC2’s action-value critic with an ensemble state-value critic. It reconstructs action-value estimates when needed and uses state value directly as the planner’s terminal estimate.

B. From Action-Values to State-Values

  • CAST learns an ensemble state-value critic and maintains a target ensemble using Polyak averaging.The state-value and action-value functions are related through the expected value of actions under the stochastic policy.Source: B. From Action-Values to State-Values · p. 3
  • When an action-value estimate is needed, CAST reconstructs a one-step estimate from the learned reward and dynamics models and the target state-value function.This composed estimate supports both the value-learning target and policy objective.Source: B. From Action-Values to State-Values · pp. 3–4
  • For planning, CAST uses the state-value estimate at the terminal state instead of sampling a terminal policy action and evaluating an action-value estimate.This removes the need to sample a terminal policy action during planning.Source: B. From Action-Values to State-Values · p. 4

C. A Hybrid Value Target: Mixing Behavior and Policy

CAST trains its critic with a real replay transition generated by planner behavior followed by an imagined transition under the current policy. This hybrid target is intended to retain the stronger behavior policy’s value while regularizing against destabilizing updates.

C. A Hybrid Value Target: Mixing Behavior and Policy

  • Bootstrapping directly from the planner behavior policy’s value performs poorly because it leads to abrupt policy updates that destabilize training.CAST therefore combines a real transition generated by β with an imagined transition generated by πθ.Source: C. A Hybrid Value Target: Mixing… · p. 4
  • The target starts with a replay transition, then uses πθ to generate the next action and learned dynamics to generate the subsequent latent state.Its alternating sequence is one real transition under β followed by one imagined transition under πθ.Source: C. A Hybrid Value Target: Mixing… · p. 4
  • The resulting critic represents neither β nor πθ alone, but their alternating composition.The replay step anchors the target to β, while the imagined policy step reintroduces πθ and prevents the target from collapsing entirely onto β.Source: C. A Hybrid Value Target: Mixing… · p. 4

D. Theoretical Analysis of Hybrid Value Targets

The analysis characterizes CAST’s hybrid value target as an alternating process between behavior-policy and learned-policy rewards. Under stated assumptions, its operator is contractive and has a unique fixed point.

D. Theoretical Analysis of Hybrid Value Targets

E. Expanded Policy Gradient Through the World Model

CAST replaces the action-value policy-improvement estimate with a model-based estimate that unrolls the current policy through learned dynamics before bootstrapping from the value critic. The primary configuration uses a one-step horizon.

E. Expanded Policy Gradient Through the World Model

F. Training Algorithm

CAST leaves data collection unchanged and makes its changes in the critic input, value target, and resulting policy objective.

F. Training Algorithm

IV. EXPERIMENTS

The experiments assess whether alternating state-value targets improve sample efficiency without sacrificing asymptotic performance, and examine the contribution of CAST’s components.

IV. EXPERIMENTS

  • The experiments ask whether alternating state-value targets improve sample efficiency.Source: §IV · p. 5
  • They test whether any sample-efficiency improvement preserves strong asymptotic performance.Source: §IV · p. 5
  • They also examine each CAST component’s contribution to overall improvement.Source: §IV · p. 5

A. Experimental Setup

CAST is evaluated on seven DeepMind Control and seven HumanoidBench tasks against five RL baselines. Across the 14 tasks, it reaches high returns earlier and achieves the best average AUC performance profile.

A. Experimental Setup

  • Evaluation covers seven DeepMind Control and seven HumanoidBench tasks, comparing CAST with five representative model-free and model-based RL algorithms.The primary comparisons are TD-MPC2, BMPC, and BOOM, which share its latent world model and planning framework.Source: A. Experimental Setup · p. 5
  • Unless otherwise stated, experiments use CAST’s alternating state-value target and one-step expanded policy gradient, with three random seeds and 95% confidence intervals.The confidence intervals are computed as mean±1.96 std-dev/√n.Source: A. Experimental Setup · p. 5

B. Sample efficiency

  • CAST matches or exceeds selected baselines on most environments and reaches high returns earlier, especially on several HumanoidBench tasks.The reported examples include H1hand-run, H1hand-pole, and Dog-run; similar trends appear across DMControl.Source: B. Sample efficiency · p. 5, Figure 3 · p. 6
  • CAST achieves the best average AUC performance profile among baselines across the 14 tasks.Fig. 4 pools the tasks and also reports each suite separately.Source: B. Sample efficiency · p. 6, Figure 4 · p. 6
  • On DMControl, CAST and BMPC remain at 1.0 through τ≈0.9; on HumanoidBench, CAST remains at 1.0 through τ≈0.85.On HumanoidBench, BMPC’s curve falls off earliest among the three planning-based methods discussed.Source: B. Sample efficiency · p. 6

C. Final Performance Comparison

CAST leads the benchmark at 1M interactions and remains competitive at 2M. Ablations favor the hybrid value target and show task-dependent effects from changing the behavior-to-policy ratio, while the one-step gradient horizon holds a small lead.

D. Ablation Study

  • On Dog-run and Humanoid-run, the purely off-policy target is consistently worst, while the purely on-policy target also lags the hybrid target.Hπ = 1 is held fixed in this value-target ablation.Source: D. Ablation Study · p. 6, Figure 5 · p. 7

C. Final Performance Comparison

  • At 1M interactions, CAST has the highest average benchmark performance, outperforming BMPC and BOOM; at 2M, it remains competitive, beating BMPC and matching BOOM.The 1M results also include the best results on several HumanoidBench environments.Source: C. Final Performance Comparison · p. 6, Table I · p. 7

D. Ablation Study

  • With HV = 2 or 3, performance remains competitive with HV = 1 on Dog-run, but both alternatives lag on Humanoid-run.The comparison varies the number of real β transitions before a single imagined πθ step.Source: D. Ablation Study · p. 7, Figure 6 · p. 7
  • On both tasks, Hπ = 2 and Hπ = 3 track each other closely and follow Hπ = 1, which holds a small lead with smaller confidence intervals.The hybrid value target is fixed across this gradient-horizon ablation.Source: D. Ablation Study · p. 7, Figure 7 · p. 7

E. Real-World Deployment

A simulation-trained CAST policy transfers to a Unitree Go2 and performs a dynamic handstand without online planning. The conclusion also reports competitive control performance and describes asynchronous predictive MPC as hiding planning latency without additional supervision.

E. Real-World Deployment

  • A policy trained entirely in simulation transfers to the Unitree Go2 and executes stable dynamic handstand motions in real time.The deployment uses domain randomization and a curriculum, and is presented as a proof of deployment rather than a comparative evaluation.Source: E. Real-World Deployment · p. 7, Figure 8 · p. 8

V. CONCLUSION

  • CAST combines a state-value critic trained toward β with a regularized hybrid Bellman target mixing real β transitions and imagined πθ transitions.The conclusion describes the target as having a characterized fixed point.Source: §V · p. 8
  • Across high-dimensional continuous-control tasks, CAST shows strong sample efficiency and competitive final performance against established model-free and model-based baselines.The Go2 demonstration uses asynchronous predictive MPC to hide planning latency from the control loop without additional supervision.Source: §V · p. 8

Related papers

Loading 2609.08853v1…