PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models

Kowndinya Boyalakuntla, Yuhan Liu, Abdeslam Boularias

arXiv:2609.39751v1cs.RO

TL;DR

Learned-world-model control couples planning with learning, but policy alignment addresses only part of that loop. PL-MPC modifies critic targets, planner terminal-value evaluation, and policy distillation; its largest HumanoidBench gains are task dependent, and it also shows higher observed success in zero-shot real-robot transfer.

  • Problem

    In learned-world-model control, planner-generated experience and learned critic and actor outputs feed back into one another, while policy alignment addresses only the planner–policy interface.

    Source: §I · p. 1

  • Method

    PL-MPC adds hybrid multi-step critic targets, disagreement-aware penalties on planner terminal values, and return-weighted distillation of planner-executed actions.

    Source: §I · pp. 1–2

  • Results

    PL-MPC’s largest HumanoidBench gains occur on balance-hard and hurdle, broader performance is task dependent, and it remains competitive on DMControl.

    Source: B. Benchmark performance · p. 5, §VII · p. 8

  • Takeaways & Limitations

    The approach modifies three interfaces in the planning–learning loop without changing the underlying world-model architecture or MPPI optimizer.

    Source: §I · p. 2

    Performance remains task dependent; on maze, useful planner trajectories remain rare, leaving return-weighted distillation with little signal to amplify.

    Source: §VII · p. 8

Abstract

from arXiv · show

Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on \texttt{balance-hard}, where Total Average Return (TAR) increases from $98\pm18$ to $387\pm255$, and \texttt{hurdle}, from $199\pm13$ to $466\pm200$; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.

I. INTRODUCTION

PL-MPC extends policy-constrained TD-MPC by modifying critic supervision, planner terminal-value evaluation, and planner-to-policy transfer. Its largest reported HumanoidBench gains are task dependent, and the paper also evaluates zero-shot deployment on a real robot.

I. INTRODUCTION

  • Planning and learning form a feedback loop: the planner generates experience for learning, while the learned critic and actor score and propose future plans.The paper identifies critic supervision, terminal-value evaluation, and planner-to-policy transfer as interfaces that can affect this loop.Source: §I · p. 1
  • PL-MPC modifies critic supervision with hybrid multi-step targets, penalizes uncertain terminal values during planning, and weights planner-action distillation by realized episode return.The latent world model and MPPI optimization procedure remain unchanged.Source: §I · pp. 1–2
  • On balance-hard, Total Average Return (TAR) rises from 98 ± 18 to 387 ± 255, and on hurdle from 199 ± 13 to 466 ± 200, gains over TD-M(PC)2; broader benchmark performance remains task dependent.Controlled ablations reveal different component interactions across the two tasks.Source: §I · p. 2
  • PL-MPC achieves higher observed success than TD-M(PC)2 on the training object size and two unseen sizes in zero-shot deployment on a 7-DoF KUKA IIWA14.Policies are trained entirely in Isaac Gym and deployed without real-world fine-tuning.Source: Figure 1 · p. 1, §I · p. 2

A. Planning with learned models and value expansion.

The background describes learned-model planning and value expansion, planner–policy alignment methods, and outcome-weighted learning under uncertainty. It also introduces the latent world-model components used by TD-MPC.

A. Planning with learned models and value expansion

  • MTD uses planner-generated replay rewards where available and adds learned-model predictions only when the target extends beyond the sampled replay slice.Prior work reports diminishing gains from longer value-expansion horizons even with oracle dynamics, and comparable model-free value expansion in its experiments.Source: A. Planning with learned models and… · p. 2

B. Planner–policy alignment

  • TD-M(PC)2, BMPC, BOOM, and PO-MPC address planner–policy mismatch through distinct forms of constraining, imitating, or regularizing policy behavior.PL-MPC builds on TD-M(PC)2 and additionally changes critic supervision and the terminal values used to rank MPPI trajectories.Source: B. Planner–policy alignment · p. 2

C. Outcome-weighted policy learning and uncertainty

  • RAD weights imitation of planner-executed replay actions by realized episode return, while ATE penalizes terminal critic values rather than changing the TD regression target.The passage contrasts ATE with STEVE, which uses uncertainty to weight value-expansion horizons.Source: C. Outcome-weighted policy learning… · p. 2

A. Control objective and latent world model

  • TD-MPC learns a control-oriented latent world model with an encoder, dynamics model, and reward model, alongside an actor and an ensemble of action-value functions.These components are trained for control rather than observation reconstruction.Source: A. Control objective and latent world… · p. 2

B. Planning and temporal-difference learning

TD-MPC uses MPPI to optimize candidate action sequences with learned-model rollouts and terminal values. Replay trains the critic, while online critics score planner rollouts and target critics provide TD bootstraps.

B. Planning and temporal-difference learning

  • MPPI scores candidate action sequences using predicted rewards and a learned terminal value, refines its sampling distribution, and executes only the first optimized action before replanning.The actor supplies proposal trajectories for planning.Source: B. Planning and temporal-difference… · pp. 2–3
  • The baseline terminal value averages decoded values from two randomly sampled online critic heads, while the random-min target uses the smaller value from two target-critic heads.Target critics provide the TD bootstrap; online critics provide the planner terminal score.Source: B. Planning and temporal-difference… · p. 3
  • Replay supplies planner-selected actions and training transitions, while the learned world model and online critic provide rollout predictions and terminal scores to the planner.The continuation action in the one-step TD target is sampled from the actor.Source: B. Planning and temporal-difference… · p. 3

C. Policy-constrained TD-MPC

The policy-constrained TD-MPC backbone steers actor updates toward planner behavior stored in replay. PL-MPC retains this backbone while modifying other planning–learning interfaces.

C. Policy-constrained TD-MPC

  • TD-M(PC)2 constrains actor updates toward the planner distribution stored in replay.Its actor objective combines online-critic value maximization, entropy regularization, and a planner-prior term based on replay proposals.Source: C. Policy-constrained TD-MPC · p. 3

IV. METHOD

  • PL-MPC changes the critic target, MPPI terminal value, and actor objective while leaving the latent world model, critic-ensemble architecture, and MPPI optimization procedure unchanged.The training algorithm applies ATE during environment interaction and computes multi-step targets from replay slices.Source: §IV · pp. 3–4
  • After episodes finish, training associates realized returns with transitions and uses return-weighted distillation after warmup when return statistics are available.The actor is updated with the backbone objective and, when activated, the distillation loss; target parameters are Polyak-updated.Source: §IV · p. 4

A. Hybrid multi-step TD targets (MTD)

MTD builds critic targets from observed planner rewards before bootstrapping, using model predictions only when the target extends beyond the sampled replay slice.

A. Hybrid multi-step TD targets (MTD)

  • MTD includes observed planner rewards before the terminal bootstrap, allowing later realized rewards to enter earlier critic targets directly.This differs from one-step TD, where later rewards affect earlier values only through subsequent updates.Source: A. Hybrid multi-step TD targets (MTD) · p. 4
  • Rewards within the sampled replay slice are observed; rewards and latent states beyond it are generated by rolling learned dynamics and reward models under the current actor.Source: A. Hybrid multi-step TD targets (MTD) · p. 4
  • For the default n = H = 3, replay positions 0, 1, and 2 contain (3, 0), (2, 1), and (1, 2) observed and model-predicted rewards, respectively.Source: A. Hybrid multi-step TD targets (MTD) · p. 4
  • The stop-gradient target updates critic predictions without propagating gradients through the actor, target critic, or world-model computations, and its observed prefix remains off-policy.The observed prefix follows historical planner actions, while the predicted tail and terminal bootstrap follow the current actor.Source: A. Hybrid multi-step TD targets (MTD) · p. 4

B. Adaptive terminal estimates (ATE)

ATE adjusts MPPI terminal scoring using target-critic disagreement, while RAD adds return-weighted imitation of planner-executed actions from replay.

B. Adaptive terminal estimates (ATE)

V ATE

  • ATE modifies MPPI trajectory scoring only; ensemble disagreement is an uncertainty proxy, not a calibrated estimate of critic error.It does not alter the TD target.Source: V ATE · p. 4

C. Return-weighted actor distillation (RAD)

V. EXPERIMENTS

The experiments evaluate PL-MPC on HumanoidBench and DMControl, then examine component effects, RAD activation timing, and physical deployment.

V. EXPERIMENTS

  • PL-MPC is evaluated on thirteen HumanoidBench locomotion tasks and four high-dimensional DMControl tasks.Source: §V · p. 4
  • The benchmark figure reports Total Average Return (TAR) as mean±std across seeds, distinct from the environment-defined success metric.Source: Figure 3 · p. 5
  • The experiments test whether changes to value learning, MPC terminal-value estimation, and policy distillation improve tasks where the TD-M(PC)2 backbone remains weak.The world-model architecture and MPPI planner are unchanged.Source: §V · p. 5

A. Experimental setup

PL-MPC improves mean TAR on several HumanoidBench tasks, with its largest gains on balance-hard and hurdle, while results remain task dependent and variable across seeds.

A. Experimental setup

  • HumanoidBench comparisons use three million locally trained environment steps for PL-MPC and several baselines, while DreamerV3 and TD-MPC2 use released two-million-step logs.Scores are means over each seed’s final five evaluations; reported table values are mean±std across seeds.Source: A. Experimental setup · p. 5

B. Benchmark performance

  • TAR increases on balance-hard from 98±18 to 387± 255 and on hurdle from 199 ± 13 to 466 ± 200; both tasks exhibit substantial across-seed variability, reflecting mean improvements rather than consistent gains across all seeds.Source: B. Benchmark performance · p. 5
  • PL-MPC improves mean performance over TD-M(PC)2 on eight of thirteen HumanoidBench tasks and is lower on five.Source: B. Benchmark performance · p. 5
  • PL-MPC has the highest mean TAR on stand, maze, slide, hurdle, and balance-hard, and ties BOOM on pole.Source: B. Benchmark performance · p. 5
  • On the four DMControl tasks with complete baseline coverage, PL-MPC is competitive and has the highest mean on humanoid-walk.Source: B. Benchmark performance · p. 5

C. Component ablations and training dynamics

The ablations show that component effects differ across balance-hard and hurdle, with targeted comparisons revealing conditional interactions rather than a uniform benefit from every component. On balance-hard, MTD is especially consequential; on hurdle, removing MTD can increase mean performance.

C. Component ablations and training dynamics

  • On balance-hard, removing any component reduces mean TAR; removing MTD causes the largest drop, from 387 ± 255 to 186 ± 51.The variants stay relatively close early, then diverge as some full-PL-MPC seeds reach higher returns.Source: C. Component ablations and training… · p. 6, Figure 5 · p. 6
  • On hurdle, removing MTD yields 530 ± 275 mean return and 0.47 success, while removing ATE or RAD lowers mean TAR to 296 ± 21 or 231 ± 72, respectively.The leave-one-out effects are task dependent and non-additive.Source: C. Component ablations and training… · p. 6

D. Targeted ablations and training diagnostics

  • With ATE and RAD fixed, the hybrid n = 3 target raises balance-hard mean return from 186±51 to 387±255; success averages 19% versus 0% without MTD.The best full-PL-MPC seed succeeds on 50.9% of evaluation episodes; success requires balancing for the full 1000-step episode.Source: D. Targeted ablations and training… · p. 6
  • On hurdle, adding ATE and RAD jointly to MTD raises mean return from 288 ± 116 to 465 ± 201; full PL-MPC averages 19% success versus 0% for matched baseline seeds.The best full-PL-MPC seed reaches 58.0% success; the comparison shows different conditional effects from removing MTD in the leave-one-out study.Source: D. Targeted ablations and training… · p. 6

E. Sensitivity to the RAD warmup

RAD warmup changes performance differently across tasks, so neither tested schedule dominates. In zero-shot wrench–nut hardware trials, PL-MPC has higher observed success than TD-M(PC)2 on the training size and both unseen sizes.

E. Sensitivity to the RAD warmup

  • Delaying RAD activation closes the gap to TD-M(PC)2 on crawl and substantially narrows it on sit-hard, but the 200K setting remains the main-benchmark default.The study tests 200K versus 750K warmups on six tasks.Source: E. Sensitivity to the RAD warmup · p. 6
  • A 750K RAD warmup improves crawl, balance-simple, and sit-hard, changes walk and maze little, and decreases return and success on hurdle.No fixed warmup dominates; the main benchmark retains 200K, while adaptive scheduling is left for future work.Source: E. Sensitivity to the RAD warmup · p. 6, Figure 7 · p. 7

VI. REAL-ROBOT WRENCH–NUT ALIGNMENT

  • Policies are trained entirely in Isaac Gym and deployed without real-world fine-tuning; hardware evaluation covers training size 5 and unseen sizes 3 and 1, with 31 trials on size 5 and 15 per unseen size.FoundationPose provides 6D object-pose estimates from a fixed RealSense D435 RGB-D camera.Source: §VI · p. 7
  • PL-MPC has higher observed success on all tested sizes: 74.2% versus 61.3% on the training size, 80.0% versus 73.3% on unseen size 3, and 33.3% versus 20.0% on unseen size 1.Pooled success across unseen sizes is 56.7% versus 46.7%; the figure reports success-rate confidence intervals and successful-trial steps and pose errors.Source: §VI · pp. 7–8, Figure 8 · p. 8

VII. CONCLUSIONS

PL-MPC changes critic supervision, MPC terminal-value evaluation, and planner-to-policy transfer within policy-constrained TD-MPC. Its largest gains occur on balance-hard and hurdle, while benchmark effects and component interactions remain task dependent.

VII. CONCLUSIONS

  • PL-MPC’s largest gains occur on balance-hard and hurdle, while maze improvements are less consistent when useful planner trajectories remain rare.HumanoidBench survival rewards can obscure behavioral differences, and successful seeds can occupy a distinct higher-return regime, contributing to high variability on hard tasks.Source: §VII · p. 8
  • RAD warmup effects vary by task: delaying distillation improves several tasks but degrades hurdle, motivating adaptive rather than fixed schedules.On balance-hard, MTD has the largest leave-one-out effect, but outcomes remain seed dependent; hurdle shows different component interactions.Source: §VII · p. 8
  • Zero-shot deployment demonstrates transfer to the physical wrench–nut task and to object sizes unseen during simulation training.Source: §VII · p. 8
  • The modifications are evaluated only on TD-M(PC)2, although they act at interfaces shared by other TD-MPC-style methods.Systematic transfer across planning-learning backbones is identified as future work.Source: §VII · p. 8

Related papers

This paper cites

Loading 2609.39751v1…