PaperlayerView exact revision on arXiv

Source-linked AI summary

VAMPS: Visual and Motor Policies from Sampling-Based Planning

Mohamed Yassine Kabouri, Pietro Noah Crestaz, Quang-Nam Nguyen, Qilong Cheng, Ludovic Righetti, Nicolas Mansard

arXiv:2610.05331v1cs.ROcs.AIcs.LG

TL;DR

Learning directly on physical robots remains difficult, while demonstration-based learning and exploration each face constraints. VAMPS uses MPPI to train reusable policies through iterative simulation-based refinement or offline learning from autonomously collected real-robot data. The paper reports iterative refinement outperforming frozen-data training and demonstrates locomotion transfer and real-robot visuomotor tasks.

  • Problem

    Learning directly on physical robots remains difficult; demonstrations are costly and bound by operator performance, while contact-rich reinforcement learning faces exploration bottlenecks.

    Source: §I · p. 1

  • Method

    VAMPS uses MPPI to generate supervision for reusable policies through iterative refinement in simulation or offline ACT training on autonomously collected real-robot data.

    Source: §I · pp. 1–2

  • Results

    VAMPS-Iterative outperforms frozen-MPPI training and transfers locomotion to a Go2; VAMPS-Batch demonstrates pick-and-place and force-aware whiteboard erasing.

    Source: §I · p. 2

  • Takeaways & Limitations

    Sampling-based planning can train reusable policies through simulation followed by hardware transfer or directly from autonomously collected real-robot data.

    Source: §I · p. 2, §VII · p. 8

Abstract

from arXiv · show

Learning robot policies directly on physical systems remains difficult because data collection is costly and policy exploration can be unsafe. We introduce Visual and Motor Policies from Sampling-Based Planning (VAMPS), a framework that uses Model Predictive Path Integral (MPPI) control to train reusable policies without human demonstrations. VAMPS supports two training modes. For one-step proprioceptive policies, it operates iteratively in simulation: the policy warm-starts MPPI, and the refined trajectories provide new supervision as the policy changes. A learned terminal value improves short-horizon planning, while an Implicit Q-Learning (IQL) critic guides the policy update. Iterative refinement outperforms training once on frozen MPPI data, and we transfer the learned locomotion policy to a Unitree Go2. For visuomotor policies, VAMPS operates directly from real-robot data. MPPI uses task-specific state estimates to plan and execute trajectories while recording RGB and sensor observations on a Flexiv Rizon 10S. Action Chunking with Transformers predicts action chunks, reducing the effective prediction horizon, and is trained offline on this fixed dataset. We demonstrate visuomotor pick-and-place and force-aware whiteboard erasing. In the latter task, the policy additionally observes the measured $6$-D wrench and desired normal force. These results show that VAMPS can learn policies either in simulation followed by hardware transfer or directly from autonomously collected real-robot data.

I. INTRODUCTION

VAMPS uses MPPI to train reusable robot policies without human demonstrations, addressing the difficulty of learning directly on physical systems. It combines iterative policy refinement in simulation with offline visuomotor learning from autonomously collected real-robot data.

II. RELATED WORK

  • Imitation learning clones demonstrations,, while DAgger queries an expert and reinforcement learning explores without demonstrations.Sampling-based planners handle non-smooth contact dynamics but require online replanning and do not directly yield reusable policies,.Source: §I · p. 1

I. INTRODUCTION

  • VAMPS distills MPPI-generated trajectories into reusable policies without human demonstrations or simulator gradients.MPPI plans using a model, task cost, and task-specific state estimates; deployed policies map available observations directly to actions.Source: §I · p. 1
  • VAMPS-Iterative repeatedly lets a one-step proprioceptive policy warm-start MPPI, then uses refined trajectories as new supervision.A learned terminal value improves short-horizon planning, while an IQL critic guides policy updates through AWR.Source: §I · p. 1
  • VAMPS-Batch records RGB, sensor, and action episodes while MPPI plans on the real robot, then trains ACT offline on the fixed dataset.At deployment, ACT uses images and robot sensors rather than task-specific state estimates.Source: §I · p. 1, Figure 1 · p. 2
  • Iterative refinement outperforms training once on frozen MPPI data, and the resulting locomotion policy transfers to a Unitree Go2.VAMPS-Batch also demonstrates pick-and-place and force-aware whiteboard erasing on a Flexiv Rizon 10S.Source: §I · pp. 1–2

III. PRELIMINARIES

MPPI optimizes a finite-horizon control sequence by sampling perturbations and repeatedly replanning as the system advances.

A. Model Predictive Path Integral (MPPI)

  • MPPI is a sampling-based trajectory optimization method that solves a finite-horizon optimal control problem from the current state.The dynamics constrain each next state to f(x_t, u_t).Source: A. Model Predictive Path Integral (MPPI) · p. 2
  • MPPI samples Gaussian perturbations around a nominal control sequence and uses each perturbed trajectory’s cost to update that sequence.Trajectory states are obtained by rolling out the dynamics from the current state.Source: A. Model Predictive Path Integral (MPPI) · p. 2
  • In receding-horizon control, MPPI applies a short portion of the optimized sequence, shifts the sequence, and replans from the resulting state.The planner repeats this process after each applied segment.Source: A. Model Predictive Path Integral (MPPI) · pp. 2–3

B. Implicit Q-Learning

The section describes IQL, ACT, and how the VAMPS framework couples sampling-based planning with reusable policies.

B. Implicit Q-Learning

  • IQL learns state-value and action-value functions from transitions while avoiding action-value maximization outside the dataset.It approximates the in-sample maximum with an asymmetric expectile loss and forms advantages as Q minus V.Source: B. Implicit Q-Learning · p. 3

C. Action Chunking with Transformers (ACT)

IV. VAMPS: LEARNING POLICIES FROM SAMPLING-BASED PLANNING

  • VAMPS has two complementary modes: iterative MPPI refinement with policy fitting, or MPPI collection of a fixed dataset followed by offline policy training.Source: §IV · p. 3

A. VAMPS-Iterative

  • VAMPS-Iterative searches over policy performance from an initial-state distribution while requiring each planner control sequence to be reproducible by the policy.For a Gaussian policy, the constraint is expressed as a KL divergence between the MPPI trajectory distribution and the policy distribution.Source: A. VAMPS-Iterative · p. 3

1) Policy-Guided MPPI with a Learned Terminal Value:

VAMPS-Iterative uses the current policy to seed MPPI, which refines trajectories that supervise policy learning; a learned value extends the effective planning horizon.

1) Policy-Guided MPPI with a Learned Terminal Value:

2) Planner-Policy Coupling:

VAMPS-Iterative offers three ways to couple MPPI and policy learning, differing in how they enforce the planner-policy constraint.

3) Critic-Guided Policy Learning:

Critic-guided learning uses both policy warm-start rollouts and refined MPPI trajectories to estimate advantages and update the policy.

  • The replay buffer combines warm-start policy rollouts with refined MPPI trajectories for critic updates.Warm-start data reduces extrapolation when evaluating the policy, while the combined data improves action coverage for advantage estimation.Source: 3) Critic-Guided Policy Learning · p. 4
  • IQL trains the value and Q critics, and AWR converts their estimated advantage into normalized policy-update weights.Source: 3) Critic-Guided Policy Learning · p. 4
  • All coupling strategies update the policy on current MPPI trajectories using advantage weights; ADMM uses its current dual variable, unlike BC and Penalty.ADMM's dual variable accumulates the planner-policy residual.Source: 3) Critic-Guided Policy Learning · p. 4

B. VAMPS-Batch

VAMPS-Batch collects real-robot episodes with MPPI, freezes them, and trains an ACT visuomotor policy offline.

  • ACT predicts action trajectories rather than one action per step, reducing the effective decision horizon and supporting one-time training on fixed data.Source: B. VAMPS-Batch · p. 5
  • MPPI plans and executes on the robot using task-specific state estimates while synchronized images, robot observations, task sensors, and actions are recorded.Source: B. VAMPS-Batch · p. 5
  • The resulting episodes are frozen for offline ACT training, and the deployed policy acts from images and robot sensors without task-specific state estimates.Source: B. VAMPS-Batch · p. 5

C. Algorithm Summary

The two VAMPS modes share MPPI-generated supervision but differ in whether planning, policy, and value learning are iterated together or data collection is completed before offline training.

C. Algorithm Summary

  • VAMPS-Iterative updates the planner, policy, and value functions together, while VAMPS-Batch freezes collected real-robot episodes before offline ACT training.Source: C. Algorithm Summary · p. 5

A. Experimental Setup

  • Panda pick-and-place tests coupling and ablations, while Go2 evaluates locomotion coupling and real-robot policy transfer.Source: A. Experimental Setup · p. 5
  • The iterative setup uses a shared BroNet stem and residual blocks, with different block counts for the critic versus policy and terminal value.The critic uses two blocks; the policy and terminal value use one.Source: A. Experimental Setup · p. 5
  • Training uses task-specific outer iterations and policy epochs, with MPPI horizon 16 and replanning every two steps.The setup also specifies separate Panda and Go2 sampling configurations and domain randomization for Go2.Source: A. Experimental Setup · p. 5
  • Iterative experiments report mean return on 50 test initial states per seed, with shaded bands representing 95% confidence intervals across seeds.Source: A. Experimental Setup · p. 5

B. Q1: How does the coupling strategy between the trajec-

BC and Penalty outperform ADMM on Panda pick-and-place, while the dual-variable updates can pull planner and policy apart. A learned terminal value improves returns at every tested planning horizon, with larger gains at shorter horizons.

tory optimizer and the policy affect performance?

  • On Panda, BC and Penalty exceed 0.90 success and plateau just under 1.0, while ADMM plateaus at 0.88 with Gaussian and 0.90 with deterministic policies.BC and Penalty return plateaus are 1010–1060; ADMM plateaus at 860–875.Source: tory optimizer and the policy affect… · p. 5
  • At the start of Go2 runs, the untrained policy’s near-zero action yields a high return because position control defaults to the joint configuration.The action head is initialized at a scale of 0.01.Source: tory optimizer and the policy affect… · p. 6
  • ADMM’s dual variables shift the regression target to u + y; updates can move planner and policy farther apart despite a primal residual of 0.24 without consensus enforcement.The penalty lowers the residual only to 0.21, while dual updates move it to 0.26; the figure’s right panel measures the 95th percentile of dual-variable magnitudes against the policy’s action bound.Source: tory optimizer and the policy affect… · p. 6, Figure 4 · p. 6

for a shorter horizon?

  • The learned terminal value improves final return at T=16, T=8, and T=4, with gains growing as the planning horizon shrinks.The gain is large at T=4, clear at T=8, and small at T=16; T=4 with the value remains below T=8 without it.Source: for a shorter horizon? · p. 6, Figure 5 · p. 6
  • The terminal value permits shorter MPPI rollouts, reducing their computational budget, but does not replace a reasonable planning horizon.The reported T=4 policy with a terminal value remains well below T=8 without one.Source: for a shorter horizon? · p. 6

D. Q3: Does the offline RL critic improve performance?

On Panda pick-and-place, VAMPS-Iterative outperformed frozen MPPI-data learners and exceeded PPO at the matched 80-step episode length. The critic added a small gain at suitable temperatures, while batch size had no effect and large penalty coefficients hurt return.

D. Q3: Does the offline RL critic improve performance?

  • The offline RL critic gave a small Panda pick-and-place gain for β=2–50; β=1 matched no-critic, while β=0.3 lowered return.The comparison retains the learned terminal value in both conditions.Source: D. Q3: Does the offline RL critic… · p. 6

cient and batch size?

  • Batch size had no effect; ρ=0.005 and ρ=0.01 gave the same return, while ρ≥0.1 sharply lowered return from the first iterations.The authors attribute the early drop to the penalty dominating MPPI cost while the policy is untrained.Source: cient and batch size? · p. 6

F. Q5: How do standard baselines perform in this regime?

  • 1109 return / 0.98 success: VAMPS-Iterative exceeded PPO at 500 steps (975 / 0.98) and frozen BC/IQL (290 / 0.02; 240 / 0.01) at 80 steps.PPO needed 500-step training episodes to approach the task; VAMPS-Iterative used 80-step episodes and passed 1000 return after 3.6×10^9 simulator steps.Source: Table I · p. 7, F. Q5: How do standard baselines… · p. 7

G. Q6: Does the policy transfer to real hardware?

VAMPS transfers a simulation-trained locomotion policy to a Unitree Go2 and uses MPPI-collected real-robot data to train ACT policies for two Flexiv tasks. The reported hardware results include 8 successes in 10 pick-and-place trials and four consecutive successful erasing trials; the authors characterize the VAMPS-Batch results as preliminary.

G. Q6: Does the policy transfer to real hardware?

VI. VAMPS-BATCH EXPERIMENTS & RESULTS

  • For both Flexiv experiments, MPPI collected real-robot RGB, sensor, and action episodes, then ACT was trained offline to predict action chunks.At inference, the action trajectory is executed open-loop before ACT is queried again.Source: §VI · p. 7

A. Experiment 1: Visuomotor pick-and-place

  • The VAMPS-Batch pick-and-place policy succeeded in 8 of 10 trials from randomized cube positions.The cube-position estimate guides MPPI during data collection; the deployed policy does not receive that task-specific estimate.Source: A. Experiment 1: Visuomotor… · p. 7

B. Experiment 2: Visuomotor and force-aware erasing policy

  • The same VAMPS-Batch checkpoint erased the writing in four consecutive whiteboard trials.Its observations include two synchronized images, joint positions, measured 6-D wrench, and desired normal force.Source: B. Experiment 2: Visuomotor and… · pp. 7–8

VII. CONCLUSIONS

  • VAMPS offers iterative MPPI refinement for proprioceptive policies and fixed-dataset offline ACT training for visuomotor policies, without human demonstrations.The authors describe the VAMPS-Batch hardware results as preliminary and plan to extend evaluation to more sensor-based tasks.Source: §VII · p. 8
Loading 2610.05331v1…