PaperlayerView exact revision on arXiv

Source-linked AI summary

MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation

Patrick Lancaster, Nicklas Hansen, Aravind Rajeswaran, Vikash Kumar

arXiv:2309.14236v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Visual model-based reinforcement learning for real-world manipulation is hindered by sparse rewards, inefficient exploration, and unsafe behavior. MoDem-V2 combines demonstration-centered exploration, agency transfer, and actor-critic ensembles to learn contact-rich skills directly on an uninstrumented robot. Across four manipulation problems, it retains sample-efficient learning while improving safety and enables real-world learning with limited online data.

  • Problem

    Visual MBRL systems using sparse visual rewards can explore inefficiently and unsafely, limiting their application to simulated or heavily engineered environments.

    Source: §I · p. 1, §IV · p. 4

  • Method

    MoDem-V2 centers early rollouts on behavioral-cloning demonstrations, gradually transfers control to short-horizon planning, and uses actor-critic ensembles for uncertainty-aware exploration.

    Source: §I · p. 2

  • Results

    MoDem-V2 retains MoDem’s sample efficiency while exhibiting safer behavior and learns pushing, picking, and in-hand manipulation directly on a real robot.

    Source: §I · p. 2, §VIII · p. 6

  • Takeaways & Limitations

    Demonstration-augmented visual MBRL can learn contact-rich manipulation directly in the real world with minimal human intervention.

    Source: §I · p. 2, C. Real World Results · p. 6, §VIII · p. 6

    The method requires a small number of demonstrations and assumes the environment can be reset to a narrow set of starting states.

    Source: §VIII · p. 6

Abstract

from arXiv · show

Robotic systems that aspire to operate in uninstrumented real-world environments must perceive the world directly via onboard sensing. Vision-based learning systems aim to eliminate the need for environment instrumentation by building an implicit understanding of the world based on raw pixels, but navigating the contact-rich high-dimensional search space from solely sparse visual reward signals significantly exacerbates the challenge of exploration. The applicability of such systems is thus typically restricted to simulated or heavily engineered environments since agent exploration in the real-world without the guidance of explicit state estimation and dense rewards can lead to unsafe behavior and safety faults that are catastrophic. In this study, we isolate the root causes behind these limitations to develop a system, called MoDem-V2, capable of learning contact-rich manipulation directly in the uninstrumented real world. Building on the latest algorithmic advancements in model-based reinforcement learning (MBRL), demo-bootstrapping, and effective exploration, MoDem-V2 can acquire contact-rich dexterous manipulation skills directly in the real world. We identify key ingredients for leveraging demonstrations in model learning while respecting real-world safety considerations -- exploration centering, agency handover, and actor-critic ensembles. We empirically demonstrate the contribution of these ingredients in four complex visuo-motor manipulation problems in both simulation and the real world. To the best of our knowledge, our work presents the first successful system for demonstration-augmented visual MBRL trained directly in the real world. Visit https://sites.google.com/view/modem-v2 for videos and more details.

I. INTRODUCTION

MoDem-V2 addresses the safety and exploration difficulties of visual model-based reinforcement learning for contact-rich manipulation directly in the real world. It combines demonstration-centered exploration, agency transfer, and actor-critic ensembles, then evaluates the resulting system across four manipulation tasks.

Motivation

  • Visual MBRL avoids extensive instrumentation but remains limited by sample inefficiency, sparse rewards, and unsafe exploration in real-world manipulation.Accurate contact-rich simulation is difficult to create and calibrate, motivating direct learning from real-world interaction.Source: §I · p. 1

Safety challenge

  • MoDem was infeasible for direct real-world use because aggressive exploration caused repeated hardware-controller faults and dangerous joint forces.Static velocity, acceleration, and torque limits were insufficient to prevent unsafe online behavior, while torque penalties acted retrospectively.Source: §I · pp. 1–2

Approach

  • MoDem-V2 integrates policy centering, agency transfer, and actor-critic ensembles to constrain early exploration and reduce over-optimistic action evaluation.Behavior-cloned actions warm-start exploration, agency transfer shifts execution toward short-horizon planning, and ensembles estimate value uncertainty.Source: §I · p. 2

Evaluation

  • MoDem-V2’s conservative exploration improves safety while retaining MoDem’s sample-efficient learning capability across contact-rich manipulation tasks.The evaluation covers pushing, picking, and in-hand manipulation in simulation and on a real robot.Source: §I · pp. 1–2

Results

  • MoDem-V2 quickly learns pushing, picking, and in-hand manipulation directly in the real world using raw visual and proprioceptive feedback with sparse rewards.The system is evaluated on four manipulation tasks without requiring explicit state estimation or dense reward specification.Source: §I · pp. 1–2

Contribution

  • The work also open-sources MoDem-V2 and discusses practical considerations for training reinforcement-learning systems on real hardware.Source: §I · p. 2

II. PRELIMINARIES

The paper formulates real-world visual manipulation as sparse-reward MBRL using camera observations and proprioception, then builds on MoDem’s demonstration-augmented world model.

Setting

  • Real-world manipulation is modeled as a high-dimensional MDP with RGB observations, proprioception, and sparse task-completion rewards.The learned policy must achieve high performance with minimal online interaction while respecting hardware safety.Source: §II · p. 3

MoDem

  • MoDem combines model predictive control, a decoder-free TD-MPC world model, and a small number of demonstrations for continuous control.Its learned components include a state embedding, latent dynamics, reward predictor, terminal value, and policy guide.Source: §II · p. 3
  • The world model is learned end-to-end, while the deterministic policy maximizes value conditioned on the learned latent state.MoDem is effective in simulation but its aggressive exploration limits use where safety constraints matter.Source: §II · p. 3

III. SAFETY

The safety formulation treats hardware and environmental constraints as unobservable during operation, motivating a planning procedure that evaluates sampled trajectories and uncertainty.

Safety constraints

  • Safety violations are defined as real-world hardware faults requiring human intervention and simulated torque-limit or excessive-contact-force violations.The simulated excessive-contact-force threshold is 100 N at the robot end effector.Source: §III · p. 3

Planning procedure

  • MoDem-V2 planning encodes the current state, samples candidate trajectories, rolls them through latent dynamics, and evaluates rewards plus terminal values.The procedure supports both prior sampling and policy-ensemble sampling before trajectory evaluation.Source: §III · p. 3
  • The planner combines ensemble terminal-value estimates to estimate epistemic uncertainty for trajectory selection.Algorithm 1 explicitly includes ensemble mixing weights and an epistemic-uncertainty estimation step.Source: §III · p. 3

IV. METHOD

The method addresses MoDem’s unsafe real-world exploration by constraining early action selection near demonstrations and improving value-based planning.

Strengths and weaknesses

  • MoDem uses demonstrations to pretrain its policy and representation, seed model learning, and iteratively optimize on demonstrations, seeding data, and online interactions.This three-stage framework accelerates model learning in simulated tasks but faces challenges suppressed during the original simulation study.Source: §IV · p. 3
  • MoDem’s world model and value function can misestimate actions far from demonstrations, causing task failure or unsafe forces and torques.The original method samples across the entire action space, allowing consecutive actions to differ substantially.Source: §IV · pp. 3–4

MoDem-V2

  • MoDem-V2 is designed to improve safety while retaining MoDem’s autonomy and data-efficiency through three planning adaptations.The adaptations are introduced after identifying excessive forces, torques, and unreliable value estimates during real-world exploration.Source: §IV · pp. 3–4

A. MoDem-V2: Real-World Model-Based Reinforcement Learning with Demonstrations

MoDem-V2 modifies exploration and value estimation to support safer real-world learning, then evaluates sample efficiency, safety, and physical manipulation capability against baselines and ablations.

A. MoDem-V2: Real-World Model-Based Reinforcement Learning with Demonstrations

  • MoDem-V2 centers exploratory actions on the learned policy, reducing evaluation in unseen state-action regions and improving action-quality discrimination.This replaces sampling across the entire action space with more conservative policy-centered exploration.Source: A. MoDem-V2: Real-World Model-Based… · p. 4
  • Agency transfer linearly shifts control from behavioral-cloning actions to short-horizon MPC as online interaction proceeds.The transfer parameter α starts at 0 and increases to 1.0 over a fixed number of interaction steps.Source: A. MoDem-V2: Real-World Model-Based… · p. 4
  • Actor-critic ensembles reduce overestimation bias and provide epistemic-uncertainty estimates for uncertainty-aware trajectory decisions.Each critic evaluates actions from policies not directly optimized for that critic, with ensemble weights w1 > 0 and w2 < 0.Source: A. MoDem-V2: Real-World Model-Based… · p. 4

V. EXPERIMENTAL DESIGN

  • The experiments compare sample efficiency and safety against other methods, including MoDem, and test whether robots learn real-world manipulation.The study uses simulation baselines and ablations, then evaluates MoDem-V2 on four physical manipulation tasks.Source: §V · p. 4

A. Hardware

The system uses a Franka Panda with task-specific grippers and three RealSense cameras, and evaluates four manipulation tasks spanning pushing, picking, and in-hand reorientation.

A. Hardware

  • The hardware uses a Franka Panda arm, Robotiq gripper for pushing and picking, D’Manus hand for reorientation, and three RealSense D435 cameras.Source: A. Hardware · p. 4

B. Task Suite

  • The task suite includes planar pushing, inclined pushing, bin picking, and in-hand reorientation across increasingly demanding contact-rich skills.Source: B. Task Suite · pp. 4–5
  • Bin picking requires precise gripper positioning and distinguishing spatially similar states to determine whether the object has been grasped.Source: B. Task Suite · pp. 4–5
  • In-hand reorientation grasps a bottle lying sideways and manipulates it upright with a multi-fingered hand that more than doubles action-space dimensionality.Source: B. Task Suite · p. 5

VI. EXPERIMENTS

The experiments compare MoDem-V2 with baselines and ablations in simulation, then test real-world learning across four manipulation tasks. MoDem-V2 preserves sample efficiency while reducing safety violations and quickly surpassing its behavioral-cloning policy.

VI. EXPERIMENTS

  • The evaluation uses four manipulation tasks, ten demonstrations for each initial policy, and aggregates each evaluation over 30 trials.Source: §VI · p. 5

A. Simulated Comparison to Baselines

B. Ablation of design choices

C. Real World Results

  • MoDem’s aggressive exploration exceeds safety limits early on real hardware, making real-world evaluation infeasible because of excessive force and torque.Source: C. Real World Results · pp. 5–6
  • MoDem-V2 safely learns real-world manipulation with minimal human intervention and exceeds its initial behavioral-cloning policy with about two hours or less of online data.Source: C. Real World Results · p. 6, Figure 6 · p. 6

VII. RELATED WORK

The paper extends visual model-based reinforcement learning and safe real-world robot learning toward contact-rich manipulation with demonstrations. It concludes that MoDem-V2 improves safety and retains sample efficiency, while requiring demonstrations and restricted resets.

VII. RELATED WORK

  • Visual MBRL commonly targets sample-efficient simulated control, whereas this work addresses the additional practicality and safety challenges of training on real robots.Source: §VII · p. 6
  • Unlike approaches requiring user-defined state-safety functions, MoDem-V2 proposes deployable safe-exploration mechanisms that preserve MoDem’s sample efficiency.Source: §VII · p. 6
  • MoDem-V2 targets demonstration-augmented visual MBRL for real-world manipulation, distinguishing it from real-world MBRL work on simpler tasks without demonstrations.Source: §VII · p. 6

VIII. DISCUSSION

  • The method centers rollouts around behavioral cloning, gradually increases world-model actions, and uses actor-critic ensembles for uncertainty-aware planning.Source: §VIII · p. 6
  • The approach requires a small number of demonstrations and assumes environments can reset to a narrow set of starting states.Source: §VIII · p. 6

IX. APPENDIX

MoDem trains its world model on replay-buffer subtrajectories using embedding prediction, reward prediction, and temporal-difference objectives, while the policy maximizes predicted value.

A. MoDem Training Objective

  • The world model minimizes an objective over length-h subtrajectories sampled from replay buffer B.The objective combines embedding prediction, reward prediction, and temporal-difference learning terms.Source: A. MoDem Training Objective · p. 9
  • The embedding-prediction term compares predicted embeddings with stop-gradient target embeddings generated by the exponentially averaged model.The stop-gradient operator is denoted sg, and ϕ is an exponentially moving average of θ.Source: A. MoDem Training Objective · p. 9
  • The temporal-difference target is defined as qt = rt + Qϕ(z′t), with λ weighting temporally close steps more heavily.λ ∈ (0, 1] assigns larger weight to temporally close time steps.Source: A. MoDem Training Objective · p. 9
  • The policy maximizes predicted value over replay-buffer subtrajectories, with gradients taken with respect to πθ only.The latent state is zt = hθ(st).Source: A. MoDem Training Objective · p. 9

B. Additional Task Details

The appendix specifies task interfaces, resets, demonstrations, and rewards across manipulation problems, then shows that additive torque penalties do not resolve safety violations or preserve learning performance.

B. Additional Task Details

  • All experiments run at 12.5 Hz with proprioceptive and image inputs, while task-specific actions, resets, demonstrations, and rewards vary by manipulation problem.The appendix covers planar pushing, incline pushing, bin picking, and in-hand reorientation.Source: B. Additional Task Details · p. 9
  • Incline pushing is harder than planar pushing because gravity-based resets produce a wider distribution of object starting positions.The two pushing tasks otherwise use the same reward and demonstration collection procedure.Source: B. Additional Task Details · p. 9
  • Bin-picking success is detected by subtracting bin images before and after releasing the grasp, while a hand-coded policy collects demonstrations and resets the object.Gripper width alone would produce false positives because of flexible fingertips and passive gripper degrees of freedom.Source: B. Additional Task Details · p. 9
  • In-hand reorientation uses a 10-joint hand and balances sufficient grasping force against allowing the bottle to pivot during the index-finger push.The real-world task detects upright success with a top-down depth camera and uses a hand-coded demonstration policy.Source: B. Additional Task Details · p. 9

C. Ineffectiveness of Torque Penalization

  • Additive torque penalties leave safety violations similar to original MoDem and fail to prevent early torque spikes during online interaction.The penalty can reduce torque later in training, but the hardest in-hand reorientation task stagnates and produces an unsuccessful policy.Source: C. Ineffectiveness of Torque… · pp. 9–10, Figure 7 · p. 10

D. Task Trajectories and Learning Curves

This appendix presents real-world MoDem-V2 learning curves and lists the training and control hyperparameters used for the experiments.

D. Task Trajectories and Learning Curves

E. MoDem-V2 Hyperparameters

  • MoDem-V2 uses behavior cloning for pretraining and seeding, ten demonstrations, and prioritized experience replay with α = 0.6 and β = 0.4.The replay buffer is unlimited, and the demonstration sampling ratio changes from 75% to 25% over 100K steps.Source: E. MoDem-V2 Hyperparameters · p. 10
  • The listed configuration uses discount factor γ = 0.95, image resolution 224 × 224, frame stack 2, action repeat 1, and planning horizon H = 5.Seed steps are 7,500 for in-hand reorientation and 5,000 otherwise.Source: E. MoDem-V2 Hyperparameters · p. 10
  • MPC action probability increases from 0.0 to 1.0, while the critic combines mean weight 1.0 with standard-deviation weight -10.0.The transition takes 25K steps for simulated incline pushing and 100K steps otherwise.Source: E. MoDem-V2 Hyperparameters · p. 10

Related papers

Loading 2309.14236v2…