Source-linked AI summary
Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, Wojciech Zaremba
TL;DR
Simulation policies often fail after deployment because real-world dynamics differ, although their high-level behavior may remain useful. The paper adapts each simulated action through a learned deep inverse dynamics model trained with test-like physical-robot data, and reports favorable comparisons with established transfer baselines.
Problem
Simulation-based policies can perform poorly on real robots when lower-level dynamics differ, despite retaining a reasonable high-level trajectory.
Method
The method predicts the simulator’s next state from the source policy and uses a learned deep inverse dynamics model to choose the real-world action that achieves it.
Results
The approach compares favorably with output error control and Gaussian dynamics adaptation across challenging transfer experiments.
Takeaways & Limitations
Test-like, incremental data collection improves inverse-dynamics adaptation, including for contact-rich and time-correlated dynamics.
Abstract
from arXiv · showhide
Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.
I. INTRODUCTION
Simulation-based policies can fail in the real world because of model discrepancies, even when their high-level behavior remains useful. The paper adapts those policies through learned inverse dynamics and test-like data collection, evaluating transfer across simulated and physical settings.
- Motivation: Simulation policies may perform well in simulation yet poorly in the real world because the environments differ in physical details.Relevant discrepancies include contact, friction, mass, geometry, and other physical quantities that are difficult to model accurately.
- Problem setting: The approach assumes simulated policies preserve high-level behavior while missing lower-level effects such as friction, backlash, hysteresis, and precise deformation.This assumption motivates adapting actions rather than relearning the overall policy.
- Approach: At each step, the method computes the source policy’s action, predicts the simulator’s next state, and uses a learned inverse dynamics model to select the real-world action.The source policy itself can come from motion planning, optimization, control, or learning methods.
- Approach: Training data are collected by operating with a poor inverse dynamics model and then improving it from the resulting experience.Target trajectories begin short or are truncated after significant deviation to improve collection efficiency.
- Evaluation: Test-like data collection improves results significantly over collecting data by applying only random controls.The paper evaluates transfer from Sim1 to Sim2 across standard MuJoCo tasks and compares against output error control and Gaussian Dynamics Adaptation.
II. RELATED WORK
The paper situates action adaptation among simulator improvement, robust control, and learned-dynamics methods. It uses a deep inverse dynamics model to adapt actions across domains without relying on domain-specific physical models.
- Simulator modeling: More accurate simulators can reduce discrepancies but remain limited by difficult physical effects, parameter identification, and slower computation.Examples include contact, non-rigidity, friction, mass, geometry, and material properties.
- Learned dynamics: The learned inverse dynamics model is intended to represent effects such as backlash, hydraulic actuation, elastic actuation, and pneumatic actuation in one unified method.It uses histories of observed states and does not require manually designed domain-specific models and identification procedures.
- Learned dynamics: Inverse dynamics models map current and next states to an action that achieves the transition, allowing their outputs to be used directly for control.This contrasts with forward models, which map a current state and action to a next state.
- Model representations: Deep neural networks offer expressive learned dynamics representations, while linear models have limited expressive power and Gaussian processes scale poorly to large datasets.The paper uses a deep inverse dynamics model for action adaptation.
- Adaptation strategies: Unlike robust-control approaches that seek one policy across many possible environments, adaptive methods learn about the specific real world encountered.The paper focuses on adapting actions rather than policies, states, or observations.
III. METHOD
The method transfers a competent source-environment policy to a target environment by using simulated transitions as targets for action adaptation. The paper studies both simulator-to-simulator transfer and simulator-to-robot transfer.
- Setting: The evaluation first transfers between two simulators so that variation between source and target environments can be controlled.The final experiments transfer from a simulator to a physical robot.
- Setting: Each environment is described by state space S, action space A, observation space O, and forward dynamics T(s, a) = s′.States may be latent because observations can be noisy and incomplete.
- Policy transfer: Source actions are mediated through the simulator’s anticipated next observation, allowing source and target action spaces to differ.This accommodates actuator calibration differences, complex actuator dynamics, and even different action dimensionalities.
- Policy representation: Trajectories are sequences of observations and actions, and the policy maps recent observation-action history to an action.The notation distinguishes recent trajectory windows used by history-dependent policies.
- Policy transfer: The method assumes a competent source policy but is agnostic to whether it was obtained through planning, model-predictive control, optimization, or reinforcement learning.The source policy is not directly executed in the target environment.
B. Transfer to the target environment
The method transfers a source policy’s high-level behavior by simulating its expected next observation and using a learned target inverse dynamics model to select the target action. This requires a reasonable source forward model and shared actuated degrees of freedom, while allowing substantially different target actions.
- Transfer mechanism: At each time step, the source policy’s prescribed action is simulated to predict the next observation, which guides target-action selection through inverse dynamics.The learned model uses recent action-observation history and the desired next observation to produce a target-domain action.
- Assumptions: The approach requires a source forward dynamics model that can reasonably estimate the next state and observation.
- Assumptions: Accurate inverse dynamics should make the target’s next observation similar to the simulated desired observation.
- Action-space flexibility: Source and target environments must share actuated degrees of freedom, but their actions may differ substantially in calibration, dynamics, or dimensionality.Source actions are mediated through the simulator and anticipated next observation rather than directly executed in the target space.
C. Training of the inverse dynamics model
The inverse dynamics model is trained from physical-environment trajectories to predict actions that produce subsequent observations, using history to capture latent dynamics and temporal dependencies.
- Training objective: The method collects physical-environment trajectories and trains a neural network to predict the action leading to the next observation.
- Training objective: The training input contains a trajectory snippet followed by its next observation, and the output is the preceding action.
- History representation: A history window H is chosen large enough for the model to implicitly infer important latent factors and temporal dependencies.
D. Data collection / Exploration
Training data are collected by executing a preliminary target policy produced by the current inverse dynamics model, with noise added to encourage diverse exploration.
- Policy-driven collection: A preliminary inverse dynamics model is used to implement the target policy that collects additional training data.
- Exploration: Noise is added to prescribed actions to ensure the collected training data are sufficiently diverse.
- Exploration: Too much exploration noise moves data far from target trajectories, whereas too little produces insufficiently diverse data.
E. Inverse dynamics neural network architecture
The inverse dynamics network consumes normalized histories of observations and actions plus a target observation, then predicts the target action through two ReLU hidden layers. The experiments use Reacher, Hopper, Half Cheetah, and Humanoid environments.
- Input representation: The model input is a sequence of k previous observations, k −1 previous actions, and a target observation concatenated into one vector.
- Experimental environments: The simulation experiments include Reacher, Hopper, Half Cheetah, and Humanoid environments.
- Preprocessing: Inputs are normalized to mean 0 and variance 1 before processing by the network.
- Network architecture: The network uses two fully connected ReLU hidden layers with 256 units each and an output layer producing action a = φ(τ−k:, o).
IV. EXPERIMENTS
The experiments evaluate adapting policies from simulated source domains to simulated targets and to a physical Fetch robot.
- IV. EXPERIMENTS: The evaluation measures transfer from a simulated source domain to either another simulator or a physical environment.The simulated transfer isolates domain differences, while the physical experiment tests deployment on Fetch.
A. Simulated Environments – Sim1 to Sim2 transfer
Sim1-to-Sim2 experiments test adaptation across MuJoCo environments with changed physical properties, noise, and challenging dynamics. The method remains effective where expert policies and forward-model baselines degrade, while requiring substantially fewer samples than direct policy learning.
- Simulated setup: Experiments use Reacher, Hopper, Half-Cheetah, and Humanoid with altered masses, link lengths, friction, torque scales, and limits.Source and target environments share robot types but differ in physical properties.
- Simulated setup: Expert policies are trained with Trust Region Policy Optimization, and performance is normalized so the expert policy scores 1.The experiments use OpenAI Gym rewards for evaluation.
- Results: The proposed method performs well across small and large environment changes, while history-based adaptation provides a minor improvement.Figure 3 reports normalized cumulative reward averaged over ten random seeds.
- Domain variations: The method adapts across gravity changes and both independent and slowly varying motor noise, including contact-rich environments.For similar domains, it can output action corrections and may use expert-policy pretraining.
- Results: Baseline adaptation methods perform well in simple-dynamics settings but are ineffective in contact-rich Hopper, Cheetah, and Humanoid environments.Locally linear dynamics approximations struggle with contact discontinuities, and small errors can accumulate until unstable episodes terminate.
- Sample complexity: Sample requirements increase with additive noise, while including history typically reduces learning complexity and reaches 75% expert performance about two orders of magnitude faster than direct policy learning.The comparison cites approximately 70 million samples for Humanoid and about one million for Hopper in the fastest listed RL algorithms.
B. Physical interaction – Sim to Real transfer
The physical experiment transfers simulated arm trajectories to a Fetch robot executing an agile swing with a bungee-induced disturbance. Action adaptation adjusts torque to follow the simulated trajectory, with performance compared against a PD controller.
- Physical setup: The experiment runs trajectory transfer from simulation to a physical Fetch robot using position control, stock ROS firmware, and 10 Hz control.The metric compares normalized distances between simulated and physical observations.
- Task and comparison: The agile arm-swing task uses a bungee cord pulling the arm’s midpoint, creating a condition requiring adapted torque.The baseline is a PD controller whose targets are states experienced in simulation.
- Results: The action adaptation method adjusts to the bungee condition by exerting the necessary torque, and results are summarized against the PD baseline over 10 trials.Figure 5 reports average distance and variance to the desired trajectory.
V. DISCUSSION AND FUTURE WORK
The paper presents action adaptation through a deep inverse dynamics model for transferring complex policies across domains. It identifies observation adaptation as necessary for broader physical-world generalization and leaves it for future work.
- V. DISCUSSION AND FUTURE WORK: The method adapts complex policies involving contact, hysteresis-like time-correlated noise, and substantial environment differences.The inverse dynamics model is trained on physical-robot behavior.
- V. DISCUSSION AND FUTURE WORK: The current approach assumes simulator observations match physical observations, an assumption that is reasonable for joint positions but not high-fidelity visual or depth sensors.The paper focuses only on action adaptation.
- V. DISCUSSION AND FUTURE WORK: Future work will investigate observation adaptation methods to complement action adaptation for physical-world transfer.The discussion cites observation adaptation as a planned direction.