PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim-to-Real Transfer for Biped Locomotion

Wenhao Yu, Visak CV Kumar, Greg Turk, C. Karen Liu

arXiv:1903.01390v2cs.ROcs.LG

TL;DR

Biped policies trained in simulation can fail to transfer because of the reality gap, and task-relevant identification data are difficult to collect before training. The paper uses two-stage system identification linked by a Projected Universal Policy, transferring Darwin OP2 controllers for three walking directions in 25 real-world trials.

  • Problem

    Biped locomotion policies are difficult to transfer from simulation because modeling discrepancies create a reality gap, while instability makes task-relevant identification data hard to collect before training.

    Source: §I · p. 1

  • Method

    The method estimates a model-parameter range before training, jointly learns a low-dimensional projection and conditioned policy family, then uses task-relevant hardware data for post-training identification.

    Source: §I · pp. 1–2

  • Results

    The approach transferred Darwin OP2 forward, backward, and sideways walking controllers to hardware in 25 real-world trials.

    Source: §I · p. 2, §VII · p. 7

  • Takeaways & Limitations

    The two-stage procedure uses generic trajectories to guide policy learning and task-relevant trajectories afterward, without iterating between system identification and policy learning.

    Source: §I · p. 1

    The current model is not expressive enough to explain all real-world observations, and further modeling improvements may be needed for more challenging transfers.

    Source: E. Identified Model Parameter Bounds · p. 7

Abstract

from arXiv · show

We present a new approach for transfer of dynamic robot control policies such as biped locomotion from simulation to real hardware. Key to our approach is to perform system identification of the model parameters {\mu} of the hardware (e.g. friction, center-of-mass) in two distinct stages, before policy learning (pre-sysID) and after policy learning (post-sysID). Pre-sysID begins by collecting trajectories from the physical hardware based on a set of generic motion sequences. Because the trajectories may not be related to the task of interest, presysID does not attempt to accurately identify the true value of {\mu}, but only to approximate the range of {\mu} to guide the policy learning. Next, a Projected Universal Policy (PUP) is created by simultaneously training a network that projects {\mu} to a low-dimensional latent variable {\eta} and a family of policies that are conditioned on {\eta}. The second round of system identification (post-sysID) is then carried out by deploying the PUP on the robot hardware using task-relevant trajectories. We use Bayesian Optimization to determine the values for {\eta} that optimizes the performance of PUP on the real hardware. We have used this approach to create three successful biped locomotion controllers (walk forward, walk backwards, walk sideways) on the Darwin OP2 robot.

I. INTRODUCTION

Biped policies trained in simulation face a reality gap, while instability makes task-relevant system identification data difficult to collect before policy training. The paper bridges pre- and post-training system identification with a Projected Universal Policy and transfers three Darwin OP2 walking controllers.

I. INTRODUCTION

  • Biped policies are sensitive to simulation-to-hardware discrepancies, and domain randomization alone is insufficient to overcome the reality gap.Instability also makes task-relevant data hard to collect before policy training.Source: §I · p. 1
  • Pre-sysID estimates a range of model parameters from potentially task-irrelevant data; post-sysID uses task-relevant hardware trajectories to optimize policy performance.The two stages bracket policy training rather than iterating between system identification and learning.Source: §I · p. 1
  • A Projected Universal Policy (PUP) jointly learns a projection from model parameters to a low-dimensional latent variable and policies conditioned on that variable.This lets the policy adapt across environments within the trained parameter range.Source: §I · p. 2
  • The method transfers simulation-trained Darwin OP2 controllers for forward, backward, and sideways walking, succeeding on hardware in 25 real-world trials.It is compared with single-model system identification and robust-policy baselines.Source: §I · p. 2

II. RELATED WORK

Prior sim-to-real work uses system identification, robust policies, or policy fine-tuning, but extending these approaches to unstable biped locomotion remains challenging. Pre-sysID addresses the lack of task-relevant trajectories by estimating a model-parameter range from generic hardware behaviors.

II. RELATED WORK

  • Sim-to-real methods identify dynamics or train robust policies, but robust-policy approaches assume the test environment lies near training environments and broader randomization may be costly and conservative.Examples include perturbations, ensembles, sensor-noise randomization, and domain randomization.Source: §II · p. 2
  • Interleaving system identification and policy learning has transferred manipulation, but quick biped failures may leave too little data for meaningful updates.Cully et al. instead search manually specified discrete behaviors, whereas this work generates policies by varying system dynamics during training.Source: §II · p. 2
  • Direct real-robot training has succeeded for multi-legged robots, but frequent resets and biped instability make it challenging for bipeds.Sim-to-real transfer moves policies trained in simulation to real robots, despite the reality gap.Source: §II · p. 2

III. PRE-TRAINING SYSTEM IDENTIFICATION (PRE-SYSID)

  • Pre-sysID estimates a reasonable range of model parameters for domain randomization rather than fitting one true model from task-irrelevant trajectories.The hardware trajectories include joint exercise without contact and standing or falling with ground contact.Source: §III · p. 3

A. Optimizing Range of Model Parameters

The parameter-range procedure fits simulation models to hardware trajectories, allowing different models to explain different trajectory subsets. Bounds are derived from the optimized models and expanded to support policy learning.

A. Optimizing Range of Model Parameters

  • The procedure fits a nominal model to all trajectories, then refits random subsets with regularization toward the nominal parameters.The regularization weight is wreg = 0.05 in the experiments.Source: A. Optimizing Range of Model Parameters · p. 3
  • Parameter bounds use per-dimension minima and maxima across subset fits, expanded by 10% on each side.This yields a range rather than relying only on the nominal fit.Source: A. Optimizing Range of Model Parameters · p. 3
  • The trajectory fitness compares simulated and hardware motor positions, torso roll and pitch for standing or falling data, and simulated foot movement for standing data.The objective weights the torso-orientation and foot-movement terms by 10 and 20, respectively.Source: A. Optimizing Range of Model Parameters · p. 3

B. Neural Network PD Actuator

The actuator is identified as a major source of modeling error, so the paper augments a conventional PD model with a neural network. Additional identified parameters include foot-ground friction and torso center of mass.

B. Neural Network PD Actuator

  • The Neural Network PD Actuator (NN-PD) models actuator behavior because actuators are a main source of modeling error relative to mass, dimensions, and joint parameters.The model predicts proportional and derivative gains from motor position difference and velocity.Source: B. Neural Network PD Actuator · p. 3
  • The NN-PD uses a shared one-hidden-layer network with five tanh nodes and 27 weights, while Darwin OP2 cannot measure applied torque accurately enough to train a torque-output network.Motor groups receive separate learnable scaling factors and torque limits.Source: B. Neural Network PD Actuator · p. 4
  • The identified model also includes foot-ground friction and torso center of mass; the 41-dimensional nominal parameters yield 14-dimensional optimized bounds.The neural-network weights are fixed rather than independently randomized because they depend on one another.Source: B. Neural Network PD Actuator · p. 4

IV. LEARNING PROJECTED UNIVERSAL POLICY

The section formulates locomotion learning as reward-maximizing reinforcement learning, then introduces PUP to condition policies on a learned low-dimensional representation of model parameters.

IV. LEARNING PROJECTED UNIVERSAL POLICY

  • The reinforcement-learning objective is to find a policy that maximizes accumulated reward over trajectories generated from the initial-state distribution.The formulation defines the state and action spaces, transition and reward functions, initial-state distribution, and discount factor.Source: §IV · p. 4
  • A Universal Policy conditions its family of policies directly on model parameters sampled within the Pre-sysID bounds, but this poses optimization challenges on a biped robot.The passage notes prior sim-to-sim success for similar parameter dimensions [7].Source: §IV · p. 4
  • PUP maps model parameters to a lower-dimensional latent variable and jointly trains that projection with policies conditioned on the latent variable.The projection exploits redundancy in how model parameters affect policies; the latent dimension is 3 in these experiments, and training uses PPO.Source: §IV · p. 4

V. POST-TRAINING SYSTEM IDENTIFICATION (POST-SYSID)

Post-sysID searches the latent space using Bayesian Optimization to select a PUP setting for real-hardware performance, measured by distance traveled before falling.

V. POST-TRAINING SYSTEM IDENTIFICATION (POST-SYSID)

  • Bayesian Optimization searches for the latent setting that gives the conditioned PUP its best performance on real hardware.The authors chose BO because they found it generally more sample-efficient than CMA for their low-dimensional latent space; prior work used CMA-ES.Source: §V · p. 4
  • 5 initial latent samples build the Gaussian Process; BO runs for 20 iterations, using the best setting found as the final output.The initial samples are drawn uniformly, and each BO iteration selects a point balancing exploitation and exploration.Source: §V · p. 4
  • 25 hardware trajectories complete policy transfer in less than 15 minutes, with distance traveled before falling as the fitness measure.The authors say this simple fitness function could be replaced by more sophisticated ones.Source: §V · p. 4

VI. EXPERIMENT

The experiment trains and transfers forward, backward, and sideways locomotion policies for the Darwin OP2, using scripted robot motions, a stepping reference, and a shared reward design.

A. Experiment Setup

  • The Darwin OP2 has 20 motors, and the hardware setup uses motor positions with consecutive readings for velocity information and an added IMU to reach 33 Hz control.The added Bosch bno055 IMU is used for orientation estimation because the onboard IMU and motors share a communication port.Source: A. Experiment Setup · pp. 4–5
  • Pre-sysID data include suspended motor step responses at magnitudes 0.1, 0.3, and 0.6, plus stand-up and fall trajectories with orientation recordings for ground-contact motions.The robot performs manual-scripted movements from random poses while suspended to avoid ground contact during motor-response collection.Source: A. Experiment Setup · p. 5
  • The robot walks on a yoga mat over a white board, and Fig. 3 shows forward, backward, and sideways deployments on the Darwin OP2.The deformable surface was chosen to provide protection for the robot.Source: A. Experiment Setup · p. 5, Figure 3 · p. 5
  • The policy observes motor positions and estimated orientation over two timesteps and adjusts a manually generated stepping-in-place reference trajectory.The adjustment magnitude is 0.3 for forward and sideways walking, 0.2 backward, and actions are discretized into 11 bins per dimension.Source: A. Experiment Setup · p. 5
  • The shared reward combines speed, torque and work penalties, reference tracking, and a constant no-fall reward; mirror-symmetry loss improved learned gait quality.For different walking directions, the robot coordinate frame is rotated so the desired direction aligns with positive x.Source: A. Experiment Setup · p. 6

B. Baselines

The evaluation compares the proposed method with a single-model policy and a domain-randomized robust policy, while adding sensor and networking uncertainty during training.

B. Baselines

  • Nominal identifies one model during Pre-sysID and trains a policy for that model.Source: B. Baselines · p. 6
  • Robust uses the Pre-sysID parameter range to train a policy with domain randomization.Source: B. Baselines · p. 6
  • Training adds control-frequency variation from 25 to 33 Hz, orientation bias from U(−0.3, 0.3), and motor-position noise with standard deviation 0.01.PUP also receives input-parameter noise drawn from U(−0.25, 0.25).Source: B. Baselines · p. 6

C. Performance on Locomotion Tasks

Our method outperforms the baselines in distance and time before balance loss, and is the only method to reach the board edge.

C. Performance on Locomotion Tasks

D. Effect of using NN-PD Actuators

NN-PD actuators improve model fit and behavioral reproduction relative to PD-only, while identified pre-sysID parameter bounds vary in confidence and may not map cleanly to target tasks.

D. Effect of using NN-PD Actuators

E. Identified Model Parameter Bounds

VII. DISCUSSION AND CONCLUSION

The paper proposes sim-to-real transfer through two-stage system identification linked by a latent-conditioned Projected Universal Policy, and reports forward, backward, and sideways walking on Darwin OP2 using 25 hardware trials.

VII. DISCUSSION AND CONCLUSION

  • Two-stage system identification connected by a Projected Universal Policy modulated by a low-dimensional latent variable is the method’s central design.Source: §VII · p. 7
  • Figure 8 presents identified model-parameter bounds alongside nominal parameter values.Source: Figure 8 · p. 7
  • The method transferred Darwin OP2 policies for forward, backward, and sideways walking using 25 real-world trials.Source: §VII · p. 7
  • Post-sysID uses task-relevant trajectories with a fitness function based only on traveling distance and elapsed time; comparable baseline reward evaluation would require global position and orientation instrumentation.Source: §VII · p. 7
  • Fewer than 2,500 task-relevant steps were used for post-sysID, while each simulation learning iteration used 20,000 steps.Source: §VII · p. 7
  • The model-parameter set is selected manually based on prior knowledge, and its applicability to other hardware or tasks remains unclear.Source: §VII · p. 7

Related papers

Loading 1903.01390v2…