PaperlayerView exact revision on arXiv

Source-linked AI summary

Preparing for the Unknown: Learning a Universal Policy with Online System Identification

Wenhao Yu, Jie Tan, C. Karen Liu, Greg Turk

arXiv:1702.02453v3cs.LGcs.ROeess.SY

TL;DR

Robotic policies trained in simulation must handle unknown dynamics and the expense of real-world system identification. UP-OSI combines a universal policy conditioned on dynamic parameters with online identification from recent histories. Across several tasks, it operates over broad and unseen dynamic ranges, while remaining untested on real robots and requiring further study for high-dimensional parameters.

  • Problem

    Robotic simulation and real-world behavior differ, while learning accurate dynamics from real-world trials can be expensive, time-consuming, and dangerous.

  • Method

    UP-OSI trains a universal policy over simulated dynamic models and an online system identification function that predicts model parameters from recent state-action history.

  • Results

    UP-OSI successfully operates across unknown dynamic models and outperforms UP alone outside the training range, including a range 100% beyond the original training range.

  • Takeaways & Limitations

    The approach provides robust control that estimates dynamics on the fly and can adapt to changing environments without real-world samples during offline training.

  • Takeaways & Limitations

    High-dimensional model-parameter identification remains unresolved, theoretical upper bounds and convergence conditions are not established, and real-robot transfer has not yet been demonstrated.

Abstract

from arXiv · show

We present a new method of learning control policies that successfully operate under unknown dynamic models. We create such policies by leveraging a large number of training examples that are generated using a physical simulator. Our system is made of two components: a Universal Policy (UP) and a function for Online System Identification (OSI). We describe our control policy as universal because it is trained over a wide array of dynamic models. These variations in the dynamic model may include differences in mass and inertia of the robots' components, variable friction coefficients, or unknown mass of an object to be manipulated. By training the Universal Policy with this variation, the control policy is prepared for a wider array of possible conditions when executed in an unknown environment. The second part of our system uses the recent state and action history of the system to predict the dynamics model parameters mu. The value of mu from the Online System Identification is then provided as input to the control policy (along with the system state). Together, UP-OSI is a robust control policy that can be used across a wide range of dynamic models, and that is also responsive to sudden changes in the environment. We have evaluated the performance of this system on a variety of tasks, including the problem of cart-pole swing-up, the double inverted pendulum, locomotion of a hopper, and block-throwing of a manipulator. UP-OSI is effective at these tasks across a wide range of dynamic models. Moreover, when tested with dynamic models outside of the training range, UP-OSI outperforms the Universal Policy alone, even when UP is given the actual value of the model dynamics. In addition to the benefits of creating more robust controllers, UP-OSI also holds out promise of narrowing the Reality Gap between simulated and real physical systems.

I. INTRODUCTION

The paper addresses the Reality Gap and the cost of real-world system identification by training control policies across simulated dynamic models. UP-OSI combines a universal policy with online identification to operate under unknown and changing dynamics.

  • The Reality Gap reflects discrepancies between simulated and real-world robotic performance caused by modeling, hardware, sensing, and other unmodeled factors.
  • Existing strategies either rely on a robust policy under an approximate model or fit dynamics from costly, time-consuming, and potentially dangerous real-world trials.
  • UP-OSI uses extensive simulated data to train a universal control policy and an online system identification model for unknown dynamic models.
  • OSI predicts dynamic parameters from recent state-action history, and UP uses those predictions with the current state to compute actions during execution.
  • The method is designed to avoid real-world samples during offline training and adapt dynamically to changing environmental factors.
  • Across cart-pole, double-pendulum, hopper, and manipulator tasks, UP-OSI operates under unknown parameters, outside training ranges, and sudden environmental changes.

B. Transfer Learning in Reinforcement Learning

Prior work transfers simulated policies, uses system identification, or trains robust policies over model ensembles. UP-OSI instead explicitly conditions its universal policy on model parameters while estimating them online.

  • System identification is commonly interleaved with policy optimization to reduce real-world experiments, while direct simulation-to-real transfer applies or adapts simulated policies.
  • UP-OSI learns a universal policy over model parameters together with an online system identification network that estimates those parameters during execution.
  • Figure 1 depicts OSI estimating parameters from motion history and UP using those estimates with the current state to compute control.
  • Unlike ensemble-based robust policies, UP explicitly receives model parameters as input, making training across dynamic models an easier learning task according to the authors.

C. Learning Policy in Unknown Environment

The policy-learning framework targets control across a parameterized space of unknown dynamics. It separates universal policy learning from online parameter estimation and trains the policy to specialize using explicit dynamics inputs.

  • UP maps the current state and dynamic parameters to control actions, unlike conventional policies that map only state to action.
  • During execution, OSI predicts parameters from recent state-action history, then UP computes an action from the prediction and current state.
  • The framework pretrains UP to anticipate the dynamic models that OSI may explore, breaking the mutual dependence of alternating policy and identification optimization.
  • The authors motivate a universal policy because small parameter changes can require drastically different successful control policies for dynamic tasks.
  • A large neural network can directly represent a universal policy across parameterized dynamics and achieve rewards comparable to policies trained for specific parameters.
  • TRPO trains UP by augmenting the state with model parameters and modifying exploration because forward simulation does not change the parameter portion of the state.

B. Learning Online System Identification Model

OSI is trained to infer dynamic parameters from short state-action histories generated in simulation. Iterative training adds mismatched control and simulation parameters to address errors caused by initially idealized trajectories.

  • OSI maps a short recent history of states and actions to continuously identify the dynamic parameters needed by UP.
  • OSI training treats each rollout history as supervised input and its generating dynamic parameters as the target output.
  • The training buffer is built from short history segments sampled from rollouts generated across randomly sampled dynamic parameters.
  • Initial OSI training performed worse than using UP with true parameters because its examples covered only consistent, optimally controlled trajectories.
  • The method iteratively adds bad cases by controlling with OSI-predicted parameters while simulating under different true parameters, then retrains OSI.

IV. EVALUATION

The evaluation tests UP-OSI on four dynamic motor-control problems under unknown model parameters, including conditions outside training and sudden environmental changes. The reported results include successful operation without advance parameter knowledge and performance beyond the training range.

  • Evaluation: UP-OSI is evaluated on four dynamic motor control problems without knowing the true model parameters in advance.The varied parameters include mass, inertia, friction coefficient, and task-related parameters.
  • Evaluation: UP-OSI uses OSI-predicted parameters during execution to control systems with unknown dynamics.The policy relies on OSI to identify the parameters during execution rather than receiving the true values in advance.
  • Evaluation: UP-OSI operates successfully outside the dynamic-model space used for training.The evaluation specifically tests model parameters beyond the training range.
  • Experimental setup: The experiments use PyDart2, a Python wrapper for the DART multibody physics simulator supported by Gazebo.The simulation timestep is 0.002s.
  • Training: UP training uses 500 TRPO-update iterations, while OSI training uses five iterations with motion histories of length three.The history length is reported as the minimum needed to learn a second-order dynamic model.

A. Double inverted pendulum with unknown center of mass

The double inverted pendulum task tests balancing when the lower pole’s center of mass is unknown and varies over a wide range. UP-OSI improves toward UP-true, identifies the parameter, and is evaluated against a regular policy across model values.

  • Task setup: The unknown parameter is the lower pole’s center-of-mass offset, varied over µ ∈ [−0.6m, 0.6m].The vertical offset 0.2µ breaks symmetry and increases control difficulty.
  • Evaluation: UP-OSI is compared with UP-true and a regular controller across center-of-mass values.Performance is the normalized average accumulated reward over 20 rollouts from randomly perturbed initial states; values above 1.0 indicate successful balancing.
  • Evaluation: Providing model parameters to UP produces a more powerful control policy across a range of dynamic models.The regular controller uses conventional state input and control output while training across model parameters.
  • Learning process: UP-OSI performance improves over training and approaches UP-true.The reward across the training range of µ increases iteratively.
  • Online system identification: OSI identifies the double inverted pendulum’s model parameter from motion history.Figure 2(d) reports the mean and standard deviation of the identified parameter for each ground-truth µ.

B. Manipulator with unknown object mass

The manipulator task requires grabbing and throwing a block to a target height without overshooting while inferring the unknown block mass. UP-OSI is evaluated against UP-true using maximum block height and predicted mass.

  • Task setup: The robot arm must throw a grabbed block to a certain height but not beyond it.The block begins in the air near the gripper while the arm initially points downward.
  • Task setup: The unknown model parameter is the block’s mass, which the robot must infer to apply the appropriate throwing effort.The target height is htarget = 2m, and zero reward beyond the target encourages low velocity at that height.
  • Evaluation: Performance is measured by the block’s highest reached point, with values closer to htarget = 2m considered better.UP-OSI and UP-true are compared using this maximum-height measure.
  • Online system identification: OSI learns to identify the block mass in the object-throwing task.The mean and standard deviation of predicted block mass are plotted throughout testing.

C. Hopper with unknown friction coefficient

The Hopper task tests whether UP-OSI can identify unknown contact friction online while controlling a single-leg robot to hop forward quickly without falling. Performance is evaluated by distance traveled before termination across friction conditions.

  • UP-OSI identifies the Hopper’s unknown foot-ground friction coefficient online during locomotion.The task is to hop forward as fast as possible without falling, with contact friction treated as an unknown model parameter.
  • The unknown friction coefficient ranges from 0.3 to 1.0, and termination occurs when the hopper falls or reaches the rollout limit.
  • UP performs well mainly for friction coefficients µ = [0.6, 1.0], a range described as sufficient for most practical applications.

D. Cart-pole swing-up with unknown pole length and unknown attached mass

The modified cart-pole task requires swing-up and balance under force limits, with unknown pole length and attached mass. UP-OSI achieves high reward across a range of pole lengths while estimating both unknown parameters.

  • Task design: The experiment increases difficulty by limiting cart force to [−40N, 40N] and adding a tip mass to mimic weight lifting.
  • Reward: The reward combines swing-up and balance terms with a penalty encouraging the cart to remain near the track center.
  • Unknown parameters: The unknown parameters are attached mass µmass ∈ [0.1kg, 1.0kg] and pole length µlength ∈ [0.2m, 0.8m].
  • Results: UP-OSI achieves high reward across a range of unknown pole lengths while using only three time steps of history as input.The attached tip mass is identified simultaneously with pole length.

E. Generalization to varying model parameter

UP-OSI generalizes across changing and unseen dynamic parameters by estimating them online, including friction changes and parameters 100% beyond training ranges. The method also has stated limitations around high-dimensional identification, latency, sensor and actuator uncertainty, and transfer across simulators.

  • Generalization to varying model parameter: UP-OSI identifies changing friction during hopper locomotion, despite neither network being trained with temporally varying parameters.For µvary = 0.55, OSI tracks the parameter change during the task.
  • Generalization beyond training range: UP-OSI operates on cart-pole parameters 100% beyond training, using pole lengths of [0.8m, 1.4m] and attached masses of [1.0kg, 1.9kg].The extended parameters are linearly coupled and exceed the original training ranges of [0.2m, 0.8m] and [0.1kg, 1.0kg].
  • Generalization beyond training range: UP-OSI significantly outperforms UP-true on cart-pole cases with pole length and attached mass outside the training range.UP-true receives the ground-truth model parameters, whereas UP-OSI estimates them during execution.
  • Limitations: The authors identify high-dimensional model-parameter estimation as an unresolved challenge and do not establish theoretical upper bounds or convergence conditions.Their cart-pole demonstration uses parameters in R4, but sample-efficiency in higher-dimensional spaces requires further analysis.
  • Limitations: The current implementation assumes deterministic policies and dynamics, while latency may complicate OSI’s history-queue scheme.The work also focuses primarily on model-parameter uncertainty rather than sensor and actuator uncertainty.
  • Limitations: Testing UP-OSI across simulators and on real robots remains future work because real-world dynamics may not fit the chosen parameterization.The authors note that time-varying estimates could provide a more expressive model for real-world dynamics.

VI. CONCLUSION

UP-OSI couples a Universal Policy with Online System Identification to control across varied dynamic models while estimating parameters during execution. It nearly matches a ground-truth-parameter baseline, outperforms the Universal Policy outside training ranges, and adapts to changing friction.

  • VI. CONCLUSION: UP-OSI couples a Universal Policy with an Online System Identification model to estimate dynamic parameters on-the-fly.The approach uses extensive offline physical simulation and deep reinforcement learning.
  • VI. CONCLUSION: UP-OSI achieves almost the same performance as UP-true while deducing unknown dynamic models during execution.UP-true is the baseline supplied with the actual model parameters.
  • VI. CONCLUSION: UP-OSI performs better than the Universal Policy alone when dynamic models fall outside the training range.The conclusion identifies this extrapolation result as a key performance benefit.
  • VI. CONCLUSION: Continuous parameter estimation lets UP-OSI adapt to changing environments such as locomotion over surfaces with varying friction.The policy estimates the dynamic model at every time step.
Loading 1702.02453v3…