PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?

Xue Bin Peng, Michiel van de Panne

arXiv:1611.01055v1cs.LGcs.GRcs.RO

TL;DR

The paper asks how action representation affects DeepRL locomotion learning and performance. It compares four actuation models across articulated-figure motion imitation and finds that local-feedback parameterizations generally improve learning, robustness, and motion quality, especially for complex characters.

  • Problem

    The impact of action parameterization on DeepRL learning difficulty, policy quality, and robustness is not yet well understood.

  • Method

    The study compares torque, musculotendon-activation, target-angle, and target-velocity policies across locomotion-imitation tasks for planar articulated figures and multiple motions.

  • Results

    Action parameterizations incorporating local feedback improve learning speed and performance across motions and morphologies, with differences growing as characters become more complex.

  • Takeaways & Limitations

    Higher-level action parameterizations scale more gracefully to complex characters while preserving the generality afforded by torque-level control.

  • Takeaways & Limitations

    The experiments use planar articulated-figure simulations, and MTU comparisons may underestimate performance because optimized actuator parameters may not be optimal.

Abstract

from arXiv · show

The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gait-cycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.

1 INTRODUCTION

The paper examines how action parameterization affects DeepRL locomotion, comparing torque-level and higher-level representations across learning, robustness, motion quality, and query rates.

  • The study compares torques, musculotendon activations, target joint angles, and target joint velocities for dynamic articulated-figure locomotion.
  • Action parameterizations with local feedback can improve learning speed and performance while retaining the generality of torque-level control.
  • The evaluation covers learning difficulty, robustness, motion quality, and policy query rates across multiple actuation models.
  • Higher-level parameterizations support more complex body structures and subjective improvements in motion quality.
  • The framework includes a DeepRL motion-imitation setup and an optimization approach combining policy learning with actuator optimization for complex muscle models.

2 BACKGROUND

The paper formulates locomotion control as an episodic reinforcement-learning problem with stochastic policies, value estimation, and policy-gradient updates based on state transitions and advantages.

  • An agent observes state s_t, samples action a_t from policy π, receives reward r_t, and transitions to a new state through environment dynamics.
  • The policy learns parameters θ that maximize expected cumulative discounted reward over horizon T.
  • The policy-gradient theorem supplies an improvement direction using the discounted state distribution and a generalized advantage function.
  • A parameterized value function is learned by minimizing Bellman loss, while policy and value parameters are trained together in an actor-critic framework.
  • Policies use Gaussian action distributions with parameterized mean μ_θ(s) and manually specified fixed covariance, sampling actions by adding Gaussian noise.
  • The policy-gradient update shifts probability toward actions associated with higher-than-expected rewards and away from lower-than-expected rewards.

3 TASK REPRESENTATION

The task combines reference-motion imitation with state features describing the articulated figure and target motion, while comparing four actuation models under a weighted tracking reward.

  • 3.1 MOTION IMITATION: The policy imitates a cyclic reference motion represented as a sequence of kinematic poses and finite-difference reference velocities.
  • 3.1 MOTION IMITATION: Reference motions come from recorded simulations or hand-authored keyframes, requiring close reproduction while satisfying physical constraints.
  • 3.2 STATES: State features include root height, link positions relative to the root, and each link’s center-of-mass velocity.
  • 3.2 STATES: Target-motion features are appended to the state to provide motion-phase information during cyclic imitation.
  • 3.3 ACTIONS: Torque actions directly specify joint torques, whereas PD and velocity actions specify target angles or velocities that controllers convert into torques.
  • 3.4 REWARD: The reward is a weighted sum of pose, joint-velocity, end-effector, root, and center-of-mass tracking terms.
  • 3.5 INITIALIZATION: Episodes initialize pose and velocity from states sampled uniformly along the reference trajectory, guiding exploration near the target motion.

4 ACTOR-CRITIC LEARNING ALGORITHM

The learning procedure uses policy-gradient exploration with positive temporal-difference updates, while actuator optimization addresses additional parameters in complex musculotendon models.

  • Learning updates: Positive temporal-difference updates reduce sensitivity to advantage scale and avoid instabilities associated with negative temporal-difference updates.
  • Exploration and evaluation: Training uses stochastic policies for exploration and deterministic mean-action policies for runtime evaluation.
  • Exploration and evaluation: The exploration indicator λ equals 1 for exploration noise and 0 for deterministic mean actions, with training probability controlled by an annealed ε-greedy schedule.
  • Episode procedure: Episodes begin from sampled reference-trajectory states and terminate after prolonged trunk-ground contact, producing zero reward thereafter.
  • Actuator optimization: CMA-based actuator optimization jointly searches musculotendon and policy parameters because complex actuation models require additional performance-sensitive parameters.

5 RESULTS

The study evaluates four actuation models for locomotion policies on simulated planar figures, comparing learning, performance, robustness, motion quality, and query-rate sensitivity. PD performs consistently well, while local-feedback parameterizations generally outperform torque control, especially for complex characters.

  • Experimental setup: Policies are trained for a 2D biped, dog, and raptor across actuation models with 58–214 state dimensions, 6–44 action dimensions, and 0–282 actuator parameters.MTU models use at least twice as many action parameters because they include antagonistic muscle pairs.
  • Policy performance and learning speed: PD achieves comparable-to-the-best performance for all motions and learns faster than other parameterizations for 5 of 7 motions.AUC is used as a proxy for learning speed, with 0 representing no progress and 1 representing the best possible performance without training.
  • Policy performance and learning speed: Torque policies have among the poorest final performance, with deterioration relative to PD and Vel as character complexity increases.Most parameterizations perform similarly on the simple biped, whereas differences become more pronounced for the dog and raptor.
  • Policy performance and learning speed: MTU policies often learn slowest but can produce more natural and responsive motions after actuator optimization.The reward primarily measures similarity to the reference motion and may not capture natural responses under perturbation.
  • Policy robustness: PD policies are among the most robust across motions under random trunk perturbations, while torque is significantly less robust for the biped walk and MTU is least robust for the raptor run.Perturbations use random-direction forces applied for 0.1–0.4s, with 1–4s between perturbations, and performance averages 128 episodes of 20s each.
  • Query rate: PD and Vel cope more effectively with lower query rates, whereas torque performance degrades more rapidly; MTU actions and resulting torques are visibly smoother than those of PD and Vel.Separate policies are trained at 15Hz, 30Hz, 60Hz, and 120Hz.

6 RELATED WORK

Prior work has used all four action types in reinforcement learning and physics-based character control. The paper connects these existing action representations to a systematic comparison of their effects on locomotion learning.

  • Reinforcement learning and control: DeepRL studies have used torques, joint velocities, implicit velocity policies, and abstract actions for continuous-action control.Examples include OpenAI Gym torque control, PR2 effort control, joint velocities, and abstract actions.
  • Biomechanics and animation: Biomechanics research emphasizes musculotendon systems as embodied controllers that provide mechanical intelligence, energy storage, and preflex-like feedback.These ideas motivate action representations that incorporate physical properties rather than exposing only low-level torques.
  • Biomechanics and animation: Physics-based character simulation has also used the four evaluated actuation forms, including torque, velocity, PD, and muscle-based control.The paper situates its comparison within both machine-learning and computer-animation control practices.

7 CONCLUSIONS

The experiments suggest that action parameterizations with local feedback improve locomotion policy learning and performance across motions and morphologies, with larger benefits for complex characters. The conclusions remain bounded by planar simulations, actuator-tuning challenges, and sensitivity to algorithms and hyperparameters.

  • Conclusions: PD target angles, MTU activations, and target velocities can improve policy performance and learning speed across motions and character morphologies.These parameterizations include basic local feedback and better reflect the embodied role of mechanics in motion control.
  • Conclusions: The gap between low-level and high-level action parameterizations grows with character complexity, and high-level parameterizations scale more gracefully.The paper reports this as an experimental trend across different articulated morphologies.
  • Limitations: The results are demonstrated only on planar articulated-figure simulations, leaving extension to 3D as future work.This limits direct generalization of the findings to three-dimensional locomotion.
  • Limitations: MTU actuator parameters remain difficult to tune, may not be optimal, and are optimized for a specific motion rather than a larger suite.Consequently, comparisons may not reflect MTUs’ full potential with more optimal actuator parameters.
  • Limitations: Replication with other reinforcement-learning algorithms and motion-control tasks is needed because hyperparameter choices can significantly affect performance.The authors nevertheless expect the general trends to be likely to generalize.
  • Future work: Future work could optimize actuation mechanics and control capabilities jointly through co-design methods.The paper presents this as a possibility motivated by evolutionary co-design.

SUPPLEMENTARY MATERIAL

The supplementary material presents the learning algorithm, actuator-optimization procedure, and policy-training machinery used in the experiments. The framework alternates policy learning with actuator optimization for the optimized MTU setting.

  • Algorithms: Algorithm 1 is titled Actor-critic Learning Using Positive Temporal Differences.The listed procedure initializes policy and actuator parameters, then iteratively updates them through policy learning and CMA-based actuator optimization.
  • Actor-critic implementation: The actor-critic implementation collects exploratory transitions, stores them in replay memory, and updates the critic and actor from minibatches.Transitions contain states, actions, rewards, next states, and the exploration indicator λ.
  • Actor-critic implementation: The actor update uses positive temporal-difference errors to adjust actions toward the policy output before applying a bounded action gradient.The supplementary algorithm explicitly conditions actor updates on δj > 0.
  • Algorithms: Algorithm 2 is titled Alternating Actuator Optimization.It corresponds to the actuator-optimization component of the alternating training framework.
  • Algorithms: The alternating optimization procedure updates policy parameters θ with Algorithm 1 and actuator parameters ψ with CMA until completion.Each pass alternates optimization of the policy and actuator parameter sets.

MTU Actuator Optimization

The paper jointly optimizes MTU actuator parameters and policy parameters through alternating learning and evolutionary optimization. Performance improves across optimization passes and final versus initial MTU parameters are compared for three motions.

  • MTU Actuator Optimization: Actuator parameters ψ parameterize system dynamics, allowing expected cumulative reward to be optimized jointly over policy and actuator choices.The alternating procedure trains θ with fixed ψ, then optimizes ψ with CMA while keeping θ fixed.
  • MTU Actuator Optimization: Figure 5 illustrates performance improvement during actuator optimization for three different agents.
  • MTU Actuator Optimization: Figure 6 compares learning curves obtained with initial and final MTU parameters for the same three motions.

Bounded Action Space

Action bounds arise from physical properties such as torque and neural activation limits. The method preserves gradients inside the bounds and modifies them only when actions violate those bounds.

  • Bounded Action Space: Torque and neural activation limits bound the action values available to a parameterization, and improper enforcement can destabilize learning.The passage motivates explicit handling of action bounds because gradients outside the bounds may be unreliable.
  • Bounded Action Space: The adopted bounded-action method preserves empirical gradients when bounds are respected and alters gradients only when bounds are violated.This differs from inverting gradients, which scales all gradients according to proximity to the bounds.

Reward

The reward function measures imitation quality using weighted pose, velocity, end-effector, root, and center-of-mass terms. The supplied figures and tables describe reward-related experimental visualizations, policy architecture, and performance summaries.

  • Reward: The reward function is defined as a weighted sum of terms that encourage the policy to track a reference motion.
  • Reward: Pose and joint-velocity terms penalize deviations from reference motion, while end-effector and root terms penalize positional errors.
  • Reward: The center-of-mass term penalizes deviations between the character’s and reference motion’s center-of-mass velocities.
  • Reward: Figure 7 contrasts fixed initial states with reference-trajectory sampling, showing how initialization affects explored state-space regions.
  • Reward: Figure 8 depicts a three-layer policy network with 512 and 256 fully connected hidden units followed by a linear output layer.
  • Reward: Table 4 reports normalized cumulative reward and normalized learning-curve area under the curve for policies across characters and actuation models.
Loading 1611.01055v1…