Source-linked AI summary
Learning and Transfer of Modulated Locomotor Controllers
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, David Silver
TL;DR
The paper addresses how locomotion controllers can acquire reusable motor behavior while solving new tasks. It uses a pre-trained low-level controller modulated by a higher-level controller, and reports solutions across swimmer, quadruped, and humanoid transfer tasks where direct end-to-end learning failed.
Problem
The paper investigates how to learn complex locomotion while reusing low-level motor structure across tasks instead of training each behavior end to end.
Method
The method pre-trains a low-level proprioceptive controller, fixes it during transfer, and connects it to a slower high-level controller that modulates its inputs using broader observations.
Results
The method produced solutions for swimmer, quadruped, and humanoid transfer tasks, whereas direct end-to-end learning failed.
Takeaways & Limitations
Information hiding enables low-level motor behaviors to remain sheltered from task-specific information and be reused across transfer tasks.
Takeaways & Limitations
The individual contributions of the architecture’s features, including different time scales, require further investigation, and low-level behavior reliability remains to be improved for difficult conditions.
Abstract
from arXiv · showhide
We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor primitives by training on simple tasks. This pre-trained module is fixed and connected to a low-frequency, high-level "cortical" network, with access to all sensors, which drives behavior by modulating the inputs to the spinal network. Where a monolithic end-to-end architecture fails completely, learning with a pre-trained spinal module succeeds at multiple high-level tasks, and enables the effective exploration required to learn from sparse rewards. We test our proposed architecture on three simulated bodies: a 16-dimensional swimming snake, a 20-dimensional quadruped, and a 54-dimensional humanoid. Our results are illustrated in the accompanying video at https://youtu.be/sboPYvhpraQ
1 Introduction
The paper proposes hierarchical locomotor controllers that separate direct motor control from abstract behavior selection. Information hiding, modulation, and multiple time scales are intended to create an abstraction barrier supporting reusable low-level functionality.
- The proposed controller hierarchy divides control between a low-level module producing direct motor commands and a high-level module selecting abstract motor behaviors.
- The high-level controller modulates the low-level controller through a communication bottleneck rather than directly specifying every motor command.
- The low-level controller receives only task-independent proprioceptive information, while the high-level controller receives proprioceptive, exteroceptive, and task-specific information.
- The low-level controller acts at every simulation step, whereas the high-level controller may update its modulatory signal less frequently.
- The design aims to shelter low-level motor control from task objectives while separating high-level behavior from detailed physics and actuation demands.
2 Architecture
The architecture composes a recurrent high-level controller with a non-recurrent low-level controller. The high level integrates full observations and periodically supplies a control signal that conditions the low-level action distribution.
- The policy is a composition of high- and low-level networks that outputs the parameters of a factorized Normal action distribution.
- The low-level controller maps proprioceptive information and the high-level control signal to the action-distribution parameters.
- The paper omits actions from the history used by the policy in these experiments.
- The recurrent high-level controller integrates observations at every time step but produces a new control signal only every K time steps.
- The high-level controller uses the full observation, its recurrent state, a control interval K, and the most recent update time to determine modulation timing.
3 Learning locomotor controllers with policy gradients
The paper trains hierarchical controllers with actor-critic policy gradients and introduces stochasticity at the high level. Reparameterized high-level noise is transformed by the low-level controller into correlated primitive actions, improving exploration structure over independent action noise.
- The method uses actor-critic policy gradients for both pre-training and transfer in fully observed and partially observed problems.
- Learning performs gradient ascent on expected discounted return, with a learned value-function baseline used to reduce policy-gradient variance.
- The value function is trained toward λ′-weighted return estimates, while λ and λ′ may differ between policy and value updates.
- Independent Gaussian action noise can produce poor, undirected, twitchy exploration, particularly in high-dimensional action spaces.
- The high-level stochastic signal is reparameterized as ˜gH(zτ, ϵτ) = µH(zτ) + σH(zτ)ϵτ, allowing gradients to backpropagate directly into the high-level controller.
- Holding high-level noise constant over its control interval and transforming it through the low-level controller induces spatially and temporally correlated primitive-level stochasticity.
4 Experiments
Experiments evaluate the hierarchical locomotor controller on a snake, quadruped, and humanoid, using pre-training to develop reusable locomotion and transfer tasks to test exploration and control. The experiments examine target-seeking, canyon traversal, soccer, and humanoid gate traversal under varying task difficulty and initialization.
- Experimental domains: The framework is evaluated on a swimming snake, quadruped, and humanoid, with increasing action dimensionality across the three physical domains.The snake has a 5-dimensional action space, the quadruped 8 dimensions, and the humanoid 21 dimensions.
- Training and transfer: Pre-training jointly learns a reusable low-level controller with a provisional task-specific high-level controller, after which transfer tasks replace the high-level component.The pre-training task uses shaping rewards to facilitate generic locomotion skills before evaluation on one or more transfer tasks.
- Snake: 4000-step snake trajectories show coherent swimming: without high-level modulation the snake swims nearly straight, while stronger modulatory noise produces more diverse behavior.The low-level controller is modulated with i.i.d. Gaussian noise sampled every 10 time steps during this analysis.
- Quadruped: The quadruped’s randomly modulated locomotor primitives achieve better spatial coverage than i.i.d. Gaussian motor noise in the exploration analysis.The comparison is shown in Figure 6 using three random-noise plots and four pre-trained-controller plots.
- Quadruped: In the quadruped transfer evaluation, sparse soccer feedback requires learning both navigation to the ball and ball manipulation into the goal zone.The task varies initial conditions, including cases where the quadruped must move away from the goal to position itself behind the ball.
- Humanoid: Humanoid experiments use a 27-degree-of-freedom body with 21 actuators, where preventing falls is itself a non-trivial control problem.The pre-training task includes straight movement and leftward or rightward circles of fixed radius 5m, with episodes terminated when height falls below 0.9m.
- Humanoid: For the humanoid, stronger modulation increases trajectory diversity but also fall probability, while Gaussian action exploration falls without making progress.Transfer trajectories through gates vary across low-level controllers, and not all learned slalom policies are equally successful.
- Humanoid: Humanoid transfer performance is sensitive to learning parameters and initial conditions, motivating richer and more constrained pre-training for more uniformly versatile controllers.The authors identify this as unsurprising given the humanoid’s complexity and the relatively small number of constraints in pre-training.
5 Related Work
The paper situates hierarchical motor control in longstanding biological and robotics traditions, including motor synergies, spinal motor primitives, and Dynamic Movement Primitives.
- Hierarchical motor control has roots in 19th-century theories and Bernstein’s 20th-century concept of multi-joint motor synergies.
- Spinal motor primitives later extended this tradition, while Dynamic Movement Primitives brought related ideas into robotics.
6 Conclusion
The paper presents a hierarchical architecture that transfers reusable low-level motor behaviors across locomotion tasks. It reports solutions on swimmer, quadruped, and humanoid transfer tasks where direct end-to-end learning failed, while identifying unresolved design questions and task-selection assumptions.
- The architecture learns low-level motor behaviors that transfer to new tasks through separate sensory access and operating time scales.
- The method was especially effective on challenging transfer tasks with sparse rewards, where motor babbling is unlikely to accrue reward.
- Direct end-to-end learning failed on swimmer, quadruped, and humanoid transfer tasks, whereas the proposed method produced solutions.
- The detailed contributions of architectural features, including different time scales, remain to be investigated more thoroughly.
- The approach assumes that simple low-level tasks can be proposed whose mastery facilitates other high-level tasks.
A Network Parameters
The experiments use a shared hierarchical neural-network design across bodies, varying hidden-unit counts and combining high- and low-level stochastic outputs.
- The same hierarchical architecture is used for all problems, with only the number of hidden units changed.
- Both controller levels parameterize means and standard deviations, with standard deviations generated by a sigmoid-output linear layer.
- The low-level controller has three tanh hidden layers, while its second-layer outputs are concatenated with the high-level controller output.
- The high-level controller emits 10 bottleneck units; snake and quadruped layers use 150 hidden units, versus 300 for the humanoid.
- During pre-training, the high-level controller is an LSTM with a perceptual encoder, while transfer uses a typically larger-capacity LSTM configuration.
B Experimental procedure for transfer tasks
Transfer experiments first pre-train reusable low-level controllers on simple locomotion tasks, then replace the provisional high-level controller and train a new transfer controller. Transfer is more sensitive to hyperparameters and initialization.
- Pre-training uses easy but sufficiently flexible tasks: target navigation for the snake and quadruped, and shared straight and circular walking tasks for the humanoid.
- The low-level controller is trained jointly with a provisional high-level controller that supplies task-specific information and maintains controllability.
- After pre-training, the high-level controller is replaced by Gaussian noise for analysis or a randomly initialized stochastic network for transfer.
- The transfer high-level controller differs from the pre-training controller in stochasticity, capacity, and operating setup.
- Transfer experiments are more sensitive to hyperparameter choices and initial conditions, possibly because the tasks are more difficult.
C.1 Variability across low-level controllers
Transfer performance varies across pre-trained low-level controllers, with snake and quadruped controllers generally producing reasonable solutions while humanoid controllers are less reliable.
- Snake and quadruped controllers all produced at least some reasonable solutions on the transfer task.
- Humanoid controllers showed greater variability across controllers, random seeds, and high-level-policy initializations on the slalom task.
- Some humanoid controllers produced only a small number of robust slalom solutions.
- Poor-performing humanoid slalom controllers often failed at episode starts because they were not robust to random initial states.
C.2 Learned low-level behavior and high-level exploration
Pre-trained controllers generate diverse behaviors under high-level modulation, and stronger or less frequent modulation generally broadens trajectories. However, large modulation can destabilize humanoid behavior, while K = 10 steps produces a stronger effect than K = 1.
- The trajectories used random initial configurations for snake and quadruped, while the low-level action-distribution standard deviation was set to 0.3.
- Different pre-trained low-level controllers generated diverse behaviors for each creature under high-level modulation.
- Without high-level control noise, snake and humanoid behavior was often stereotyped, including nearly straight swimming or fixed-radius circles.
- Increasing modulation strength generally made snake and humanoid trajectories more diverse, although humanoids fell easily at large standard deviations.
- K = 10 steps increased the modulatory effect of high-level noise compared with K = 1 step.