Source-linked AI summary
Policies Modulating Trajectory Generators
Atil Iscen, Ken Caluwaerts, Jie Tan, Tingnan Zhang, Erwin Coumans, Vikas Sindhwani, Vincent Vanhoucke
TL;DR
Learning complex controllable locomotion is difficult when policies must represent periodic structure and memory from limited observations. The paper introduces PMTG, which lets simple policies modulate parameterized trajectory generators, and demonstrates controllable quadruped locomotion with simulation-to-real transfer.
Problem
Complex robotic behaviors such as locomotion remain difficult for learning-based methods because expressive policies are harder to train and require more data.
Method
PMTG combines a predefined, stateful trajectory generator with a learned policy that modulates its parameters and corrects its actions.
Results
PMTG learned controllable quadruped locomotion with a simple policy using only IMU readings and transferred successfully from simulation to the Minitaur robot.
Takeaways & Limitations
Simple policies can learn complex controllable locomotion when paired with trajectory generators that provide prior structure and internal state.
Takeaways & Limitations
The work relies on ad hoc trajectory generators chosen from intuition, leaving the best generator types and theoretical foundations for future study.
Abstract
from arXiv · showhide
We propose an architecture for learning complex controllable behaviors by having simple Policies Modulate Trajectory Generators (PMTG), a powerful combination that can provide both memory and prior knowledge to the controller. The result is a flexible architecture that is applicable to a class of problems with periodic motion for which one has an insight into the class of trajectories that might lead to a desired behavior. We illustrate the basics of our architecture using a synthetic control problem, then go on to learn speed-controlled locomotion for a quadrupedal robot by using Deep Reinforcement Learning and Evolutionary Strategies. We demonstrate that a simple linear policy, when paired with a parametric Trajectory Generator for quadrupedal gaits, can induce walking behaviors with controllable speed from 4-dimensional IMU observations alone, and can be learned in under 1000 rollouts. We also transfer these policies to a real robot and show locomotion with controllable forward velocity.
1 Introduction
PMTG combines a predefined, stateful trajectory generator with a learned policy that can modulate the generator and correct its actions. The approach targets complex periodic behaviors and is demonstrated on synthetic control and speed-controlled quadruped locomotion.
- Motivation: The architecture addresses robotics tasks whose complexity makes highly expressive policies difficult to train and data-intensive.The motivation includes locomotion, where recurrent or more complex policies may be harder to train.
- Architecture: PMTG combines prior trajectory knowledge with a learned policy that modulates the generator while observing its state.The policy can also directly correct the generator’s output.
- Contributions: The paper applies PMTG to a synthetic control problem and then to quadruped locomotion with desired speed as an external control input.The locomotion experiments use trajectory generators parameterized by stride length, frequency, and walking height.
- Contributions: Policies are trained with reinforcement learning or evolutionary strategies, including a one-layer linear policy using four-dimensional IMU observations.The learned policies are transferred to a real robot for controllable forward-velocity locomotion.
2 Related Work
Related work frames locomotion as periodic, structured motion and reviews optimization-based controllers, trajectory generators, and controllable-policy formulations. PMTG follows the single-controller approach of incorporating external control signals rather than switching among separate controllers.
- Optimization-based control: Optimization methods such as evolutionary algorithms, Bayesian optimization, and reinforcement learning are widely used to design locomotion controllers.Neural networks are common policy parameterizations, but architecture search can require substantial computation.
- Periodic locomotion: Locomotion controllers commonly exploit periodic structure through cyclic trajectories, central pattern generators, or policies that modulate predefined patterns.Prior work includes feedback balance control layered on user-specified pattern generators.
- Controllable locomotion: For controllable locomotion, one generic controller conditioned on external signals avoids the jerky motion and balance loss associated with abruptly switching separate controllers.The paper adopts this formulation and uses PMTG to pursue efficient learning.
3 Architecture
PMTG combines a predefined controller, a learned policy, and the controlled robot by extending the policy’s observations and actions with the trajectory generator’s state and parameters. In synthetic control, the generator supplies periodic structure and PMTG outperforms reactive PPO.
- Architecture: PMTG consists of a predefined controller, a learned policy, and the robot, with trajectory generators serving as the periodic-motion controller.The architecture can also be extended to predefined controllers such as kinematic controllers.
- Policy-TG interface: The policy receives the trajectory generator’s state and controls its parameters, while the combined action is u = u_tg + u_fb.The generator and policy outputs are summed, allowing learned correction or suppression of generator actions.
- Policy-TG interface: The trajectory generator acts as a black-box dynamical module whose state augments observations and whose controllable parameters augment the action space.Its internal state does not affect the reward.
- Synthetic control: In the synthetic task, a figure-eight generator provides prior structure while the policy controls x- and y-axis amplitudes and receives position plus generator state.The environment asks a point to follow a desired cyclic trajectory, with PPO training the policy.
- Synthetic control: PMTG + PPO reaches a reward close to zero, whereas Vanilla PPO fails to learn good behavior and remains worse even with an added time signal.The result is attributed to PMTG providing time-awareness and memory for a task with partially observable phase information.
4 Quadruped Locomotion
PMTG combines a stateful trajectory generator with a learned policy that modulates gait parameters and adds corrective actions for controllable quadruped locomotion. In simulation and robot experiments, simple policies learned speed tracking efficiently and transferred to real-robot walking, while higher-speed policies experienced occasional slipping.
- Controller Design: The trajectory generator encodes periodic leg motion using gait phases and parameters such as stride length, leg clearance, walking height, and frequency.Different phase patterns were used for walking and bounding, and the generator could be pre-optimized or hand-tuned to provide a rough gait prior.
- Controller Design: The controller receives IMU-derived state, desired velocity, and trajectory-generator phase, then outputs gait parameters and direct leg corrections.The trajectory generator uses walking height, amplitude, and frequency; the combined outputs become desired motor positions tracked by proportional-derivative controllers.
- Training: PMTG policies learned controllable walking with both PPO and ES, whereas reactive controllers without the trajectory generator failed to reach optimal rewards.Both linear and two-layer feed-forward policies achieved the desired behavior, but lower rewards indicated difficulty tracking changing target speeds without PMTG.
- Training: Fewer than 1000 rollouts were sufficient for good locomotion policies when ES used PMTG with a 77-parameter linear policy.The authors attribute this data efficiency to embedding prior knowledge in the trajectory generator and reducing policy-learning complexity.
- Results: A converged running policy tracked changing desired speeds by modulating gait amplitude and frequency, while its corrective actions changed leg-swing patterns across speeds.The robot followed the target speed during a rollout, and the resulting leg motion remained periodic but changed shape substantially with speed.
- Robot Experiments: Real-robot transfer succeeded for slower walking, while faster policies mostly completed rollouts but occasionally slipped at speeds up to 0.8 m s−1.The robot generally recovered from falls and continued walking, although slippage affected distance and direction for some policies.
5 Conclusion
PMTG combines parameterized trajectory generators with learned policies to provide prior knowledge and memory for complex behaviors. Quadruped experiments validated the approach with simple policies, low-dimensional IMU observations, multiple training algorithms and gaits, and transfer to a real robot, while leaving trajectory-generator selection as an open direction.
- 5 Conclusion: PMTG combines a parameterized Trajectory Generator with a learned policy that modulates it using environmental observations and the TG’s internal state.This combination provides prior knowledge and memory while enabling simple reactive policies to learn complex behaviors.
- 5 Conclusion: PMTG achieved quadruped locomotion with ES and RL across two gaits using a linear policy, IMU-only observations, and desired velocity as control input.The policies were successfully transferred from simulation to the Minitaur robot.
- 5 Conclusion: The approach currently relies on ad hoc trajectory generators chosen from intuition, motivating future work on generator selection, demonstration-based extraction, and theoretical foundations.The authors also identify connections to recurrent models as a future research direction.
Appendix
The appendix defines the trajectory generator’s phase progression and leg-specific phase offsets for periodic quadruped motion. The generator frequency is selected by the policy, while gait choice determines relative leg timing.
- Appendix: The trajectory generator phase ranges from 0 to 2π and advances according to its frequency, which the policy selects at each timestep.The frequency ftg is treated as a policy action in PMTG.
- Appendix: The leg trajectory generator produces swing and extension signals whose parameters encode stride amplitude, walking height, and other leg-motion characteristics.S(t) and E(t) represent swing and extension, while αtg controls swing amplitude and htg corresponds to walking height.
- Appendix: The transformed time t′ encodes the relative durations of swing and stance phases through β, the swing-to-stance duration proportion.This transformation is computed from the swing and stance phases.
- Appendix: Each leg receives a phase formed by adding its gait-specific offset Δφleg to the trajectory-generator phase modulo 2π.Offsets are defined relative to the left front leg and differ across gaits such as walking and bounding.