Source-linked AI summary
DreamWaQ: Learning Robust Quadrupedal Locomotion With Implicit Terrain Imagination via Deep Reinforcement Learning
I Made Aswin Nahrendra, Byeongho Yu, Hyun Myung
TL;DR
Quadrupedal locomotion controllers often depend on complex sensing, while proprioception-only methods have limited long-distance terrain demonstrations. DreamWaQ uses asymmetric actor-critic reinforcement learning and context-aided estimation to imagine terrain implicitly, and it was evaluated on challenging outdoor environments with a Unitree A1.
Problem
Existing locomotion methods depend on complex sensing, while proprioception-only approaches have limited demonstrations across long distances and challenging terrains.
Method
DreamWaQ uses asymmetric actor-critic reinforcement learning and a context-aided estimator to train a proprioceptive policy that implicitly infers terrain properties.
Results
DreamWaQ consistently outperforms the other compared methods in simulation, performs almost as well as an oracle height-map policy, and robustly walks on hills and unstructured yards.
Takeaways & Limitations
The framework demonstrates proprioception-only quadrupedal locomotion across challenging outdoor terrains using a Unitree A1.
Takeaways & Limitations
DreamWaQ must first contact obstacles with its legs, and handling high-rise stairs remains future work requiring exteroception for earlier gait planning.
Abstract
from arXiv · showhide
Quadrupedal robots resemble the physical ability of legged animals to walk through unstructured terrains. However, designing a controller for quadrupedal robots poses a significant challenge due to their functional complexity and requires adaptation to various terrains. Recently, deep reinforcement learning, inspired by how legged animals learn to walk from their experiences, has been utilized to synthesize natural quadrupedal locomotion. However, state-of-the-art methods strongly depend on a complex and reliable sensing framework. Furthermore, prior works that rely only on proprioception have shown a limited demonstration for overcoming challenging terrains, especially for a long distance. This work proposes a novel quadrupedal locomotion learning framework that allows quadrupedal robots to walk through challenging terrains, even with limited sensing modalities. The proposed framework was validated in real-world outdoor environments with varying conditions within a single run for a long distance.
I. INTRODUCTION
DreamWaQ addresses the difficulty of quadrupedal control and limited proprioception-only terrain demonstrations by learning robust locomotion from proprioception and validating it across challenging environments.
- Motivation: Quadrupedal robots traverse unstructured terrains but are difficult to control, while conventional controllers require complex modeling, estimation, optimization, and tuning.These requirements create a substantial human-engineering burden.
- Motivation: Exteroceptive sensing supports terrain-aware locomotion, but cameras can fail in adverse conditions and LiDAR struggles to estimate physical terrain properties accurately.Snow and tall grass illustrate mismatches between appearance and traversability.
- Motivation: Proprioceptive sensing is relatively light and robust, yet prior proprioception-based methods have limited demonstrations over long distances and varied challenging terrains.High uncertainties and estimation errors may cause failures during such operation.
- Contribution: DreamWaQ trains a robust locomotion policy using only proprioception to implicitly infer terrain properties, including height maps, friction, restitution, and obstacles.The framework uses deep reinforcement learning to support gait adaptation across terrains.
- Contribution: DreamWaQ combines asymmetric actor-critic learning, context-aided estimation, and real-world robustness evaluation on a Unitree A1 in diverse outdoor environments.The authors report sustained walking on challenging hills and yards and improved performance over existing learning-based methods.
II. DREAMWAQ
DreamWaQ formulates locomotion as an infinite-horizon partially observable Markov decision process and represents unobserved world information with a latent context vector.
- A. Preliminaries: The environment is modeled as an infinite-horizon partially observable Markov decision process defined by states, observations, actions, transitions, rewards, an initial distribution, and discounting.The state, observation, and action spaces are continuous.
- A. Preliminaries: A context vector z_t provides a latent representation of the world state and is inferred by the method described later in the framework.It is introduced to summarize information unavailable in partial observations.
B. Implicit Terrain Imagination
DreamWaQ avoids sequential teacher-student training by using asymmetric actor-critic learning, allowing a proprioceptive policy to learn implicit terrain imagination during one PPO training phase.
- B. Implicit Terrain Imagination: Teacher-student approaches can bound student performance by the teacher and become data inefficient when networks are trained sequentially.Behavior cloning also supplies only good-action supervision, limiting exposure to failure states learned during reinforcement learning.
- B. Implicit Terrain Imagination: DreamWaQ uses an asymmetric actor-critic architecture in which the policy receives temporal partial observations while the value network receives the full state.The actor-critic interplay is reported as sufficient for implicitly imagining privileged observations.
- B. Implicit Terrain Imagination: Using H = 5 temporal observations, DreamWaQ trains with PPO in one phase, improving training efficiency and allowing exploration of possible trajectories.The authors connect this exploration to robustness through generalization.
- B. Implicit Terrain Imagination: The policy receives proprioceptive observations, body velocity, and latent state; joint encoders and an IMU measure observations, while CENet estimates velocity and latent state.The policy then infers actions from these inputs.
2) Value Network:
The value network estimates state value from privileged observations, while the reward combines locomotion objectives with a motor-power distribution term intended to reduce overheating.
- Value Network: The value network outputs an estimate of V(s_t) from the privileged observation s_t.The privileged observation includes disturbance force and a height-map scan unavailable to the policy.
- Value Network: The policy action is a 12 × 1 vector of desired joint angles tracked by per-joint PD controllers.
- Reward Function: The reward combines commanded-velocity tracking and stability objectives to produce stable, natural locomotion.Its elements are defined using quantities including body-frame gravity, velocities, body height, foot motion, and joint torque.
- Reward Function: The power distribution reward penalizes high variance in motor power usage to reduce long-run motor overheating.This differs from minimizing only overall power, which may leave some motors working harder than others.
5) Curriculum Learning:
The framework combines proprioceptive estimation, shared-encoder context inference, and adaptive bootstrapping, while curriculum learning progressively exposes the policy to difficult terrain.
- Curriculum Learning: The curriculum uses smooth, rough, discretized, and stair terrains across ten inclination levels from 0° to 22°.A grid-adaptive curriculum improves low-speed turning stability and helps prevent foot tripping.
- Curriculum Learning: DreamWaQ uses proprioception to estimate body velocity and latent terrain context for locomotion control.The policy receives v_t and z_t estimated from proprioceptive observations, with learned velocity estimation reducing accumulated drift.
- Curriculum Learning: CENet jointly estimates body velocity and environmental context through a shared encoder with an auto-encoding mechanism.The shared encoder supports synchronous inference and learns forward and backward dynamics through reconstruction.
- Curriculum Learning: CENet is optimized with a hybrid body-velocity estimation and β-VAE loss.The VAE component uses reconstruction error and KL divergence, while velocity estimation uses MSE.
- Curriculum Learning: AdaBoot increases bootstrapping when reward variation is low and avoids it when large coefficient of variation indicates insufficient learning.The bootstrapping probability is bounded using p_boot = 1 − tanh(CV(R)).
A. Compared Methods
The evaluation compares proprioception-only locomotion methods that differ in adaptation, estimation, and training design.
- Compared Methods: The comparison includes a baseline policy trained without an adaptation mechanism.
- Compared Methods: AdaptationNet uses an implicit environmental-factor encoder trained with a student-teacher framework.Its policy combines 1D CNN and MLP layers.
- Compared Methods: EstimatorNet trains the policy concurrently with an estimator that explicitly estimates body state without context estimation.
B. Simulation
DreamWaQ was trained in a domain-randomized Isaac Gym simulation and consistently outperformed the other learning-based methods, approaching an oracle policy with height-map access.
- Simulation setup: DreamWaQ training used 4,096 domain-randomized agents running in parallel for 1,000 iterations.The policy, value, and CENet networks were trained synchronously with PPO.
- Simulation setup: Training DreamWaQ for approximately one hour generated data equivalent to about 46 days of real-world training.
- Learning performance: DreamWaQ consistently outperformed all other methods in learning-curve comparisons.EstimatorNet initially achieved a higher mean episodic reward than AdaptationNet but later declined after encountering more difficult terrains.
- Learning performance: Despite using no exteroception, DreamWaQ performed almost as well as an oracle policy with direct surrounding-terrain height-map access.The learning curves report means and standard deviations across ten random seeds.
D. Command Tracking
Simulation evaluations examined command tracking, state estimation, and robustness under disturbances. DreamWaQ achieved lower tracking errors, CENet was especially effective during stair stumbling, and the complete method was more robust than the baselines.
- Command tracking: DreamWaQ consistently outperformed the baselines on absolute command-tracking error, with paired t-tests indicating significant improvement.The evaluation used random commands and repeated controller runs for comparison.
- Command tracking: AdaBoot significantly improved DreamWaQ’s command-tracking performance through its statistical bootstrapping strategy during training.
- State estimation: CENet produced small estimation errors on flat terrain and remained accurate when the robot stumbled down stairs, where EstimatorNet failed to estimate body velocity accurately.The comparison used squared estimation error in a simulated stairs environment.
- State estimation: Accurate CENet estimation enabled the robot to climb stairs safely in cases where inaccurate estimation could lead to catastrophic failure.
- Robustness analysis: DreamWaQ was significantly more robust than the other methods, as verified by its survival rate and maximum push tolerance.The robot was tested under repeated random pushes until falling and during random walking.
- Robustness analysis: In real-world stumbling and slipping tests, DreamWaQ immediately adapted its gait and stabilized its pose without deterioration in body-velocity estimation.The response was attributed to the robust and accurate CENet.
G. Long-Distance Walk
DreamWaQ was evaluated on two outdoor courses with slopes, deformable terrain, and hiking-track conditions. The trajectories were recorded using onboard RTK-GPS, and both courses covered several hundred meters.
- Outdoor evaluation: Course A was an on-campus yard with many slopes and deformable terrains, while Course B was an on-campus hill with elevation gain up to 22 m.
- Outdoor evaluation: Course A and Course B had total lengths of 430 m and 465 m, respectively.
- Outdoor evaluation: The robot’s outdoor trajectories were measured with an RTK-GPS mounted on the robot at 10 Hz.Course elevations relative to the starting point were represented in the figure’s color bars.
- Outdoor evaluation: The outdoor tests demonstrated DreamWaQ’s robustness across unstructured natural and man-made terrains during long-distance walking.The reported courses included yards and a hiking track with varying slopes and surfaces.
1) Course A:
Course A exposed the robot to slopes, vegetation, stairs, deformable terrain, and wet conditions. DreamWaQ adapted its gait and joint power to traverse these obstacles and surfaces safely.
- Course A:: Thick vegetation trapped the robot’s legs, but it successfully increased joint power and adapted its speed to overcome the trap.
- Course A:: DreamWaQ safely traversed stairs and deformable slopes in both dry and wet terrain conditions.Wet testing followed rainfall and included slippery stairs and muddy slopes.
- Course A:: Course B combined asphalt, gravel, and slopes; the robot completed the 465 m climb within 10 minutes and reached the summit despite motor-heating risks.The robot was commanded to move slowly to reduce required torque during the climb.
- Course A:: The paper identifies obstacle contact as a limitation of DreamWaQ’s adaptation mechanism and proposes exteroception for future planning around more complex structures.High-rise stairs are given as an example of a structure requiring improved gait planning before contact.