PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers

Ruihan Yang, Minghao Zhang, Nicklas Hansen, Huazhe Xu, Xiaolong Wang

arXiv:2107.03996v3cs.LGcs.CVcs.RO

TL;DR

Quadrupedal locomotion research commonly relies on blind proprioceptive control, which is limited for anticipating obstacles and uneven terrain. LocoTransformer fuses proprioceptive states and depth images with cross-modal Transformers, improving locomotion and generalization in simulation and real-world transfer.

  • Problem

    Blind proprioceptive control provides immediate robot-state measurements but cannot consistently anticipate and avoid distant obstacles or terrain changes.

  • Method

    LocoTransformer uses an MLP, ConvNet, and Transformer encoders to fuse proprioceptive states with first-person depth images for end-to-end locomotion RL.

  • Results

    The method improves locomotion control and generalization across simulated obstacles and uneven terrain, unseen environments, and real-world deployment.

  • Takeaways & Limitations

    Visual observations combined with proprioception enable the robot to plan around obstacles and challenging terrain, including unseen real-world environments.

Abstract

from arXiv · show

We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) with a Transformer-based model that learns to combine proprioceptive information and high-dimensional depth sensor inputs. While learning-based locomotion has made great advances using RL, most methods still rely on domain randomization for training blind agents that generalize to challenging terrains. Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn to proactively maneuver environments with obstacles and uneven terrain by anticipating changes in the environment many steps ahead. In this paper, we introduce LocoTransformer, an end-to-end RL method that leverages both proprioceptive states and visual observations for locomotion control. We evaluate our method in challenging simulated environments with different obstacles and uneven terrain. We transfer our learned policy from simulation to a real robot by running it indoors and in the wild with unseen obstacles and terrain. Our method not only significantly improves over baselines, but also achieves far better generalization performance, especially when transferred to the real robot. Our project page with videos is at https://rchalyang.github.io/LocoTransformer/ .

1 INTRODUCTION

The paper argues that blind proprioceptive control is insufficient for consistently avoiding large obstacles and proposes LocoTransformer, which fuses proprioception with depth vision through cross-modal Transformers. It evaluates this approach in simulation and real environments, emphasizing obstacle avoidance, uneven terrain, and generalization.

  • Motivation: Blind quadrupedal RL policies rely on proprioceptive state and domain randomization, but may not consistently avoid large obstacles.Proprioception supports immediate reaction, whereas distant obstacles require perception and trajectory adjustment.
  • Approach: LocoTransformer combines proprioceptive states and first-person depth images with a cross-modal Transformer for locomotion control.The model uses an MLP for proprioception, a ConvNet for depth images, and Transformer encoders for multimodal fusion and action prediction.
  • Evaluation: The experiments cover simulated obstacles, moving obstacles, and rough mountainous terrain, followed by sim-to-real deployment indoors and in forests.The real environments include hallway boxes, forest trees, and uneven terrain.
  • Findings: The method improves locomotion control and generalizes better to unseen environments, particularly during transfer to a real robot.The stated contributions include vision-based end-to-end RL and deployment on a running quadrupedal robot avoiding obstacles and trees in the wild.

2 RELATED WORK

The related work contrasts manually tuned model-based locomotion with learning-based approaches and situates this paper at the intersection of vision-based RL and multimodal Transformers. Its stated novelty is applying cross-modal Transformers to fuse proprioceptive and visual inputs for locomotion.

  • Learning Legged Locomotion: Model predictive control and trajectory optimization can require detailed environmental knowledge and substantial manual parameter tuning.These requirements make such methods challenging to apply in complex environments.
  • Vision-based Reinforcement Learning: Vision-based RL extends reinforcement learning beyond state inputs toward real-world applications using visual observations.The cited work includes representation learning and RL methods using visual inputs.
  • Transformers and Multi-modal Learning: Transformers support multimodal reasoning by treating different modality inputs as tokens for self-attention.Prior applications span language, vision, and joint video-caption modeling.
  • Transformers and Multi-modal Learning: This paper applies cross-modal Transformers to fuse proprioceptive states and visual inputs for locomotion, which it identifies as a first use in that setting.The approach extends multimodal Transformer use beyond language and vision.

3 REINFORCEMENT LEARNING BACKGROUND

The paper models robot-environment interaction as a finite-horizon Markov decision process and trains a parameterized policy to maximize expected discounted episode return.

  • MDP Formulation: The robot-environment interaction is modeled as an MDP with states, actions, transitions, rewards, horizon, and discount factor.The formulation is (S, A, P, R, H, γ), where P is the transition function and R is the reward function.
  • Policy Learning: The agent learns a policy πθ that outputs an action distribution conditioned on the current state.The policy is parameterized by θ.
  • Objective: The learning objective is to maximize the expected discounted episode return over trajectories generated by the policy.The return sums rewards over the finite episode horizon using discount factor γ.

4 METHOD

LocoTransformer uses separate encoders to project proprioceptive states and depth images into tokens, then applies cross-modal Transformer attention for locomotion decisions. Its projection head aggregates modality-specific features to predict actions or values while preserving spatial visual reasoning.

  • Separate Modality Encoders: Separate MLP and ConvNet encoders project proprioceptive states and depth images into a shared latent feature space.The proprioceptive input is encoded as one feature, while visual inputs become spatial feature maps.
  • Separate Modality Encoders: The visual feature map is split into N × N regional tokens, while the proprioceptive feature becomes one C-dimensional token.Together, these form N × N + 1 tokens for Transformer processing.
  • Transformer Encoder: Scaled dot-product attention transforms tokens into queries, keys, and values, computes normalized pairwise weights, and combines values for each token.The attention weights use dot products scaled by the self-attention dimension before Softmax normalization.
  • Transformer Encoder: Residual connections help preserve proprioceptive information as attention operates across many visual tokens and a single proprioceptive token.This addresses the risk that proprioceptive information gradually vanishes through multiple Transformer layers.
  • Transformer Encoder: Stacked Transformer encoder layers apply self-attention across proprioceptive and visual tokens to fuse modalities and support spatial reasoning.Each layer uses self-attention, LayerNorm, residual connections, and a two-layer MLP.
  • Transformer Encoder: The projection head separately pools visual and proprioceptive tokens, concatenates their feature vectors, and predicts actions or values.Separate pooling prevents the larger number of visual tokens from diluting proprioceptive information.

5 EXPERIMENTS

Experiments evaluate LocoTransformer across simulated obstacles, moving objects, spheres, and mountainous terrain, then test sim-to-real transfer. Across these settings, visual-proprioceptive fusion with a Transformer improves generalization, spatial reasoning, and real-world locomotion.

  • Experimental setting: Experiments cover six simulated environments with varying terrain, obstacles, and reward-bearing spheres, plus indoor and forest real-world scenarios.Policies are evaluated using mean episode return, distance moved, and obstacle collisions after 15M training samples across five seeds.
  • Attention analysis: Figure 3 visualizes attention between the proprioceptive token and visual tokens, with warmer colors indicating larger attention weights over image regions.The attention analysis examines how the model uses spatial information for decision-making at different time steps.
  • Generalization: 69% and 56%: LocoTransformer improves transfer episode return over State-Depth-Concat in wide- and thin-obstacle environments, respectively.The improvements are measured when transferring policies between unseen obstacle widths.
  • Complex simulated environments: In more complex environments, LocoTransformer improves final performance and sample efficiency, while State-Depth-Concat offers no episode-return gain over State-Only on mountainous terrain.The Transformer better preserves spatial structure and fuses global and local terrain information with proprioceptive inputs.
  • Real-world transfer: 92% farther indoors and approximately 90% farther in forests: LocoTransformer surpasses the real-world baseline while colliding less or not at all.The forest policy avoided all obstacle collisions, whereas the baseline frequently collided with trees and became stuck in potholes.

6 CONCLUSION

The paper combines proprioceptive and visual information with LocoTransformer for quadrupedal locomotion. It reports improved obstacle navigation, challenging-terrain locomotion, unseen-environment generalization, and real-world performance.

  • Conclusion: LocoTransformer incorporates proprioceptive and visual information for locomotion control.The model uses cross-modal Transformer fusion to combine both modalities.
  • Conclusion: Visual inputs support navigation through obstacles, including obstacles of different sizes and moving obstacles, and assist locomotion on mountainous terrain.These capabilities are demonstrated across the paper’s simulated tasks.
  • Conclusion: Cross-modal Transformer policies achieve better generalization on unseen environments and in the real world.The conclusion attributes this to effective fusion of proprioceptive and visual information.

A DETAILED EXPERIMENT SETUP

Experiments use a Unitree A1 with proprioceptive and depth inputs, shared reward terms, domain randomization, and onboard real-world deployment settings.

  • Robot and Inputs: The Unitree A1 provides joint angles, IMU information, base displacement, and the previous action as proprioceptive inputs.These vectors are stacked over the last three steps to retain historical state information.
  • Robot and Inputs: The 12-dimensional action controls joint-angle changes and is bounded by 0.5 for locomotion stability.Position control sets the robot actions.
  • Reward Definition: The reward combines forward motion, energy usage, survival, and sphere collection when applicable.Forward reward follows the task direction, energy penalizes large motor-torque norms, alive reward is positive until termination, and sphere reward applies per collected sphere.
  • Real-World Setup: Real-world deployment uses an Intel RealSense depth camera, onboard computation, 25 Hz control, and 400 Hz PD-controller torque updates.The Unitree A1 has 18 links and 12 degrees of freedom; action repeat is 16, with PD gains KP=40 and KD=0.6.
  • Domain Randomization: Training uses domain randomization for proprioception and depth inputs, including randomly masked depth readings to simulate noisy visual observations.At each time step, 3 to 30 values in the 64×64 depth input are replaced by the maximum reading.

B.2 HYPERPARAMETERS SHARED BY ALL METHODS

All methods share PPO-related training settings, while baselines and LocoTransformer use matched network components apart from their perception or fusion design.

  • Shared Training Settings: Shared training uses horizon 1000, discount factor .99, batch size 256, three optimization epochs, Adam, and 1e-4 policy and value learning rates.The shared clip parameter is 0.2, with 8192 samples per iteration.
  • Baseline Architecture: The state-only baseline uses four fully connected layers with 256 units to provide sufficient learning capacity for fair comparison.Adding more layers produced only minor performance differences.
  • Baseline Architecture: State-Depth-Concat and LocoTransformer keep all components identical except for the perception encoder.This isolates the effect of their multimodal representation and fusion designs.
  • LocoTransformer Configuration: The shared proprioceptive encoder and projection head each use two fully connected layers with 256 units.LocoTransformer uses token dimension 128 and two Transformer encoder layers.

C MORE ATTENTION VISUALIZATION RESULTS

Attention visualizations indicate that LocoTransformer shifts focus among obstacles, goals, and rugged terrain across an episode, supporting spatially guided maneuvering.

  • Attention Visualization: LocoTransformer attends to newly appearing obstacles and changes direction to escape threats in the Thin Obstacle environment.One sequence shows attention moving from a closed obstacle to a wall, followed by a left turn and body reorientation.
  • Attention Visualization: In the Mountain environment, attention covers both the final goal and rugged terrain that is difficult to step on.The visualized regions vary with the required direction and terrain hazards.
  • Attention Visualization: The attention sequences are presented row-wise to show how the agent’s focus evolves over time.Figure 7 provides additional attention-map sequences for interpreting the model’s behavior.

D.2 RESULTS IN SIMULATION

Simulation comparisons test Transformer and end-to-end RL designs against classical-controller alternatives, finding stronger multimodal performance but slower training for the controller baseline.

  • Experimental Questions: Simulation evaluates whether Transformer fusion outperforms CNN fusion and whether end-to-end RL outperforms visual policy plus classical control.These are the two stated experimental questions.
  • Network Architectures: LocoTransformer outperforms the baseline in the multimodal vision-guided whole-body-controller setting.For vision-only input, the Transformer and CNN show only a minor difference.
  • Network Architectures: The authors speculate that attention improves multimodal fusion more than image representation learning.This interpretation is based on the contrast between multimodal and vision-only results.
  • Training Cost: The controller requires several times more samples to train, which the authors attribute to a low-level controller that is insufficiently agile to varying high-level commands.The proposed explanation concerns the controller’s ability to provide explicit information for RL optimization.
  • Deployment Computation: End-to-end learned policies can use the onboard GPU for real-time inference, contrasting with customized traditional controllers reported at around 120Hz on Unitree A1.The comparison concerns deployment computation rather than training performance.

D.3 RESULTS IN REAL WORLD

In real-world tests, LocoTransformer handled obstacles with smoother, more flexible motion than the vision-guided whole-body controller, which collided because of limited agility and poor obstacle-shape generalization.

  • End-to-end vision-and-motion training enabled smoother transitions between forward motion and turning while adjusting speed around obstacles.
  • The end-to-end policy produced more diverse motions that transitioned according to visual observations.
  • The vision-guided whole-body controller collided with walls in narrow paths because it could not turn quickly enough.
  • It also collided with trees whose shapes were outside the training distribution, reflecting limited ability to adjust quickly.

D.4 QUANTITATIVE COMPARISON IN REAL WORLD

Quantitative real-world comparisons found that the vision-guided whole-body controller incurred more collisions and performed worse in indoor environments with larger obstacles.

  • The vision-guided whole-body controller created more collisions, even though it enabled the robot to walk longer distances.
  • Figure 10 compares the method and vision-guided whole-body controller in indoor obstacle and forest environments.
  • In indoor environments with larger obstacles, the controller performed worse in both collision times and distance moved.
Loading 2107.03996v3…