Learning Visual Locomotion with Cross-Modal Supervision

I. INTRODUCTION

• The policy uses visual look-ahead to estimate terrain height before the robot reaches it, and the authors report better performance than blind policies.The paper frames this as a robotics study of the role of look-ahead in walking.Source: §I · p. 2
• The authors choose monocular RGB because it is scientifically interesting and may be favored in engineering settings such as cost, power, or passive sensing.They note that prior walking systems have used depth, while RGB-only walking has been little studied.Source: §I · p. 2
• Simulated RGB differs substantially from real-world RGB, while high-fidelity rendering is computationally costly for sample-intensive reinforcement learning.The paper contrasts this with depth inputs, for which the sim-to-real gap is described as small.Source: §I · p. 2
• CMS trains vision to predict terrain height at look-ahead points using proprioceptive estimates obtained later, when the robot reaches those points.A blind policy can walk clumsily, providing a starting point for collecting supervision.Source: §I · pp. 2–3
• Across stairs, slippery slopes, curbs, tall steps, and discrete terrain, the visual policy reaches 100% performance on four staircases with less than 30 minutes of real-world data.The data were collected over four days; the evaluated stairs reached 19 cm, slopes 35 degrees, and curbs and tall steps 20 cm.Source: §I · p. 3
• CMS continually improves the visual system from onboard experience, with consistent real-world gains and adaptation to large visual-field shifts using limited data.The paper also reports precise foot placement with vision.Source: §I · p. 3

II. OVERVIEW

• The policy uses terrain geometry, environment parameters, and the current proprioceptive state to predict target actions, with visual terrain estimates supporting adaptation to complex terrain.The visual module estimates terrain in front of and below the robot from egocentric frames.Source: §II · p. 3
• CMS trains vision to predict future proprioceptive terrain estimates from egocentric visual input by minimizing the CMS error.Proprioception provides an estimate of terrain also recorded ahead of time by the camera.Source: §II · p. 3
• The process starts with a blind policy trained in simulation and transferred to the real world, then uses the learned terrain predictor with a look-ahead policy also trained in simulation.Onboard experience is used to continue training the visual predictor and improve the policy.Source: §II · p. 3

A. Simulation Training

• The look-ahead policy takes a terrain estimate 15 cm ahead in addition to the state, extrinsics, and terrain information used by both policies to predict target joint angles at 100 Hz.The target angles are converted to torques using a PD controller.Source: A. Simulation Training · p. 3
• Simulation training uses fractal terrains, parameterized inclines and stairs, with stair heights sampled from 10–21 cm and lengths from 30, 40, 50, or 60 cm.The reward encourages user-defined forward and angular speeds while penalizing lateral speed and jerky motion.Source: A. Simulation Training · pp. 3–4
• The vision-based policy anticipates stair geometry and adapts its gait, speeding the initial climb by approximately two times and achieving a higher overall success rate than the blind policy.The blind policy touches the stairs multiple times before estimating their height and climbing.Source: Figure 3 · p. 4
• Training starts on flat terrain for 500M steps, then progresses through increasingly difficult stairs while retaining flat-terrain samples and matching actions from a flat-trained policy.Environment parameters such as payload, friction, and motor strength are randomly sampled.Source: A. Simulation Training · p. 4
• For sim-to-real transfer, a network predicts environment and terrain estimates from proprioception and action history for the blind policy.The procedure follows the structure and training approach proposed in RMA.Source: A. Simulation Training · p. 4
• Using CMS, the real-world walking policy improves from approximately 50% to 100% in the considered settings with less than 30 minutes of data collected over four days.Collecting data with the latest vision policy leads to faster convergence than collecting it with the initial blind policy.Source: Figure 4 · p. 5

B. Vision-Based Locomotion via CMS

• Real-world rollouts pair egocentric camera input and current proprioceptive state with a proprioceptive terrain estimate.The deployed blind policy provides the setting for collecting this data.Source: B. Vision-Based Locomotion via CMS · p. 5
• A convolutional network is trained to estimate current extrinsics and future terrain geometry from camera input using supervised learning.The future geometry target is the terrain estimate at a time offset.Source: B. Vision-Based Locomotion via CMS · p. 5
• With a visual predictor of upcoming terrain, the system can use the look-ahead policy, which performs better than the blind policy.Source: B. Vision-Based Locomotion via CMS · p. 5

C. Lifelong Learning via CMS

• CMS uses the proprioceptive γ estimate to continuously train the visual lookahead terrain predictor during execution.The training data are collected through real-world experience.Source: C. Lifelong Learning via CMS · p. 5
• The predictor is trained on the continuously collected real-world dataset D.Source: C. Lifelong Learning via CMS · p. 5
• Continual predictor training is used to improve the performance of the overall policy.Source: C. Lifelong Learning via CMS · p. 5

IV. EXPERIMENTAL SETUP

• Simulation uses RaiSim with the Unitree A1 URDF; episodes last at most 1200 steps, and control runs at 100 Hz with a 0.025-second simulation timestep.Episodes terminate early if roll or pitch exceeds a threshold or the robot base is very close to the ground.Source: §IV · p. 5
• The vision estimator predicts future terrain γ from the latest three grayscale camera frames, 50 IMU measurements, and desired velocity commands.The camera operates at 15 Hz.Source: §IV · p. 5
• The estimator processes image features with ShuffleNet-V2 and projects them to a two-dimensional channel space before predicting γ.An MLP with hidden dimensions [128, 64] predicts γ.Source: §IV · p. 5

V. RESULTS AND ANALYSIS

• The evaluation is designed to measure the quantitative and qualitative benefits of vision-based locomotion over blind locomotion in the real world.Source: §V · pp. 5–6
• The prism-adaptation figure compares nominal performance, performance immediately after camera rotation, and performance after 80 seconds of experience; behavior recovers with about one minute of fine-tuning.The bottom row shows the nominal and rotated camera positions and their views.Source: Figure 5 · p. 6
• A second evaluation question compares deploying the latest policy during continual training with collecting all data under a blind policy and using it at once.Source: §V · p. 6

A. Lifelong Vision Learning

• Vision-based anticipation produces a smooth, agile gait and speeds the initial stair climb by approximately two times compared with the blind policy.The blind policy taps the terrain twice before walking, while the vision policy adapts its stride to anticipated geometry.Source: A. Lifelong Vision Learning · p. 6
• On four stairs, performance metrics improve with experience, and the three-day policy climbs all stairs without failure after 26 minutes of walking data.Five trials were performed on each stair; a “day” is a policy trained with all data collected by its predecessors.Source: A. Lifelong Vision Learning · p. 6
• The vision policy is 26% faster in average speed and 50 percentage points higher in success rate than its blind counterpart.After 26 minutes of walking data, a policy trained on three days of experience climbed all tested stairs without a failure.Source: A. Lifelong Vision Learning · p. 6
• Vision increases the average experience collected during a real-world experiment by 21%, and equal amounts of vision-collected data outperform blind-collected data.The authors attribute the data-quality difference to stumbling or terrain-tapping by the blind controller, versus more precise gait from a vision controller.Source: A. Lifelong Vision Learning · p. 6
• The final vision policy succeeds on 80% of previously unseen staircases, 100% of curbs, and 60% of inclines, although the vision module was never trained for them.The tested inclines reached 35 degrees.Source: A. Lifelong Vision Learning · pp. 6–7
• After a roughly 30-degree camera yaw shift, the policy learns the visual bias and walks straight after three trials, or about 80 seconds of data.Fine-tuning minimizes CMS error and updates only the last three layers of the terrain predictor for 10 epochs after each trial.Source: A. Lifelong Vision Learning · p. 7
• After the camera returns to its original position, the policy adapts back to the original visual field after two trials.Source: A. Lifelong Vision Learning · p. 7

• Performance rises from 58% to 75% as ground-truth lookahead increases from 0 to 15cm.Source: Table I · p. 7, B. The value of visual lookahead · p. 7
• Table I compares motor policies with different amounts of ground-truth visual lookahead.Source: Table I · p. 7
• The value of lookahead saturates after 15cm.Source: Table I · p. 7

A. Blind Locomotion

• Blind controllers react to terrain but cannot anticipate discontinuities, leading to clumsy behavior on such terrain.Model-based controllers require accurate robot models and task-specific engineering; optimization-based methods can be brittle outside lab-controlled conditions.Source: A. Blind Locomotion · p. 7

B. Vision-Based Locomotion

• Vision-based methods use elevation maps to predict motion primitives or directly predict commands, but similar real-world performance is not yet established broadly.A cited real-world system uses up to three lidars or eight stereo modules and has computational costs that make it infeasible for smaller robots.Source: B. Vision-Based Locomotion · p. 7

C. Self-supervised learning

• Prior self-supervised work uses haptic terrain identification or traversability estimates, generally for high-level trajectories tracked by separate low-level controllers.Source: C. Self-supervised learning · p. 7
• This work uses self-supervision to improve low-level controller quality and practically increase traversable terrain through experience.Source: C. Self-supervised learning · p. 8

VII. CONCLUSION

• CMS uses onboard data and time-shifted proprioception to continually improve visual walking, but the current system does not improve the motor system in the real world.The authors identify coupling CMS with techniques that improve the motor policy as future work.Source: §VII · p. 8