Advanced Skills by Learning Locomotion and Local Navigation End-to-End

I. INTRODUCTION

• Current policies still lack animal- or human-level agility and precision despite needing to handle model mismatch, noisy sensing, disturbances, missteps, and slippage.Greater dexterity requires switching gaits, planning consecutive steps, jumping, and using other body contacts.Source: §I · p. 1

A. Related Work

• Velocity-command pipelines constrain behaviors such as slowing before a jump or stopping in danger, and constant-speed tracking can favor less energy-efficient trotting.These constraints make behaviors with variable speed harder to achieve.Source: A. Related Work · pp. 1–2
• Earlier agile-locomotion methods have demonstrated jumps and climbing, but report constraints including costly trajectory optimization, multi-stage training, or limited real-world transfer evidence.Selected dynamic maneuvers have been shown on real robots, but the cited examples address specific tasks.Source: A. Related Work · p. 2
• Prior local-navigation approaches either steer a locomotion controller with a high-level policy or combine navigation and locomotion end to end, but both retain velocity-command-trained locomotion.Trajectory optimization can directly use a target state, but is computationally expensive and less robust than reinforcement-learning policies.Source: A. Related Work · p. 2

B. Contribution

• The policy is rewarded for reaching a target location after a defined time, with task reward based on final distance and independent of the trajectory.This decouples goal completion from additional costs, allowing the policy to optimize its path and gait while reaching the target.Source: B. Contribution · p. 2
• Compared with velocity tracking and continuous-position baselines, the method handles harder stairs and slopes and enables gap jumps, box climbing, and obstacle avoidance.The learned gait is associated with lower failure rates and energy consumption; policies run in real time on the real robot without a local planner.Source: B. Contribution · p. 2

II. METHOD

• A neural-network policy maps robot sensor measurements to motor commands and is trained in simulation with deep reinforcement learning before direct transfer to the real robot.The setup uses Isaac Gym and PPO; transfer requires no further training or adaptation.Source: §II · p. 2

A. Observation and Action Space

• Observations include proprioception, commands, prior actions, and terrain measurements, while actions are target joint positions sent through a PD controller.Commands specify a three-dimensional target in the base frame and the remaining time to reach it.Source: A. Observation and Action Space · p. 2

B. Rewards

• Rewards combine task rewards activated only at episode end with penalties provided throughout the episode.The task reward targets goal reaching, while ongoing terms penalize motion or contact costs.Source: B. Rewards · p. 2

1) Task Reward:

• The target-position reward is active only during the final T_r interval, rewarding proximity to the target base position and otherwise returning zero.T_r must be long enough for the policy to stop stably at the target; a shorter window can encourage a last-moment jump or lean.Source: 1) Task Reward · p. 2

2) Penalties:

• Penalties cover joint and foot accelerations, joint torques, collisions, and abrupt action changes; minimizing them does not counteract the main reward if the target is reached in time.The foot-acceleration penalty reduces ground impacts, which can cause noise and potential damage.Source: 2) Penalties · p. 3

3) Exploration Reward:

• The exploration reward encourages target-directed base velocity early in training and is removed once the task reward reaches 50% of its maximum.This temporary reward encourages exploration without constraining the final learned solution.Source: 3) Exploration Reward · p. 3

4) Stalling Penalty:

• Because PPO discounts episode rewards, the policy can defer walking penalties, leading it to wait until the last moment and then run toward the target; a small penalty for waiting far away counters this effect.Source: 4) Stalling Penalty · p. 3
• Fig. 2 presents curricula for three new terrain types, from each easiest version to the hardest terrain seen during training.Source: Figure 2 · p. 3

• The gap-crossing task requires stepping or jumping across a distance that makes constant base velocity too constraining.The robot inevitably slows after the jump because it barely reaches the far side.Source: C. Terrains · p. 3
• The climbing task requires a sequence that places the front feet up before using the back knees to help reach the top of a pit.Source: C. Terrains · p. 3
• The high-obstacle task tests local planning by requiring the policy to leave the direct path to the target and find a route around obstacles.Source: C. Terrains · p. 3

D. Episode Process

• Each episode samples a target 1–5 m away, provides the task reward after 5 seconds of a 6-second episode, and counts finishing within 0.5 m as success.Targets are sampled uniformly in polar coordinates and invalid targets overlapping holes or high obstacles are resampled.Source: D. Episode Process · p. 3

III. SIMULATION EXPERIMENTS

• Table I reports maximum terrain difficulty solved at 95% success for velocity tracking, continuous position tracking, and final position tracking.Source: Table I · p. 4

E. Training Stability

• The temporally sparse task reward makes value prediction harder and training more sensitive to instabilities, including failed runs after random-seed changes.Source: E. Training Stability · p. 4
• Training stability measures include larger batches, shorter episodes, and removing value-function bootstrapping because termination time is defined and given to actor and critic.The policies are trained for 2,000 iterations, taking approximately one hour.Source: E. Training Stability · p. 4

III. SIMULATION EXPERIMENTS

• The simulation comparison uses velocity tracking and continuous position tracking baselines, with continuous position reward supplied at every time step.Velocity tracking removes sideways commands, retaining forward/backward and yaw commands.Source: §III · p. 4

A. Maximum Terrain Difficulty

• Step height and slope angle rise 80% and 35%; maximum obstacle height, obstacle size, gap length, and pit depth rise 40%, 140%, 700%, and 850% versus velocity tracking.The comparison evaluates the highest difficulty each policy solves at a success rate of at least 95%.Source: A. Maximum Terrain Difficulty · p. 4, Figure 3 · p. 5
• Continuous position tracking reaches higher difficulty than velocity tracking on most terrains but is less reliable on obstacles and inferior to final-position tracking on every terrain.Final-position tracking also enables jumps, knee-assisted box climbing, and avoidance of nontrivial obstacles.Source: A. Maximum Terrain Difficulty · p. 4

B. Energy Consumption

• Fixing the target and time lets final-position control walk continuously, and reducing the time makes its gait faster; continuous position tracking is instead commanded by changing target distance.The final-position policy was not trained for continuous walking; the fixed target is set 3 m ahead.Source: B. Energy Consumption · pp. 4–5
• Sum of squared torques per traveled distance is similar across approaches at high speeds, but velocity tracking uses more at low speeds; continuous tracking outperforms final-position tracking at higher speeds.The authors use squared torque as an energy proxy because it is penalized during training and proportional to motor heat losses; final-position policies are mostly trained around 1 m/s.Source: B. Energy Consumption · pp. 4–5, Figure 4 · p. 5
• On hardware, final-position control outperforms velocity tracking at low speeds and converges to similar torques at higher speeds.Hardware torques are higher than in simulation, which the authors attribute to more restrictive actuator limits in simulation; continuous tracking was not transferred because its aggressive motions could damage the robot.Source: B. Energy Consumption · p. 5, Figure 4 · p. 5

C. Emergent Gaits

• Removing the constant-base-velocity constraint permits gait variation, and the authors report larger, smoother steps and motion perceived as more natural than velocity tracking.Velocity tracking favors trotting; with final-position control, base oscillations are not penalized and a foot-air-time or clearance reward is unnecessary.Source: C. Emergent Gaits · pp. 5–6
• Final-position control learns a three-phased asymmetric gait toward the front left, while continuous position tracking uses an inverted schedule toward the right hind side.The final-position gait contacts the right hind leg first, then the left hind and right front legs, and finally the left front leg briefly.Source: C. Emergent Gaits · p. 5
• Real-robot demonstrations include running on stairs, climbing a 0.55 m box, jumping over a 0.6 m gap, and navigating around obstacles.Source: Figure 6 · p. 6
• Each training run selects a random but fixed walking direction, so repetitive back-and-forth walking can be problematic.The policy tends to turn rather than walk sideways or backward.Source: C. Emergent Gaits · p. 6

D. Hardware Deployment

• The actuator model was inaccurate at high velocity, so the authors clipped policy torques to an affine maximum-torque limit based on motor velocity.At maximum velocity, the model still allowed torques that brushless DC motors cannot produce.Source: D. Hardware Deployment · p. 6
• Operators could command the robot with position targets or a joystick whose direction sets the target direction and magnitude scales the remaining time.The policy responded well to rapidly changing joystick commands despite not being trained for them.Source: D. Hardware Deployment · p. 6
• The robot demonstrated flat-terrain locomotion, high-speed stair climbing, crossing a 0.6 m gap, climbing a 0.55 m box, and obstacle navigation.The authors did not use the most complicated terrains for practical reasons.Source: D. Hardware Deployment · p. 6
• Deployment success varied by terrain: it was similar to I on stairs but lower on harder terrains, and climbing required a motion-capture system.The authors attribute the harder-terrain results to state-estimation and perception modules not designed for jumping or climbing.Source: D. Hardware Deployment · p. 6

IV. DISCUSSION

• Specifying a target location instead of base velocity reduces solution-space constraints, decouples the task from optimizing other quantities, and supports a more energy-efficient gait and complex maneuvers.The authors also report greater susceptibility to training instability.Source: §IV · p. 6
• The approach could extend beyond challenging-terrain locomotion; in fall recovery, an end-of-episode stand-up reward would leave the recovery trajectory open.The policy would not need to recover as fast as possible.Source: §IV · p. 6
• The authors argue that relatively small training changes improve agility, while emphasizing that the robots' capabilities remain underused.They encourage applying similar techniques to other tasks.Source: §IV · p. 6