PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response

Junfeng Long, Zirui Wang, Quanyi Li, Jiawei Gao, Liu Cao, Jiangmiao Pang

arXiv:2312.11460v3cs.ROcs.AIcs.CVcs.LGcs.SYeess.SY

TL;DR

Partial, noisy robot observations make it difficult to estimate environmental conditions needed for robust locomotion. HIM estimates disturbances from a robot-response embedding using proprioception and contrastive learning. In real-world benchmarks, it outperforms the compared methods on all tasks and is reported to handle difficult and unseen terrain cases.

  • Problem

    Most legged-robot sensors provide partial and noisy observations, making accurate estimation of external states such as terrain friction and elevation maps challenging.

    Source: §1 · p. 2

  • Method

    HIM estimates environmental disturbances from an embedding of the robot’s explicit velocity and implicit stability response, optimized toward the successor state with contrastive learning.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    HIM outperforms RMA, MoB, and Built-in MPC on all real-world benchmark tasks and is reported to perform well on high-difficulty and unseen terrain cases.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Joint encoders and an IMU provide sufficient information for the framework, and contrastive learning provides a more noise-robust, sample-efficient representation than regression.

    Source: §5 · p. 9

    Without external perception such as vision, legged robots retain only a primitive skill set.

    Source: §5 · p. 9

Abstract

from arXiv · show

Robust locomotion control depends on accurate state estimations. However, the sensors of most legged robots can only provide partial and noisy observations, making the estimation particularly challenging, especially for external states like terrain frictions and elevation maps. Inspired by the classical Internal Model Control principle, we consider these external states as disturbances and introduce Hybrid Internal Model (HIM) to estimate them according to the response of the robot. The response, which we refer to as the hybrid internal embedding, contains the robot's explicit velocity and implicit stability representation, corresponding to two primary goals for locomotion tasks: explicitly tracking velocity and implicitly maintaining stability. We use contrastive learning to optimize the embedding to be close to the robot's successor state, in which the response is naturally embedded. HIM has several appealing benefits: It only needs the robot's proprioceptions, i.e., those from joint encoders and IMU as observations. It innovatively maintains consistent observations between simulation reference and reality that avoids information loss in mimicking learning. It exploits batch-level information that is more robust to noises and keeps better sample efficiency. It only requires 1 hour of training on an RTX 4090 to enable a quadruped robot to traverse any terrain under any disturbances. A wealth of real-world experiments demonstrates its agility, even in high-difficulty tasks and cases never occurred during the training process, revealing remarkable open-world generalizability.

1 INTRODUCTION

Learning-based quadruped locomotion has advanced, but partial and noisy observations make external-state estimation difficult. HIM estimates environmental disturbances from the robot’s response, using proprioceptive observations, and is reported to perform across varied and unseen terrain conditions.

1 INTRODUCTION

  • Learning-based quadruped controllers trained on simulated data have demonstrated agility and complex skills including manipulation and parkour.The cited work reports deployments in real environments, including the wild and low-cost hardware platforms.Source: §1 · p. 2
  • Prior two-stage methods train an oracle with external states before mimicking or inferring them, but noise, information loss, and platform-specific exteroceptive sensors constrain deployment.Domain-randomization noise can restrict optimization, especially when regressing absolute environmental parameters.Source: §1 · p. 2
  • HIM uses a hybrid internal embedding of velocity and stability, optimized toward the robot’s successor state with contrastive learning, to infer environmental disturbances without external-state inputs.The policy receives partial observations and the embedding; the embedding is extracted from historical observations.Source: §1 · p. 2
  • HIM is reported to traverse varied terrain under disturbances using joint encoders and an IMU, with training taking 1 hour on an RTX 4090.The method is also reported to handle long-range stairs and unseen compositional terrains and deformable slopes.Source: §1 · p. 2

2 RELATED WORK

Legged-locomotion research has moved from model-based control toward learned controllers that adapt to dynamics, but sim-to-real transfer and representation learning remain challenges. HIM distinguishes itself by using contrastive learning rather than regression objectives or computationally expensive model-based reinforcement learning.

2 RELATED WORK

  • Traditional model-based controllers have difficulty adapting to widely varying physical properties, while reinforcement-learning controllers trained in simulators can adapt to dynamics and deploy on real robots.The passage cites examples including ice, rough terrains, and deformable materials as varying physical conditions.Source: §2 · p. 3
  • Mimic-learning approaches use adaptation or teacher-student frameworks to address missing environmental information, but performance can drop substantially from reference to deployable policies.The passage names Kumar et al. (2021), Agrawal et al. (2022), and Margolis & Agrawal (2023) for adaptation, and Lee et al. (2020), Chen et al. (2019), Margolis et al. (2022), and Wu et al. (2022) for teacher-student methods.Source: §2 · p. 3
  • Regression-based dynamics-learning methods require fitting targets, which the authors say can fit domain-randomization noise and lead to representation collapse.Examples named are DayDreamer (Wu et al., 2023) and DreamWaQ (Nahrendra et al., 2023).Source: §2 · p. 3
  • Unlike model-based reinforcement learning and regression objectives, HIM uses prototypical contrastive learning to distinguish situations from historical observations and make fuller use of samples.The passage connects sample-driven regularization with better robustness.Source: §2 · p. 3

3 METHODOLOGY

HIM uses a response-based internal model to support legged locomotion without policy access to external states, and trains that model alongside the policy. Its embedding combines estimated velocity and implicit response, with the latter learned by contrastively aligning observation histories and successor states.

3.1 FRAMEWORK OVERVIEW

  • The policy receives partial observations and the hybrid embedding; training alternates HIO, which updates HIM, with PPO, which updates the actor and critic.HIM is frozen while the actor and critic are optimized.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 HYBRID INTERNAL MODEL

  • Treating environmental dynamics as disturbances, HIM estimates explicit velocity and implicit response to represent locomotion’s tracking and stability demands.These estimates and robot observations are fed to the policy as part of a closed-loop control system.Source: §3.2 · p. 5

3.3 HYBRID INTERNAL OPTIMIZATION

  • HIM regresses explicit velocity to simulation ground truth and learns a 16-dimensional implicit response by contrastively aligning histories with successor observations, using within-trajectory positives.Pairs from different trajectories are negatives; the framework uses SwAV-like swapping assignments.Source: §3.3 · p. 5
  • The learned latent representation separates environments more than regression, which the authors associate with richer terrain information and stronger terrain identification.The analysis uses t-SNE visualizations of HIM and regression outputs.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4 TRAINING DETAILS

  • Training uses 4,096 parallel Isaac Gym environments and 1,000 rollouts (one hour), with dynamics randomization and a 200-terrain difficulty curriculum.Performance continues improving until 2,000 rollouts; terrain levels change according to tracking reward and traversal progress.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate HIM against alternative controllers in simulation and on three Unitree robots, then ablate its components. HIM leads the reported real-world comparisons, performs better than other methods on almost all simulation tasks, and its internal model contributes to performance.

4.2 MAIN RESULTS

  • HIM outperforms RMA, MoB, and Built-in MPC on all real-world tasks while maintaining more natural gaits.The benchmarks include long-range stairs and compositional terrains and deformable slopes not encountered in training.Source: Table 2 · p. 7, §4.2 · p. 7

4.1 EVALUATION SETUPS

  • Real-world evaluations compare methods on Unitree A1, Go1, and Aliengo, while simulation evaluations compare tracking errors under varied terrains and motion ranges.The compared methods include ablations, Regression, Oracle, RMA, and MoB; Built-in MPC is included only in real-world experiments.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • HIM outperforms other methods on almost all simulation tasks and significantly outperforms Regression.It also performs better in 66.67% of the velocity range on easy terrain.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3 ABLATION STUDIES

  • In simulation, HIM’s results are most similar to Oracle despite requiring no terrain information, and removing either its internal embedding or velocity estimation degrades policy performance.The learning curves average 10 seeds; the ablations also report that hybrid internal optimization outperforms Regression and RMA does not imitate all Oracle performance.Source: §4.3 · p. 8, Figure 5 · p. 9
  • Real-world ablations show that HIM contributes to performance, with the internal model playing a critical role.Table 4 reports average performance over 20 trials with 95% confidence intervals.Source: §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSION

The conclusion links learning-based locomotion with Internal Model Control and reports that joint encoders and an IMU provide sufficient information. It also favors contrastive learning over regression for robustness to noise and sample efficiency, while identifying absent external perception as a boundary.

5 CONCLUSION

  • Joint encoders and an IMU provide sufficient information, while contrastive learning gives more noise-robust representations and higher sample efficiency than regression.The authors describe their use of batch-level information in locomotion tasks as a first.Source: §5 · p. 9

A.1 REWARD FUNCTIONS

A.2 DOMAIN RANDOMIZATIONS

  • Table 6 presents the domain randomizations and their respective ranges used to train HIM and the other methods.The randomizations are attributed to Wu et al. (2022) and Nahrendra et al. (2023); specific ranges are not given in the supplied passage.Source: §A.2 · p. 13, Table 6 · p. 13

A.3 TRAINING TERRAINS

The training terrains comprise slopes, rough slopes, stairs, and discrete obstacles, assigned difficulty levels from 0 to 9. The appendix also identifies a hyperparameter table.

A.3 TRAINING TERRAINS

  • Terrain difficulty levels run from 0 to 9, with slopes up to 40° and rough-slope noise from ±1 cm to ±8 cm.Stair widths range from 20–40 cm and heights from 5–23 cm; discrete-obstacle heights range from ±5–15 cm.Source: §A.3 · p. 14
  • Terrain-level formulas scale slope inclination, rough-slope amplitude and inclination, stair height, and discrete-obstacle height with levelterrain.The four terrain proportions are 0.1, 0.2, 0.6, and 0.1, respectively, for slopes, rough slopes, stairs, and discrete obstacles.Source: §A.3 · p. 14

A.4 HYPER PARAMETERS

  • Table 7 is identified as the table of hyperparameters for training.The supplied passage gives no hyperparameter values.Source: Table 7 · p. 14

B.1 REAL-WORLD BENCHMARK SETUP

The real-world benchmark covers stairs, unseen terrains, and resistance to disturbances. Figure 6 provides scene snapshots for the quantitative analysis.

Stairs

  • The benchmark includes short- and long-range stair tests, measuring passage success on two five-step stair sets and the number of stairs ascended without stopping.The short-range test uses an A1 on stairs 16 cm high and 27 cm wide; the long-range test uses an Aliengo on stairs 17 cm high and 25 cm wide.Source: §B.1 · p. 15

Unseen terrains

  • Unseen-terrain tests measure success on gravel-and-stair alternations and on a deformable slope made from two soft beds.Aliengo is used for the compositional terrain test and A1 for the deformable-slope test.Source: §B.1 · p. 15

Anti-disturbance

  • Anti-disturbance tests measure maximum tolerated load during a leg-dragging obstacle, a lateral hit, and walking with a payload, plus balance success after a 42 cm drop.The tests use A1 except the missing-step test, which uses Aliengo.Source: §B.1 · p. 15

Benchmark scenes

  • Figure 6 shows snapshots of the real-world benchmark scenes used for quantitative analysis.The figure covers the benchmark scenes described in the text.Source: Figure 6 · p. 15

B.2 SIMULATION BENCHMARK SETUPS

Simulation benchmarks test velocity tracking across three command ranges and terrain levels 1–4. Tracking tests isolate velocity components or measure them together.

Command ranges

  • Tracking commands span three ranges: slopes and rough slopes include ranges up to ±2.0 m/s forward and ±2.0 rad/s yaw, while other tests use ±1.0 m/s and ±1.0 rad/s.The command dimensions are forward and lateral linear velocity and yaw angular velocity.Source: §B.2 · p. 16

Test conditions

  • Linear-velocity tests set angular velocity to zero, angular-velocity tests set linear velocity to zero, and combined tests set neither to zero.Commands are uniformly sampled; terrain levels 1–4 are tested with equal 25% representation.Source: §B.2 · p. 16

B.3 ABLATION ON NUMBER OF PROTOTYPES

The prototype ablation finds that tracking performance improves as prototype count rises to 64, then declines. The results point to a trade-off between early training speed and final performance, with the appropriate count still requiring investigation.

B.3 ABLATION ON NUMBER OF PROTOTYPES

  • Performance steadily improves as prototype count increases to 64, then starts to decline.Figure 7 shows normalized linear tracking score over prototype count.Source: §B.3 · p. 16, Figure 7 · p. 16
  • Higher prototype counts accelerate initial training but can compromise final performance, while lower counts may train more slowly and finish better.The ablation suggests environmental dynamics need neither dense regression-like representation nor an overly sparse one.Source: §B.3 · p. 16
  • Choosing a prototype count that balances training speed and final performance remains an open issue for further study.The default configuration uses 16 prototypes, corresponding to the square of terrain types.Source: §B.3 · p. 16

B.4 OUTDOOR EXPERIMENTS

The policy was evaluated in additional real-world outdoor scenes, with scene illustrations provided for the agility evaluation.

B.4 OUTDOOR EXPERIMENTS

  • The policy was further evaluated in additional real-world scenes.The paper points to videos of the outdoor experiments.Source: §B.4 · p. 17
  • Videos of the additional outdoor experiments are referenced by the paper.Source: §B.4 · p. 17
  • Figure 8 presents snapshots of outdoor scenes used for the quadruped robot’s agility evaluation.Source: Figure 8 · p. 17
Loading 2312.11460v3…