PaperlayerView exact revision on arXiv

Source-linked AI summary

Loco-Loco-RL: Low-Cost Terrain Mapping for Humanoid Locomotion with Reinforcement Learning

Jordan Dowdy, Gryffin Reizian, Jean Chagas Vaz

arXiv:2609.19041v1cs.RO

TL;DR

Humanoid terrain locomotion needs informative perception, but dense sensors can provide redundant data while adding cost and overhead. This paper uses sparse ToF terrain measurements with a token-compressed temporal transformer policy. Hardware experiments on the Robotis OP3 show sinusoidal command tracking and rough-terrain walking, with conservative motion near larger vertical changes.

  • Problem

    Dense terrain sensors can produce more raw data than policies need while increasing computational, memory, power, and integration overhead, motivating a low-cost sparse-sensing alternative.

    Source: §I · p. 1

  • Method

    The policy tokenizes proprioception and sparse terrain observations, uses self-attention to fuse current inputs, compresses tokens, and applies temporal cross-attention over a rolling history.

    Source: §I · p. 2, 2) Token-Compressed Temporal… · p. 3

  • Results

    Hardware experiments on the Robotis OP3 showed sinusoidal velocity-command tracking and rough-terrain walking, with the policy sometimes slowing or avoiding terrain changes to preserve stability.

    Source: §VI · p. 5, A. Velocity Tracking Metrics · p. 5, B. Multi-Terrain Locomotion Metrics · p. 5

  • Takeaways & Limitations

    The work demonstrates sim-to-real use of low-resolution ToF terrain sensing within a terrain-based locomotion benchmark on the Robotis OP3.

    Source: Paper Contributions · p. 2

    Near larger vertical terrain changes, the policy often reduced or ignored commanded motion to preserve stability; improving traversal over such discontinuities remains future work.

    Source: §VI · p. 6

Abstract

from arXiv · show

Informative terrain perception is important for robust reinforcement learning policies in humanoid locomotion. Still, common sensors such as depth cameras and LiDARs incur high cost, power, and processing overhead while often producing redundant, high-resolution data. This work uses a low-cost time-of-flight sensor to provide a compact 3D local terrain representation for humanoid locomotion. To efficiently use this sparse exteroceptive input, we introduce a token-compressed temporal transformer policy. Proprioceptive and terrain observations are tokenized and processed by a self-attention multi-head transformer to capture within-timestep relationships between observation terms. The attended tokens are then compressed through an MLP-based latent-space token compression module before being stored in a rolling 15-timestep history. A second cross-attention multi-head transformer extracts temporal locomotion features from this compact history for policy learning. By compressing tokens before temporal aggregation, the architecture preserves important terrain-observation structure while limiting the dimensional growth of attention over observation histories. We validate our method through sim-to-real transfer on physical hardware using a terrain-based locomotion benchmark, demonstrating robust humanoid terrain walking with low-cost local terrain sensing.

I. INTRODUCTION

Humanoid locomotion on unstructured terrain depends on useful terrain observations, yet dense sensors can provide more data than policies need while adding computational, power, and integration overhead. The paper investigates sparse, low-cost ToF sensing and introduces a token-compressed temporal transformer policy, then reports sim-to-real transfer on the Robotis OP3.

I. INTRODUCTION

  • Dense depth cameras and LiDAR can produce more raw data than policies need and add computational, memory, power, and integration overhead.These costs motivate investigating low-cost terrain sensing for humanoid locomotion.Source: §I · p. 1
  • The study asks whether a 64-point, limited-field-of-view ToF sensor can provide enough terrain information for robust RL locomotion.The sensor offers low cost, reduced power use, and compact integration despite its lower resolution than depth cameras or LiDAR.Source: §I · pp. 1–2
  • The policy tokenizes proprioception, actions, commands, and sparse height scans, fuses current observations with self-attention, compresses tokens, and queries rolling history with cross-attention.The compact latent representation is intended to preserve task-relevant terrain structure while reducing sensing and computational overhead.Source: §I · p. 2

Paper Contributions

  • The contributions include low-resolution ToF terrain representation, tokenized multimodal observations, pre-history token compression, and sim-to-real transfer on the Robotis OP3 using a terrain-based locomotion benchmark.Source: Paper Contributions · p. 2

Article Structure

The article outlines related work, its RL framework and methodology, experiments and results, and a conclusion discussing problems and future improvements.

Article Structure

  • The article proceeds from literature review to formal RL methodology, experimentation, results, and a conclusion with future improvements.Source: Article Structure · p. 2

1) Classical Humanoid Locomotion:

  • Classical humanoid locomotion work uses MPC with ZMP, Linear Inverted Pendulum, and Cart-Table methods for fine bipedal motion control, while flexible joints and linkages pose limitations addressed by more flexible MPC formulations,,.Source: 1) Classical Humanoid Locomotion · p. 2

2) Reinforcement Learning Humanoid Locomotion:

3) Transformer-Based Policy Architectures:

The paper situates transformer policies and terrain-aware RL among prior approaches, then describes its methodology and formalizes the RL policy objective.

3) Transformer-Based Policy Architectures:

4) Terrain-Based Features in Reinforcement Learning Locomotion:

  • Prior terrain-aware locomotion work combines exteroceptive and proprioceptive inputs, while humanoid parkour methods use vision-based whole-body policies for larger terrain changes and obstacles [10],,, [13].Source: 4) Terrain-Based Features in… · p. 2

III. METHODOLOGY

  • The methodology section covers system hardware and architecture, policy definition, RL rewards, simulation, domain randomization, observations, and training parameters.Source: §III · p. 2

1) Formal Definition:

  • The formal RL definition maps states to possible actions through a policy and defines an optimal policy by expected reward maximization.Source: 1) Formal Definition · p. 3

2) Token-Compressed Temporal Transformer Policy:

The policy tokenizes deployable proprioceptive and terrain observations, fuses current-timestep terms, then compresses them before temporal aggregation. Cross-attention over the stored history produces the feature used by the actor, while the critic can use privileged training signals.

2) Token-Compressed Temporal Transformer Policy:

  • Proprioceptive observations and ToF terrain measurements form the deployable actor input, with each observation term represented as a learned token.The actor’s semantic observation terms are split into 11 terms for tokenization.Source: 2) Token-Compressed Temporal… · p. 3
  • Self-attention fuses relationships among observation terms within the current timestep before an MLP compresses the attended tokens into four latent tokens.Each latent token has dimension 64; this compression limits dimensional growth when storing tokens across time.Source: 2) Token-Compressed Temporal… · p. 3
  • Cross-attention uses current compressed tokens to query a 15-timestep history, producing a temporal feature for the actor head.The current-token summary is added as a residual to the temporal summary.Source: 2) Token-Compressed Temporal… · p. 3
  • The critic follows the token-compressed temporal structure with additional privileged observations that are available during training but not required for deployment.Source: 2) Token-Compressed Temporal… · p. 3

B. Hardware, Simulation Environment, and Training

The system pairs an OP3 humanoid with an 8 × 8 ToF terrain scan and trains the policy in Isaac Sim. For hardware use, sensor measurements are streamed to a Jetson Nano for local policy inference.

B. Hardware, Simulation Environment, and Training

  • Training used Isaac Sim and Isaac Lab with 4096 parallel environments, a 500.0 Hz physics solver, a 50.0 Hz policy rate, and approximately 80 hours of GPU training.The platform was the Robotis OP3, and training used an Nvidia RTX 5090 GPU.Source: B. Hardware, Simulation Environment,… · p. 3
  • The 8 × 8 ToF height scan is treated as one 64-value observation term and tokenized with an MLP; the actor uses 11 current-timestep tokens.An MLP compresses the attended token set into four 64-dimensional latent tokens, which are stored in a 15-timestep rolling history and processed with cross-attention.Source: B. Hardware, Simulation Environment,… · pp. 3–4
  • For deployment, an ESP32 streams ToF measurements to the onboard Jetson Nano, where ROS formats the data as an 8 × 8 scan for local policy inference.Source: B. Hardware, Simulation Environment,… · p. 4

1) Observations, Actions, and Rewards:

The actor uses onboard observation groups, while the critic also receives privileged training-only signals. The policy outputs changes to 18 joint positions, and its reward design follows the authors’ prior work.

1) Observations, Actions, and Rewards:

  • Eight deployable semantic observation groups are split into 11 actor terms, while eight privileged training-only groups give the critic 19 terms.Center of mass, centroidal angular momentum, and capture point are critic-only signals and are not required for deployment.Source: 1) Observations, Actions, and Rewards · p. 4
  • The policy predicts changes from a nominal pose for 18 joint positions: 12 leg joints and 6 arm joints.Output scalers are 0.15 for the legs and 0.25 for the arms.Source: 1) Observations, Actions, and Rewards · p. 4
  • The reward design follows prior works, –, with the rewards, expressions, and weights listed in Table II.Source: 1) Observations, Actions, and Rewards · p. 4

C. Sim-to-Real

The sim-to-real setup combines domain randomization, varied terrain, and delayed actuator commands. The referenced tables cover observation access, reward and penalty terms, and randomization quantities.

C. Sim-to-Real

  • Joint properties were randomized by ±50.0% from their initialized quantities, alongside friction, armature, link mass, and link center of mass.The randomized joint properties included maximum torque and maximum velocity.Source: C. Sim-to-Real · p. 4, Table III · p. 5
  • The referenced tables cover observation terms and privileged access, reward and penalty terms, and domain randomization.The supplied table passages identify their topics but do not provide their entries.Source: Table I · p. 4, Table II · p. 4, Table III · p. 5
  • The terrain curriculum included rough, boxed, sloped, and custom uneven terrain types, with actuator-command delays of 0.0–0.02 seconds.The maximum delay equaled one policy control cycle.Source: C. Sim-to-Real · pp. 4–5

IV. EXPERIMENTATION AND VALIDATION

The hardware experiments were designed to validate the locomotion policy through velocity tracking and rough-terrain walking.

IV. EXPERIMENTATION AND VALIDATION

  • The experiments assessed the policy through real-world deployment and hardware testing.The stated aim was validation of the proposed humanoid locomotion policy.Source: §IV · p. 5
  • Velocity-tracking performance was one of the two evaluation targets.The other target was rough-terrain capability.Source: §IV · p. 5
  • Rough-terrain capability was assessed through a multi-terrain locomotion experiment.This evaluation accompanied the velocity-tracking assessment.Source: §IV · p. 5

A. Velocity Tracking Performance

The validation covered flat-ground sinusoidal velocity tracking and rough/rocky-terrain walking, while the results section introduces the collected performance measurements.

A. Velocity Tracking Performance

  • Hardware velocity tracking used 10 trials of 0.25 Hz sinusoidal commands, with amplitudes of 0.75 m/s for linear and 1.0 rad/s for angular velocities.The evaluation was conducted on flat terrain.Source: A. Velocity Tracking Performance · p. 5

B. Multi-Terrain Locomotion Validation

  • The rough/rocky-terrain experiment tested whether local ToF height scans provided enough information for terrain-aware walking while following velocity commands.It assessed stability on uneven ground using sparse local terrain features.Source: B. Multi-Terrain Locomotion Validation · p. 5

V. RESULTS AND DISCUSSION

  • Figure 3 presents velocity-tracking performance averaged over 10 trials for sinusoidal vx, vy, and ωz commands.The results section describes the collected experimental data and performance measurements.Source: Figure 3 · p. 5, §V · p. 5

A. Velocity Tracking Metrics

Hardware results show rough-terrain locomotion with conservative responses to larger vertical changes, alongside sinusoidal velocity tracking with weaker lateral accuracy.

A. Velocity Tracking Metrics

  • The policy followed the general sinusoidal command structure, with clearer periodic tracking for vx and ωx than for lateral vy.Responses were smoothed and attenuated rather than matching commanded amplitudes.Source: A. Velocity Tracking Metrics · p. 5

B. Multi-Terrain Locomotion Metrics

  • On rough terrain, the policy maintained locomotion over moderate surface variation but often slowed, stepped away, or ignored commands near larger vertical changes to preserve stability.The conclusion identifies conservative behavior near larger vertical terrain changes as the main observed limitation.Source: B. Multi-Terrain Locomotion Metrics · p. 5, §VI · p. 6

VI. CONCLUSION

  • Robotis OP3 hardware experiments demonstrated sinusoidal velocity tracking and rough-terrain walking using local terrain features.The conclusion summarizes these as outcomes of the low-cost ToF sensing and token-compressed temporal-transformer approach.Source: §VI · p. 5
  • Future work targets traversal over larger height discontinuities, improved velocity tracking, and testing whether richer terrain coverage improves robustness without increasing sensing cost.The conclusion also names richer local sensing as a future direction.Source: §VI · p. 6
Loading 2609.19041v1…