PaperlayerView exact revision on arXiv

Source-linked AI summary

PGMT: Perceptive General Motion Tracking for Humanoid Robots

Hongyi Li, Li Peizhuo, Yucheng Tao, Ze Wang, Fangzhou Xu, Jinyi Chen, Yanyan Yuan, Dapeng Jia, Yongbin Jin, Mingfeng Fan, Guillaume Sartoretti, Hongtao Wang

arXiv:2609.08511v2cs.RO

TL;DR

Terrain-agnostic humanoid references can become physically infeasible on complex terrain. PGMT learns a general tracking prior and injects motion-conditioned terrain perception, allowing terrain-aware adaptation while preserving motion intent. Its reported evaluations show higher benchmark completion than the pretraining-only tracker and execution of diverse whole-body motions on uneven terrain.

  • Problem

    Terrain-agnostic references can become physically infeasible on complex terrain, requiring adaptation to local terrain while retaining the commanded motion.

    Source: §I · p. 1

  • Method

    PGMT fuses raw motion references with local terrain observations and uses terrain glimpses to select regions relevant to the current motion.

    Source: §I · p. 2, 1) Perception Injection Process · p. 4

  • Results

    PGMT-Ours achieves 87.81% full-benchmark completion and 83.33% L9 completion, versus 40.02% and 35.31% for PGMT-Pretrain, respectively.

    Source: B. Comparison with Motion-Tracking… · p. 5

  • Takeaways & Limitations

    The same policy executes diverse whole-body references on stairs and uneven surfaces while adapting contacts and posture to local terrain.

    Source: 4) Disturbance Robustness and Fall… · p. 7

Abstract

from arXiv · show

Humanoid motion trackers can reproduce diverse whole-body motions, but their performance degrades on complex terrain where terrain-agnostic references become physically infeasible. We present PGMT, a Perceptive General Motion Tracking pipeline for humanoid robots that learns terrain adaptation from independently selected motion references and terrains. PGMT first learns a general tracking and recovery prior, then incorporates terrain perception through motion-conditioned terrain glimpses that selectively encode regions relevant to the current motion. Terrain-aware tracking relaxation allows necessary deviations from the reference while preserving its motion intent. Zero-shot deployment on a Unitree G1 demonstrates robust terrain-adaptive locomotion and whole-body motion execution over real-world terrain with obstacles up to 37 cm high, while supporting teleoperation, dynamic motion tracking, and fall recovery. PGMT extends general humanoid motion tracking beyond flat ground, providing a unified policy for terrain-adaptive locomotion, diverse whole-body behaviors, and teleoperation in complex environments. Project homepage: https://luyili.github.io/pgmt/

I. INTRODUCTION

PGMT addresses terrain-agnostic motion references that can be infeasible on complex terrain by making adaptation part of the tracking policy. It combines motion intent with local terrain constraints so the robot can adapt execution while preserving the intended motion.

I. INTRODUCTION

  • Human operators and higher-level planners cannot anticipate exact terrain geometry, so onboard perception must resolve local constraints during execution.The reference specifies what motion to perform; environmental observations determine how to realize it physically.Source: §I · p. 1
  • Motion and terrain data are naturally unpaired, paired demonstrations become harder to collect as diversity grows, and terrain-adapted references limit scalability.Existing approaches that construct adapted references depend on the quality and generalization of that intermediate stage.Source: §I · pp. 1–2
  • PGMT fuses raw motion references with local terrain observations, adapting footholds and posture when strict tracking is infeasible while preserving motion intent.The policy learns this adaptation directly, without an upstream terrain-matched reference generator.Source: §I · p. 2
  • PGMT is presented as a general motion tracker that follows terrain-agnostic references over complex terrain without terrain-matched trajectories.The same policy is described as a low-level interface between terrain-agnostic commands and terrain-aware execution.Source: §I · p. 2
  • PGMT decouples motion and perception encoding so new perceptual modalities can be injected while preserving learned motion-tracking capability.The architecture is designed to support progressive extension.Source: §I · p. 2
  • Simulations and real-world experiments across diverse terrains and motion-reference command sources demonstrate PGMT's zero-shot generalization.The introduction frames applications including teleoperation, search and rescue, and transportation through complex environments.Source: §I · pp. 1–2

II. RELATED WORKS

Prior humanoid trackers and perception-aware controllers address complementary parts of terrain adaptation, but diverse whole-body references are rarely paired with terrain-specific adaptations. The paper frames perceptive tracking using proprioception, future motion references, and an elevation map as policy inputs.

II. RELATED WORKS

  • Existing perceptive controllers adapt well to terrain, but their objectives remain tied to locomotion or obstacle-crossing tasks rather than diverse whole-body references.Perception-aware methods have used terrain representations, global-local reasoning, or active gaze to focus on relevant regions [19],,, [22], [23], [24].Source: §II · p. 2
  • Large motion collections are rarely paired with terrain-specific adaptations; methods add missing information through pre-learning supervision or generation-tracking adaptation, but intermediate-signal errors can propagate to the tracker.Pre-learning supervision uses terrain-conformal references, scene geometry, motion-matched experts, or inferred contacts [3], [4], [27], [28]; generation-tracking methods use execution feedback, [29], while guided diffusion builds behaviors from motion primitives collected by a real-robot tracker [30].Source: §II · p. 2
  • Tracking depends on reference quality: retargeting artifacts and infeasible poses pass to the policy, while references lack deployment geometry and may conflict with terrain or contact constraints [18].Source: §II · p. 2

III. PROBLEM FORMULATION

  • The policy is formulated as a POMDP: it receives proprioception, a ten-frame proprioception history, future reference frames, and an elevation map, and outputs target joint positions.The future frames convey motion intent; the elevation map covers a yaw-aligned 2 m × 2 m region and is introduced during the second training stage.Source: §III · pp. 2–3
  • Future reference frames are sampled at exponentially increasing temporal offsets, and collectively provide future motion intent to the policy.Figure 2 summarizes a tracking-pretraining stage followed by perception injection, which combines local geometry with motion intent.Source: §III · p. 3, Figure 2 · p. 3

IV. PERCEPTIVE GENERALIZED MOTION TRACKING

PGMT first learns a terrain-agnostic whole-body tracking prior, then adds elevation-map perception for terrain-aware adaptation. Its pretraining combines a state-conditioned intent-fusion architecture with reward decomposition, recovery training, and adaptive motion sampling.

IV. PERCEPTIVE GENERALIZED MOTION TRACKING

  • PGMT uses two stages: tracking pretraining learns a terrain-agnostic prior, then perception injection adds terrain perception without relearning tracking from scratch.The second-stage observation includes the elevation map in addition to the non-perceptive input.Source: §IV · p. 3

A. Tracking Pretraining Stage

  • The first stage trains a general whole-body motion prior on flat terrain, with an Intent Fusion Module combining proprioception and future reference frames for the actor and critic.The actor uses the resulting state-conditioned representation for low-level control, while the critic uses it for value estimation.Source: A. Tracking Pretraining Stage · p. 3
  • Body-part-based reward decomposition, a recovery curriculum, and adaptive motion sampling support learning a unified policy for diverse whole-body motions.These training components are part of the terrain-agnostic pretraining stage.Source: A. Tracking Pretraining Stage · p. 3

1) Terrain-agnostic Policy Architecture:

The terrain-agnostic policy compresses proprioceptive history and conditions future motion intent on the robot’s current state. The actor uses that intent for control, while a multi-head critic models reward groups separately.

1) Terrain-agnostic Policy Architecture:

  • Cross-attention uses current proprioception to query historical observations, producing a compact token for recent robot dynamics and temporal context.The current observation is also retained by the actor for instantaneous low-level feedback.Source: 1) Terrain-agnostic Policy Architecture · p. 3
  • The Intent Fusion Module uses a history-conditioned state token to attend to current state and future references, producing motion intent conditioned on the robot’s state and recent dynamics.This lets the policy selectively emphasize future reference information; the intent token is shared by actor and critic.Source: 1) Terrain-agnostic Policy Architecture · p. 3
  • The actor combines current proprioception with the motion-intent token to produce control actions.Target joint positions are converted to torques by a low-level PD controller.Source: 1) Terrain-agnostic Policy Architecture · p. 3
  • A shared-backbone multi-head critic estimates separate values for upper-body, lower-body, and auxiliary reward groups.Its privileged observations are available only during simulation training.Source: 1) Terrain-agnostic Policy Architecture · p. 4

2) Training Optimization:

PGMT first trains a terrain-agnostic tracking prior on flat terrain using retargeted LAFAN1 motions. Root-centric tracking, recovery initialization, and adaptive motion sampling support diverse behavior and recovery.

2) Training Optimization:

  • Flat-ground pretraining uses retargeted LAFAN1 motions sampled across sequences and starting frames to train one policy over diverse whole-body behaviors.Root-centric errors emphasize relative body motion and reduce dependence on absolute position and heading, supporting transfer across initial poses and headings.Source: 2) Training Optimization · p. 4
  • A recovery curriculum progressively increases postfall-state initialization as episode-survival performance improves.The offline pool contains postfall robot states.Source: 2) Training Optimization · p. 4
  • Adaptive sampling raises the probability of motion segments associated with frequent tracking failures while retaining uniform coverage of the motion dataset.Source: 2) Training Optimization · p. 4

B. Perception Injection Stage

The perception-injection stage adds motion-conditioned terrain information to the tracking policy. Sparse learned terrain glimpses enter the Intent Fusion Module, while an added critic head focuses learning on terrain contact.

1) Perception Injection Process:

  • The Terrain-Glimpse Encoder predicts four locomotion-relevant locations from state, terrain, and future motion, then samples 5×5 terrain patches for compact control inputs.It targets regions around potential footholds and terrain boundaries rather than densely encoding the full elevation map.Source: 1) Perception Injection Process · p. 4
  • Bilinear sampling makes terrain-patch cropping differentiable, allowing glimpse locations to be learned without region or foothold annotations.The resulting local terrain tokens are passed to the IFM.Source: 1) Perception Injection Process · p. 4
  • The IFM incorporates local terrain tokens by augmenting its keys and values, allowing its motion-intent representation to include terrain-aware information.The first-stage IFM conditions on proprioception and future motion references; the injection adds terrain tokens to that input.Source: 1) Perception Injection Process · p. 5
  • A new terrain-contact reward group assesses touchdown quality and contact consistency, with penalties for slippage, stumbling, rapid contact switching, and excessive contact forces.Touchdown quality uses local height variation; offline terrain-mesh queries provide contact labels.Source: 1) Perception Injection Process · p. 5
  • An additional critic value head estimates terrain-contact return, providing a focused learning signal for terrain interaction and foothold selection.The perception-injection stage extends the critic from three output values to four.Source: 1) Perception Injection Process · p. 5

2) Terrain-Adaptive Training Design:

Terrain-adaptive training relaxes selected tracking objectives where terrain requires deviation, while correcting planar position drift and sampling feasible motion–terrain combinations through a curriculum.

2) Terrain-Adaptive Training Design:

  • Terrain-aware relaxation subtracts terrain- and difficulty-scaled tolerance from selected lower-body tracking errors, while upper-body objectives and remaining motion constraints stay strict.It applies to lower-body link position, link orientation, and joint position objectives; flat or other terrain families do not activate the relaxation.Source: 2) Terrain-Adaptive Training Design · p. 5
  • Planar position error is computed in the reference-anchor frame and fed back into the reference velocity command.The error uses reference and robot world-frame anchor positions and the reference anchor rotation to extract planar components.Source: 2) Terrain-Adaptive Training Design · p. 5
  • A level-based curriculum trains on flat terrain, slopes, stairs, boxes, and random rough terrain, advancing after success and regressing after tracking failures.Terrain level jointly controls geometric difficulty, tracking relaxation, and selected termination delays; behavior–terrain compatibility excludes clearly invalid combinations without paired demonstrations or foothold targets.Source: 2) Terrain-Adaptive Training Design · p. 5

V. EXPERIMENTS

PGMT is evaluated in simulation against general motion-tracking baselines and alternative terrain-perception designs, then tested across real-world control modes.

  • Simulation comparisons cover general motion-tracking baselines and alternative terrain-perception designs across terrain types and difficulty levels.Source: §V · p. 5
  • Real-world evaluation tests PGMT across multiple control modes.Source: §V · p. 5
  • The evaluation combines simulation comparisons with real-world deployment tests.Source: §V · p. 5

A. Experimental Setup

The experiments evaluate terrain-adaptive tracking across matched terrain conditions, compare PGMT with motion-tracking baselines, and ablate perception and critic designs.

A. Experimental Setup

  • Each terrain family has ten difficulty levels, and policies are evaluated on 9,600 matched 30-s episodes with 192 episodes per terrain–level pair.Matched episodes share motion references, starting frames, terrain assignments, randomization, observation corruption, and actuator-delay settings.Source: A. Experimental Setup · p. 5

B. Comparison with Motion-Tracking Baselines

C. Ablation Studies

  • Across four non-flat terrain families at L9, PGMT-Ours averages 79.56% success versus 76.43% for PGMT-CNN, with a 78.12% stair success rate versus 64.58–69.79% for ablated variants.PGMT-CNN remains comparable over the full evaluation schedule, while the larger L9 stair separation highlights the value of motion-conditioned spatial selection on difficult terrain and motions.Source: C. Ablation Studies · p. 6, Figure 3 · p. 6
  • During Perception Injection, the split-return critic continues progressing, reaching about 380 mean episode steps and nearly 80% timeout rate, while aggregate-return metrics remain near zero.Both critics learn comparably during Tracking Pretraining; the split-return critic uses separate heads for upper-body, lower-body, and auxiliary reward groups.Source: Figure 4 · p. 6, Figure 5 · p. 7

D. Real-World Evaluation

PGMT is deployed zero-shot on a Unitree G1 and demonstrates terrain-adaptive locomotion indoors and outdoors, including on unseen stairs, uneven ground, and high boxes.

D. Real-World Evaluation

  • Zero-shot deployment uses a 29-DoF Unitree G1 with LiDAR and onboard elevation mapping, with perception and control running on a Jetson Orin NX.Source: D. Real-World Evaluation · p. 7

1) Terrain-Adaptive Locomotion:

  • PGMT traverses uneven terrain, ascends and descends staircases with unseen geometries, and climbs onto boxes up to 37 cm high while following flat-ground motion references.It adapts foot placement, swing clearance, and body posture according to the local elevation map.Source: 1) Terrain-Adaptive Locomotion · p. 7
  • PGMT also handles irregular outdoor surfaces and jogging or sprinting on stairs, including clearing multiple steps in one stride.Source: 1) Terrain-Adaptive Locomotion · p. 7
  • Stable locomotion persists on soft surfaces and under substantial perception noise without real-world fine-tuning.Source: 1) Terrain-Adaptive Locomotion · p. 7

2) Teleoperation:

PGMT supports whole-body and locomotion teleoperation, terrain-aware whole-body motion, and disturbance recovery with a unified policy.

2) Teleoperation:

  • Whole-body teleoperation executes operator-commanded behaviors including punching, squatting, lying down, and returning to standing, with operator motion captured by PICO and retargeted online using GMR.Source: 2) Teleoperation · p. 7
  • In locomotion teleoperation, the operator controls upper-body motion while motion matching generates the lower-body reference, allowing stair traversal without explicitly specifying terrain-dependent lower-body motions.Source: 2) Teleoperation · p. 7
  • PGMT executes dynamic whole-body motions, including cartwheels, on stairs and uneven surfaces while adapting contacts and posture to local terrain.Terrain adaptation preserves the structure of whole-body references while making terrain-dependent adjustments for stable execution.Source: 4) Disturbance Robustness and Fall… · p. 7

4) Disturbance Robustness and Fall Recovery:

  • After substantial perturbations, PGMT can recover from a fallen configuration to stable standing, with disturbance rejection and recovery handled by the same policy.The recovery occurs without switching to a dedicated recovery controller.Source: 4) Disturbance Robustness and Fall… · p. 8

VI. CONCLUSION

  • A current limitation is that elevation maps encode geometry but not semantics or affordances, making it difficult to distinguish obstacles from objects intended for interaction.Source: §VI · p. 8

Related papers

Loading 2609.08511v2…