Source-linked AI summary
Trajectory-Level Automatic Curriculum Learning for Legged Locomotion on Unstructured Terrain
Rocky Liu, Tengyu Liu, Baoxiong Jia, Fangwei Zhong, Xinyi Tong, Hongzhao Xie, Siyuan Huang
TL;DR
Curriculum learning for legged locomotion lacks a principled way to order tasks on unstructured terrain. TACL learns task difficulty from current-policy rollouts and samples matched trajectories, improving traversal capability and trajectory-level generalization over non-curriculum and handcrafted curricula.
Problem
Unstructured terrain lacks explicit criteria for selecting tasks suited to the current policy or ordering their difficulty, so existing methods impose structured curriculum spaces.
Method
TACL learns a current-policy difficulty evaluator from rollout outcomes and uses it in a closed loop to sample capability-matched trajectories directly from unstructured terrain maps.
Results
TACL improves traversal capability over non-curriculum training and robustness across diverse approach directions versus handcrafted curricula, including a 39.5% trajectory-level generalization improvement.
Takeaways & Limitations
TACL provides effective curricula directly from raw terrain maps without handcrafted terrain templates or predefined trajectories.
Takeaways & Limitations
Highly structured maneuvers may require directed exploration, such as weak task priors or demonstration-guided proposals, beyond generic trajectory sampling.
Abstract
from arXiv · showhide
Training locomotion policies for complex unstructured terrain requires a curriculum to avoid early exploration failures. However, since unstructured terrain lacks explicit difficulty ordering for curriculum design, existing methods resort to heuristic curricula over parameterized terrains. This abstraction limits generalization, as policies can overadapt to near-fixed perceptual patterns. To address this, we propose \textbf{\ourname{}}, an \textbf{T}rajectory-level \textbf{A}utomatic \textbf{C}urriculum \textbf{L}earning framework that generates training tasks directly from unstructured terrain maps. At each curriculum update, the evaluator learns a difficulty function for the current policy that maps a given trajectory task to a difficulty score. The sampler then proposes new trajectories guided by the learned evaluator as the curriculum for the next policy update. This forms a closed loop in which the curriculum is iteratively matched to the evolving policy. Quantitative and qualitative experiments show that \ourname{} continuously provides effective curricula on unstructured terrain, improving trajectory success rate by \(56.3\%\) over direct training without curriculum. Compared with handcrafted curriculum learning, our method improves success rate by \(18.5\%\) on the hardest terrain tasks and by up to \(39.74\%\) when evaluating traversal from diverse approach directions on the same obstacle type.
1 Introduction
TACL is a closed-loop curriculum framework that generates trajectory-tracking tasks directly from raw unstructured maps, addressing the lack of explicit difficulty ordering and limitations of handcrafted curricula.
- Motivation and problem: Unstructured terrain lacks an explicit criterion for selecting tasks suitable for the current policy or progressing task difficulty.This creates early-stage exploration failures that can produce conservative locomotion policies.
- Motivation and problem: Handcrafted curriculum learning uses parameterized geometric templates and heuristic progressions, requiring substantial manual effort while limiting task distributions.These limitations can cause policies to overadapt to nearly fixed perceptual patterns.
- TACL framework: TACL automatically generates trajectory-tracking curricula from raw unstructured maps to match the evolving competence of the policy.Its evaluator and sampler form the framework’s two primary modules.
- Experiments: 39.5% trajectory-level generalization improvement was achieved while producing more robust terrain traversal performance than HCL.Evaluation covered an unstructured terrain map and the handcrafted curriculum map from Extreme Parkour, with comparisons against random trajectory sampling and HCL.
- TACL framework: TACL uses a context-aware transition encoding and evaluator to predict task difficulty for the current policy and guide sampling of capability-matched training tasks.The resulting curriculum is generated in a closed loop rather than through manually parameterized terrain templates.
2 Related Works
Prior locomotion curricula either manually schedule increasingly difficult tasks or adapt task selection to the policy’s learning state. Handcrafted methods parameterize terrain and trajectory difficulty, while automatic methods use learning signals to select tasks suited to the current policy.
- Handcrafted Curriculum Learning in Robot Locomotion: Handcrafted curricula manually schedule training tasks from easy to hard to reduce mismatch between task difficulty and policy capability.In standard locomotion, scheduling can expand command ranges such as target velocity, heading, or motion amplitude.
- Handcrafted Curriculum Learning in Robot Locomotion: For complex terrain traversal, prior methods parameterize obstacle families and advance policies through manually specified difficulty sequences using heuristic performance thresholds.Obstacle families include stairs, gaps, hurdles, and ramps.
- Automatic Curriculum Learning in Robot Locomotion: Automatic curriculum learning adaptively selects training tasks according to the policy’s learning state, reducing the need for manual curriculum design.The central challenge is choosing tasks that are neither too easy nor too difficult for the current policy.
- Automatic Curriculum Learning in Robot Locomotion: Existing automatic approaches use reward prediction, temporal-difference (TD) error, regret, surprise, or learning progress as task-selection signals.The passage identifies these signals as approaches for selecting training tasks automatically.
3 Method
The method builds a closed-loop curriculum that samples waypoint trajectories from unstructured terrain maps, evaluates their policy-conditioned difficulty, and generates subsequent tasks accordingly. It combines transition descriptors, balanced success/failure supervision, and constrained difficulty-guided trajectory search.
- Closed-loop curriculum: Each curriculum iteration couples policy rollout collection, difficulty-evaluator training, and evaluator-guided generation of the next trajectory task set.The evaluator is updated from transition-level outcomes produced by the current policy, then guides the sampler in the next curriculum iteration.
- Task representation: A task is a waypoint sequence on terrain map M, with the robot starting at p0 and sequentially tracking p1 through pN.Consecutive waypoints define sub-trajectories for individual rollout episodes.
- Task representation: Transition descriptors combine preceding and current height patches, validity masks, and their relative yaw angle.Patches are cropped along waypoint directions and start-referenced, while masks identify cells covered by the actual transition length.
- Difficulty evaluation: The evaluator learns policy-conditioned difficulty scores from latent transition codes using balanced queues of successful and failed traversal outcomes.Its score di = Dk(zi) lies in [0, 1], with larger values indicating greater difficulty; difficulty reflects the current policy rather than intrinsic terrain difficulty.
- Evaluator-guided generation: The sampler searches waypoint locations by minimizing hard feasibility penalties plus a soft objective that matches sub-trajectories to target difficulties sampled from a moderate truncated-Gaussian interval [α, β].Hard penalties reject boundary violations, excessive transition lengths, and invalid or non-traversable waypoint regions.
- Evaluator-guided generation: MH-MCMC uses symmetric Gaussian waypoint proposals, so proposal terms cancel in the acceptance ratio while proposal noise and sampling temperature decay over iterations.The target interval [α, β] remains fixed across curriculum updates.
4 Experiment
Experiments evaluate TACL’s adaptive curriculum on unstructured terrain against random sampling and handcrafted progression, including capability growth, generalization, robustness, and CVAE-based difficulty estimation. TACL progressively targets harder terrain, improves traversal performance over baselines, and is tested in a CVAE-ablation study.
- Experimental setup: Experiments compare TACL with RandST and HCL-E using the same policy implementation but different training-task sources.TACL samples adaptively, RandST samples feasible trajectories randomly, and HCL-E follows Extreme Parkour’s handcrafted curriculum.
- Capability expansion: TACL progressively shifts sampled tasks toward harder terrain as the policy capability improves.Evaluator-predicted difficulty spans the full range, while trajectory boundaries move toward harder terrain during training.
- Capability expansion: By 16k epochs, TACL achieves above 70% success across all tested platform heights, while RandST remains conservatively reliable around 0.3 m.TACL has nonzero success across all heights by 4k and over 50% success below 0.65 m by 8k.
- Obstacle-family generalization: TACL generalizes better than HCL-E across most obstacle families, whereas HCL-E’s success declines from 5k epochs to the final checkpoint.HCL-E changes from 94%→82% on step, 78%→30% on gap, 81%→62% on parkour, and 93%→89% on hurdle; TACL improves steadily without handcrafted trajectories.
- Approach-direction robustness: 98.42% overall success is achieved by TACL versus 58.68% for HCL-E on step tasks approached from diverse directions within the training height range.The highest-difficulty training step terrain is 0.5 m, and the metric averages success over diverse approach directions.
- CVAE ablation: TACL’s CVAE-based transition encoding is evaluated as critical by comparing it with a raw-transition evaluator variant while keeping sampling and policy training unchanged.The ablation predicts binary difficulty from the raw transition descriptor using separate CNN encoders fused with the direction component.
5 Conclusion
TACL enables robot locomotion policies to learn complex traversal skills on unstructured terrain without handcrafted terrain templates or predefined trajectories. It closes the curriculum loop by learning difficulty from current-policy rollouts and guiding new training-trajectory generation.
- TACL enables policies to learn complex traversal skills on unstructured terrain without handcrafted terrain templates or predefined trajectories.
- The framework learns a difficulty evaluator from current-policy rollouts to guide sampling of new training trajectories for subsequent policy updates.
- Experiments show that TACL generates effective curricula directly from raw terrain maps.
6 Limitations
The framework learns robust locomotion without manually predefined trajectories, but generic trajectory sampling may be insufficient for highly structured maneuvers. Weak task priors or demonstration-guided proposals could improve sampling efficiency while retaining direct curriculum generation on unstructured maps.
- 6 Limitations: Generic trajectory sampling may provide insufficiently directed exploration for highly structured maneuvers.The framework otherwise learns robust locomotion behaviors without manually predefined trajectories.
- 6 Limitations: Weak task priors could improve sampling efficiency for highly structured behaviors.This addition is proposed as a way to provide more directed exploration.
- 6 Limitations: Demonstration-guided proposals could improve sampling efficiency while preserving direct curriculum generation on unstructured maps.