PaperlayerView exact revision on arXiv

Source-linked AI summary

Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation

Fukang Liu, Zhaoyuan Gu, Yilin Cai, Ziyi Zhou, Hyunyoung Jung, Jaehwi Jang, Shijie Zhao, Sehoon Ha, Yue Chen, Danfei Xu, Ye Zhao

arXiv:2409.20514v6cs.RO

TL;DR

Humanoid loco-manipulation remains challenging because of high-dimensional dynamics and contact-rich interactions, while dynamically feasible whole-body motion learning is an open problem. Opt2Skill uses DDP-based trajectory optimization to generate references for RL tracking. It reports improved tracking and task success over human-data and inverse-kinematics references, torque-related gains in contact-force tracking, and successful hardware transfer.

  • Problem

    Humanoid robots face high-dimensional, underactuated dynamics and complex contact-rich interactions, and learning dynamically feasible whole-body motions for diverse loco-manipulation remains an open problem.

    Source: §I · p. 1

  • Method

    Opt2Skill uses DDP-based trajectory optimization to generate whole-body motions that obey robot dynamics and task requirements, then trains RL policies to track those references.

    Source: §I · p. 1, A. Whole-Body Trajectory Optimization · p. 3

  • Results

    Opt2Skill outperforms human-data and inverse-kinematics references in tracking accuracy and task success, while torque information improves contact-force tracking in contact-rich tasks.

    Source: §I · pp. 1–2, 1) Performance Analysis · p. 7

  • Takeaways & Limitations

    On Digit, the approach transfers to hardware for versatile loco-manipulation skills, including stair traversing, multi-contact box manipulation, and door traversing.

    Source: §V · p. 8

Abstract

from arXiv · show

Humanoid robots are designed to perform diverse loco-manipulation tasks. However, they face challenges due to their high-dimensional and unstable dynamics, as well as the complex contact-rich nature of the tasks. Model-based optimal control methods offer flexibility to define precise motion but are limited by high computational complexity and accurate contact sensing. On the other hand, reinforcement learning (RL) handles high-dimensional spaces with strong robustness but suffers from inefficient learning, unnatural motion, and sim-to-real gaps. To address these challenges, we introduce Opt2Skill, an end-to-end pipeline that combines model-based trajectory optimization with RL to achieve robust whole-body loco-manipulation. Opt2Skill generates dynamic feasible and contact-consistent reference motions for the Digit humanoid robot using differential dynamic programming (DDP) and trains RL policies to track these optimal trajectories. Our results demonstrate that Opt2Skill outperforms baselines that rely on human demonstrations and inverse kinematics-based references, both in motion tracking and task success rates. Furthermore, we show that incorporating trajectories with torque information improves contact force tracking in contact-involved tasks, such as wiping a table. We have successfully transferred our approach to real-world applications.

I. INTRODUCTION

Humanoid loco-manipulation is difficult because of high-dimensional dynamics and contact-rich interactions, while existing reference sources can be infeasible or hard to obtain at scale. Opt2Skill uses full-body trajectory optimization to guide RL, reporting better tracking and task success than human-data and inverse-kinematics references, with hardware transfer across diverse tasks.

I. INTRODUCTION

  • Humanoid control is challenging because robots are high-dimensional and underactuated, with complex contact-rich interactions.Model-based whole-body planning for versatile tasks also faces computational demands and a need for precise contact-state feedback.Source: §I · p. 1, A. Model-Based Trajectory Optimization… · p. 2
  • Learning dynamically feasible whole-body motions from model-based methods for diverse humanoid loco-manipulation remains an open problem.Retargeted human motions may not be kinematically or dynamically feasible, and high-quality human data for diverse tasks is challenging to acquire.Source: §I · p. 1
  • Opt2Skill outperforms human-data and inverse-kinematics references in tracking accuracy and task success, and transfers to hardware across diverse loco-manipulation tasks.The demonstrated tasks include bulky-object handling, door traversal, desk-object manipulation, and uneven-terrain locomotion.Source: §I · pp. 1–2
  • Full-body trajectory optimization provides higher-quality motion data than mocap and IK baselines; torque information further improves tracking in contact-rich scenarios.The contribution claims connect these reference qualities to better tracking accuracy and task success.Source: §I · pp. 1–2

B. Reinforcement Learning for Humanoids

RL motion imitation can improve data efficiency and naturalness over learning from scratch, but human references require retargeting and can be dynamically infeasible. Opt2Skill instead uses full-order trajectory optimization to generate task-specific references and train whole-body RL policies for hardware transfer.

B. Reinforcement Learning for Humanoids

  • RL motion imitation simplifies reward tuning and improves data efficiency over training from scratch while producing more natural motions.Human-motion retargeting remains a challenge; whole-body loco-manipulation retargeting can yield dynamically infeasible motions requiring substantial post-processing.Source: B. Reinforcement Learning for Humanoids · p. 2
  • Full-order-dynamics trajectory optimization specifies both lower-body leg motions and upper-body contact sequences, enabling references beyond periodic motions for humanoid loco-manipulation.The authors report hardware success for policies learned from full-order-dynamics optimization and identify joint torques as critical for high-dimensional contact-rich skills.Source: B. Reinforcement Learning for Humanoids · p. 2
  • Compared with the related quadruped study, Opt2Skill targets a 30-DOF humanoid and evaluates seven diverse tasks with simulation and hardware experiments.The related work uses a simplified rigid-body model, whereas Opt2Skill uses full-order dynamics.Source: B. Reinforcement Learning for Humanoids · p. 2
  • The framework’s references contain joint states, body motion, torques, and interaction forces, which supervise RL policies whose joint targets are tracked by a low-level PD controller.The framework is described as remaining reactive to disturbances, sensor noise, and dynamics variability.Source: Figure 2 · p. 3

III. METHODS

  • Opt2Skill uses DDP to generate dynamically feasible, task-specific motions that encode contact-rich dynamics and torque limits, then trains RL policies to predict joint-level targets.The policies track diverse offline references and are intended to transfer to real hardware.Source: §III · p. 3

A. Whole-Body Trajectory Optimization

Opt2Skill builds whole-body trajectories by optimizing a floating-base humanoid model under task and contact constraints. DDP in Crocoddyl produces task-specific references with states, torques, and end-effector motions for downstream policy training.

A. Whole-Body Trajectory Optimization

  • The trajectory optimization uses full-body states and joint torques, with hybrid dynamics, contact, joint-limit, torque-limit, and friction-cone constraints.The model has an unactuated 6-DoF base and fully actuated limbs; contact constraints distinguish stance or impact feet from non-contact feet.Source: A. Whole-Body Trajectory Optimization · p. 3
  • DDP in Crocoddyl solves the whole-body optimization using timestep-level end-effector targets for precise foot tracking or sparse subgoals for more flexible tasks.Task-specific costs and contact schedules produce feasible motions; optimized references include joint and base states, torques, and end-effector motions.Source: A. Whole-Body Trajectory Optimization · pp. 3–4
  • The framework varies gait phase, contact mode, speed, foot clearance, center-of-mass position, and target location to generate motion datasets across walking, stair climbing, pickup, drawing, and reaching.These optimized references include joint torques and interaction forces often absent from kinematic demonstrations.Source: A. Whole-Body Trajectory Optimization · p. 4

B. RL-Based Imitation of Dynamically Feasible Trajectories

Opt2Skill uses trajectory-optimization references, including torque and contact-force information, to train RL policies for whole-body loco-manipulation. The policies track references while adapting to disturbances during execution.

B. RL-Based Imitation of Dynamically Feasible Trajectories

  • TO references satisfy full-body dynamics, contact constraints, and torque limits, and provide joint torques and contact forces for policy observations and rewards.The trajectory information is used to learn robust, transferable whole-body behaviors.Source: B. RL-Based Imitation of Dynamically… · p. 4
  • Each task uses its own TO trajectory dataset; deployment supplies partial references from a new offline-generated trajectory without reusing training data or running TO online.A trajectory is randomly sampled at each training episode's start to initialize the environment and compute tracking rewards.Source: B. RL-Based Imitation of Dynamically… · p. 4
  • RL policies track dynamically feasible nominal trajectories and adapt to real-world disturbances, addressing TO's modeling inaccuracies, contact uncertainties, and real-time computational cost.The passage contrasts offline trajectory generation with direct real-time use of DDP for control.Source: B. RL-Based Imitation of Dynamically… · p. 4

1) Problem Formulation:

The policy is trained as an RL controller to track TO-generated references, with separate actor and critic observations and joint-target actions sent to a PD controller. Training uses PPO in simulation and domain randomization to support deployment with noisy observations.

1) Problem Formulation:

  • The problem is formulated as an MDP for learning a policy that maximizes expected discounted reward, while real deployments are partially observable because of sensor noise and unobserved environmental factors.The reward encourages tracking trajectories generated by trajectory optimization.Source: 1) Problem Formulation · p. 4
  • Table I catalogs reward components and weights; the walking task excludes the “Base Pos.” reward.Tracking rewards encourage following TO-generated references.Source: Table I · p. 4, 1) Problem Formulation · p. 4
  • Table II catalogs domain-randomization parameters used in the training setup.Source: Table II · p. 4
  • PPO trains an actor-critic MLP policy in MuJoCo Playground for fewer than 300 million steps and under seven hours on one RTX4090 GPU.The networks use hidden sizes [512, 512, 256, 256].Source: 1) Problem Formulation · p. 4

2) Imitation Policy:

  • The critic has privileged state information in simulation, while the actor receives noisy proprioceptive inputs available at deployment.The actor also receives partial reference information to avoid reliance on noisy or redundant global-base-state and end-effector-pose inputs.Source: 2) Imitation Policy · pp. 4–5
  • The critic observation includes privileged proprioception, including body translation, projected gravity, joint and action histories, end-effector position, contact force, torque, and PD gains.Joint-position and action histories use ten samples at four-timestep intervals.Source: 2) Imitation Policy · pp. 4–5
  • The actor observation combines noisy proprioception with partial reference information, including reference state and torque information.The notation uses hats to indicate reference-trajectory information.Source: 2) Imitation Policy · p. 5
  • The policy outputs offsets from a default standing pose for 20 actuated joints, which are converted into torque commands by a PD joint controller.The policy runs at 200 Hz; the PD controller runs at 1 kHz in simulation and 2 kHz on hardware, using the same gains in both settings.Source: 2) Imitation Policy · p. 5

3) Reward Functions, Domain Randomization, and Curriculum:

The reward design combines task tracking and regularization, while domain randomization and a noise curriculum are used to support robust learning and sim-to-real transfer.

3) Reward Functions, Domain Randomization, and Curriculum:

  • Task rewards guide tracking of commands such as target velocity, end-effector positions, pickup targets, and contact events; structured foot trajectories make heuristic reward shaping unnecessary.The TO solver parameterizes foot trajectories with controllable step height and timing, supporting end-effector tracking.Source: 3) Reward Functions, Domain… · p. 5
  • Joint-torque and contact-force tracking rewards improve stable contact and adaptation to varying contact conditions, while regularization penalizes large torques and abrupt actions.Regularization promotes smooth, physically plausible, energy-efficient motions and supports robustness and sim-to-real transfer.Source: 3) Reward Functions, Domain… · p. 5
  • Training randomizes robot dynamics, observation noise, control delays, and external disturbances, and gradually increases observation noise and penalty strength through a curriculum.The curriculum supports more stable early learning and better later generalization.Source: 3) Reward Functions, Domain… · p. 5

IV. RESULTS

The evaluation uses Digit and compares trajectory optimization against human-retargeted and IK-based motion sources.

A. Robot System

  • Digit is a 30-DoF bipedal humanoid, evaluated in a high-fidelity physics simulator before hardware deployment.It has 20 actuated joints; the simulator shares many features with the hardware.Source: A. Robot System · p. 5

B. Comparison with Different Datasets

1) Dataset Generation:

Opt2Skill achieves the best tracking accuracy across most reported metrics, while the evaluation also examines terrain robustness and observation-history length.

1) Dataset Generation:

  • The human-retargeted dataset has about 70 trajectories, while IK and TO datasets each contain 2,000 flat-ground trajectories with matched speed and clearance ranges.The TO trajectories use DDP on the same target keypoints used to generate IK trajectories.Source: 1) Dataset Generation · p. 5

2) Evaluation Metrics:

  • Tracking evaluation includes per-frame joint, velocity, and end-effector errors, plus accumulated position, lateral, and yaw drift.Source: 2) Evaluation Metrics · p. 5
  • Terrain evaluation compares policy success across stair heights and slope inclinations; the figure places stairs on the left and slopes on the right.The supplied passages describe the evaluation scenarios but do not report their outcomes in this section's evidence.Source: 2) Evaluation Metrics · pp. 5–6, Figure 3 · p. 6

3) Tracking Performance:

  • 2.00 cm hand error and 5.23 cm foot error; Opt2Skill also has the lowest reported global drift across the compared policies.The IK policy has the lowest joint-angle error (0.02 rad), but higher task-space errors; end-effector differences versus IK and human-retargeted policies are significant (p < 0.001).Source: 3) Tracking Performance · p. 6, Table III · p. 6
  • A 10-step observation history achieves the best overall tracking accuracy; overly long histories can deteriorate performance.Source: 3) Tracking Performance · p. 6

4) Task Success on Rough Terrain:

Opt2Skill policies show stronger rough-terrain success than human-retargeted and IK-based baselines, and the paper describes an ablation design for contact-force and torque information.

4) Task Success on Rough Terrain:

  • Opt2Skill maintains high stair-climbing success up to 12 cm steps, while the other policies drop sharply beyond 8 cm, and it is more robust on steeper slopes.Trials used varying initial yaw angles and similar trajectory parameters; no task-specific test-time tuning was used.Source: 4) Task Success on Rough Terrain · p. 6

C. Evaluating the Role of Torque Information in Contact-Rich Loco-Manipulation

  • The wiping-task ablation compares position-only tracking with policies adding reference force, reference torque, or both to observations and rewards.Training uses 1,400 TO-generated trajectories with varied desk heights and contact normal forces.Source: C. Evaluating the Role of Torque… · p. 6

1) Performance Analysis:

Torque and force references improve contact-task tracking, and hardware experiments demonstrate a range of loco-manipulation behaviors with tracking and environmental variation.

1) Performance Analysis:

  • The position-only policy tracks position best but produces similar force patterns across reference levels, sometimes insufficient or excessive for the target.Source: 1) Performance Analysis · p. 7
  • Pos+T improves contact-force tracking (2.7 N average error) and position tracking (2.9 cm average error) over Pos+F.Reference torque provides dynamically grounded information about when and how to apply force; the overall pipeline requires an offline TO stage before policy training.Source: 1) Performance Analysis · p. 7
  • Pos+F+T achieves the best overall force tracking and the most stable force application, with position tracking competitive with Pos.It adapts to table heights of 0.9 m and 1.0 m despite a reference defined for 0.95 m; differences from other baselines are significant (p < 0.001).Source: 1) Performance Analysis · p. 7, Table IV · p. 7, Figure 4 · p. 7

D. Hardware Experiments

  • On hardware, desk-object pushing yields average hand tracking errors of 3.7 cm (left) and 3.8 cm (right), while shelf box pick-up yields 3.6 cm for each hand.At box contact, a y-axis deviation is attributed to an approximately 2 cm y-direction size mismatch between reference and real-world boxes for each hand.Source: D. Hardware Experiments · p. 8
  • Hardware demonstrations include locomotion, multi-contact reaching, box pickup, pickup-and-walk, door opening, and drawing and wiping; policies use over 1,000 randomized reference trajectories per task.The robot also walks up stairs and a ramp using a reference designed for flat ground.Source: D. Hardware Experiments · pp. 7–8

V. CONCLUSION

Opt2Skill uses trajectory optimization to guide RL policies for humanoid loco-manipulation. The conclusion reports better tracking with high-quality, dynamically feasible references, especially when torque information is used, and sim-to-real results on Digit.

V. CONCLUSION

  • The quality of motion references affects RL tracking, and full-body-dynamics trajectory optimization provides high-quality, dynamically feasible trajectories.Motion imitation based on these trajectories yields better tracking, especially when torque information is included.Source: §V · p. 8
  • Opt2Skill demonstrates sim-to-real results on Digit for dynamic stair traversing, multi-contact box manipulation, and door traversing.Source: §V · p. 8

Related papers

Loading 2409.20514v6…