PaperlayerView exact revision on arXiv

Source-linked AI summary

WoCoCo: Learning Whole-Body Humanoid Control with Sequential Contacts

Chong Zhang, Wenli Xiao, Tairan He, Guanya Shi

arXiv:2406.06005v2cs.ROcs.GRcs.SYeess.SY

TL;DR

Sequential-contact tasks challenge motion-planning methods that can be time-consuming and rely on simplified dynamics, while RL work often targets standard locomotion or scenario-specific footholds. WoCoCo decomposes control into contact stages and demonstrates end-to-end RL policies on real-world humanoid tasks and a dinosaur robot task.

  • Problem

    Sequential-contact tasks are commonly addressed with time-consuming model-based planning that relies on simplified dynamics, while RL work has focused on standard locomotion or scenario-specific constrained footholds.

    Source: §1 · p. 1

  • Method

    WoCoCo reformulates whole-body control as sequential fulfillment of contact stages and combines dense contact, stage-count, and task-agnostic curiosity rewards.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    WoCoCo-trained end-to-end RL policies were validated in the real world on four humanoid tasks and applied to a 22-DoF dinosaur robot task.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The demonstrations support WoCoCo’s stated versatility across humanoid and non-humanoid robot tasks.

    Source: §1 · p. 3

    Switching contact stages requires explicit contact feedback from sensors or human observers.

    Source: §6 · p. 8

Abstract

from arXiv · show

Humanoid activities involving sequential contacts are crucial for complex robotic interactions and operations in the real world and are traditionally solved by model-based motion planning, which is time-consuming and often relies on simplified dynamics models. Although model-free reinforcement learning (RL) has become a powerful tool for versatile and robust whole-body humanoid control, it still requires tedious task-specific tuning and state machine design and suffers from long-horizon exploration issues in tasks involving contact sequences. In this work, we propose WoCoCo (Whole-Body Control with Sequential Contacts), a unified framework to learn whole-body humanoid control with sequential contacts by naturally decomposing the tasks into separate contact stages. Such decomposition facilitates simple and general policy learning pipelines through task-agnostic reward and sim-to-real designs, requiring only one or two task-related terms to be specified for each task. We demonstrated that end-to-end RL-based controllers trained with WoCoCo enable four challenging whole-body humanoid tasks involving diverse contact sequences in the real world without any motion priors: 1) versatile parkour jumping, 2) box loco-manipulation, 3) dynamic clap-and-tap dancing, and 4) cliffside climbing. We further show that WoCoCo is a general framework beyond humanoid by applying it in 22-DoF dinosaur robot loco-manipulation tasks.

1 Introduction

WoCoCo addresses the lack of a systematic RL method for real-world humanoid control with desired contact sequences, where task-specific tuning and long-horizon exploration remain challenges. It decomposes tasks into contact stages and combines task-agnostic rewards with a sim-to-real pipeline; the authors report real-world demonstrations across four humanoid tasks and a 22-DoF dinosaur robot task.

1 Introduction

  • Existing RL methods for task-aware contact sequences use policies tuned to particular tasks or state-machine transitions, leaving a systematic general method for real-world humanoids missing.Prior approaches also include a shared hierarchical architecture without sim-to-real treatment and contact tracking that requires human motion references.Source: §1 · p. 2
  • Sequential-contact tasks challenge RL through sparse contacts, risks that deter long-horizon exploration, and sim-to-real modules that can hinder exploration.The paper identifies domain randomization and task-irrelevant regularization rewards among the sim-to-real approaches that can hinder exploration.Source: §1 · p. 2
  • WoCoCo reformulates control as sequential fulfillment of contact stages, breaking the exploration burden into separate stages.The framework frames its design questions around reaching desired contact states, exploring across stages, and creating a compatible sim-to-real pipeline.Source: §1 · p. 2
  • Its reward combines dense contact, stage-count, and task-agnostic curiosity terms to guide contact achievement and exploration across stages.Dense rewards count correct and incorrect contacts; stage-count rewards encourage progression, while curiosity further facilitates exploration.Source: §1 · p. 2
  • A three-stage training curriculum introduces domain randomization and then increases regularization weights to reduce the exploration burden from sim-to-real modules.Training begins without domain randomization, resumes with it, and finally increases regularization weights.Source: §1 · p. 3
  • WoCoCo policies were validated in the real world on four humanoid tasks, and the framework was also applied to a 22-DoF dinosaur robot task.The authors describe these as the first instances of each humanoid task being solved by a single end-to-end RL policy.Source: §1 · p. 3

2 Overview: Learning with Sequential Contacts and Task Decomposition

WoCoCo represents active-contact tasks as sequences of contact stages based on desired contact sequences. Each stage combines a contact goal with a task goal, and the framework formulates policy learning around those goals.

2 Overview: Learning with Sequential Contacts and Task Decomposition

  • Active-contact tasks are decomposed into sequential contact stages based on the desired contact sequences.The paper gives parkour jumping and loco-manipulation as examples of tasks represented this way.Source: §2 · p. 3
  • A stage is fulfilled by simultaneously meeting its contact goal and task goal.Contact goals specify end-effector contact states, while task goals capture additional task-specific requirements.Source: §2 · p. 3
  • The studied tasks use predefined contact stages, which can be advanced after fulfillment is checked by sensors or human observation.For parkour, a stepping stone is a stage: correct foot contacts form the contact goal, and maintaining upper-body posture forms the task goal.Source: §2 · p. 3
  • Policy learning is formulated as an extended Markov decision process with contact and task goals included in the learning problem.The policy uses observations that may include proprioception, optional exteroception, and goal-related information, and outputs joint targets for low-level PD controllers.Source: §2 · p. 3

3 WoCoCo Rewards and Sim-to-Real Transfer

WoCoCo combines task-agnostic rewards with a sim-to-real training curriculum. Its reward design uses contact, stage-count, and curiosity terms, while the curriculum introduces domain randomization and progressively stronger regularization.

3.1 WoCoCo Rewards

  • The reward design combines contact, stage-count, and curiosity terms to support contact reaching and exploration across stages and the state space.The paper reports that its count-based curiosity rewards facilitate exploration with task-agnostic observations and constrain random-network overfitting through numerical decay as bucket visits increase.Source: §3.1 · p. 4
  • Contact rewards provide denser feedback than 0-1 rewards by rewarding correct end-effector contacts and penalizing incorrect ones.The incorrect-contact penalty is masked at the first stage to encourage exploration and avoid penalties immediately after reset.Source: §3.1 · p. 4
  • Stage-count rewards encourage progress to new contact stages when staying at the current stage can yield positive rewards.The task-goal condition prevents intentional task-goal failure from accumulating stage-count rewards.Source: §3.1 · p. 4
  • Task-agnostic count-based curiosity uses random neural-network hashing to assign observations to buckets and bases reward on bucket visit counts.The authors contrast this with RND over-exploration of states that do not generate meaningful behaviors and report stable exploration across challenging whole-body tasks.Source: §3.1 · p. 4

3.2 Sim-to-Real Transfer

  • The sim-to-real curriculum trains first without domain randomization, then with it, and finally increases regularization weights by 20% per 2000 iterations until they double.Curiosity rewards are activated only during the first stage; previous joint states and actions are stacked to enhance robustness through temporal memory.Source: §3.2 · p. 5

4 Case Studies

The case studies apply WoCoCo’s shared reward and regularization terms across tasks, with only one or two intuitive task-specific rewards. They cover five settings, including four humanoid tasks and dinosaur loco-manipulation.

4 Case Studies

  • Across the case studies, the same framework rewards and regularization terms were used, with one or two intuitive task-specific rewards adjusted for each task.The case studies span tasks with different contact sequences.Source: §4 · p. 5

4.1 Case I: Versatile Parkour Jumping

  • In simulation, jumping handled continuous varied contacts and postures despite unseen gravel; real-world tests covered double-foot jumps at different heights and hug postures.Real-world trials were limited by facility constraints to double-contact sequences with one or two continuous jumps.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 Case II: Anywhere-to-Anywhere Box Loco-Manipulation

  • The humanoid turned, transitioned between walking and picking, approached the destination while carrying the box, and recovered after stepping on a tethered belt.The end-to-end policy approached and transported the box from its current to goal position without posture priors.Source: §4.2 · p. 6

4.3 Case III: Dynamic Clap-and-Tap Dancing

  • WoCoCo learned a dancing policy deployed in the real world, with accurate tapping and optional clapping as task capabilities.The task assigns contact stages to feet and hands.Source: §4.3 · pp. 6–7

4.4 Case IV: Bidirectional Cliffside Climbing

  • Cliffside climbing was robust to pushes and unseen gravel in simulation; in the real world, the humanoid adapted to varying hand contact forces.Hands touched wall goal regions while feet stood on ground goal regions.Source: §4.4 · p. 7

4.5 Beyond Humanoid: Dinosaur Loco-Manipulation

  • A 22-DoF dinosaur robot pushed a ball toward destinations using its head, tail, or four feet, producing trajectories that spell “WoCoCo.”The task adapts box loco-manipulation by defining the desired contact point on the ball toward the destination.Source: §4.5 · pp. 7–8, Figure 6 · p. 8

5 Analyses and Ablations

The jumping-task analyses show that WoCoCo’s rewards support exploration across contact stages, while learned behaviors remain agile, adaptable, and robust to perturbations. Ablations expose distinct failure patterns when dense contact, stage-count, or curiosity rewards are removed or replaced.

5 Analyses and Ablations

  • The authors base analyses and ablations on parkour jumping because it is arguably the most challenging task and training every task in the same setting is costly.Source: §5 · p. 8

Ablating Dense Contact Rewards

  • With binary 0-1 contact rewards, the humanoid fails to explore jumping over stones and instead tracks upper-body postures without moving.The authors contrast this behavior with their dense contact rewards.Source: §5 · p. 8

Ablating Stage Count Rewards

  • Without stage-count rewards, the humanoid avoids fulfilling the contact goal to preserve other rewards rather than advance to later stages.This behavior illustrates the role of stage-count rewards in encouraging progress through the sequence.Source: §5 · p. 8

Ablating Curiosity Rewards

  • Without curiosity rewards, the humanoid under-explores; RND curiosity produces risky backward leaning, while WoCoCo curiosity explores effectively without behavior-specific overfitting.The RND behavior matches the reported risk of over-exploring rare states while remaining alive [36].Source: §5 · p. 8

Empirical Benefits of WoCoCo

  • Across diverse task configurations, WoCoCo policies achieve agile, efficient motions without simplified models or motion priors and withstand unseen gravel perturbations.The policies fulfill versatile contact goals.Source: §5 · p. 8

Training Stability

  • Learning remained stable across five random seeds despite stochastic curiosity-driven exploration.The learning curves plot average curiosity values and task progress, defined as fulfilled contact stages divided by the maximum.Source: §5 · p. 8

6 Limitation and Future Works

The authors identify uncertainty about controller failure and reliance on motion capture, predefined contact sequences, and explicit feedback to switch stages as limitations or current constraints.

6 Limitation and Future Works

  • The framework does not indicate when its controller will fail, unlike model-based methods that can explicitly assess whether a feasible solution exists.The authors suggest exploring failure predictors and other safety-assessment methods.Source: §6 · p. 8
  • The prototype relies on motion capture, heuristic predefinition of contact sequences, and explicit sensor or human feedback to switch contact stages.Onboard sensing and high-level planners are proposed as future work; implicit stage switching by the policy is also a possibility.Source: §6 · p. 8

A An Illustrative Example of Symbols in Contact Rewards

The example applies the contact-reward calculation to a parkour-jumping timestep, distinguishing the task goal from the current contact goal and illustrating the symbols used.

A An Illustrative Example of Symbols in Contact Rewards

B Ablation Baseline Behaviors

The baseline-behavior figure contrasts outcomes from reward ablations, while corresponding learning curves show average curiosity and normalized task progress alongside WoCoCo’s convergent progress.

B Ablation Baseline Behaviors

  • Baseline behaviors differ by ablation: binary contact rewards and no curiosity fail exploration, no stage-count reward leads to contact-goal avoidance, and RND curiosity produces risky backward leaning.Source: Figure 8 · p. 13
  • The corresponding curves plot average curiosity and task progress, defined as fulfilled contact stages divided by the maximum; WoCoCo’s convergent average progress is marked in red as successful learning.Source: Figure 9 · p. 13, B Ablation Baseline Behaviors · p. 13

C Training Stability

The section reports parkour-jumping training curves and describes symmetry augmentation and the reward-weight reference. It specifies how curiosity rewards are symmetrized, but the supplied passages do not report curve outcomes or reward-weight values.

C Training Stability

D Symmetry Augmentation

E WoCoCo Reward Weights

  • Supplementary Table 1 lists WoCoCo reward weights, but the supplied caption gives no specific values.Source: Table 1 · p. 14

F Regularization Rewards

The regularization-reward design includes style shaping for tasks that do not require jumping. A supplementary table is identified as listing the regularization rewards.

F Regularization Rewards

  • When jumping is not desired, two style-shaping terms—feet air time and no fly—are used.Source: F Regularization Rewards · p. 14
  • Supplementary Table 2 is identified as listing regularization rewards, but the supplied caption gives no further details.Source: Table 2 · p. 15

G Task-Related Rewards

The appendix specifies task-related reward terms, policy observations, and implementation choices for the four humanoid tasks. Its curriculum ablations report learning and real-world behavior issues when domain randomization starts earlier or regularization is omitted.

G.1 Versatile Parkour Jumping

  • Parkour jumping uses one task-related reward based on elbow rotation and position errors relative to desired orientations and positions in the base frame.Source: §G.1 · p. 15

G.2 Anywhere-to-Anywhere Box Loco-Manipulation

  • Box loco-manipulation uses two reward terms to encourage approaching and moving the box toward its destination; dinosaur ball loco-manipulation omits direction-angle conditions.Source: §G.2 · p. 15

G.3 Dynamic Clap-and-Tap Dancing

  • Clap-and-tap dancing uses two terms to encourage arm spread and precise footholds.Source: §G.3 · p. 15

G.4 Bidirectional Cliffside Climbing

  • Cliffside climbing uses two terms to encourage facing the wall while maintaining precise end-effector placement.Source: §G.4 · p. 16

H Policy Observations and Architechtures

I.1 Curiosity Observations

  • Curiosity-observation preprocessing options showed no significant outcome differences; reported results use the sine-and-cosine normalization option.Source: §I.1 · p. 17

K Significance of Sim-to-Real Curriculum

  • Moving domain randomization into phase 1 hindered parkour learning, with average progress below 0.6 versus above 0.75 for standard WoCoCo; omitting phase 3 produced aggressive behaviors and jerky cliffside climbing.The figure description attributes stepping beyond the cliff edge to jerky motions despite contact goals remaining within the boundary.Source: §K.1 · p. 17, §K.2 · p. 17, Figure 11 · p. 18

L Control and Real-to-Sim

  • The policy runs at 50 Hz and the PD controller at 200 Hz, with a 4-Hz low-pass filter on PD targets; the robot’s torso mass and foot geometry were measured because its URDF values were biased or wrong.Source: §L.1 · p. 18, §L.2 · p. 18
Loading 2406.06005v2…