Source-linked AI summary
Co${}^{2}$Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction
Jeonghwan Kim, Hyeonwoo Kim, Hanbyul Joo
TL;DR
Coordinating whole-body scene interaction and dexterous object manipulation across long tasks remains a challenge for controllers that address them separately. Co2Skill combines both in a goal-conditioned policy with curricula for precision and cross-task execution. It outperforms MaskedMimic across five scene tasks and matches or exceeds specialists on four of five, while reliable interaction completion remains a challenge.
Problem
Physics-based controllers largely address scene interaction and dexterous object manipulation independently, despite everyday activities requiring their coordination across changing task states.
Source: §1 · p. 1Method
Co2Skill uses one goal-conditioned policy with task- and phase-dependent observations, partial reference guidance, varied-start exploration, and cross-task training.
Source: §1 · pp. 2–3Results
Across five scene tasks, Co2Skill outperforms MaskedMimic; Stair SR is 69.23% versus 7.81%, and it matches or exceeds specialists on four of five tasks.
Source: §1 · p. 2, §5.3 · p. 7Takeaways & Limitations
The experiments demonstrate scene interaction and object manipulation under varied initial conditions, with qualitative results showing coordinated motion across task transitions.
Source: §6 · p. 9Reliable interaction completion remains a challenge, although placement precision among successful trials is comparable across evaluation splits.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Achieving human-level dexterity in complex, unstructured environments requires the seamless integration of whole-body scene interaction and dexterous object manipulation skills. While existing physics-based controllers generate physically plausible behaviors in each domain, they largely address these two capabilities independently. In this paper, we present Co${}^{2}$Skill that integrates scene interaction and dexterous manipulation through a unified policy formulation. Built on a pretrained motion prior, the policy uses task and phase dependent observation masks to select information relevant to the current interaction goals. We introduce a goal-conditioned loco-manipulation curriculum that combines partial reference guidance for precision with exploration from varied initial states while allowing goal-directed execution beyond the demonstrated trajectories. We further introduce a cross-task curriculum that jointly trains individual skills and selected task sequences, preserving physical states across task boundaries and maintaining grasps during subsequent scene interactions. Together, these support sequential task execution and simultaneous scene interaction with object manipulation. We evaluate sitting, standing, climbing, stair traversal, and goal-directed manipulation, together with sequential execution and with random different conditions. Additionally, we demonstrate skill compositions in indoor environments, illustrating their integration within the same control formulation.
1 INTRODUCTION
Long-horizon human-environment interaction requires coordinating locomotion, scene contact, and grasping while remaining stable across changing task states. Co2Skill addresses this with one policy and curricula for precise interactions and cross-task execution.
1 INTRODUCTION
- Everyday tasks require coordinating locomotion, whole-body scene interaction, and finger-level grasping across room-scale spaces and changing physical states.Each skill must begin stably from the state left by the preceding skill.Source: §1 · p. 1
- Scene interaction and dexterous manipulation use different inputs, while fixed training starts and reference motions limit transfer to new states and positions.Separate specialists also do not support simultaneous scene tasks and grasping.Source: §1 · pp. 1–2
- Co2Skill combines scene interaction and dexterous manipulation in one goal-conditioned policy built on a pretrained motion prior.Task and phase-dependent observation selection provides information relevant to current interaction goals.Source: §1 · p. 2
- The loco-manipulation curriculum combines partial reference guidance for precise interactions with varied-start exploration beyond reference trajectories.The cross-task curriculum trains individual tasks and selected sequences, preserving physical states across transitions and maintaining grasps during later scene interaction.Source: §1 · pp. 2–3
2 RELATED WORK
Related work develops datasets and physics-based controllers for scene interaction, object manipulation, and reusable motion skills. Co2Skill builds on these directions to integrate locomotion and whole-body manipulation through staged training.
2.1 HUMAN-SCENE INTERACTIONS
- Human-scene interaction datasets progress from static environments to partially dynamic scenes and synthesized locomotion-manipulation behaviors.Later work also diversifies indoor environments through captured behavior and VR rendering.Source: §2.1 · p. 3
2.2 HUMAN-OBJECT INTERACTIONS
- Human-object datasets span dynamic object interaction, finger-only grasping, and articulated whole-body manipulation, though some have limited action diversity.ParaHome captures indoor sequences combining locomotion and finger-level manipulation.Source: §2.2 · p. 3
2.3 PHYSICS-BASED CHARACTER CONTROL
- Physics-based character control includes reference tracking, reusable latent skills, dynamic grasp synthesis, and scene-level contact handling.Recent scene-interaction work also explores multi-task composition through task tokenization.Source: §2.3 · p. 3
2.4 BEHAVIOR FOUNDATION MODELS
- Behavior foundation models and task-specific tokens provide reusable motion representations, motivating staged integration of locomotion and whole-body manipulation datasets.The paper aims to train a general motion prior and then steer it toward varied tasks.Source: §2.4 · p. 3
3 PRELIMINARIES
The controller targets joint-level actions that coordinate locomotion, scene contact, and object interaction across selected skills without requiring reference data for an entire sequence. Its formulation is goal-conditioned reinforcement learning with a simulated humanoid action space.
3 PRELIMINARIES
- The goal is to produce joint-level actions that coordinate locomotion, scene contact, and hand-object interaction to reach goals and move between selected skills.The approach does not require a large-scale reference dataset covering the entire demonstration sequence.Source: §3 · p. 4
3.1 FORMULATION
- Character control is formulated as goal-conditioned reinforcement learning, where task conditions specify active interactions and targets and may change as the state progresses.The policy selects actions from the state and task condition to maximize expected discounted return over an episode.Source: §3.1 · p. 4
- The SMPL-X humanoid has 52 joints and a 153-dimensional joint-level action space, with joint position targets tracked by PD controllers.The root's global translation and rotation evolve through simulated dynamics.Source: §3.1 · p. 4
4 METHOD
Co²Skill combines scene interaction and dexterous manipulation in a shared policy built on a pretrained motion prior. Its task-conditioned curricula extend goal-directed behavior across varied starts and task transitions while preserving physical state and grasps.
4 METHOD
- The method adapts a shared task encoder on a fixed pretrained whole-body prior and decoder, predicting latent residuals that become joint-level action targets.The overview describes learning the prior and decoder first, then adapting the task encoder.Source: §4 · p. 4, Figure 2 · p. 5
4.1 UNIFIED INTERACTION TASK REPRESENTATION
- Shared, scene, manipulation, and per-hand tokens encode character state, goals, object geometry, and grasp targets; task masks activate relevant tokens while shared tokens stay active.Sitting while holding an object activates scene-interaction tokens together with object and grasp tokens.Source: §4.1 · pp. 4–5
4.2 GOAL-CONDITIONED LOCO-MANIPULATION CURRICULUM
- The loco-manipulation curriculum pairs reference guidance for precise contact with goal-driven exploration from varied starts, extending execution beyond demonstrated approaches.Reference guidance is scoped according to task, while exploration samples different starting positions and directions around furniture.Source: §4.2 · p. 5
- Sit and climb-up jointly train reference-initialized tracking and exploration from varied distances and directions; standing, box descent, and stair traversal are Reach tasks with waypoints as needed.Source: §4.2 · p. 5
- Pick-move-place uses an end-target object pose plus grasp and release conditions, with object-relative wrist targets expressed relative to the character to preserve alignment across placements.The object goal alone does not specify grasp choice or release timing.Source: §4.2 · pp. 5–6
- Reference motion guides initial body coordination and close-range grasp formation; tracking ends at reference lift time, actual lift completion starts transport, and a dwell separates placement from release.Source: §4.2 · p. 6
4.3 CROSS-TASK CURRICULUM
- Selected training sequences follow grasp-and-lift with a scene task, activating its goal without resetting the character or object so the next task inherits the lifting state.Without a reference spanning the sequence, the policy explores continuation under task rewards.Source: §4.3 · p. 6
- During the subsequent scene task, object and grasp tokens remain active with grasp-retention conditions, while placement and release stay inactive.The carried-object target is updated with the character’s motion.Source: §4.3 · p. 6
5 EXPERIMENTS
The experiments evaluate scene interaction, reference-conditioned lifting, goal-directed pick-move-place tasks, and composed behavior. The shared policy performs strongly across scene tasks, while results also show limits in transfer to unseen objects.
5.1 SETUPS
- The experiments use AMASS and GRAB data, including synthetic interaction sequences; the pretrained motion prior and decoder stay frozen while a four-layer transformer task encoder is trained.The simulation runs policies at 30 Hz and physics at 120 Hz.Source: §5.1 · pp. 6–7
5.2 METRICS
- The evaluation reports scene-task success rate and target-position error, lifting success rate, MPJPE and tracking length, and pick-move-place phase success and release error.Scene-task success criteria vary by task; stair success requires completing both waypoint stages.Source: §5.2 · p. 7
5.3 QUANTITATIVE RESULTS
- The shared policy reaches 69.23% Stair SR versus 7.81% for MaskedMimic and improves Stair SR over specialists from 54.41% to 69.23%.It also achieves 100.00% SR on Climb-up, Stand-up, and Climb-down, and 93.22% on Sit; compared with specialists, Sit SR decreases from 99.48% to 93.22%.Source: §5.3 · pp. 7–8, Table 1 · p. 8
- Test-set tracking SR rises from 56.67% to 90.00% with synthetic-data augmentation, while MPJPE falls from 3.29 to 3.11 cm and tracking length grows from 1.78 to 1.99 s.The authors suggest added sequence diversity reduces overfitting and improves generalization to unseen motions with known object identities.Source: §5.3 · p. 8
- The pick comparison with the reference-conditioned tracker is contextual, not direct, because the methods differ in conditioning and success criteria.The tracker measures reference completion, whereas picking requires maintained contact and proximity to the lift target.Source: §5.3 · p. 8
- The policy completes the full pick-move-place sequence in 63.39% of training episodes and 64.80% of test episodes; unseen-object pick, move, and release SRs fall to 47.90%, 40.25%, and 25.19%.Among successful release episodes, placement errors remain between 2.34 and 2.87 cm across splits.Source: §5.3 · p. 8
5.4 QUALITATIVE RESULTS
- Qualitative rollouts compose navigation, object transport, and body-scene interaction, with transitions that maintain balance and coordinated full-body motion.The figure illustrates task composition in an environment with static structures and manipulable objects.Source: §5.4 · pp. 8–9, Figure 3 · p. 9
6 CONCLUSION
Co2Skill unifies scene interaction and dexterous manipulation through a shared policy, using task- and phase-specific observations, partial reference guidance, task-driven exploration, and cross-task training. Experiments demonstrate varied-condition interactions and coordinated transitions, while reliable completion remains a challenge.
6 CONCLUSION
- A task-conditioned encoder predicts residual actions in a frozen motion-prior latent space, while task- and phase-specific masks select observations for each interaction.Partial reference guidance and task-driven exploration support individual interactions; cross-task training supports manipulation-to-scene transitions with grasp pose maintained.Source: §6 · p. 9
- Experiments demonstrate scene interaction and object manipulation under varied initial conditions, coordinated full-body transitions, placement precision comparable across evaluation splits, and generalization to unseen objects, while reliable completion remains challenging.The conclusion also identifies expanding trained transitions and improving robustness to unseen objects as future directions.Source: §6 · p. 9
- Future work could add visual and language-based task conditioning, broaden trained transitions, improve robustness to unseen objects, and investigate adaptation to real-world humanoids.Source: §6 · p. 9