PaperlayerView exact revision on arXiv

Source-linked AI summary

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions

Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, Mengdi Xu

arXiv:2609.16683v1cs.ROcs.AIcs.LG

TL;DR

Learning humanoid-object interactions requires coordinating locomotion, balance, and dexterous contact, while transferring human demonstrations across embodiments remains challenging. WEAVE constructs contact-preserving robot-object references and trains a unified contact- and geometry-aware policy; it reports 92.5% success on trained interactions and 65.0% on unseen sequences.

  • Problem

    Humanoid-object interaction requires coordinated balance, locomotion, object motion, and finger-level contact, while prior approaches leave contact unmodeled or target different embodiments.

    Source: §I · p. 1

  • Method

    WEAVE completes and contact-aware retargets captured human-object interactions, then trains a unified policy to track robot and object motion across objects and clips.

    Source: §I · p. 2, A. Overview · p. 3

  • Results

    92.5% success rate on Dtrain and 65.0% SR on Dtest, with no additional training for the unseen interactions.

    Source: B. Whole-Body Dexterous… · p. 7, §VI · p. 8

  • Takeaways & Limitations

    WEAVE produces approximately 23 hours of physically executed rollouts for downstream interaction-policy learning and humanoid-object interaction modeling.

    Source: §VI · p. 8

    Evaluation is entirely in simulation using ground-truth odometry and object pose, and the sim-to-real gap for contact-rich interaction is not assessed.

    Source: §VI · p. 8

Abstract

from arXiv · show

Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations. Weave first converts captured human-object interactions into executable robot-object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 29 body joints and 12 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~9,000 physically executed rollouts spanning ~23 hours, providing robot-object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Project website: https://xiaohu-art.github.io/Weave/

I. INTRODUCTION

WEAVE addresses the challenge of learning humanoid-object interaction that couples balance, locomotion, and dexterous contact. It builds contact-preserving robot-object references from human demonstrations and trains one contact- and geometry-aware policy across objects and interaction clips.

Motivation

  • Learning this behavior is challenging because grasping, object motion, locomotion, and balance are tightly coupled, while body and finger control operate at different scales.Pose matching alone does not ensure that the robot can sustain the forces and torques needed to move an object.Source: §I · pp. 1–2
  • A shared policy must adapt to object geometry and sequence-specific approach, posture, and object motion, unlike prior methods that often train one policy per object or clip.The goal is to capture common coordination patterns while adapting whole-body motion and finger contact to each reference.Source: §I · p. 2

Contributions

  • WEAVE trains one policy jointly across objects and interaction clips to coordinate whole-body motion, object transport, and finger-level grasping.The policy tracks robot and object motion while explicitly learning hand-object contact.Source: §I · p. 2
  • The framework converts human-object demonstrations into executable robot-object trajectories by completing approach motions and preserving task-relevant contacts across the morphology gap.The pipeline combines approach-motion completion with contact-aware retargeting.Source: §I · p. 2
  • WEAVE evaluates interaction execution and joint multi-object learning, and releases a physically plausible rollout dataset for downstream learning and interaction modeling.The evaluation also examines how policy architecture and optimization affect learning efficiency.Source: §I · p. 2

II. RELATED WORKS

The related work spans interaction-preserving retargeting, humanoid loco-manipulation, and humanoid vision-language-action systems. WEAVE combines contact-aware reference construction with physics-based policy learning and is presented as complementary to VLA systems.

A. Interaction-Preserving Retargeting and Motion Generation

  • WEAVE combines contact-aware retargeting with Kimodo-based approach-motion completion, using kinematic references to supervise reinforcement learning for executable interactions.Prior work includes retargeting that preserves whole-body interaction structure, hand-object topology, or contact-force consistency [3], [16]–.Source: A. Interaction-Preserving Retargeting… · p. 2

B. Humanoid Loco-Manipulation

  • WEAVE supervises finger-level contact and learns one policy across objects and clips, whereas related approaches include modular control, reference tracking, visually guided manipulation, and interaction-data generation [24]–[37].The cited literature includes policies for robot and object motion, dexterous interaction, and approaches for diversifying humanoid interaction data.Source: B. Humanoid Loco-Manipulation · pp. 2–3

C. Humanoid Vision–Language–Action Models

  • WEAVE uses physics-based reinforcement learning on human-object motion to produce grounded finger-level behavior, complementing VLA systems that can provide semantic specifications and high-level motion commands.The paper notes that humanoid VLA research faces the challenge of acquiring data combining locomotion, precise object interaction, and articulated finger motion.Source: C. Humanoid Vision–Language–Action… · p. 3

III. PROBLEM FORMULATION

  • The task is formulated as a POMDP whose policy outputs desired joint positions for a low-level PD controller.After retargeting and motion completion, captured sequences supply robot-object reference trajectories for policy learning.Source: §III · p. 3

IV. METHOD

WEAVE constructs robot-object references from human interactions, then trains a unified reference-tracking policy across objects and clips. Its objective combines tracking, contact-aware grasping, and motion regularization.

A. Overview

  • WEAVE first constructs references through whole-body retargeting, contact-aware hand refinement, and approach-motion completion, then trains a unified policy to coordinate body, fingers, and object motion.Training uses reference tracking jointly across multiple objects and interaction clips.Source: A. Overview · p. 3
  • The reference-tracking reward includes pelvis and link orientation, object position, and object orientation terms, with hatted quantities denoting references.The table caption states that tracking terms contribute weighted exponential functions of tracking error.Source: Table I · p. 4, A. Overview · p. 4
  • Contact-related terms encode expected fingertip contact and normalized fingertip force, alongside regularization terms for foot sliding and soft joint limits.The expected-contact gate and surface-to-fingertip direction are specified for each hand and fingertip.Source: A. Overview · p. 4

B. Interaction-Preserving Reference Construction

The reference-construction pipeline turns paired human–object motion into synchronized robot–object trajectories with approach motion and contact annotations. It uses whole-body retargeting, contact-aware hand refinement, and approach-motion completion.

B. Interaction-Preserving Reference Construction

  • The pipeline converts paired SMPL-X motion and object trajectories into robot–object references spanning approach, grasping, and object transport, with contact annotations for policy training.References initialize simulation states and guide policy learning under simulator dynamics.Source: B. Interaction-Preserving Reference… · pp. 4–5
  • Whole-body inverse kinematics aligns selected robot links with human counterparts while penalizing joint-limit violations and promoting feasible, smooth motion.The pelvis is constrained horizontally, with height optimized from ground contact rather than scale calibration.Source: B. Interaction-Preserving Reference… · p. 4
  • Hand refinement jointly adjusts arm and finger configurations because whole-body inverse kinematics alone does not ensure a stable fingertip grasp.The refinement leaves the IK pelvis and leg configuration fixed; contact attraction is supplemented by a force-closure objective because surface contact alone may not hold the object.Source: B. Interaction-Preserving Reference… · p. 4
  • Kimodo generates an approach prefix whose path ends at the initial interaction pose and heading, then concatenates it with the retargeted interaction trajectory.Sampling approach azimuths yields multiple references from one interaction sequence, using K = 3 for training and K = 5 for test.Source: B. Interaction-Preserving Reference… · p. 5
  • Each robot link receives a separated, neutral, or contact label based on its distance from the object surface, specifying the desired interaction pattern.The labels accompany completed trajectories used to initialize and guide policy learning.Source: B. Interaction-Preserving Reference… · p. 5

C. Contact-Rich Skill Acquisition via Reference Tracking

The section describes a unified reference-conditioned policy trained across objects and interaction clips, including its observations, tracking objectives, and simulation training setup.

C. Contact-Rich Skill Acquisition via Reference Tracking

  • A unified reference-conditioned policy is trained jointly across objects and interaction clips using robot proprioception, object state and geometry, and reference motion.The short-horizon reference command includes joint configuration, pelvis and object poses, and contact labels.Source: C. Contact-Rich Skill Acquisition via… · p. 5
  • The policy is optimized with PPO using reference-state initialization and reference-based early termination, with training at 200 Hz simulation and 50 Hz control.The policy uses SimBaV2 networks, Muon for two-dimensional weight matrices, and AdamW for biases and other parameters.Source: C. Contact-Rich Skill Acquisition via… · pp. 5–6
  • The tracking reward combines robot and object tracking with geometry-aware grasping, contact matching, and motion regularization.Contact matching applies only to non-neutral labels; other penalties include foot sliding, action variation, and non-finger joint-limit violations.Source: C. Contact-Rich Skill Acquisition via… · p. 5
  • Training randomizes robot and object friction and restitution, torso center of mass, and finger actuator properties; episodes terminate when tracking errors or missing expected contacts exceed stated thresholds.The provided termination criteria include pelvis-position error above 0.25 m, object-position error above 0.30 m, and absent expected hand–object contacts for ten consecutive control steps.Source: C. Contact-Rich Skill Acquisition via… · pp. 5–6

V. EXPERIMENTS

The experiments are organized around policy execution across objects and sequences, multi-object versus specialist training, and the effects of architecture and optimizer on learning efficiency.

V. EXPERIMENTS

  • The experiments ask how effectively the learned policy executes whole-body dexterous loco-manipulation across objects and interaction sequences.Source: §V · p. 6
  • They compare joint multi-object training with a single-object specialist on interaction completion and tracking accuracy.Source: §V · p. 6
  • They examine how policy architecture and optimizer affect learning efficiency.Source: §V · p. 6

A. Experimental Setup

The experiments use disjoint training and test reference splits and evaluate joint and object tracking with clip-balanced metrics. The setup specifies the tracked bodies and rotation-error measure.

Embodiment and dataset

  • The Unitree G1 setup uses 29 actuated body DoFs and two Inspire hands with 12 actuated finger DoFs, with 7,869 training and 1,605 test trajectories.The splits contain 19.56h and 3.67h of references, respectively.Source: A. Experimental Setup · p. 6

Evaluation metrics

  • Evaluation reports joint and object tracking errors using a clip-balanced average across N evaluation clips.For each clip, Ti denotes the number of executed frames.Source: A. Experimental Setup · p. 6
  • The tracking notation uses a for the pelvis anchor, B for tracked robot bodies, and geodesic rotation error for ∠(·, ·).Source: A. Experimental Setup · p. 6

B. Whole-Body Dexterous Loco-Manipulation

A single policy trained on Dtrain performs well on trained interactions and transfers to unseen sequences, though test completion remains lower. The test split includes more generated approach variations, which may contribute to that gap.

Results

  • 92.5% success rate and 96.3% progress rate are reached on Dtrain at 100k iterations, while object rotation error is 7.18° and position error is 4.27 cm.At 10k iterations, success rises from 78.0% and progress from 89.5%; object rotation and position errors fall from 11.17° and 5.17 cm.Source: B. Whole-Body Dexterous… · p. 7
  • On Dtest, the policy reaches 65.0% SR and 84.5% PR, with tracking errors close to Dtrain but a substantial completion gap.Source: B. Whole-Body Dexterous… · p. 7
  • The test split spans wider approach variation: approach-motion completion generates K = 3 variations per captured interaction for Dtrain and K = 5 for Dtest.The authors identify this split-distribution mismatch as a likely cause of the completion gap.Source: B. Whole-Body Dexterous… · p. 7
  • One policy fits 7,869 reference trajectories spanning nine objects, and its performance is still improving at the end of training.Source: B. Whole-Body Dexterous… · p. 7

C. Joint Multi-Object Skill Learning

Joint multi-object training outperforms object-specific specialists in pooled completion without regressions on any object. Its advantage accompanies a trade-off between imitation fidelity and interaction completion, with shared whole-body structure and shape-related contacts offered as possible mechanisms.

Results

  • 95.3% pooled success is achieved by the unified policy, compared with 91.5% for specialists, under an equal aggregate training-iteration budget.Joint training matches specialist success on three objects and improves it on six, with no object regressing; pooled rates are weighted by evaluation-trajectory count.Source: Figure 5 · p. 7, C. Joint Multi-Object Skill Learning · p. 7
  • Specialists often have slightly lower tracking errors while completing fewer interactions, so lower imitation error does not imply more reliable completion.The authors argue tracking error alone should not be the primary metric for contact-rich interaction.Source: C. Joint Multi-Object Skill Learning · p. 7

Why joint training helps

  • The authors propose that shared approach and transport structure lets clips mutually augment one another, while related object shapes share contact patterns that favor geometry-dependent grasp selection.These are mechanisms presented as consistent with the evidence, not established causal findings.Source: C. Joint Multi-Object Skill Learning · p. 7

D. Architecture and Optimizer Ablation

The ablation compares MLP and SimBaV2 architectures with AdamW and Muon on the small-table task under a shared iteration budget. Muon improves earlier than AdamW, while SimBaV2 adds a smaller complementary benefit.

Results

  • Both Muon configurations improve earlier than their AdamW counterparts, while SimBaV2 provides a smaller additional improvement.The comparison uses four architecture-optimizer combinations on the small-table task under a common 3,000-iteration budget; Figure 6 plots mean episode length and reward.Source: Figure 6 · p. 7, D. Architecture and Optimizer Ablation · p. 8
  • Muon’s orthogonalized momentum updates are proposed to distribute updates across coupled reward objectives, while SimBaV2 limits policy changes as the state distribution shifts at contact.The authors describe these optimizer and architecture effects as complementary.Source: D. Architecture and Optimizer Ablation · p. 8

VI. CONCLUSIONS AND LIMITATIONS

WEAVE converts captured interactions into robot skills using contact-aware references and a unified policy, achieving high success on trained interactions and untrained sequences. Evaluation and capabilities remain bounded by simulation-only testing, embodiment and object coverage, and reference-conditioned behavior.

VI. CONCLUSIONS AND LIMITATIONS

  • 92.5% success rate on trained interactions and 65.0% on sequences beyond training data distribution without additional training; physically executed rollouts total ∼23h.The rollouts are produced by the unified policy.Source: §VI · p. 8
  • The reference pipeline preserves task-relevant hand-object contacts and completes missing approach phases, while one contact- and geometry-aware policy coordinates locomotion, balance, and finger-level grasping.Source: §VI · p. 8
  • Simulation-only evaluation uses ground-truth odometry and object pose, leaving the sim2real gap untested and deployment dependent on state estimation or perception-based distillation.Source: §VI · p. 8
  • The underactuated Inspire hand and proximal-joint commands bound finger-level fidelity; heavy or highly articulated objects fall outside reference and domain-randomization coverage.Source: §VI · p. 8
  • The policy is reference-conditioned, so selecting and composing interactions with a higher-level planner remains future work.Source: §VI · p. 8

Related papers

Loading 2609.16683v1…