PaperlayerView exact revision on arXiv

Source-linked AI summary

Gated Residual Body-Hand Coordination for Whole-Body Humanoid Teleoperation

Ruiming Wu, Shuang Li, Liding Zhang, Alois Knoll, Zhaopeng Chen

arXiv:2609.18763v1cs.RO

TL;DR

Independent body and hand commands may fail to preserve bimanual geometry. The paper coordinates frozen modular controllers with bounded, gated residual corrections and prepares robot-specific motion references. On held-out GRAB motions, the residual reduces wrist and fingertip geometry errors by 39.2–56.3% relative to direct composition while maintaining comparable whole-body tracking ability.

  • Problem

    Independently generated body and hand commands do not explicitly preserve cross-chain geometry, allowing wrist-placement and finger-retargeting errors to accumulate during bimanual interaction.

    Source: §I · p. 1

  • Method

    A residual policy applies separate bounded corrections to frozen body and hand commands, using motion-conditioned action gating and reference-geometry-dependent reward gating.

    Source: §I · p. 2

  • Results

    39.2–56.3%: Full Residual reduces all four reported wrist and fingertip geometry errors relative to direct composition.

    Source: C. Residual Ablation · p. 6

  • Takeaways & Limitations

    Simulation results demonstrate improved bimanual wrist and fingertip geometry while maintaining comparable whole-body tracking ability.

    Source: §VII · p. 7

    The released G1 policy remains a stronger non-data-matched reference, trained at a substantially larger scale.

    Source: §VI · p. 7

Abstract

from arXiv · show

Whole-body humanoid teleoperation commonly combines a motion-tracking policy with a separate dexterous-hand retargeter. However, independently generated commands do not explicitly preserve body-hand geometric relations, leading to mismatches in relative wrist poses and fingertip positions during bimanual interaction. We present a gated residual coordination framework that keeps both modules frozen and applies bounded corrections to their outputs. A motion-conditioned action gate allocates correction authority across joint groups, while reference-geometry-dependent reward gates emphasize relevant interaction objectives during training. To establish the nominal body controller on Agile One, we introduce multi-pose morphology calibration that jointly estimates triaxial scales and effector-local offsets, together with staged motion dataset curation for training a SONIC-based tracker. The residual policy uses human motion references, initial commands, and robot proprioception without explicit object or contact observations. In simulation, it reduces wrist and fingertip geometry errors by 39.2-56.3% over direct composition on held-out GRAB motions, while preserving whole-body tracking on AMASS, with success rates of 89.03% without residual coordination and 89.29% with it. Ablations characterize the contributions of reward gating, adaptive correction authority, and separate body and hand correction heads.

I. INTRODUCTION

Separately generated body and hand commands can fail to preserve cross-chain geometry, even when each module tracks its own reference. The paper preserves the modular controllers while coordinating their commands with bounded residuals, context-dependent gates, and robot-specific motion preparation.

I. INTRODUCTION

  • Independent body and hand commands can accumulate wrist-placement and finger-retargeting errors, producing inaccurate relative wrist poses and fingertip positions in bimanual interaction.The mismatch can occur even when each module tracks its own reference well.Source: §I · p. 1
  • Replacing the modular stack with a monolithic controller would enlarge action and optimization spaces, couple subsystem development, and require joint retraining when either component changes.Robot-specific references also pose a challenge because single-pose calibration cannot distinguish body scaling from effector-local offsets.Source: §I · p. 1
  • A motion-conditioned action gate allocates correction authority across joint groups, while reference-geometry-dependent reward gates emphasize interaction objectives when their geometric relations are relevant.The action gate operates during execution; the reward gate shapes training.Source: §I · p. 2
  • The framework jointly refines frozen body and hand commands with separate, joint-group-bounded corrections while retaining the modular tracker and hand retargeter.The residual policy determines where and when to apply corrections rather than replacing the nominal modules.Source: §I · p. 2
  • Multi-pose calibration jointly estimates root-frame triaxial scales and effector-local offsets, and staged curation prepares robot-specific references for a SONIC-based tracker.Multiple paired human–robot poses resolve the scale–offset ambiguity of single-pose fitting.Source: §I · pp. 1–2

II. RELATED WORK

Prior work addresses embodiment retargeting, whole-body tracking, dexterous hand mapping, and interaction-aware control through several complementary approaches. The cited work leaves command-level coordination between independently generated body and hand commands as a distinct setting.

A. Motion Data, Retargeting, and Curation

  • Retargeting and curation methods address cross-embodiment mapping and reference quality, including feasibility filtering and curated motion subsets that can outperform larger unfiltered corpora [5], [14].The cited approaches also address temporal artifacts and human–object–environment relations when that information is available [12], [13].Source: A. Motion Data, Retargeting, and… · p. 2

B. Whole-Body Tracking and Dexterous Teleoperation

  • Whole-body trackers use human references and proprioception, while hand methods map human hand geometry to robot targets; modular combinations coexist with object- and contact-aware coupling [1], [2], [4], [18], [19].The related work distinguishes these approaches from command-level coordination of independently generated body and hand commands.Source: B. Whole-Body Tracking and Dexterous… · p. 2

C. Residual Coordination and Gating

Residual control and gating provide context for coordinating modular body–hand commands, while the paper also prepares embodiment-specific references for Agile One. The work distinguishes its teleoperation setting from approaches that use task-specific object or contact observations.

C. Residual Coordination and Gating

  • Residual methods add corrective actions around existing controllers, and prior interaction-specific approaches may condition on object, contact, goal, or task-phase signals [7], [8], [22]–[25].The present setting instead infers coordination from human references, initial commands, and robot proprioception without task-specific object or contact observations.Source: C. Residual Coordination and Gating · pp. 2–3

III. MULTI-POSE MORPHOLOGY CALIBRATION AND MOTION DATASET CURATION

  • Seven manually paired human and Agile One poses jointly estimate root-frame triaxial scales and effector-local offsets.The paired poses provide calibration data for morphology compensation.Source: Figure 2 · p. 3

C. Residual Coordination and Gating

  • Prior gates activate objectives by contact stage, motion phase, event window, or robot–target distance; this paper derives reward relevance and residual authority from the human-motion reference [26]–.The two mechanisms provide complementary context-dependent control and reward modulation.Source: C. Residual Coordination and Gating · p. 3

III. MULTI-POSE MORPHOLOGY CALIBRATION AND MOTION DATASET CURATION

  • The Agile One reference corpus begins with 52,663 paired candidates from five SEED categories, preserving human–robot correspondence through calibration and curation.Agile One has 29 body degrees of freedom.Source: §III · p. 3

A. Multi-Pose Morphology Calibration

The method calibrates retargeting across human and robot morphologies, then curates the resulting motion references for tracker training. Calibration uses multiple paired poses; curation addresses reference quality, temporal discontinuities, ground alignment, and redundancy.

A. Multi-Pose Morphology Calibration

  • Multiple paired human–robot poses jointly estimate root-frame triaxial scales and effector-local offsets to compensate for kinematic discrepancies.Root-frame scaling rotates longitudinal, lateral, and vertical corrections with the human; the seven-pose fit is identifiable when its 21 × 6 design matrix has rank six.Source: A. Multi-Pose Morphology Calibration · p. 3
  • 46,363 trajectories remain after frame-wise position and orientation checks remove motions exceeding configured limits from 52,663 calibrated candidates.The check covers pelvis, head, wrist, and ankle targets.Source: A. Multi-Pose Morphology Calibration · p. 3

B. Corpus-Scale Curation

  • Curation combines quality screening, transient IK repair, foot-height alignment, and density-aware selection because geometrically valid targets may still contain discontinuities, ground misalignment, or redundant motion.Fig. 3 depicts the four operations and marks rejection, alignment, or retention.Source: B. Corpus-Scale Curation · p. 3, Figure 3 · p. 4
  • 31,878 motions remain after quality screening; transient IK repair reconstructs eligible abrupt joint switches and retains 28,523 motions in total.Changes above 3° per frame at 120 Hz flag abrupt switches; events affecting at most two joints in one chain are interpolated and rechecked.Source: B. Corpus-Scale Curation · p. 3
  • 24,821 motions remain after foot-height alignment, and density-aware selection yields 20,000 references totaling 43.67 hours.Selection prunes dense neighborhoods across 153 motion descriptors; the median retained-to-input percentile-range ratio across 150 non-constant descriptors is 100.53%.Source: B. Corpus-Scale Curation · p. 4

IV. GATED RESIDUAL COORDINATION

The framework trains a SONIC-based nominal tracker, then coordinates frozen body and hand command generators with bounded, motion-gated residual corrections. Separate correction heads preserve their distinct output spaces while a shared representation supports coordination.

IV. GATED RESIDUAL COORDINATION

  • A shared-trunk, two-head residual policy coordinates frozen body and hand commands, with a human-motion action gate modulating bounded correction authority.The body and hand generators remain frozen; geometry-dependent reward gates affect residual training only.Source: §IV · p. 4, Figure 4 · p. 5

A. Base Tracker Training

  • The SONIC-based nominal tracker is trained with PPO for 30,000 iterations, then frozen before command-level coordination learning.Its curriculum increases default-pose initialization, dynamics randomization, root-velocity perturbations, and motion/joint-limit regularization.Source: A. Base Tracker Training · p. 4

B. Residual Command Interface

  • A shared representation feeds separate 29-dimensional body and 32-dimensional hand correction heads, modifying normalized tracker actions and physical joint-angle targets, respectively.Joint-group bounds constrain corrections; leg corrections are more tightly bounded and head corrections are disabled.Source: B. Residual Command Interface · p. 4

C. Human-Motion Action Gate

A human-motion action gate continuously reallocates correction authority between manipulation and locomotion rather than switching between controllers.

C. Human-Motion Action Gate

  • The action gate uses 25 reference frames—15 past, the current frame, and nine future samples—to assess manipulation and locomotion motion features.The future samples come from SONIC’s existing 180 ms buffer, adding no further lookahead.Source: C. Human-Motion Action Gate · p. 5
  • A nonzero gate floor retains support correction during manipulation and upper-body and hand correction during locomotion.The resulting authority is modulated continuously rather than selecting a single active controller.Source: C. Human-Motion Action Gate · p. 5

D. Geometry-Gated Learning

Geometry-gated learning weights task-space relations according to human-reference geometry, emphasizing different objectives as interaction distance changes. The gates use human references and require neither object states nor contact labels.

D. Geometry-Gated Learning

  • The task-space objective compares root, wrist, fingertip, and inter-wrist relations using position, distance, rotation, and direction errors.The relation terms accommodate differences in human and robot limb lengths, hand dimensions, and reachable workspaces.Source: D. Geometry-Gated Learning · p. 5
  • Logistic close gates downweight inter-hand fingertip and inter-wrist objectives as human-reference distances increase through 0.10–0.20 m and 0.20–0.40 m, respectively.A far gate increases wrist-direction weight over 0.20–0.40 m.Source: D. Geometry-Gated Learning · p. 5
  • Gate distances come exclusively from human references, so evaluating them requires neither object states nor contact labels.Residual training uses 2,304 GRAB motions, including mirrors, and 1,006 PICO references.Source: D. Geometry-Gated Learning · p. 5

V. EXPERIMENTS

The experiments assess retargeting calibration and SONIC tracker adaptation on held-out AMASS motions. SONIC AO performs comparably to a data- and budget-matched G1 policy, while the larger-scale released G1 policy remains stronger.

A. Retargeting Calibration

  • Triaxial seven-pose calibration reduces post-IK root and wrist target errors by 11.6% and 28.4%, respectively, while ankle target error increases by 0.36 mm, from 1.67 to 2.03 mm.The comparison applies both calibration configurations to the same 52,663 paired motion candidates.Source: A. Retargeting Calibration · pp. 5–6, Table I · p. 5

B. SONIC Tracker Evaluation

  • Tracker evaluation uses 8,802 held-out AMASS motions with GRAB excluded, each with ten observation-corrupted replicas.Success runs use root and distal-height terminations; tracking runs retain all frames to motion end.Source: B. SONIC Tracker Evaluation · p. 6
  • SONIC AO has tracking errors comparable to the data- and budget-matched G1 policy, while SONIC G1 Released achieves the highest success and lowest error in every reported tracking measure.The released policy is a non-data-matched reference trained with a substantially larger corpus and compute budget.Source: Table II · p. 6, B. SONIC Tracker Evaluation · p. 6

C. Residual Ablation

The held-out GRAB ablation shows that residual coordination improves body–hand geometry over direct composition, while the gates and correction heads expose trade-offs among coordination, root execution, and residual use.

C. Residual Ablation

  • 100% root-tracking success across conditions makes continuous geometric errors the primary ablation comparison.The benchmark uses 50 held-out motions across five regimes, with matched replicas and shared command generators and references.Source: C. Residual Ablation · p. 6
  • In Table III’s root and body–hand coordination errors, Full Residual reduces root XY error by 24.2% and all four reported wrist and fingertip geometry errors by 39.2–56.3% relative to direct composition.Source: C. Residual Ablation · p. 6
  • Constant Reward Gate has the lowest aggregate coordination errors, while Full Residual improves close-bimanual wrist distance and rotation.In close-bimanual windows, Full Residual reduces wrist-distance error from 6.54 to 6.11 cm and wrist-rotation error from 23.57° to 22.33°; its raw saturation is 0.96% versus 1.62% outside those windows, and 3.15% versus 1.67% within them.Source: C. Residual Ablation · p. 6
  • Constant Action Gate and Upper-Body Only increase root XY error, while Single Head raises raw residual saturation without a consistent geometric advantage.Constant Action Gate increases root XY error from 0.110 to 0.129 m; Upper-Body Only raises it to 0.153 m. Raw residual saturation is the fraction of active residual channel–time entries near tanh saturation before gating and scaling.Source: C. Residual Ablation · p. 6

D. Whole-Body Tracking Preservation

On the same articulated-hand plant, adding residual coordination slightly raises success while reducing root-orientation error; other tracking errors increase modestly.

D. Whole-Body Tracking Preservation

  • The synchronized unseen-GRAB comparison places the human reference, direct composition, and residual coordination in separate rows.The figure offers a qualitative comparison alongside the AMASS tracking-preservation results.Source: Figure 5 · p. 7
  • 89.03% to 89.29%: enabling the residual changes height-scaled success on the same articulated-hand plant.Table IV reports the AMASS tracking-preservation comparison.Source: D. Whole-Body Tracking Preservation · p. 6, Table IV · p. 7
  • Root-orientation error falls from 0.2089 to 0.2020 rad, while root XY, body-position, and body-orientation errors increase modestly.The residual and base conditions share the articulated-hand plant and neutral hand references; they differ only by the residual policy.Source: D. Whole-Body Tracking Preservation · pp. 6–7

VI. DISCUSSION AND LIMITATIONS

The discussion frames residual coordination as improving held-out body–hand geometry while trading off bimanual coordination, root execution, and residual use. The conclusion summarizes the framework and identifies broader evaluation and feedback directions for future work.

VI. DISCUSSION AND LIMITATIONS

  • Held-out GRAB results support improved body–hand geometric coordination over direct composition, but the full model does not minimize every aggregate metric.The reward gate localizes interaction-specific learning objectives, while the action gate modulates correction authority by motion context.Source: §VI · p. 7
  • Evaluation is limited to simulation, one humanoid embodiment, one training seed, and an action gate reliant on the base system’s 180 ms reference buffer.The policy also lacks explicit object-state, contact-force, and tactile feedback.Source: §VI · p. 7

VII. CONCLUSION

  • The paper concludes that bounded, motion-conditioned residuals coordinate frozen whole-body and hand command generators while improving bimanual geometry and maintaining comparable whole-body tracking.The nominal Agile One body stream uses multi-pose morphology calibration and staged motion curation for a SONIC-based tracker.Source: §VII · p. 7
  • Future work will investigate tactile feedback, neural hand retargeting, stronger pretrained trackers, and generalization across a broader range of motions.The stated aims include contact-aware residual coordination and improved human–robot hand correspondence.Source: §VII · p. 8

Related papers

Loading 2609.18763v1…