Source-linked AI summary
HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction
Jihwan Shin, Adrià López Escoriza, Junzhe He, Matthias Heyrman, Marco Hutter
TL;DR
Robot-compatible interaction references are limited, and pose-based retargeting can miss functionally important object contacts. HOI-Retarget transfers human-object interactions using object-frame contact targets and windowed optimization; on 13 OMOMO categories, it lowers mean contact-point gap from 18.3 cm to 0.5 cm versus the closest interaction-aware baseline. The released dataset provides thousands of robot interaction references, though the current palm-contact representation does not cover articulated grasping.
Problem
Robot-compatible interaction references are limited, while pose-only retargeting can shift hands away from objects or onto functionally different surfaces.
Source: §I · p. 1Method
HOI-Retarget transfers human-object interactions by optimizing object-frame contact targets over windows while balancing body tracking, foot support, and smoothness under robot limits.
Source: §I · p. 1, B. Windowed Trajectory Optimization · p. 4Results
0.5 cm mean contact-point gap versus 18.3 cm for the closest interaction-aware kinematic baseline across 13 OMOMO categories, at less than one quarter of its compute.
Source: §V · p. 6Takeaways & Limitations
The released pipeline includes 6,952 robot interaction references spanning 75 objects, with support for object-scale augmentation and synchronized multi-robot motions.
Source: §I · p. 2, §V · p. 6The palm-contact representation omits finger closure, contact distribution, and grasp forces, so tasks requiring articulated grasping remain kinematic references.
Source: A. Limitations · p. 6
Abstract
from arXiv · showhide
Learning from demonstration (LfD) has enabled humanoid robots to acquire diverse whole-body skills, but extending this paradigm to human-object interaction (HOI) is limited by the availability of robot-compatible interaction references. We present HOI-Retarget, a contact-centric retargeting method that transfers HOI onto a humanoid robot for large-scale motion-data generation. Its windowed trajectory optimization uses every labeled contact as a target in the object frame, balancing body tracking, foot support and smoothness under the robot's kinematic limits. The method can augment a single demonstration across object sizes, absorb contacts reconstructed from monocular video, and extend to several robots manipulating one object. We publicly release the code and the retargeted motion dataset.
I. INTRODUCTION
Robot-compatible human-object interaction references are scarce, and pose-only retargeting can shift contacts away from functionally important object locations. HOI-Retarget preserves object-frame contacts through windowed optimization; on 13 OMOMO categories it reduces contact-point gap from 18.3 cm to 0.5 cm versus the closest interaction-aware baseline and supports large-scale motion-data generation.
I. INTRODUCTION
- Pose-only retargeting can shift a hand away from the object or onto a functionally different surface, especially on thin or spatially localized grasp regions.Such interaction errors produce poor references for downstream control.Source: §I · p. 1
- HOI-Retarget preserves contact location and timing by expressing active contacts as object-frame targets and optimizing overlapping windows for tracking, contact, foot support, and smoothness under robot limits.The formulation begins from an IK retarget and couples consecutive frames while remaining tractable for long clips.Source: §I · p. 1
- 0.5 cm mean source contact-point gap versus 18.3 cm for the closest interaction-aware baseline, while running at less than one quarter of its compute.The comparison covers 13 OMOMO object categories retargeted to a Unitree G1.Source: §I · p. 2
- The released dataset contains 6,952 robot HOI clips across five source datasets and 75 objects, including multiple humanoid platforms and synchronized multi-robot interactions.The dataset totals 13.8 hours.Source: §I · p. 2
A. Human-to-Robot Motion Retargeting
Prior retargeting work addresses body-motion fidelity, interaction geometry, or dynamic feasibility, while human and video HOI datasets supply increasingly diverse but human-specific motion and contact estimates. HOI-Retarget is distinguished by directly targeting labeled object contacts in robot-compatible references.
A. Human-to-Robot Motion Retargeting
- Body-retargeting methods optimize pose fidelity or feasibility, but do not explicitly represent the location and duration of contacts with manipulated objects.Examples include keypoint optimization, inverse kinematics, modular kinematic objectives, learned motion projection, and RL-based execution.Source: A. Human-to-Robot Motion Retargeting · p. 2
B. Interaction-Aware Retargeting
- OmniRetarget preserves broad scene relationships through interaction-mesh deformation, whereas HOI-Retarget directly minimizes labeled object-frame contact error over temporal windows.Physics-based methods such as DynaRetarget, SPIDER, and ULTRA instead refine existing references for dynamic execution.Source: B. Interaction-Aware Retargeting · p. 2
C. Human-Object Interaction Data
- Captured and reconstructed HOI sources include motion-capture datasets and monocular-video methods, but their human-specific motions and estimated contacts can be unreliable.HOI-Retarget converts these sources to a common robot representation and exposes contact segments for refinement.Source: C. Human-Object Interaction Data · p. 3
III. HOI-RETARGET
HOI-Retarget first closes the embodiment gap with IK and object scaling, then refines the resulting interaction trajectory with windowed optimization. Object-frame contact targets remain tied to the object as it moves or changes size.
III. HOI-RETARGET
- The two-stage pipeline uses IK and object rescaling to bring a human-object clip within robot reach, then windowed optimization to recover contacts while preserving motion, smoothness, and kinematic limits.The object trajectory remains fixed during the optimization stage.Source: §III · p. 3
A. Inverse-Kinematics Retargeting with Object Scaling
- A common contact-labeling criterion transfers contact and stance flags to the robot’s two palms and two feet, with active contact positions expressed in the object frame.Contacts are based on body-mesh proximity to the object; foot stance additionally requires low height and velocity.Source: A. Inverse-Kinematics Retargeting with… · p. 3
- Scaling the object and its trajectory by the robot-to-human height ratio keeps it within reach, while object-frame targets stay fixed on its surface and enable object-size augmentation from one source clip.A Savitzky–Golay filter removes reconstruction noise and rescaling-related setdown discontinuities.Source: A. Inverse-Kinematics Retargeting with… · p. 3
B. Windowed Trajectory Optimization
The method refines an inverse-kinematics reference by optimizing robot configurations over short, overlapping windows. Its grouped costs preserve body tracking and object contacts while maintaining foot support and smoothness within kinematic limits.
B. Windowed Trajectory Optimization
- The nonlinear program optimizes configurations warm-started from IK; joint velocities are finite differences, and the object trajectory is a fixed parameter.This makes the robot's configurations responsible for recovering the interaction.Source: B. Windowed Trajectory Optimization · p. 3
- Overlapping H-frame windows pin their first p frames to the accumulated trajectory, preserving continuity while coupling frames for contact resolution and jerk penalties.Window size keeps each optimization problem independent of clip length.Source: B. Windowed Trajectory Optimization · pp. 3–4
- The tracking group anchors joint angles, base position, head, and ankles to the IK reference; a base-relative torso term constrains base orientation.Without the torso term, the pelvis can drift while the other costs remain satisfied.Source: B. Windowed Trajectory Optimization · p. 4
- The contact group aligns active links and palm orientations to object-frame targets, holds stance feet at reference height and sole orientation, and penalizes jerk.A small smoothness weight removes IK jitter, while larger weights begin to erase source-motion detail.Source: B. Windowed Trajectory Optimization · p. 4
C. Contact Refinement
The contact refinement addresses noisy source annotations by replacing shifting per-frame targets with a static target for each contiguous contact segment. This assumes the contact does not slip on the object.
C. Contact Refinement
- Noisy framewise reconstruction can make a stationary grasp appear to move across the object surface, driving the arm to follow the shifting target.Source: C. Contact Refinement · p. 4
- Assuming no slip, the method averages each contiguous contact segment into one static point that can be repositioned by hand before rerunning optimization.The web interface exposes one point per segment, useful when monocular-video occlusion or depth ambiguity has displaced estimated contacts.Source: C. Contact Refinement · p. 4
IV. RESULTS
The experiments assess contact preservation, object-size augmentation, monocular-video retargeting, and dynamic refinement across datasets and embodiments. The results also demonstrate breadth in kinematic references while delimiting multi-robot execution claims.
IV. RESULTS
- The kinematic benchmark uses 4,421 OMOMO clips across 13 object categories on Unitree G1, with additional datasets retargeted to G1 and H2.Dynamic refinement is evaluated on G1 using an Isaac Lab RL tracker and MuJoCo-based SBTO.Source: §IV · p. 4
- The experiments ask whether object-frame contacts persist across object sizes and monocular-video retargeting, and whether the kinematic advantage persists after physics refinement.Source: §IV · p. 4
- Kinematic references span six HOI sources, two humanoid embodiments, and objects with substantially different geometry and scale.Two-person demonstrations are retargeted against a shared object trajectory to produce synchronized multi-agent references.Source: §IV · p. 4
- The multi-agent results establish kinematic input and embodiment coverage, not mutual collision avoidance or dynamically coupled execution.Source: §IV · p. 4
A. HOI Preservation
On the common OMOMO subset, HOI-Retarget sharply reduces contact error versus OmniRetarget while also improving several motion measures and runtime. Qualitative examples show that its contact-centered objective recovers labeled object regions.
A. HOI Preservation
- Figure 3 presents source SMPL-X motion alongside Unitree G1 and H2 retargets.Source: Figure 3 · p. 5
- OmniRetarget fails to find a feasible solution for 424 of 4,421 clips (9.6%), predominantly involving large, extended objects.The reported means use clips solved by all three methods; contact metrics use the subset containing hand–object contact.Source: A. HOI Preservation · p. 5
- HOI-Retarget lowers body-pose deviation from 25.0° to 21.0°, body jerk from 193.5 to 49.9 m/s3, and runtime from 156.2 to 33.7 seconds per clip.Source: A. HOI Preservation · p. 5
- GMR has the lowest body-pose and link-velocity-direction errors but leaves a 36.8 cm contact gap, illustrating the object-agnostic control’s trade-off.The comparison disables HOI-Retarget mesh scaling so all methods manipulate the same object geometry; object trajectory adjustments only bring it into robot workspace.Source: A. HOI Preservation · pp. 4–5
- 0.5 cm versus OmniRetarget’s 18.3 cm mean contact-point gap, with hand-orientation-change error of 6.1° versus 37.6° for OmniRetarget.Table III reports the common subset: 3,997 clips overall and 3,604 clips for contact metrics.Source: Table III · p. 5, A. HOI Preservation · p. 5
- In the qualitative comparison, HOI-Retarget places palms at labeled source regions on the coat rack and chair, unlike OmniRetarget’s broader body–object arrangement.Source: Figure 4 · p. 5, A. HOI Preservation · p. 5
B. Data Augmentation
The section tests object-scale augmentation, monocular-video input, and dynamic refinement of retargeted references. Contact alignment persists across tested object scales, while references retain advantages in object path, contact location, and hand orientation after two refinement procedures.
B. Data Augmentation
- With object scale varied from ×0.25 to ×1.50, palms follow corresponding surface regions on box and chair interactions rather than staying at their original world positions.This is qualitative evidence of geometric contact consistency, not proof of dynamic feasibility at every tested size.Source: B. Data Augmentation · p. 5, Figure 5 · p. 6
C. Retargeting from Video
- For a monocular-video reconstruction of a person moving a table, contact segments were collapsed to one point and remaining displaced targets manually corrected before retargeting.Occlusion and depth ambiguity caused estimated hand contact to alternate between penetration and separation; the example shows pipeline compatibility, not reconstruction accuracy across a video dataset.Source: C. Retargeting from Video · p. 5
D. Dynamic Refinement
- Under RL refinement, object-path error falls from 0.285 to 0.213 m, contact-point gap from 9.6 to 8.6 cm, and hand-orientation-change error from 31.5° to 25.3°; under SBTO, the values fall from 0.269 to 0.194 m, 14.1 to 9.9 cm, and 38.4° to 28.1°, respectively.The comparison uses 68 commonly refined clips, with contact metrics on 63 clips; SBTO link-velocity direction and jerk differences were not statistically significant.Source: D. Dynamic Refinement · p. 6
V. DISCUSSION
HOI-Retarget transfers human–object interactions to humanoid robots using object-frame contact targets and windowed trajectory optimization. Its reported results include a smaller contact-point gap than the closest interaction-aware kinematic baseline, advantages after dynamic refinement, and generated references across multiple interaction settings.
V. DISCUSSION
- Fig. 6 depicts HOI-Retarget applied to video reconstruction, with contact refinement used to address monocular reconstruction artifacts.The discussion also describes an open release of 6,952 kinematic humanoid interaction references spanning 75 unique objects.Source: Figure 6 · p. 6, §V · p. 6
- The method reduces mean contact-point gap from 18.3 to 0.5 cm versus the closest interaction-aware kinematic baseline, using less than one quarter of its compute.The comparison covers 13 OMOMO object categories.Source: §V · p. 6
- The relative advantage in object path, contact location, and hand orientation remains after both RL- and optimization-based dynamic refinement.The references are intended for refinement, and the discussion also reports object-scale augmentation, monocular-video input, and synchronized multi-agent references.Source: §V · p. 6
A. Limitations
The limitations concern the quality of input references and the palm-only representation of hand contact. Future work proposes grasp-aware retargeting and optional collision-aware objectives.
A. Limitations
- Input pose, motion, and contact errors can propagate to retargeted references; contact-segment refinement addresses erroneous annotations but not underlying human or object motion.Potential source issues include unnatural poses, jitter, inaccurate contacts, and body–object penetration.Source: A. Limitations · p. 6
- Palm contacts encode where and when a hand meets an object, but omit finger closure, contact distribution, and grasp forces; articulated-grasp tasks therefore remain kinematic references.The proposed extension is to condition optimization on grasp configurations coupling finger posture, object-relative contacts, and whole-body motion.Source: A. Limitations · p. 6, B. Future Work · p. 6
B. Future Work
- Future work proposes an optional body–object collision objective, while noting that dynamic refinement can resolve many small penetrations and that severe source penetration may yield unsuitable initialization.The proposed collision cost is framed as valuable for severe source penetration, not as necessary to eliminate every small kinematic-stage penetration.Source: B. Future Work · p. 6
APPENDIX A NOTATION AND EVALUATION METRICS
The appendix provides symbol and metric tables, then evaluates the optimization horizon with and without a collision cost. Windowing trades a small reported motion-quality margin for lower resource use and reliable completion under the tested memory allocation.
APPENDIX A NOTATION AND EVALUATION METRICS
- Table V lists HOI-Retarget symbols, and Table VI defines the metrics reported in Tables III and IV.The horizon experiment uses the ten longest clips for each of 13 OMOMO objects, totaling 130 clips and 44,974 frames, with one CPU and 16 GB per solve.Source: Appendix A · p. 7, Table V · p. 7, Table VI · p. 7
APPENDIX B OPTIMIZATION HORIZON
- Windowing gives up 0.9° of link-velocity direction and 66 rad/s^3 of joint jerk versus full-trajectory optimization, while the full horizon uses 2.2× the memory for 1.3× the time on the 87 clips both complete.With collision cost, all windowed solves complete, while 43 of 130 full-trajectory solves exceed the memory budget; at 64 GB, all 43 complete, so the finding concerns feasibility under a fixed allocation.Source: Appendix B · p. 7, Table VII · p. 7
- Per-frame solves are cheapest but perform worst because a single frame cannot trade contact against neighboring frames and degrades the jerk term to a first-order velocity penalty.The jerk term is the RMS third difference in joint space.Source: Appendix B · p. 7