Source-linked AI summary
InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions
Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, Liang-Yan Gui
TL;DR
InterMimic targets the challenge of learning physically plausible, diverse human-object interactions from imperfect MoCap data. It trains subject-specific teachers to refine and retarget motion, distills them into a student, and fine-tunes that student with RL. The resulting policy produces realistic interactions across datasets, generalizes zero-shot, and connects with kinematic generators.
Problem
Scaling physics-based human-object interaction imitation is difficult because MoCap contains contact and hand-detail errors across varied objects.
Method
InterMimic trains subject-specific teacher policies to refine and retarget MoCap, then distills their supervision and references into a student policy with RL fine-tuning.
Results
InterMimic produces realistic, diverse interactions across multiple HOI datasets, with zero-shot generalization and integration with kinematic generators.
Takeaways & Limitations
The framework supports whole-body interaction skills beyond traditional grasping and connects physics-based control with interaction prediction and text-to-interaction generation.
Takeaways & Limitations
The method struggles to fully correct MoCap data with significant errors, so severely corrupted samples may be filtered out.
Abstract
from arXiv · showhide
Achieving realistic simulations of humans interacting with a wide range of objects has long been a fundamental goal. Extending physics-based motion imitation to complex human-object interactions (HOIs) is challenging due to intricate human-object coupling, variability in object geometries, and artifacts in motion capture data, such as inaccurate contacts and limited hand detail. We introduce InterMimic, a framework that enables a single policy to robustly learn from hours of imperfect MoCap data covering diverse full-body interactions with dynamic and varied objects. Our key insight is to employ a curriculum strategy -- perfect first, then scale up. We first train subject-specific teacher policies to mimic, retarget, and refine motion capture data. Next, we distill these teachers into a student policy, with the teachers acting as online experts providing direct supervision, as well as high-quality references. Notably, we incorporate RL fine-tuning on the student policy to surpass mere demonstration replication and achieve higher-quality solutions. Our experiments demonstrate that InterMimic produces realistic and diverse interactions across multiple HOI datasets. The learned policy generalizes in a zero-shot manner and seamlessly integrates with kinematic generators, elevating the framework from mere imitation to generative modeling of complex human-object interactions.
1 University of Illinois Urbana-Champaign 2 Electronic Arts † Equal Advising
InterMimic enables physically simulated humans to perform highly dynamic, multi-object interactions with diverse objects. The framework also supports scalable skill learning and downstream uses including robot transfer, MoCap refinement, and kinematic generation.
- InterMimic enables physically simulated humans to interact with dynamic and diverse objects.
- The framework supports highly dynamic, multi-object interactions and scalable skill learning.
- Downstream applications include humanoid-robot transfer, interaction MoCap refinement, and kinematic interaction prediction or generation.
1. Introduction
The paper addresses the difficulty of scaling physically plausible whole-body human-object interaction imitation from imperfect MoCap data. InterMimic progressively perfects motion with subject-specific teachers, then integrates skills through student distillation and reinforcement learning.
- MoCap-based human-object animation requires correcting contact errors while preserving captured motion and physical plausibility.
- Physics-based interaction imitation faces MoCap artifacts, unrealistic simulated dynamics, and challenges from missing hand detail and diverse object interactions.
- InterMimic trains multiple teacher policies on subsets of imperfect interactions to imitate, retarget, and refine diverse motor skills.
- Teacher rollouts become refined, unified references that reduce the impact of original MoCap errors during student training.
- The student uses demonstration-based distillation to bootstrap PPO updates, then increases reinforcement-learning updates to surpass demonstration memorization.
- InterMimic is presented as a framework for whole-body skills with diverse dynamic objects, realistic motion recovery, zero-shot generalization, and integration with kinematic generators.
2. Related Work
Prior work covers human-human, hand-object, static-scene, and humanoid-control interactions, while many methods remain limited in interaction scope or hand-centric assumptions. InterMimic is positioned as a physics-based approach for universal whole-body loco-manipulation with diverse dynamic objects, trained through a teacher-student pipeline across multiple datasets.
- Existing interaction animation and control: Existing physics-based interaction research spans human-human, hand-object, static-scene, and real-world humanoid-control settings.
- Kinematic interaction generation: Many character-animation methods support increasingly diverse objects, but approaches integrating object motion often assume interactions occur primarily through the hands.
- Physics-based interaction animation: Physics-based methods use motor-control policies in simulators, with prior complex-interaction studies often targeting specific scenarios such as sports.
- InterMimic: InterMimic uses a two-stage pipeline that trains teacher policies on small subsets and distills them into a scalable student policy.
- InterMimic: Experiments process OMOMO, BEHAVE, HODome, IMHD, and HIMO data to evaluate scalability across diverse interactions and robustness to MoCap artifacts.
3. Methodology
InterMimic formulates HOI imitation as an MDP in which policies match reference human-object motion while compensating for imperfect data and embodiment differences. Its methodology trains RL teachers with retargeting- and contact-aware rewards, then distills them into a scalable student policy.
- Task Formulation: The policy generates simulated human-object trajectories matching ground-truth references while compensating for missing or inaccurate dataset details.
- Task Formulation: The MDP comprises states, actions, simulator transition dynamics, and a reward function, supporting both teacher and student policies.
- Policy Representation: States combine human proprioception, object observations, geometry, whole-body haptic sensing, and future reference poses.Haptic inputs include joint-to-surface distance vectors and contact markers indicating applied forces.
- Imitation as Perfecting: Reference contact markers are inferred from object acceleration and discretized into promotion, neutral, and penalty levels to accommodate variable contact distances.Neutral regions avoid enforcing or penalizing potentially inaccurate contacts.
- Imitation as Perfecting: Teacher policies are trained with RL using embodiment-aware kinematic rewards and embodiment-agnostic interaction dynamics, with distance-dependent weighting near objects.The reward includes joint-position, joint-rotation, interaction-tracking, object-tracking, and contact-tracking costs.
- Imitation as Perfecting: Reference State Initialization can expose policies to erroneous contacts and hand motions, motivating rollout initialization with simulation-corrected references.The training framework uses PPO-based RL and separately describes teacher training followed by teacher-to-student distillation.
4. Experiments
Experiments evaluate InterMimic’s teacher policies and student framework on imperfect references, large-scale datasets, generated interactions, and unseen objects. Results show improved tracking, refined physical interactions, zero-shot integration, and benefits from reference distillation, RL fine-tuning, and Transformer policies.
- Experimental Setup: OMOMO contains approximately 10 hours across 15 objects; InterMimic trains 17 subject-specific teacher policies, reserving subject 14 for testing.OMOMO is the primary dataset for evaluating the full teacher-student distillation framework at scale.
- Quantitative Evaluation: InterMimic maintains reference tracking longer than the baseline and produces interactions that more closely match imperfect MoCap references.The baseline struggles with artifacts such as incorrect hand positioning, leading to shorter tracking durations.
- Qualitative Evaluation: The student consistently outperforms baselines on training imitation and out-of-distribution generalization, including test-set and kinematic-generated interactions.Table 2 evaluates OMOMO, HOI-Diff generations, InterDiff generations, and test objects with weights ten times greater than training objects.
- Qualitative Evaluation: Teacher policies correct incorrect hand positioning, floating contacts, and implausible rotations of symmetric objects while preserving multi-part interactions.Figures 4 and 5 illustrate correction of reference errors, including objects incorrectly depicted as sliding along the ground.
- Qualitative Evaluation: Zero-shot integration supports text-to-interaction synthesis and future interaction prediction, while generalization extends to novel objects from BEHAVE and HODome.Figure 6 shows integration with HOI-Diff and InterDiff; Figure 7 shows generalization to unseen objects.
- Ablation Study: Reference distillation improves all metrics, PPO fine-tuning resolves conflicts among teachers, and Transformer policies outperform MLP approaches on test and generated cases.Without reference distillation, the policy struggles with unseen shapes; supervised distillation alone or training from scratch does not achieve optimal performance.
5. Conclusion
InterMimic synthesizes physically grounded and generalizable whole-body human-object interactions from imperfect motion-capture data. Its teacher-student framework combines motion refinement, retargeting, and scalable distillation, supporting zero-shot generalization and integration with kinematic generation.
- Conclusion: InterMimic synthesizes realistic, physically grounded, and generalizable human-object interactions from diverse imperfect MoCap data.The framework targets a wide range of whole-body skills involving dynamic and varied objects.
- Conclusion: Contact-guided rewards and optimized trajectory collection help teacher policies recover missing physical details and correct inaccuracies in MoCap data.Teacher policies refine interaction motion before student-policy training.
- Conclusion: Combining policy distillation with reference distillation enables efficient skill scaling from multiple teachers into student policies.The approach uses refined teacher outputs as training references.
- Conclusion: Zero-shot generalization and integration with kinematic generation bridge interaction imitation with generative modeling.The conclusion describes this capability as extending the framework beyond direct demonstration replication.
Supplementary Material
The supplementary material provides demonstrations, simulation and reward details, contact-label construction, implementation information, additional experiments, and discussion of limitations and societal impact.
- Supplementary Contents: Figure A depicts simulated humans performing physical interactions with scalable skill learning across diverse objects.The figure is associated with the supplementary demonstration materials.
- Supplementary Contents: The supplement includes a demonstration video and details of the physical human-object interaction simulation setup.These materials are provided in Sections A and B, respectively.
- Supplementary Contents: Additional material explains reference contact labels, reward formulation, physical state initialization, and interaction early termination.These mechanisms are discussed in Sections C through E.
- Supplementary Contents: Implementation details cover interaction prediction, text-guided interaction generation, and translation of MoCap interactions into humanoid-robot skills.The implementation discussion appears in Section F.
- Supplementary Contents: Additional experiments provide qualitative results and analyze failure cases, followed by discussion of limitations and societal impact.These topics are covered in Sections G and H.
A. Demo Video
The demo video highlights InterMimic’s ability to imitate dynamic, long-term interactions, retarget MoCap skills, correct physical artifacts, and scale across diverse objects and skills.
- The teacher policy imitates highly dynamic and long-term human-object interactions.
- Teacher-based retargeting transfers human MoCap interactions to a humanoid robot despite embodiment differences.
- Teacher policies correct contact penetration, floating, and jittering, producing cleaner references for student training.
- The baseline PhysHOI fails on sequences that InterMimic successfully imitates.
- The student policy learns from hours of data spanning diverse objects and interaction skills.
- Zero-shot generalization supports text-to-HOI, interaction prediction, and interactions with unseen skills, objects, and multiple objects.
E.2. Physical State Initialization
Physical State Initialization addresses failures of reference-state initialization on imperfect MoCap by selecting states from prior references and rollouts, while early termination can make training inefficient.
- Reference State Initialization can fail on imperfect MoCap, reducing experience collection and undermining performance.
- Failed initialization regions can become unreachable when fixed-length rollouts cannot cover them.
- Interaction early termination filters poor initial states but excessive failures cause frequent resets and slow training.
- Physical State Initialization stores reference and prior simulation states in a buffer for future rollout initialization.
- The framework also integrates with HOI-Diff for text-guided interaction generation.
F. Additional Implementation Details
The supplementary implementation details describe generator integration, training settings, initialization procedures, humanoid-hand modeling, and unchanged extension to multiple objects.
- InterMimic uses kinematic generator outputs as input references for imitation and integrates with InterDiff for interaction prediction.
- OMOMO evaluation uses Subject 9 as the base model while teachers retarget interactions from other subjects.
- The first simulation frame is initialized to match the first reference frame from MoCap or kinematic generation.
- For humanoid-robot learning, the framework directly learns from SMPL-X data without external retargeting, but its hand model omits the real setup’s mimic joint.
- Multiple-object interactions require changes only to object-related state and reward components.
- Table C lists hyperparameters for training teacher and student policies.
G. Additional Experiemental Results
Additional experiments report teacher-policy failures when reference interactions contain too many errors, despite demonstrated adaptability to correcting some reference errors.
- Teacher policies can fail when reference interactions contain too many errors.
- The reward design prioritizes reference tracking, limiting correction when reference errors become excessive.
H. Discussion
InterMimic can retarget interactions across different body shapes and filter severely corrupted MoCap samples, but it does not fully resolve penetration, large motion errors, or detailed finger dexterity.
- Teacher-policy failures on severely corrupted MoCap data lead those samples to be excluded from student training.An incorrectly flipped hand in an OMOMO reference is given as an example.
- InterMimic successfully retargets an interaction across different body shapes, achieving the same task with different trajectories.The comparison contrasts a green reference human with a yellow simulated human.
- The method does not entirely prevent unnatural object support involving penetration rather than friction.High depenetration velocity and contact-based energy mitigate but do not eliminate the issue.
- Hand-interaction recovery may provide limited benefits for tasks requiring detailed finger motions.The stated limitation concerns dexterous tasks beyond those explored in the paper.
- Performance could improve with a larger and more diverse dataset, despite scalability to hours of MoCap data and unseen skills and object geometries.The authors specifically suggest incorporating more diverse objects to enhance zero-shot generalization.