PaperlayerView exact revision on arXiv

Source-linked AI summary

InterMimicGen: Scaling Humanoid Loco-Manipulation through Self-Evolving Motion Imitation

Yucheng Zhang, Sirui Xu, Jinhong Li, Liuyu Bian, Anatulya Nandi, Derek Zhang, Xiangchen Liu, Xueting Li, Umar Iqbal, Yu-Xiong Wang, Liang-Yan Gui

arXiv:2610.06850v1cs.ROcs.CVcs.GR

TL;DR

Humanoid loco-manipulation demonstrations are heterogeneous and sparse, and small changes can disrupt contact or balance. InterMimicGen retargets interactions into robot references and couples a physics-based tracker with iterative, task-preserving augmentation. The reported results include a growing set of executable motions and transfer to real robots.

  • Problem

    Captured human-object demonstrations are sparse samples of feasible loco-manipulation, while small posture or object-placement changes can break contact or balance.

    Source: §1 · p. 2

  • Method

    InterMimicGen retargets heterogeneous human-object captures into robot references, then edits verified interactions, fine-tunes a physics-based tracker, and retains successful simulated rollouts.

    Source: §1 · p. 2

  • Results

    The verified reference set grows across rounds, the evolved tracker executes nearly all accepted references while retaining original-reference execution, and generated trajectories execute on real robots.

    Source: §4.4 · p. 9, §4.5 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    The tracker and executable interaction references can grow together, providing an expanding motion resource for subsequent whole-body controller learning.

    Source: §5 · p. 9

    The loop densifies coverage around interactions already represented in source motions rather than creating new task semantics, and selection may favor easier objects or compound reference errors.

    Source: H LIMITATIONS · p. 28

Abstract

from arXiv · show

Captured human-object interactions provide rich supervision for humanoid loco-manipulation, but they are sparse, heterogeneous, and not directly executable by robots. We introduce InterMimicGen, a self-evolving motion-imitation framework in which robot motion data and a tracking policy improve each other. First, we consolidate motion-captured human-object interaction datasets and retarget them into humanoid robot references while preserving whole-body coordination and dexterous hand-object relationships. This produces a large and diverse humanoid robot reference collection for dexterous whole-body loco-manipulation. Second, we train a physics-based generalist tracker that executes these references in simulation on a humanoid with dexterous hands, covering a scale and diversity beyond prior humanoid tracking systems for loco-manipulation. Third, we close a data flywheel: each round makes small, task-preserving changes to where an interaction takes place and how the body performs it, fine-tunes the tracker on them, and keeps only the variants whose simulated execution completes the task, which seed the next round. With more iterations, these small edits compound into broader coverage around the sparse original demonstrations while preserving task semantics and motion quality. Experiments show contact-preserving retargeting across robot configurations, broad tracking with a single generalist policy, executable motions that keep growing over augmentation rounds, and transfer to real robots. InterMimicGen provides a unified path from heterogeneous human demonstrations to a continually expanding motion resource for humanoid robot learning.

1 INTRODUCTION

Captured demonstrations are sparse samples of feasible humanoid loco-manipulation, where small changes can disrupt contact or balance. InterMimicGen retargets human-object interactions into robot references and iteratively expands them with a tracker that learns to execute the variations.

1 INTRODUCTION

  • Sparse demonstrations leave feasible interaction variations uncovered, and small posture or object-placement shifts can break contact or balance.A fixed reference set therefore cannot cover the variations; the paper motivates data that grows with the tracker.Source: §1 · p. 2
  • InterMimicGen combines heterogeneous human-object motion capture, robot retargeting, physics-based tracking, and iterative data expansion.Its retargeter preserves long-horizon body-object relations and refines finger contact at millimeter scale.Source: §1 · p. 2
  • Each round makes small, task-preserving edits, fine-tunes the tracker, and retains only simulated rollouts that complete the task as seeds for the next round.Successful corrected motions expand the reference set, which supports progressively larger variations.Source: §1 · p. 2

2 RELATED WORK

Related work spans human-object interaction data, whole-body and dexterous retargeting, physics-based imitation, and demonstration augmentation. InterMimicGen combines these lines by expanding humanoid loco-manipulation references through motion imitation and real deployment.

Human-object interaction data

  • Human-object capture datasets cover complementary forms of interaction, while InterAct (Xu et al., 2025c) consolidates sources and HiPHI (Ji et al., 2026) adds extensive body-object capture.The paper combines their motion into a robot reference collection and expands executable coverage through retargeting and simulation.Source: §2 · p. 2

Whole-body and dexterous motion retargeting

  • Retargeting work uses learned mappings or optimization over skeletons, contacts, and geometry; whole-body interaction also requires hand contacts to remain consistent with locomotion and balance.InterMimicGen separates coarse whole-body interaction solving from local dense contact refinement.Source: §2 · p. 3

Physics-based imitation and whole-body manipulation

  • Physics-based imitation has progressed from individual skills to shared controllers and object-centric dynamics, including real-world humanoid loco-manipulation.InterMimicGen starts from reference tracking and extends it into a self-evolving data engine with real deployment.Source: §2 · p. 3

Demonstration augmentation and iterative generation

  • Demonstration augmentation reuses task intent under adapted conditions, while InterMimicGen expands dexterous humanoid loco-manipulation from motion-capture references through motion imitation rather than a separate kinematic synthesizer.The paper notes that separate kinematic motion synthesis can be brittle for object interaction.Source: §2 · p. 3

3 METHOD

InterMimicGen consolidates human-object captures, retargets them to humanoid embodiments, and trains a physics-based generalist tracker. Its self-evolving loop edits verified interactions, fine-tunes the tracker, and retains executable variants to grow the reference collection.

3 METHOD

  • InterMimicGen turns captured human-object interactions into robot references, tracks them under robot dynamics, and repeatedly expands the verified set with executable edits.The reference stores target robot motion, object motion, and contact intent; Figure 2 sketches reference construction and the iterative loop.Source: §3 · p. 3, Figure 2 · p. 4

3.1 A CONSOLIDATED INTERACTION REFERENCE COLLECTION

  • InterAct and HiPHI provide complementary human-motion, object-geometry, and interaction coverage, and are converted to a shared body-object-hand representation.Missing hand annotations are treated as unknown; references unsupported by a target robot or simulation are removed.Source: §3.1 · pp. 3–4

3.2 RETARGETING FOR DEXTEROUS WHOLE-BODY INTERACTION

  • Retargeting first solves long-horizon body-object geometry and grasp opposition, then refines local contact while preserving hand-object relationships.The coarse solve uses an interaction mesh and feasible constrained solvers; cleanup addresses collisions, jitter, and contact drift while checking motion quality.Source: §3.2 · pp. 4–5

3.3 A GENERALIST FOR INTERACTION MOTION IMITATION

  • A single PPO policy shares parameters across motions and objects, observing robot state, upcoming references, and object-relative geometry to output PD joint targets.Training balances body and object motion, interaction geometry, and contact fidelity, with reference-state initialization and adaptive sampling.Source: §3.3 · p. 5

3.4 SELF-EVOLVING MOTION IMITATION

  • Variants preserve interaction type, intended contacts, object, and task outcome while editing either object placement or body posture.Object edits change the interaction location or trajectory; body edits change posture while keeping the object path and hand targets.Source: §3.4 · p. 5
  • Each round fine-tunes on proposals, validates their simulated execution, and carries successful variants forward; learning lets the tracker reach variations beyond its base capability.Validation checks completion, falls, smoothness, and task outcome; failed proposals are not edited again, preventing errors from accumulating across rounds.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate retargeting quality, physics-based tracking, iterative expansion of executable references, and transfer to real robots. Across these evaluations, InterMimicGen produces cleaner references, broadens executable coverage through tracker learning, and demonstrates real-robot execution.

4 EXPERIMENTS

  • The experiments ask whether retargeting preserves body-object relations and finger contact, trackers generalize across objects and embodiments, iterative learning expands executable skills, and motions transfer to real robots.Tracking comparisons use per-object specialists as a practical upper bound because no publicly available tracker is trained on a comparable interaction collection.Source: §4 · p. 6, §4.1 · p. 6

4.1 EXPERIMENTAL SETUP

  • The evaluation uses a collection of 16,059 motions, 140.68 reference hours, and 157 object entries, retargeted to six robot configurations.Metrics assess retargeting plausibility and fidelity, tracking success and errors, and augmentation growth, success, and execution quality.Source: §4.1 · p. 6

4.2 RETARGETING QUALITY AND DEXTEROUS CONTACT

  • Retargeting with dexterous hands improves several motion-quality measures over Weave, while explicit grasp modeling preserves demonstrations’ hand-object relationships that rigid hands only approximate.Compared with Weave, references slightly improve hand-contact preservation and greatly reduce penetration, foot sliding, and joint acceleration, at the cost of slightly higher in-hand slip.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 PHYSICS-BASED INTERACTION TRACKING

  • One generalist nearly matches specialists in body and object position error, but trails them in success rate and object rotation, especially on grasping.Bimanual interactions are tracked most reliably; sitting and grasping are harder, and sharing one policy mainly costs object control rather than body tracking.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 DOES ITERATIVE GENERATION EXPAND LEARNED COMPETENCE?

  • By round five, verified references grow more than a hundredfold, while body edits span several times the posture range of ULTRA’s scene-scaling variants.A frozen tracker accepts almost no new candidates after round one, indicating that expansion depends on learning rather than filtering alone.Source: Figure 6 · p. 8, Table 3 · p. 9, §4.4 · p. 9
  • The evolved tracker executes nearly all accepted references while retaining every original reference, and augmented executions remain close in body acceleration and hand jitter.Foot sliding rises from about 1% to about 2% of contact frames; both trackers execute every original reference.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.5 CROSS-EMBODIMENT EXECUTION AND REAL-WORLD TRANSFER

  • InterMimicGen trajectories execute on real robots across platforms using low-level controllers that track robot motion without object feedback.The demonstrated tasks include box lifting and placement, suitcase pulling, tripod carrying, and chair relocation.Source: §4.5 · p. 9, Figure 7 · p. 9

5 DISCUSSION AND CONCLUSION

InterMimicGen treats the humanoid tracker as both a consumer and producer of interaction data. Its evolving loop enlarges the reference archive and extends tracker capability while retaining seed skills.

5 DISCUSSION AND CONCLUSION

  • A consolidated human-object motion collection is retargeted across six robot configurations, then physics-based tracking and iterative validation turn successful variations into new references.The tracker learns interactions beyond the base tracker’s reach while preserving its seed skills; a frozen tracker stops accepting new variations after the first round.Source: §5 · p. 9

A DATA AND EMBODIMENTS

The section consolidates interaction references across robot embodiments, retargets them with whole-body and contact-aware geometry, and trains a generalist tracker to follow body, object, and interaction features.

A DATA AND EMBODIMENTS

  • The consolidated collection contains 16,059 sequences, and each retargeted reference stores robot state, object pose and twist, and contact intent.The sources contribute 9,030 InterAct and 7,029 HiPHI sequences.Source: A DATA AND EMBODIMENTS · p. 15, Table 4 · p. 15

A.1 EMBODIMENT SCOPE

  • Studies focus on G1 with dexterous hands, while other platforms use only motions they can support.Unsupported motions are excluded from each platform’s evaluation.Source: §A.1 · p. 15, Table 5 · p. 15

B RETARGETING

  • The retargeter transfers body-object structure through an interaction mesh, then cleans geometry while preserving contact.Its whole-body solve includes finger opposition; cleanup uses penetration repair and collision-aware smoothing.Source: B RETARGETING · pp. 15–17

C.1 OBSERVATION, ACTION, AND REWARD

  • The tracker observes robot proprioception, upcoming reference differences, object-relative state, and body-object geometry, then outputs joint-position offsets for PD torque control.The geometric feature is computed for current and reference configurations and applies across different hand joint spaces.Source: §C.1 · p. 18
  • The reward multiplies body, object, interaction, and contact tracking terms, so strength in one cannot offset failure in another.Interaction error compares body-to-surface vectors, weighting close pairs more heavily.Source: §C.1 · p. 18

C.2 TRAINING AND EVALUATION PROTOCOL

The tracker samples compatible references adaptively, while self-evolution edits verified interactions and retains candidates that pass execution and quality checks.

C.2 TRAINING AND EVALUATION PROTOCOL

  • Adaptive sampling gives every compatible reference a uniform probability share and shifts the remaining probability toward clips the policy rarely completes.The uniform share is ρ = 0.25; completion is measured by a moving average.Source: §C.2 · p. 18

D.1 EDIT SPACE AND INVARIANTS

  • Each edit changes where or how a verified interaction is performed while retaining its interaction type, intended contacts, object, and task outcome.Edits are blended gradually and recompute motion and interaction features from the changed geometry.Source: §D.1 · pp. 19–20, D SELF-EVOLVING MOTION IMITATION · p. 19

D.2 OBJECT EDITS

  • Object edits move the interaction in the seed’s release frame, leaving the approach unchanged and blending the displacement across contact until release.The edits preserve the type of outcome, such as placement on a support, rather than the exact terminal pose.Source: §D.2 · p. 20

D.3 BODY EDITS

  • Pelvis, stance, and arm edits vary posture while keeping task-specific quantities fixed; arm edits move in the palm-pose constraint’s nullspace.The constructed arm pose is projected back to the original palm target before rollout.Source: §D.3 · p. 20

D.4 BUILDING, EXECUTING, AND KEEPING EDITS

  • Candidates are executed in three seeded rollouts and checked for terminal contact, sustained falls, motion smoothness, and stable object placement.Object edits enter fine-tuning before judgment; verified body edits join training in later rounds.Source: §D.4 · p. 20

D.5 GROWTH OVER ROUNDS

  • Each round proposes edits around verified parents, extending object edits from successful outward-reaching variants and body edits from extreme successful postures.Figure 8 shows further captures with original references and verified augmentations accumulated through rounds 1 and 5.Source: §D.5 · pp. 20–21, Figure 8 · p. 21

E EVALUATION METRICS

The section defines metrics for data coverage, retargeted references, tracking, and augmentation, using shared features, thresholds, and frame rates.

E EVALUATION METRICS

  • Metrics are grouped by data coverage, retargeted references, tracking, and the augmentation loop.The section defines each metric used in the experiments once.Source: E EVALUATION METRICS · p. 21
  • All compared methods use the same features, thresholds, and frame rates.Source: E EVALUATION METRICS · p. 21

E.1 DATA COVERAGE

The evaluation specifies feature constructions and metrics for coverage, retargeting, tracking, and self-evolution, then lists additional comparisons and a frozen-tracker ablation.

E.1 DATA COVERAGE

  • Coverage visualizations encode human motion, object geometry, and human-object interaction separately; external coverage matches motion probes to the source library.The human-motion feature omits object information, and coverage scaling averages 20 random orderings with 5th–95th percentile shading.Source: §E.1 · pp. 21–22

E.2 RETARGETED REFERENCES

  • Retargeted references are evaluated on shared source clips using contact preservation, slip, penetration, foot sliding, fidelity, smoothness, and joint-limit metrics.The section specifies operational definitions and thresholds for each measure.Source: §E.2 · p. 22

E.3 TRACKING

  • The tracking benchmark covers 15 objects across bimanual, sitting, and grasping categories; success is reaching the final reference frame without termination.Tracking errors are measured on successful clips against matching reference frames, excluding terminal holds.Source: §E.3 · pp. 22–23

E.4 SELF-EVOLUTION

  • Self-evolution is measured by verified-reference growth, motion quality, tracker success before and after evolution, and frozen-tracker rollout yield.Motion quality includes body-joint acceleration, foot sliding, and hand jitter.Source: §E.4 · pp. 23–24

F ADDITIONAL RESULTS

  • Additional results extend the retargeting, tracking, and loop comparisons and include a frozen-tracker ablation to isolate fine-tuning’s role.Source: F ADDITIONAL RESULTS · p. 24

F.1 RETARGETING COMPARISON ON G1 WITH NON-DEXTEROUS HANDS

On G1 with non-dexterous hands, retargeters make different trade-offs across fidelity, geometric constraints, and smoothness. The paper’s dexterous retargeting addresses the rigid-hand limitation by preserving grasp relationships with articulated fingers.

F.1 RETARGETING COMPARISON ON G1 WITH NON-DEXTEROUS HANDS

  • The comparison uses 537 OMOMO clips shared by four methods, covering largebox, smallbox, plasticbox, and suitcase.Contact preservation is evaluated on the 479 clips with positive supplied left/right hand-contact labels.Source: §F.1 · p. 24
  • With rigid hands, no method wins across fidelity, geometric constraints, and smoothness: the authors’ references are smoothest and most accurate at end effectors, OmniRetarget penetrates least, and UMR preserves most hand contact.Rigid hands cannot close around objects, so contact and penetration trade off directly.Source: §F.1 · p. 24, Table 10 · p. 25
  • Figure 14 shows a box-lifting reference executed by specialist trackers on Booster K1 and G1 configurations, including G1 with non-dexterous, Inspire, or Dex3 hands.Source: Figure 14 · p. 25
  • Articulated fingers let the authors’ references preserve the demonstration’s grasp and hand–object relationship.The cited figures show contact-preserving retargeting across robot configurations.Source: §F.1 · p. 25

F.2 ABLATION ON LEARNING IN THE LOOP

Fine-tuning keeps the self-evolution loop productive after the frozen tracker stalls; verified references grow across rounds, and body edits extend beyond scene scaling. The paper also reports deployment on humanoids and a mobile manipulator.

F.2 ABLATION ON LEARNING IN THE LOOP

  • Fine-tuning sustains high candidate yield in rounds 2 and 3, while the frozen tracker accepts almost nothing and its reference growth stalls.Both trackers have similar first-round yields; the frozen run stops after round 3. Figure 15 shows a tripod grasp executed on G1 with Dex3 and Inspire hands.Source: §F.2 · pp. 25–26, Table 11 · p. 26, Figure 15 · p. 26

F.3 SELF-EVOLUTION BY ROUND

  • Verified references grow in every setting and round, while all three quality measures rise in later rounds, most clearly foot sliding.Later rounds accept edits farther from captures and a different mix of object and body edits, so the trend also reflects harder motions.Source: Table 12 · p. 26, §F.3 · p. 26

F.4 BODY-MOTION RANGE OF AUGMENTED REFERENCES

  • Body edits span a region several times wider than ULTRA’s scene-scaling variants, mostly along directions that scaling does not vary.Figure 16 embeds body-joint-angle DCT coefficients relative to each seed and projects them with PCA; all seeds sit at the origin.Source: §F.4 · p. 26, Figure 16 · p. 27

G HARDWARE AND REAL-WORLD DEPLOYMENT

  • Hardware deployment spans Unitree G1 and Booster K1 humanoids and the Dexmate Vega wheeled mobile manipulator.G1 uses SONIC for whole-body control, while Vega sends retargeted upper-body trajectories to onboard joint controllers and converts base motion to wheel commands.Source: G HARDWARE AND REAL-WORLD DEPLOYMENT · pp. 27–28

H LIMITATIONS

The loop densifies coverage around demonstrated interactions but does not create new task semantics; iterative selection may favor easier objects or compound reference errors.

H LIMITATIONS

Related papers

Loading 2610.06850v1…