PaperlayerView exact revision on arXiv

Source-linked AI summary

LessMimic: Long-Horizon Humanoid Interaction with Unified Distance Field Representations

Yutang Lin, Jieming Cui, Yixuan Li, Baoxiong Jia, Yixin Zhu, Siyuan Huang

arXiv:2602.21723v1cs.RO

TL;DR

Humanoid interaction still lacks a unified representation that supports reference-free inference, geometric generalization, and long-horizon skill composition. LESSMIMIC uses Distance Field cues to train a single whole-body policy, achieving strong results across object scales and composed tasks.

  • Problem

    Existing approaches rely on reference motions or task-specific rewards, while a unified representation for reference-free inference, geometric generalization, and long-horizon skill composition remains an open challenge.

    Source: §I · p. 2

  • Method

    LESSMIMIC encodes DF-derived geometric and velocity cues in an interaction latent, trains a single whole-body policy with behavior cloning and AIP-guided RL, then distills it for vision-only deployment.

    Source: §I · p. 2

  • Results

    80–100% success on PickUp and SitStand across object scales from 0.4ˆ to 1.6ˆ; 62.1% success on 5-task trajectories, with viability up to 40 sequential task instances.

    Source: §I · p. 2

  • Takeaways & Limitations

    LESSMIMIC demonstrates a single reference-free policy for generalizable, long-horizon humanoid interaction across object scales and composed skills.

    Source: §V · p. 8

Abstract

from arXiv · show

Humanoid robots that autonomously interact with physical environments over extended horizons represent a central goal of embodied intelligence. Existing approaches rely on reference motions or task-specific rewards, tightly coupling policies to particular object geometries and precluding multi-skill generalization within a single framework. A unified interaction representation enabling reference-free inference, geometric generalization, and long-horizon skill composition within one policy remains an open challenge. Here we show that Distance Field (DF) provides such a representation: LessMimic conditions a single whole-body policy on DF-derived geometric cues--surface distances, gradients, and velocity decompositions--removing the need for motion references, with interaction latents encoded via a Variational Auto-Encoder (VAE) and post-trained using Adversarial Interaction Priors (AIP) under Reinforcement Learning (RL). Through DAgger-style distillation that aligns DF latents with egocentric depth features, LessMimic further transfers seamlessly to vision-only deployment without motion capture (MoCap) infrastructure. A single LessMimic policy achieves 80--100% success across object scales from 0.4x to 1.6x on PickUp and SitStand where baselines degrade sharply, attains 62.1% success on 5 task instances trajectories, and remains viable up to 40 sequentially composed tasks. By grounding interaction in local geometry rather than demonstrations, LessMimic offers a scalable path toward humanoid robots that generalize, compose skills, and recover from failures in unstructured environments.

I. INTRODUCTION

LESSMIMIC addresses the lack of a unified, geometry-aware interaction representation with a reference-free policy grounded in Distance Fields. It combines DF-derived cues with a staged training pipeline to support generalization and long-horizon skill composition.

  • Existing reference-based policies couple object geometry to reference trajectories, while reference-free methods without principled interaction signals rely on task-specific rewards and produce isolated policies.Source: §I · p. 2
  • Distance Fields provide continuous, differentiable surface distances and gradients, with local cues largely invariant to object size and approach direction and velocity components encoding ongoing contact.Source: §I · p. 2
  • LESSMIMIC encodes history-dependent, velocity-decomposed DF interaction features in a VAE latent and trains one whole-body policy through behavior cloning, AIP-guided RL, and visual distillation.At inference, it uses a target root trajectory and current DF observation without motion references, task-specific rewards, or separate planners.Source: §I · p. 2
  • 80–100% success on PickUp and SitStand across object scales from 0.4ˆ to 1.6ˆ; 62.1% success on 5-task trajectories, with viability up to 40 task instances.Both reference-based and reference-free baselines degrade sharply on PickUp and SitStand across scales.Source: §I · p. 2

II. RELATED WORK

Prior approaches trade interaction fidelity against autonomy and geometric or multi-skill generalization. LessMimic uses a unified Distance Field representation within a reference-free framework for humanoid interaction.

A. Reference-based Humanoid-Object Interaction

  • Reference-conditioned policies use reference motions as explicit supervision to stabilize learning and produce physically plausible behaviors, but object-property deviations from training data invalidate prescribed motions and cause brittleness on novel objects [38] [37].Teleoperation and human-in-the-loop guidance improve robustness under object and contact variation, but add supervision overhead and constrain policies to observed-motion topology [5].Source: A. Reference-based Humanoid-Object… · p. 3

B. Reference-free Humanoid-Object Interaction

  • Model-based and optimization methods avoid prerecorded motions but rely on accurate models and short planning horizons, while learned reference-free methods often remain task-specific and do not compose skills over extended horizons.The lack of a unified interaction representation limits robust multi-skill behavior within one policy.Source: B. Reference-free Humanoid-Object… · p. 3

C. Representation for Human-Object Interaction

  • Voxel and occupancy methods are costly, point- and mesh-based representations lack continuous distance information, and implicit neural representations can be too slow for high-frequency control.Distance Fields provide continuous, differentiable geometry and analytical gradients at negligible query cost for contact-aware control.Source: C. Representation for Human-Object… · p. 3

III. THE LESSMIMIC FRAMEWORK

  • LessMimic encodes DF-derived interaction features into a VAE latent and trains a whole-body policy through interaction-skill pre-training and discriminative post-training.The framework uses MoCap or depth-image observations and deploys actions through a whole-body controller.Source: §III · pp. 3–4, Figure 2 · p. 3, Figure 3 · p. 4

A. DF-based Interaction Representation

LessMimic represents interaction through local Distance Field geometry and motion relative to object surfaces. A temporal history of these cues is encoded into a compact latent for policy learning.

A. DF-based Interaction Representation

  • Local DF relationships describe contact and relative motion rather than global coordinates, supporting reasoning invariant to object shape and scale.This representation captures interaction structure instead of memorizing absolute trajectories.Source: A. DF-based Interaction Representation · p. 4
  • A Distance Field gives each point’s distance to an object surface, while its gradient supplies the local surface orientation for contact-aware control.The field acts as a geometry-anchored local reference frame.Source: A. DF-based Interaction Representation · p. 4
  • Velocity is decomposed into normal motion for approach or force application and tangential motion for sliding or surface traversal.These components express motion relative to the object’s local surface.Source: A. DF-based Interaction Representation · p. 4
  • A temporal window of per-link geometric features forms the interaction representation, which is encoded by a VAE into a smooth latent to improve sensor-noise robustness and training convergence.The features include DF distance, gradient, and velocity components.Source: A. DF-based Interaction Representation · pp. 4–5
  • During sitting, mean DF distance decreases as mean gradient magnitude rises near chair contact, with dashed lines marking interaction-phase transitions.The plotted distance curve includes the joint-wise range as a shaded region.Source: Figure 4 · p. 5

B. Interaction Skill Pre-Training

Interaction-skill pre-training uses physically feasible teacher-generated trajectories to initialize a policy for the inference-time observation setting. DAgger-style correction addresses rollout covariate shift.

B. Interaction Skill Pre-Training

  • A physics-tracking mimic policy generates feasible state-action trajectories from retargeted human motions, with a residual module compensating for dynamic mismatches during object interaction.The paper notes that purely retargeted MoCap data can violate simulation physics, while collecting high-quality interaction data at scale is costly.Source: B. Interaction Skill Pre-Training · p. 5
  • The base policy is behavior-cloned from mimic-policy data using proprioception, a sparse root-trajectory command, and the DF interaction latent rather than privileged reference motions at inference.DAgger rollouts query the teacher for corrective actions to mitigate covariate shift; training minimizes action MSE.Source: B. Interaction Skill Pre-Training · p. 5
  • Pre-training provides a stable initialization that grounds the DF interaction representation in whole-body control for later geometric generalization.The method uses DAgger-style correction between the base policy’s rollouts and teacher actions.Source: B. Interaction Skill Pre-Training · p. 5

C. Discriminative Post-Training

Discriminative post-training fine-tunes the pre-trained policy with RL across randomized object geometries. AIP supplies geometric interaction supervision without motion-tracking rewards or geometry-specific shaping terms.

C. Discriminative Post-Training

  • RL fine-tuning randomizes object scale, shape, and surface properties because fixed-object pre-training can encourage memorized kinematic trajectories.Motion-tracking rewards and hand-crafted shaping terms are excluded under procedural object variation.Source: C. Discriminative Post-Training · p. 5
  • AIP trains a discriminator to distinguish policy interaction latents on novel objects from latents in a reference interaction buffer, regularizing interaction validity in the geometric domain.The discriminator uses a least-squares GAN objective.Source: C. Discriminative Post-Training · p. 5
  • The policy maximizes a composite reward combining root-command tracking, discriminator-derived interaction reward, and AMP-style motion-naturalness reward.The interaction reward is derived from the AIP discriminator output.Source: C. Discriminative Post-Training · p. 5
  • Conditioning the discriminator on the DF latent rather than full robot state encourages valid geometric interaction without fixing a kinematic template, allowing novel poses on unseen geometries.The resulting policy is denoted πfull.Source: C. Discriminative Post-Training · p. 5

D. Visual-Motor Policy Distillation

LESSMIMIC is distilled from a MoCap-dependent full policy to a policy using egocentric depth, with domain randomization intended to support transfer. The evaluation spans four interaction tasks and includes real-world examples of generalization to an unseen object shape and varied chair heights.

D. Visual-Motor Policy Distillation

  • The vision policy uses a history of egocentric depth frames to recover geometric cues as a compact latent, while retaining the full policy’s control head.The student is trained by DAgger-style distillation from the frozen full-policy teacher, which supplies action supervision at encountered states.Source: D. Visual-Motor Policy Distillation · p. 6
  • Domain randomization perturbs camera extrinsics, depth observations, and physical properties to support robustness to sensor noise and modeling discrepancies.The stated goal is reliable execution of contact-rich interactions from onboard perception alone.Source: D. Visual-Motor Policy Distillation · p. 6

IV. EXPERIMENTS

  • The evaluation compares MoCap-based and closed-loop visual-motor policies against reference-based and reference-free baselines across four interaction tasks.The tasks are PickUp, SitStand, Push, and Carry; evaluation uses task-specific geometric and contact criteria.Source: §IV · p. 6, A. Experimental Setup · p. 6

A. Experimental Setup

  • In real-world examples, the policy generalizes from a training box to an unseen soccer ball and maintains stable pelvis contact across 12 cm and 46 cm chairs.These examples illustrate shape generalization and SitStand performance across seat geometries.Source: Figure 5 · p. 7

B. Generalization on Object Sizes

LESSMIMIC is evaluated for object-scale generalization and long-horizon skill composition. It remains robust across scale variations and achieves non-zero success on sequences of up to 40 tasks, while ablated variants collapse beyond short sequences.

B. Generalization on Object Sizes

  • Scale generalization tests objects from 0.4x to 1.6x the training scale, with PickUp also varying among box, spherical, and cylindrical shapes.The setting probes whether policies adapt to local geometric relations or rely on memorized motion patterns.Source: B. Generalization on Object Sizes · p. 7

C. Long-Horizon Skill Composition

  • 62.1% success on 5-task trajectories and 2.1% viability at 40 tasks; all ablated variants collapse to zero success beyond short sequences.The tasks are randomly composed and executed under one policy without environment resets; the paper attributes implicit transitions to changing geometric contexts.Source: Table III · p. 7, C. Long-Horizon Skill Composition · p. 7

B. Generalization on Object Sizes

  • LESSMIMIC maintains robust success across scale variation on four tasks, while reference-based methods degrade sharply away from the reference scale.PickUp success remains above 90% across most tested scales, and LESSMIMIC outperforms the strongest baselines by 15–20% on SitStand and Push at larger object sizes.Source: B. Generalization on Object Sizes · p. 7

D. Distilling LESSMIMIC to Visual Input

The vision-based policy retains the full policy’s performance trends but shows lower success, while ablations expose contributions from the interaction prior, physicalization, randomization, RL, and Transformer backbone. Real-world tests report successful MoCap-based PickUp and SitStand trials, and the conclusion identifies articulated and deformable objects as future work.

D. Distilling LESSMIMIC to Visual Input

  • The vision-based policy achieves 63.7–99.7% PickUp success at scales where all baseline methods exhibit near-zero performance, though performance is reduced from the MoCap-based model.SitStand is excluded from vision evaluation because back-side contacts have limited egocentric observability; the paper attributes moderate reductions to depth-perception uncertainty.Source: D. Distilling LESSMIMIC to Visual Input · pp. 7–8

E. Ablation Study

  • Removing AIP reduces scale robustness, synthetic physicalization most affects contact-rich Carry, and disabling geometry randomization causes near-zero success outside training scales.Removing RL fine-tuning yields limited performance and poor generalization, while replacing the Transformer with an MLP degrades PickUp and Carry performance.Source: E. Ablation Study · p. 8

F. Real-World Deployment

  • The MoCap-based model achieves 10/10 success on PickUp (22 cm3) and SitStand (46 cm), with root tracking accuracy above 90% in both cases.The vision-based variant achieves 8/10 and 7/10 on the two PickUp conditions, and is not evaluated on SitStand due to limited observability of back-side contacts.Source: F. Real-World Deployment · p. 8, Table IV · p. 8

V. CONCLUSION

  • The conclusion proposes extending DF-based representation to articulated and deformable objects and improving robustness under partial observability.It reports simulation success of 80–100% on PickUp and SitStand from 0.4x to 1.6x scales, plus 62.1% success on 5-task trajectories and viability up to 40 sequential tasks.Source: §V · p. 8

APPENDIX A THE LESSMIMIC FRAMEWORK

The appendix details a three-stage training pipeline and shared policy architecture. Interaction skill pre-training uses teacher supervision while keeping reference motions out of the student’s observations, and encodes DF interaction history with a VAE.

APPENDIX A THE LESSMIMIC FRAMEWORK

  • The appendix documents network architectures, training procedures, reward formulations, and shared policy architecture across three stages with different optimization configurations.Training hyperparameters are summarized in Table A1.Source: Appendix A · p. 12, Table A1 · p. 12

A. Interaction Skill Pre-Training

  • The student policy is behavior-cloned from a ResMimic teacher that uses privileged reference information to generate physically valid interaction trajectories.The student itself excludes reference motion information, so its behavior is grounded in the DF-based interaction latent rather than reference cues.Source: A. Interaction Skill Pre-Training · pp. 12–13
  • A Transformer models temporal dependencies across interaction history, and the same architecture is retained through pre-training, post-training, and visual distillation.Architectural consistency allows pre-training weights to carry over without modification.Source: A. Interaction Skill Pre-Training · p. 12
  • The interaction-history VAE encodes each DF feature into a compact latent sequence, smoothing raw-measurement noise and yielding fixed-dimensional policy input.The history captures recent geometric evolution over a temporal window.Source: A. Interaction Skill Pre-Training · p. 12
  • Pre-training minimizes the action discrepancy between the student and teacher without a task-specific reward, using on-policy rollouts and corrective teacher labels to address covariate shift.The DAgger-style collection exposes the student to states encountered during its own execution.Source: A. Interaction Skill Pre-Training · p. 13

B. Discriminative Post-Training

Discriminative post-training replaces motion-reference supervision with reinforcement learning and geometry-aware adversarial guidance across procedurally varied conditions. It separates geometrically valid interaction patterns from full-body motion naturalness.

B. Discriminative Post-Training

  • Post-training targets robustness to novel object geometries by combining procedural variation with geometry-aware adversarial supervision rather than fixed-dataset behavior cloning.The authors argue that behavior cloning can encourage memorization of specific kinematic trajectories instead of underlying interaction rules.Source: B. Discriminative Post-Training · p. 13
  • Training randomizes object geometries, scales, and physical properties, removes reference motions and motion-tracking losses, and optimizes the policy with reinforcement learning.The post-training setup preserves the pre-training architecture, observation structure, and DF-based representation.Source: B. Discriminative Post-Training · pp. 13–14
  • AIP discriminates using DF-based interaction latents, encouraging valid approach-contact-release patterns in local geometric coordinates without encoding the kinematic template.Its reference samples come from the pre-training interaction buffer, and the least-squares GAN formulation is chosen for more stable gradient behavior.Source: B. Discriminative Post-Training · pp. 13–14
  • The composite reinforcement-learning reward combines task tracking, AIP-derived interaction quality, and AMP-derived motion style to balance valid contact patterns with natural motion.The interaction and motion-style terms operate on different inputs: the geometric interaction latent and the full robot state, respectively.Source: B. Discriminative Post-Training · p. 14

APPENDIX B EXPERIMENTS

Appendix B details experimental randomization and reward design. Randomization spans interaction conditions, while the reward combines task and interaction objectives with regularizers for stable, safe execution.

APPENDIX B EXPERIMENTS

  • Appendix B provides supplementary experimental details on domain randomization, reward design, baseline implementations, and evaluation metrics.Source: Appendix B · p. 14

A. Domain Randomization

  • Domain randomization covers object geometry, physical properties, initial conditions, command perturbations, actuation noise, and perceptual inputs for vision-based distillation.Perceptual augmentation includes camera-pose jitter, depth quantization, dropout, sensor noise, and camera-extrinsic randomization.Source: A. Domain Randomization · p. 14

B. Reward Design

  • The reward design pairs task-completion and interaction-quality rewards with auxiliary regularizers for training stability and safe long-horizon execution.Primary terms include root tracking, AIP interaction style, AMP motion style, and object tracking; regularizers address abrupt actions, falls, and excessive joint excursions.Source: B. Reward Design · pp. 14–15

C. Baselines

The baselines share the simulator, control frequency, and low-level controller, but differ in reference use and task-specific design. Evaluation uses task success, Push contact rate, and real-world root tracking accuracy.

C. Baselines

  • All baselines use the same simulator, control frequency, task definitions, object configurations, and termination conditions as LESSMIMIC, but lack its DF-based representation.Baseline results are averaged over three random seeds under identical evaluation protocols.Source: C. Baselines · p. 15
  • HDMI and ResMimic track task-specific motion references, while VisualMimic adds egocentric observations but still tracks reference motions.The references are unmodified under object-scale variation; VisualMimic uses the authors’ official pre-trained checkpoints.Source: C. Baselines · p. 15
  • PhysHSI is reference-free and uses handcrafted humanoid-state and object-relative features, with task-specific rewards and separate designs that do not directly support the evaluated multi-task setting.Source: C. Baselines · p. 15

D. Evaluation Metrics

  • Evaluation reports task-specific success criteria, continuous end-effector contact rate for Push, and root trajectory tracking accuracy for real-world deployment.Root tracking accuracy is the proportion of time steps within 0.6 m of the commanded trajectory.Source: D. Evaluation Metrics · pp. 15–16
Loading 2602.21723v1…