Source-linked AI summary
ContactMimic: Humanoid Object Interaction via Contact Control
Xinyao Li, Xialin He, Runpei Dong, Saurabh Gupta
TL;DR
Keypoint tracking alone does not specify the meaningful physical contacts required by many humanoid object-interaction tasks. CONTACTMIMIC trains a policy on keypoint trajectories and explicit contact commands, using contact-aware rewards and augmented motion pairs. The policy controls contact across simulated and real motions, outperforms a keypoint-only baseline on contact metrics at comparable tracking accuracy, and supports manipulation without task-specific rewards.
Problem
Keypoint trajectories alone incompletely specify object-interaction tasks, while current humanoid trackers do not represent the contacts the robot should make.
Source: §1 · p. 2Method
CONTACTMIMIC conditions keypoint tracking on explicit binary body-contact commands and trains with contact-aware rewards and motion-pair augmentation that breaks keypoint-contact correlations.
Source: §1 · p. 2Results
Takeaways & Limitations
Explicit contact conditioning provides a controllable interface for contact-rich tasks, with runtime contact state encoded from proprioception alone.
Source: §5 · p. 8The approach trains a separate policy per motion, relies on HUMOTO data that limits interaction diversity, and has been evaluated in the real world on five motions using one robot.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Keypoint tracking alone is insufficient for object interaction tasks such as sitting on a chair, wiping a board, or pushing furniture, where the robot can reach the correct pose without making meaningful physical contact with the object. We present CONTACTMIMIC, a learning framework that tracks explicit partlevel binary contact commands alongside keypoint trajectories. CONTACTMIMIC is made possible through the use of contact-following rewards and a trajectory augmentation scheme aimed at breaking the correlations between keypoint trajectories and contact labels. The resulting policy successfully decouples contact behavior from keypoint geometry, and achieves precise physical contact as well as contact-controllability (produce or suppress contact during deployment as desired). Simulation experiments across 10 diverse human-object interaction motions confirm that CONTACTMIMIC exhibits contact controllability that enables it to complete manipulation tasks without task-specific rewards, while also outperforming keypoint-only trackers on contact-relevant tasks. Ablations confirm the necessity of the proposed trajectory augmentation scheme and sim2real deployment validates contact controllability in the real world across 5 different motions. Video results are available on https://lixinyao11.github.io/contactmimic-page/.
1 Introduction
Keypoint-only tracking can reproduce motion shape while missing the contacts that make object interactions useful. CONTACTMIMIC conditions tracking on per-body contact commands and trains the policy to follow them, enabling contact to be controlled independently of keypoint geometry.
1 Introduction
- Contact-conditioned tracking addresses interactions where success depends on which body parts contact objects and when, not just on keypoint trajectories.Keypoint-only trackers may miss task-relevant contacts and require task-specific reward fine-tuning.Source: §1 · p. 2
- Augmented motion pairs with similar keypoints but different contact patterns encourage the policy to attend to contact commands instead of inferring contact from geometry.Raw human-object interaction data often correlates keypoints and contact patterns strongly.Source: §1 · p. 2
- CONTACTMIMIC injects contact commands into the policy and rewards matching reference contacts while guiding body parts toward or away from object surfaces.The pipeline extracts keypoint trajectories and per-body contact labels, augments motion pairs, and trains a contact-conditioned policy (Figure 2).Source: §1 · p. 2, Figure 2 · p. 3
- Across simulation and real-world experiments, changing the contact command under the same keypoint trajectory changes behavior, including whether the robot leans on a chair backrest.Compared with BeyondMimic [1], CONTACTMIMIC achieves more consistent task-relevant contacts and enables manipulation without task-specific rewards; proprioception provides cues about runtime contact state.Source: §1 · pp. 2–3
2 Related Work
Prior humanoid trackers and loco-manipulation systems largely treat contact as incidental to geometry or as a fixed planning choice. CONTACTMIMIC instead exposes contact as a runtime-controllable policy input.
2 Related Work
- Humanoid trackers imitate kinematic targets, while loco-manipulation systems add teleoperation, vision, task-specific rewards, or force adaptation; contact remains incidental to matching geometry.This limits performance on tasks distinguished by contact, such as wiping a board versus waving above it.Source: §2 · p. 3
- Existing HOI retargeting methods preserve interaction structure, but pipelines generally provide one canonical trajectory per clip, limiting data for decoupling contact from keypoint geometry.Naive joint-angle retargeting can cause foot-skating, penetration, or hands floating above intended surfaces.Source: §2 · p. 3
- Prior contact-aware planning and synthesis either fix contact schedules at planning time or use contact as an intermediate representation, rather than a runtime-controllable policy input.ResMimic adds a contact-tracking reward but conditions on motion and object trajectories, not fine-grained contact commands.Source: §2 · p. 3
3 Method
CONTACTMIMIC trains a humanoid tracker on reference keypoints and binary body-contact commands. Its method combines contact-aware rewards with augmented motion pairs designed to separate contact labels from keypoint geometry.
3 Method
- The policy takes proprioception, reference keypoint targets, and binary contact labels for contact-capable robot bodies, then outputs target joint angles for PD control.The method preprocesses and retargets human-object reference data; its technical contributions are contact-aware rewards and data generation for genuine contact-conditioned behavior.Source: §3 · p. 4
3.1 Reward Design
- Contact-label matching rewards agreement between reference and actual body-part/object-part contacts, using TPR and TNR or TPR and FPR for sparse-contact motions.Contact-distance rewards encourage intended part-pairs to be close and unintended pairs to be far from object surfaces.Source: §3.1 · p. 4
3.2 Generating Motion Pairs to Break Correlations between Keypoints and Contacts
- Augmentations break correlations between keypoint positions and task-relevant contacts by flipping labels, removing objects, or inflating target-object geometry.Label flipping retains the trajectory but penalizes contact; object removal retains keypoints while zeroing contact labels; inflated geometry perturbs keypoints and zeroes the corresponding labels. These augmentations can be composed.Source: §3.2 · pp. 4–5
4 Experiments
Across simulation, hardware, baseline comparisons, and ablations, the experiments test whether CONTACTMIMIC follows explicit contact commands and whether keypoint-only tracking suffices. They also examine augmentation and whether policy representations encode runtime contact state.
4 Experiments
- The experiments address command following, hardware transfer, keypoint-only sufficiency, augmentation necessity, and runtime contact-state representation.Source: §4 · p. 5
4.1 Experimental Setup
- Experiments use a Unitree G1 in Isaac Lab and ten HUMOTO motions, with contact, impulse, torque, MPJPE, and—when objects move—object displacement as metrics.Table 1 identifies each motion’s task-relevant robot-body/object-part contact pair; all but the final two motions use fixed objects.Source: §4.1 · p. 5, Table 1 · p. 5
4.2 Contact Control Works in Simulation
- Contact counts and impulses decrease when the command is turned off across near and augmented far-keypoint trajectories, while contact-relevant torque changes as expected.For box lifting and chair kicking, object displacement is greater when contact is commanded; keypoint-plus-contact tracking achieves this without task-specific rewards.Source: §4.2 · p. 6, Figure 4 · p. 6
4.3 Contact Control Transfers to the Real World
- On the real Unitree G1, the policy follows contact commands across five motions, making contact when commanded and staying off the surface when contact is suppressed.Examples include sustained hand pressure while wiping and transferring weight onto a seat when sitting.Source: §4.3 · pp. 6–7, Table 2 · p. 7
4.4 Keypoint Control Alone Is Insufficient
- BeyondMimic [1] has lower contact counts and impulses despite comparable MPJPE, while CONTACTMIMIC succeeds at object manipulation with similar keypoint errors.The comparison indicates that keypoint-only control fails to manipulate objects in these tasks.Source: §4.4 · p. 7, Table 3 · p. 7
4.5 Our Proposed Data Augmentation Scheme Is Important
- With augmentation, contact control is better than without it for all but one motion, indicating that policy inputs and rewards alone are insufficient to learn contact control.The training data must be engineered to break correlations between keypoint positions and contact labels.Source: §4.5 · pp. 7–8, Figure 5 · p. 8
4.6 Policy Representations Encode Runtime Contact State
- Linear probes on policy observations and layer-2 representations predict runtime contact state with F1 scores well above chance and the reference label.The policy receives no explicit contact sensing at inference time.Source: §4.6 · p. 8, Table 4 · p. 8
5 Discussion and Limitations
CONTACTMIMIC combines keypoint tracking with a test-time controllable contact label. The paper reports limitations in policy scope, training data diversity, and real-world evaluation.
5 Discussion and Limitations
- The current approach trains a separate policy per motion rather than one universal contact-conditioned tracker.Source: §5 · p. 8
- Use of high-quality HUMOTO interaction data limits the diversity of interactions currently covered.Source: §5 · p. 8
- Real-world evaluation covers five motions on a single robot, leaving broader hardware validation for future work.Source: §5 · p. 8
APPENDIX
The appendix points readers to supplementary material on presentation, implementation, training, deployment, and additional results.
APPENDIX
- Appendix A describes the overview video and project page, while Appendix B gives PPO hyperparameters, policy observations and architecture, and domain randomization details.Source: APPENDIX · p. 12
- Appendices C and D provide reward-term settings and details of paired-motion generation, including contact-label extraction, augmentation parameters, and per-motion training configuration.Source: APPENDIX · p. 12
- Appendices E and F cover Unitree G1 deployment and additional quantitative results, including per-motion controllability and augmentation-ablation results.Source: APPENDIX · p. 12
A Overview Video and Project Page
The project page provides supplementary videos and raw hardware recordings, alongside method, simulation, baseline, and ablation comparisons.
A Overview Video and Project Page
- The page includes the pipeline overview, from retargeting and paired-motion augmentation through contact-conditioned training and real-world deployment.Source: A Overview Video and Project Page · p. 12
- Real-world clips show five motions on the Unitree G1 with the same policy toggling contact commands, while simulation clips cover ten motions and four evaluation-time trajectory conditions.Source: A Overview Video and Project Page · p. 12
- The page includes comparisons with BeyondMimic [1] and a no-augmentation variant, illustrating test-time command control and contact-controlled task outcomes.Source: A Overview Video and Project Page · p. 12
B Implementation Details
Each motion is trained as a separate PPO policy using reference keypoints and contact labels alongside robot state. Training randomizes physical properties and includes augmented trajectories.
B Implementation Details
- Each motion has its own PPO policy, trained in 4096 parallel environments at 50 Hz with resets randomly selecting the default or an augmented trajectory.This exposes the policy to both contact and no-contact target labels during training.Source: B Implementation Details · p. 12
- The actor observes proprioception, reference keypoint targets, and a binary reference contact-label map.The critic additionally receives base linear velocity and noise-free actor observations as privileged information.Source: B Implementation Details · p. 12
- PPO hyperparameters for training each contact-conditioned policy are listed in Table 5.Source: Table 5 · p. 13
- Robot link masses, joint friction, and object friction and mass are randomized at each episode start to facilitate sim-to-real transfer.Source: B Implementation Details · p. 13
C Reward Details
The per-step reward combines motion tracking, contact-aware objectives, and regularization. Contact rewards measure label agreement and guide relevant body parts toward or away from object surfaces.
C Reward Details
- The reward combines tracking, contact-aware, and regularization terms; tracking terms follow BeyondMimic [1] and use Gaussian kernels on error signals.Table 6 lists reward terms, functional forms, weights, and tunable shaping parameters.Source: C Reward Details · p. 13, Table 6 · p. 13
- The contact label-matching reward defaults to balanced accuracy, but sparse-contact motions use TPR − λFPR with λ = 1.0 to strengthen gradients when most reference labels are zero.Source: C Reward Details · p. 13
D Paired Motion Generation Details
The paper generates contact labels and paired motion variants to train contact-conditioned policies. Real-world deployment uses the onboard policy with a pre-recorded reference and commanded contact labels.
D Paired Motion Generation Details
- Retargeted trajectories are replayed in MuJoCo, labeling a robot body part as contacting an object when it is within 1 cm of the surface.The semantic object part is assigned from the nearest surface point.Source: D Paired Motion Generation Details · p. 13
- Three augmentation strategies include inflated target geometry, random contact-label flipping, and random interaction-object removal.Inflated geometry expands target collision surfaces by 5–10 cm during a second retargeting pass and zeros the corresponding contact labels; per-motion probabilities vary.Source: D Paired Motion Generation Details · p. 14, Table 7 · p. 14
- Augmentation settings and label-matching reward modes vary by motion; sit-and-squat replaces inflated geometry with chair removal, while sit-on-table omits object removal.For sit-and-squat, the no-contact variant is a squat at the same height; removing the table for sit-on-table would cause failure.Source: D Paired Motion Generation Details · p. 14
E Real-World Deployment
- The G1 policy runs onboard at 50 Hz, with target joint positions tracked by a 1000 Hz joint-level PD controller.At deployment, object placement is measured by hand; the policy uses onboard proprioception, the pre-recorded reference, and the contact-label command, without external motion capture or vision.Source: E Real-World Deployment · p. 14
F Additional Quantitative Results
Across interaction motions, contact bodies and impulse generally follow the commanded contact label rather than keypoint variant. The results also show motion-specific limits, while the augmentation ablation indicates that paired-motion augmentation is important for learning contact control.
Per-motion takeaways
- On the sit-on-table motion, contact-off drives contact bodies and impulse near zero, while contact-on restores them to the reference level.Source: F Additional Quantitative Results · p. 14
- For step-foot-on-chair, contact bodies remain near 0.6 because balance requires chair contact, but impulse still changes with the command.The reported near-keypoint impulses are 1.16 N·s for contact-off and 1.62 N·s for contact-on.Source: F Additional Quantitative Results · p. 15
- For lean-on-backrest I and whiteboard wiping, contact-off leaves residual contact because avoiding it would incur keypoint error that outweighs the contact-aware penalty.Impulse nevertheless drops by a factor of 3–4×.Source: F Additional Quantitative Results · p. 15
Per-motion contact controllability
- With the pickup trajectory’s near keypoints, contact-off barely displaces the box (0.20), whereas contact-on produces the largest displacement (0.49).This is presented as an example of contact conditioning enabling manipulation that keypoints alone cannot produce.Source: F Additional Quantitative Results · p. 15
Per-motion augmentation takeaways
- Augmentation reduces unwanted box grabbing under contact-off from 1.95 contact bodies and 1.85 N·s impulse to 0.29 and 0.17, respectively.Step-foot-on-chair and kick-chair show little ablation effect because the keypoint motion forces contact or leaves little contact decision to learn.Source: F Additional Quantitative Results · p. 15
Data augmentation ablation
- Without augmentation, contact metrics are largely insensitive to the contact label, indicating that the policy ignores the conditioning signal.Table 9 and Figure 6 report the ablation; the figure contrasts the full method with the no-augmentation baseline.Source: F Additional Quantitative Results · p. 15, Table 9 · p. 16, Figure 6 · p. 16
Per-motion contact controllability
- Across motions, contact bodies and impulse track the commanded contact label rather than the keypoint variant.Table 8 reports contact-label controllability across all interaction motions; the arrow annotations indicate expected torque changes when contact is turned off.Source: F Additional Quantitative Results · pp. 14, 17, Table 8 · p. 17