Source-linked AI summary
Object Motion Guided Human Motion Synthesis
Jiaman Li, Jiajun Wu, C. Karen Liu
TL;DR
The paper addresses full-body motion synthesis for manipulating large-sized objects while maintaining hand-object contact. OMOMO uses two conditional diffusion stages with hand positions as an intermediate representation, and the authors report effective synthesis that generalizes to unseen objects, alongside smartphone capture and a dataset.
Problem
Full-body manipulation synthesis must handle uncertain human motions and precisely maintain hand-object contact during interaction.
Method
OMOMO uses two conditional diffusion processes: one predicts hand positions from object motion and geometry, and the other synthesizes full-body poses from those positions.
Results
The authors demonstrate effective synthesis on their dataset and generalization to unseen objects.
Takeaways & Limitations
The framework supports realistic human motion generation from object motion, with smartphone-based capture and a dataset for human-object interaction research.
Takeaways & Limitations
The dataset inadequately represents dexterous hand movements, and the framework cannot effectively handle intermittent object contacts.
Abstract
from arXiv · showhide
Modeling human behaviors in contextual environments has a wide range of applications in character animation, embodied AI, VR/AR, and robotics. In real-world scenarios, humans frequently interact with the environment and manipulate various objects to complete daily tasks. In this work, we study the problem of full-body human motion synthesis for the manipulation of large-sized objects. We propose Object MOtion guided human MOtion synthesis (OMOMO), a conditional diffusion framework that can generate full-body manipulation behaviors from only the object motion. Since naively applying diffusion models fails to precisely enforce contact constraints between the hands and the object, OMOMO learns two separate denoising processes to first predict hand positions from object motion and subsequently synthesize full-body poses based on the predicted hand positions. By employing the hand positions as an intermediate representation between the two denoising processes, we can explicitly enforce contact constraints, resulting in more physically plausible manipulation motions. With the learned model, we develop a novel system that captures full-body human manipulation motions by simply attaching a smartphone to the object being manipulated. Through extensive experiments, we demonstrate the effectiveness of our proposed pipeline and its ability to generalize to unseen objects. Additionally, as high-quality human-object interaction datasets are scarce, we collect a large-scale dataset consisting of 3D object geometry, object motion, and human motion. Our dataset contains human-object interaction motion for 15 objects, with a total duration of approximately 10 hours.
1 INTRODUCTION
This work targets full-body human manipulation of large-sized objects in dynamic environments, introducing OMOMO, a dataset, and smartphone-based motion capture.
- Dynamic manipulation behaviors require full-body human motion synthesis for applications including embodied AI, character animation, VR/AR, and robotics.
- OMOMO uses conditional diffusion to synthesize full-body movements from object motion and object geometry.
- OMOMO predicts hand positions before synthesizing full-body poses, enabling explicit contact constraints between hands and objects.
- An iPhone-based application captures object motion and generates corresponding full-body human poses for everyday interactions.
- The work introduces a dataset pairing 3D object geometry, object motion, and full-body motion for human-object interaction.
2 RELATED WORK
Prior work spans human-scene and human-object motion datasets, contextual motion synthesis, reaching, and full-body manipulation. However, existing research has generally limited dynamic interaction with large objects, motivating synthesis for diverse objects in contextual environments.
- Human Motion and Interaction Datasets: Existing datasets pair human motion with scenes, video, or object motion, but their interaction coverage varies from static scenes to small or large objects.PROX, HPS, and EgoBody provide scene-related data, while GRAB focuses on small objects and BEHAVE records larger-object interactions.
- Human Motion and Interaction Datasets: SAMP and COUCH primarily collect motions interacting with static furniture, including sitting and lying down.Their object-interaction settings center on chairs and sofas rather than dynamic manipulation.
- Contextual Human Motion Synthesis: Contextual motion-synthesis methods generate full-body motion from scene information, planned paths, or action keypoints, with pipelines covering navigation and furniture interaction.Prior systems include root-trajectory and pose modules, collision-free path planning, and action-keypoint scene abstractions.
- Contextual Human Motion Synthesis: Reaching methods synthesize poses aimed at grasping objects, while other work jointly synthesizes human and object motion or models human-scene interaction.These approaches generally target specific objects and, as described here, do not address navigation in cluttered scenes.
- Full-body Manipulation: The paper distinguishes its focus from dexterous hand-motion synthesis and prior kinematic or physics-based full-body methods by targeting dynamic manipulation of diverse large-sized objects.The authors summarize prior work as rarely considering human interaction with large dynamic objects, with some full-body manipulation studies constrained to boxes.
3 METHOD
OMOMO synthesizes full-body human poses from object geometry through two conditional diffusion stages: hand positions are generated and contact-corrected before driving full-body pose synthesis. The method combines compact BPS object features, diffusion denoising, and explicit hand-object contact handling to model varied manipulation behaviors.
- Problem: The task is to generate full-body poses from time-varying object geometry while handling pose ambiguity and precise hand-object contact.The same object motion can correspond to different human movements, while naive diffusion does not precisely enforce contact.
- Data Representation: BPS encodes object geometry as compact fixed-length features that an MLP projects into 256-dimensional vectors.The representation samples 1024 points, computes nearest-mesh-vertex differences, appends object location, and projects the resulting features with an MLP.
- Hand Position Generation: Stage 1 uses conditional diffusion to generate left- and right-hand joint positions from sequences of object geometry features.A transformer denoising network conditions on object features and incorporates noise-level embeddings.
- Contact Constraints: A post-processing step enforces contact by detecting hand proximity below a threshold of 0.03 and propagating consistent hand-object offsets through later frames.The method uses the nearest object vertex and object rotations to update subsequent hand positions; both hands near the object indicate two-handed manipulation.
- Full-body Pose Generation: Stage 2 conditions the same denoising architecture on rectified hand positions to synthesize full-body human poses.This stage is trained using human motion data only.
- Pipeline: The complete pipeline models a one-to-many mapping from object motion to human poses while maintaining realistic object contact.The method first generates hand positions, applies contact constraints, and then produces full-body poses.
4 DATASET
The dataset pairs 3D object geometry with synchronized human and object motions to support full-body human manipulation research. It covers 15 everyday objects and uses separate object splits to evaluate generalization.
- Dataset construction: The dataset contains paired 3D object geometry, human motion, and object motion for full-body manipulation learning.The collection includes object geometry acquisition, motion capture, and data processing.
- Object geometry: 15 everyday objects were selected, reconstructed from monocular videos, manually denoised, and downsampled for training.Objects include furniture, tools, containers, and household equipment.
- Motion capture: Motion capture used 12 Vicon cameras at 120 FPS, five markers per object, and 17 participating subjects.Human and object motions were captured simultaneously during instructed interactions.
5 EXPERIMENT
Experiments compare OMOMO with adapted baselines and ablations using pose, physical-plausibility, qualitative, and perceptual evaluations. Results support the two-stage design and its contact constraints, including on unseen objects.
- Evaluation protocol: The evaluation measures pose errors against ground truth together with contact correctness, object penetration, and foot sliding.Metrics include HandJPE, MPJPE, MPVPE, root errors, collision percentage, and contact precision, recall, and F1.
- Baselines and splits: OMOMO is compared with adapted GOAL and OMOMO-single-stage across 15-object and 5-unseen-object test splits.The conditional diffusion model is sampled 20 times per input for quantitative comparison.
- Metric interpretation: Baseline GOAL’s lower collision percentage is attributed to characters often not attempting to manipulate the object.Its foot positions may also drift above the floor without being counted as foot sliding.
- Quantitative results: OMOMO outperforms GOAL and OMOMO-single-stage, with superior contact evaluations among the OMOMO variants.The reported comparisons cover both evaluation settings.
- Perceptual study: 31% of perceptual-study preferences favored OMOMO over the compared sequences, while ground truth remained preferred in some cases.The study sampled 100 sequences per approach across all 15 objects and used 20 workers per question.
- Ablation study: Ground truth hand positions further improve synthesis results relative to predicted hand positions in the ablation study.This comparison directly tests the effect of hand-position accuracy on full-body pose generation.
6 APPLICATION
The application captures object motion with an iPhone attached to the object and generates human motion for reconstructed unseen objects. It demonstrates smartphone-based interaction capture while exposing a contact limitation.
- Motion capture: An iPhone XR mounted on the target object uses ARKit camera poses to derive object motion during interaction.Visual-inertial odometry combines camera imagery and sensor information.
- Limitation: The contact constraint cannot generate motions involving intermittent object contacts and may produce implausible penetration when hand positions are fixed.The limitation is illustrated by generated results compared with corresponding ground truth motion.
- Motion synthesis: Object motion is combined with Luma-reconstructed 3D geometry and passed through OMOMO to generate full-body human poses.The showcased objects were not used during model training.
7 CONCLUSION
The paper concludes that OMOMO synthesizes moving-object-guided human motion through a two-stage contact-constrained framework, complemented by smartphone capture and a large paired dataset. Its main limitations concern dexterous hand realism, intermittent contacts, and kinematic artifacts.
- Contributions: The two-stage paradigm enforces contact constraints while generating realistic full-body human motion for interactions with moving objects.The conclusion presents this as the central synthesis approach.
- Application: A smartphone-only application captures human interaction motion by attaching the phone to the manipulated object.The system uses object motion to support the generated interaction motion.
- Dataset: The introduced dataset provides 3D object geometry, high-quality object motion, and human motion for human-object interaction research.The dataset is intended to facilitate further work on these interactions.
- Limitations: The dataset inadequately represents dexterous hand movements, which can yield implausible hand motions.The authors suggest hand priors and optimization as possible improvements.
- Limitations: The contact constraints do not effectively handle intermittent object contacts, while the kinematic method can produce artifacts.The authors point to contact-state prediction and physics-based components as future directions.