Source-linked AI summary
IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, Philipp Slusallek
TL;DR
IMoS addresses the limited synthesis of plausible full-body human-object interactions from simple textual instructions. It uses decoupled variational autoregressors for body parts and optimizes object poses, establishing improved performance and perceptual quality while remaining limited for dynamic grasping and long-term action sequences.
Problem
Existing full-body motion synthesis generally omits object interactions, while human-object methods mainly address hand or finger grasping rather than post-grasp full-body actions.
Method
IMoS encodes instruction-object pairs and uses separate conditional variational autoregressors for the arms and body, followed by 6-DoF object-pose optimization.
Results
IMoS synthesizes plausible intended actions, advances the state of the art on multiple quantitative evaluations, and improves perceptual results through decoupled arm and body modeling.
Takeaways & Limitations
The framework provides a practical pipeline for large-scale character animation involving diverse instructed object interactions, including both-handed exchanges.
Takeaways & Limitations
The method is limited for rare high-frequency wrist intents, dynamic grasping with hand-object relative motion, imprecise contacts, and minute-scale sequences of actions.
Abstract
from arXiv · showhide
Can we make virtual characters in a scene interact with their surrounding objects through simple instructions? Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions? Inspired by these questions, we present the first framework to synthesize the full-body motion of virtual human characters performing specified actions with 3D objects placed within their reach. Our system takes textual instructions specifying the objects and the associated intentions of the virtual characters as input and outputs diverse sequences of full-body motions. This contrasts existing works, where full-body action synthesis methods generally do not consider object interactions, and human-object interaction methods focus mainly on synthesizing hand or finger movements for grasping objects. We accomplish our objective by designing an intent-driven fullbody motion generator, which uses a pair of decoupled conditional variational auto-regressors to learn the motion of the body parts in an autoregressive manner. We also optimize the 6-DoF pose of the objects such that they plausibly fit within the hands of the synthesized characters. We compare our proposed method with the existing methods of motion synthesis and establish a new and stronger state-of-the-art for the task of intent-driven motion synthesis.
1. Introduction
IMoS addresses instruction-driven synthesis of diverse full-body human-object interactions, extending motion generation beyond grasping or object-free actions. It combines decoupled arm/body synthesis with object-pose optimization and evaluates the resulting motions quantitatively and perceptually.
- Motivation and contribution: Textual instructions specify object types and intended actions for synthesizing diverse full-body character-object interactions.The framework targets interactions such as using, yielding, or eating with objects placed within reach.
- Method: IMoS uses separate conditional variational autoregressors for the arms and the rest of the body.The decoupled design supports more precise arm and body synthesis while allowing diverse samples from latent space.
- Method: The model optimizes object 6-DoF positions after synthesizing motion to satisfy grasping constraints.This explicitly incorporates object placement into the generated interaction rather than synthesizing character motion alone.
- Evaluation: IMoS is evaluated on the GRAB dataset of approximately 1.3K human-object interaction sequences using motion, diversity, multimodality, recognition, and perceptual metrics.The evaluation compares synthesized sequences with recent conditional motion synthesis methods.
- Contributions: The contributions include two-handed interactions, hand-to-hand object exchanges, and instruction-conditioned full-body motion for objects within reach.The paper identifies these settings as extending prior character-object interaction work.
2. Related Work
Prior work spans human-motion forecasting, joint human-object modeling, hand-object grasp synthesis, and full-body grasp synthesis. IMoS addresses the more challenging setting of intent-conditioned human and object motion with diverse generated sequences, using separate arm and body synthesis modules within an autoregressive generator.
- Research areas: Related work covers human pose forecasting and synthesis, human-object 3D interaction modeling, hand-object grasp synthesis, and full-body grasp synthesis.These four areas define the main methodological context for IMoS.
- Human pose forecasting and synthesis: Stochastic pose-forecasting methods use VAEs or GANs to introduce variability into predicted motion sequences.The cited forecasting literature models future motions from past joint positions or rotations.
- Human-object 3D interaction modeling: Human-object interaction methods jointly model people and objects, including approaches that reconstruct them from scene evidence or synthesize their motion.Examples include silhouette-based joint optimization and neural state-machine approaches.
- Hand-object grasp synthesis: Hand-object grasp synthesis includes analytical constrained-optimization methods and data-driven approaches that learn grasp representations.This literature spans both computer graphics and robotics.
- Full-body grasp synthesis: Full-body grasp methods such as GOAL and SAGA commonly estimate a whole-body grasping pose and use motion infilling to reach it.This strategy differs from IMoS’s intent-conditioned autoregressive synthesis of arms and the rest of the body.
- Research gap: Conditioning human and object motions on intended actions while preserving sequence diversity requires learning their intent-based mutual interactions.The setup becomes especially challenging when the object is grasped.
3. Intent-Driven Full-Body Motion Generator
The generator conditions full-body motion on character shape, object information, and textual intent, using separate variational autoregressors for arms and the remaining body. An object optimizer then estimates 6-DoF object poses and hand configurations to preserve plausible interaction and contact.
- Problem formulation: The pipeline synthesizes full-body motion conditioned on character shape, an in-reach 3D object, and a textual instruction specifying the intended action.It assumes interaction with one object at a time, using either one or both hands.
- Condition Encoder: Object categories are encoded with a one-hot vector, while action labels are embedded with CLIP before concatenation with body shape parameters.The CLIP embeddings provide semantics-based action representations; semantically related actions can induce similar arm movements.
- Arm Synthesis Module: A conditional variational autoregressor synthesizes arm movements from the encoded conditions, previous arm poses, and recent object positions.Its latent sampling supports diverse arm motions during inference.
- Body Synthesis Module: A separate variational autoregressor synthesizes the remaining body parts while self-attention models correlations among joints across recent frames.These correlations capture coordinated movements such as tilting the head while bringing a cup to the mouth.
- Object Optimizer Module: The Object Optimizer estimates object rotation, translation, and hand pose so the object follows the synthesized body motion plausibly.It uses initial grasp estimates, preserves hand-object distances, maintains first-frame contacts, and regularizes changes between consecutive frames.
- Object Optimizer Module: The contact term identifies first-frame vertex pairs within a threshold and forces those pairs to remain in contact during subsequent frames.The distance-preservation term alone does not guarantee continued contact, motivating the additional contact constraint.
4. Implementation
The implementation represents motions with fixed key frames and trains autoregressive arm and body modules using reconstruction and regularization losses. At inference, the model generates motion and corresponding object positions for novel intent-object pairs and body shapes.
- Training and Inference Routines: Ground-truth sequences are represented with T = 15 key frames sampled at 8-10 fps to stabilize computation and reduce redundant-frame overhead.The fixed representation also limits parameter load and helps avoid overfitting to redundant frames.
- Training and Inference Routines: The model uses k = 4 past frames and trains the arm and body autoregressors with KL-divergence and pose-related losses.The implementation uses fully connected layers with skip connections, LeakyReLU activations, and batch normalization.
- Training and Inference Routines: During inference, the system autoregressively synthesizes full-body motion for novel intent-object pairs and novel body shape parameters.Inputs include an initial body pose, an in-reach 3D object, and the intended action.
- Implementation Details: The model is trained for 1,600 epochs with Adam using a base learning rate of 5 × 10^-4 and batch size 64.Training takes roughly four hours on an NVIDIA A100-PCIE-40GB GPU.
- Implementation Details: Generating full-body poses and corresponding object positions for a 15-frame sequence takes approximately 1-1.5 minutes during inference.The learning rate uses a reduce-on-plateau scheduler, and the loss weights are explicitly fixed in the implementation.
5. Experiments and Results
The experiments evaluate IMoS on GRAB against motion-synthesis baselines and ablations using quantitative, perceptual, and qualitative analyses. Results show improvements across reported metrics, stronger perceived realism and instruction fidelity, diverse full-body coordination, and plausible two-handed and head-hand interactions.
- Dataset and evaluation: GRAB contains whole-body interactions from ten subjects with 51 objects and four intents, including use, pass, lift, and offhand.The use intent includes 26 object-related actions such as drinking, pouring, photographing, and browsing.
- Dataset and evaluation: IMoS is compared with ACTOR, Action2Motion, and TEMOS, which are retrained for this setting under different conditioning protocols.ACTOR and Action2Motion use action labels without object information, while TEMOS receives sentences containing the action and object.
- Dataset and evaluation: The evaluation uses MPJPE, AVE, FID, recognition accuracy, diversity, and multimodality, with repeated experiments and 95% confidence intervals.MPJPE measures mean joint error over time; AVE measures variance error, while the remaining metrics assess naturalness, recognition, diversity, and multimodality.
- Quantitative evaluation: IMoS significantly improves all reported metrics over existing methods and its ablated versions on the test set.Table 2 compares ACTOR, Action2Motion, TEMOS, and three ablations across the six evaluation metrics.
- Perceptual evaluation: In the perceptual study, participants selected IMoS as most realistic in 80% of responses and as having the best instruction fidelity in 81.6%.The study collected judgments for 30 sequences from 75 participants, comparing IMoS with ACTOR and Action2Motion.
- Qualitative evaluation: Qualitative analyses show diverse head, arm, and torso configurations, plausible both-handed interactions, and coordinated head-hand motion without explicit head supervision.Sampling the variational latent space produces variations for the same instruction, while the model synthesizes full-body coordination and offhand interactions.
6. Discussion and Limitations
IMoS synthesizes plausible full-body human-object interactions, but its current scope is bounded by rare high-frequency wrist actions, static-grasp assumptions, imperfect contact data, and no long-term action sequences.
- Observed performance: The method produces plausible intended motions, but rare high-frequency wrist actions such as “squeeze” and “shake” remain failure cases.These failures occur despite performance across varied intents and objects.
- Limitations: The object optimizer assumes an initial hand-object distance, which works well for most static-grasp intents but limits dynamic grasping.Dynamic grasping includes hand slipping and relative motion, such as rotating a cube or stretching an elastic band.
- Observed performance: Offhanding is challenging because the object must be optimized while switching between hands.The figure illustrates two interactions in which the character passes an object from one hand to the other.
- Limitations: Sparse marker-based capture causes imprecise contacts in some GRAB sequences, including fingers missing the object or interpenetrating it.These contact artifacts limit the precision of synthesized body-object interactions.
- Limitations: The method does not address long-term motion synthesis involving sequences of object-related actions over minutes.Its scope therefore excludes extended multi-action behavior with an object.
- Ethical considerations: The method is not photo-realistic and lacks texture and fine appearance details, reducing its suitability for deceptive-content production.Combining it with more realistic texture methods could raise future ethical concerns.
7. Conclusion and Future Work
IMoS presents a full-body motion synthesis method for character-object interactions and models arm and body motions separately with conditional variational autoregression. It also supports two-handed exchanges, while future work targets dynamic grasps and more precise sentence-conditioned control.
- Conclusion: IMoS is presented as the first full-body motion synthesis method for character-object interactions.The authors position it as a practical pipeline for applications requiring large-scale character animations.
- Conclusion: Decoupling arms and body motions through conditional variational autoregression yields measurable perceptual improvements and advances the state of the art on multiple quantitative evaluations.The conclusion attributes the improvement to separately modeling the arms and body motions.
- Conclusion: The system synthesizes two-handed interactions, including sequences in which an object exchanges hands.This extends the interaction scope beyond single-hand object use.
- Future work: Future work will extend the model to dynamic grasps and full-body poses that allow the virtual character to change its grasp within a sequence.This directly targets a limitation of the current setting.
- Future work: The authors also plan to use descriptive sentence embeddings to generate more precise and controllable motions.The example instruction specifies that a person passes a bowl using the right hand.