Source-linked AI summary
LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model
Quankai Gao, Jiawei Yang, Qiangeng Xu, Le Chen, Yue Wang
TL;DR
Human–object manipulation requires fine-grained motion and physical interaction modeling, while existing approaches generalize poorly or remain constrained to simulation. LOME fine-tunes a pretrained video diffusion model to generate egocentric interactions from images, text, and per-frame human actions by jointly modeling actions and environment. It achieves stronger action following and visual quality, produces realistic physical consequences, and supports real-world scenarios without explicit 3D/4D reconstruction, while still struggling with some multi-object interactions.
Problem
Human–object manipulation demands modeling physical dynamics, contact-rich motion, and action-dependent object motion beyond what reconstruction methods and simulated approaches support.
Method
LOME fine-tunes a pretrained video diffusion model using an input image, text prompt, per-frame body and hand actions, and joint action-environment modeling.
Results
LOME achieves 66.85% PCK@20 versus 51.33% for the best baseline, improves FVD from 59.83 to 39.58, and receives 97% and 94% user preference for action following and visual quality.
Takeaways & Limitations
LOME generates realistic, contact-rich human–object manipulation videos with coherent physical consequences across diverse real-world scenarios without explicit 3D/4D reconstruction.
Takeaways & Limitations
LOME struggles with coordinating multiple objects, including grasping and lifting a cup while directing ice cubes into the specified target.
Abstract
from arXiv · showhide
Learning human-object manipulation presents significant challenges due to its fine-grained and contact-rich nature of the motions involved. Traditional physics-based animation requires extensive modeling and manual setup, and more importantly, it neither generalizes well across diverse object morphologies nor scales effectively to real-world environment. To address these limitations, we introduce LOME, an egocentric world model that can generate realistic human-object interactions as videos conditioned on an input image, a text prompt, and per-frame human actions, including both body poses and hand gestures. LOME injects strong and precise action guidance into object manipulation by jointly estimating spatial human actions and the environment contexts during training. After finetuning a pretrained video generative model on videos of diverse egocentric human-object interactions, LOME demonstrates not only high action-following accuracy and strong generalization to unseen scenarios, but also realistic physical consequences of hand-object interactions, e.g., liquid flowing from a bottle into a mug after executing a ``pouring'' action. Extensive experiments demonstrate that our video-based framework significantly outperforms state-of-the-art image based and video-based action-conditioned methods and Image/Text-to-Video (I/T2V) generative model in terms of both temporal consistency and motion control. LOME paves the way for photorealistic AR/VR experiences and scalable robotic training, without being limited to simulated environments or relying on explicit 3D/4D modeling.
1 Introduction
LOME addresses the difficulty of generating generalizable, contact-rich human–object manipulation by conditioning a pretrained video model on explicit human actions and environmental context. It achieves accurate action following, improved video quality, and realistic physical consequences across real-world scenarios.
- Human–object manipulation requires modeling physical dynamics, contact-rich motion, and how object motions evolve in response to human actions.
- LOME conditions video generation on an input image, text prompt, and per-timestep body poses and hand gestures for continuous action control.
- LOME jointly models action and environment latents to decouple human actions from environmental dynamics and improve generalization to novel actions and unseen environments.
- 66.85% PCK@20 action-following accuracy versus 51.33% for the best baseline, while FVD improves from 59.83 to 39.58.User studies report 97% preference for action following and 94% preference for visual quality.
- LOME generates coherent interactions across diverse real-world scenarios, including multiple objects and realistic physical consequences.The reported examples include manipulation outcomes that evolve consistently over time.
2 Related Work
Prior work relies heavily on reconstruction, simulation, or generic video-generation conditioning, leaving realistic motion control and physical interaction challenging. LOME instead uses action-conditioned fine-tuning of a video generative model with explicit spatial action signals.
- Reconstruction-based human–object interaction methods use detection, pose estimation, fitting, and 3D parametric models or object templates.
- Existing action-conditioned video generation has been applied to navigation and hand or agent–object interaction.
- Video generative models commonly combine text, image, and action conditions to enable controllable generation through diffusion.
- Data scaling and RLHF do not necessarily yield realistic motion dynamics or physical interactions, and RLHF can induce motion degradation.
- LOME integrates human actions as explicit spatial control signals through action-conditioned fine-tuning.
3 LOME
LOME generates human-object manipulation videos from images, text, and per-frame actions using spatial action maps and joint action-environment modeling. Its design preserves scene content, incorporates camera conditions, and modifies guidance because actions are jointly denoised with video.
- Inputs and Conditions: LOME conditions video generation on an input image, text description, camera information, and per-frame human actions.The input image anchors the scene while text and camera conditions define the manipulation and viewpoint.
- Action Maps: 3D skeleton and hand keypoints are projected into full-resolution, background-masked 2D action maps for pixel-level spatial control.Keypoints outside the camera frustum are discarded to prevent scene information from leaking into the action signal.
- Joint Action-Environment Modeling: LOME jointly denoises video and action latents while keeping the input-image latent clean, thereby modeling their joint action-environment distribution.The concatenated representation contains the scene anchor, camera-conditioned video latent, and encoded action latent.
- Modified Guidance: Standard classifier-free guidance is modified because the action is denoised by the model and is not independent of its output.The sampling distribution accounts for both independent text-camera conditions and the non-independent action.
- Video Generation: After denoising, LOME discards the recovered image and action latent frames before decoding the remaining latents into the final video.The first frame and latter half of the latent sequence correspond to the recovered input-image and action latents, respectively.
4 Experiments
LOME is evaluated on diverse egocentric human-object manipulations using curated datasets, baselines, quantitative metrics, qualitative comparisons, user studies, and ablations. Results show stronger motion consistency, action following, and physical realism, while exposing failures in noisy or multi-object settings.
- Implementation Details: LOME is fine-tuned from Wan2.1-VACE-14B with LoRA adaptation while preserving the pretrained model’s components and motion priors.The method uses the pretrained VAE, decoder, and text encoder without modification, freezes diffusion DiT blocks, and applies rank-128 LoRA to the VACE module.
- Dataset: EgoDex provides 338,234 egocentric videos with approximately 800 hours of diverse human-object manipulation footage and detailed 3D pose annotations.Training uses five EgoDex training sets, with one non-overlapping test set reserved for inference; ten additional in-the-wild videos assess generalization.
- Dataset: Temporal resampling converts variable-length clips to six frames by uniform downsampling or back-and-forth upsampling.Longer clips preserve the first and last frames, while shorter clips are repeatedly resampled until reaching the target length.
- Results and Analysis: LOME qualitatively outperforms CoSHAND, Wan-I2V, and GwtF in object and hand motion consistency and produces more realistic hand-object interactions.CoSHAND struggles with the correct object in multi-object scenes, while Wan-I2V fails to complete manipulation sequences specified by text prompts.
- Results and Analysis: Only LOME generates a coherent pouring sequence with progressively increasing liquid in the cup, reflecting the instructed physical consequence.CoSHAND and GwtF fail to produce meaningful hand-object interactions, while Wan-I2V does not complete the pouring sequence or fill the cup progressively.
- Ablation Study: Removing joint action-environment modeling degrades hand realism, motion consistency, and action-following performance.Temporal concatenation also outperforms channel concatenation, which the authors associate with bidirectional temporal attention between corresponding video and action frames.
- Results and Analysis: The noisy dataset contains projection misalignment from 3D pose and camera estimation errors, and LOME struggles with simultaneous multi-object interactions.A reported failure causes ice cubes to fall into a tray rather than the intended cup after an incorrect grasp.
5 Limitations and Future Works
LOME remains constrained by imperfect headset-based pose and camera estimates, which can misalign action maps and hand poses, and by difficulties coordinating multiple objects.
- Imperfect 3D human- and camera-pose estimates from headsets can misalign action maps with hand poses, lowering PCK@20 scores.These projection errors contribute to lower-than-expected action-following accuracy.
- LOME still struggles to coordinate interactions among multiple objects.In one failure case, it fails to grasp and lift a yellow cup, causing ice cubes to fall into a tray instead of the target cup.
6 Conclusion
LOME adapts video diffusion models into an egocentric world model for general human–object manipulation. By jointly modeling actions and environments, it generates realistic contact-rich videos with accurate action following and high visual fidelity without 3D/4D reconstruction or parametric fitting.
- LOME adapts video diffusion models to learn general human–object manipulation in an egocentric setting.
- Jointly modeling human actions and environments enables realistic, contact-rich manipulation videos with accurate action following and high visual fidelity.
- LOME produces these manipulation videos without relying on 3D/4D reconstruction or parametric model fitting.
- The results demonstrate LOME's potential for photorealistic simulation of real-world human–object manipulation.