Source-linked AI summary
Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, Xiaolong Wang
TL;DR
The paper addresses the limited integration of human motion synthesis with scene affordance reasoning. It proposes a hierarchical, scene-conditioned framework with mesh-scene geometric optimization, and reports more natural and physically plausible long-term motion than competing approaches.
Problem
Existing methods often neglect human-scene interaction, while affordance reasoning has mainly focused on static poses rather than motion.
Method
A hierarchical framework predicts scene-conditioned SMPL-X bodies at sub-goals, synthesizes short-term motion between them, and optimizes the segments using human-scene geometry constraints.
Results
The framework generates more natural and physically plausible long-term motion in 3D scenes than other baselines.
Takeaways & Limitations
Combining hierarchical motion generation with scene affordance and geometric constraints supports realistic long-term human motion and interaction in varied 3D environments.
Takeaways & Limitations
The method assumes each short-term motion has a fixed length of k + 1 steps.
Abstract
from arXiv · showhide
Synthesizing 3D human motion plays an important role in many graphics applications as well as understanding human activity. While many efforts have been made on generating realistic and natural human motion, most approaches neglect the importance of modeling human-scene interactions and affordance. On the other hand, affordance reasoning (e.g., standing on the floor or sitting on the chair) has mainly been studied with static human pose and gestures, and it has rarely been addressed with human motion. In this paper, we propose to bridge human motion synthesis and scene affordance reasoning. We present a hierarchical generative framework to synthesize long-term 3D human motion conditioning on the 3D scene structure. Building on this framework, we further enforce multiple geometry constraints between the human mesh and scene point clouds via optimization to improve realistic synthesis. Our experiments show significant improvements over previous approaches on generating natural and physically plausible human motion in a scene.
1. Introduction
The paper targets long-term human motion synthesis in 3D scenes, addressing both realistic motion and human-scene interaction. It bridges motion synthesis with affordance learning through a hierarchical, scene-conditioned framework and geometric optimization.
- The goal is to generate long-term human motion in 3D scenes, including navigation around furniture and transitions such as walking, turning, and sitting.
- Existing motion-synthesis methods emphasize natural pose sequences but rarely model human-scene interaction, while affordance research mainly studies static human poses.
- The paper introduces a two-level hierarchy that predicts scene-conditioned poses at intermediate sub-goals, then synthesizes short-term motion between consecutive poses.
- The framework uses SMPL-X representations, a CVAE for sub-goal body synthesis, and a bidirectional LSTM for short-term motion generation.
- Experiments on PROX and MP3D show realistic and physically plausible motion, with advantages over state-of-the-art motion and pose-generation methods across multiple metrics and human evaluation.
2. Related work
Related work spans scene affordance learning, environment-aware human-dynamics prediction, and motion synthesis. The paper differs by generating motion directly from realistic scene point clouds while modeling the human as a mesh.
- Affordance learning: Scene-affordance research predicts human placement or pose in scenes, increasingly using 3D body shape rather than only skeletons.
- Human dynamics prediction: Human-dynamics research includes trajectory and pose prediction, increasingly incorporating 3D information or bird’s-eye-view images.
- Motion synthesis: Motion-synthesis research develops representations, blending methods, and sequential models for generating or combining human motion.
- Positioning: The framework organizes long-term synthesis by generating sub-goal bodies, synthesizing short-term segments, and connecting them through optimization.
- Positioning: Unlike approaches using predefined motions or simple hand-crafted environments, this method generates motion directly from realistic scene point clouds and uses human meshes instead of skeletons.
3. Method
The method synthesizes long-term scene-conditioned human motion hierarchically, representing humans with SMPL-X and refining generated sequences through geometric optimization. It combines sub-goal body generation, short-term route-and-pose synthesis, and constraints for contact, collision, foot stability, and temporal smoothness.
- Overview: The framework represents each human body with modified sequential SMPL-X parameters covering translation, continuous rotation, body shape, body pose, and hand pose.SMPL-X is used instead of skeletons to support flexible geometry constraints and realistic contacts.
- Hierarchical generation: A two-level hierarchy first generates bodies at sub-goal positions, then synthesizes short-term motion between consecutive sub-goals and connects the sequences.Sub-goals reduce prediction uncertainty and allow diverse long-term motion through latent-variable sampling.
- Static human body synthesis: The static body synthesis stage uses a CVAE conditioned on body shape, sub-goal location and orientation, and scene point-cloud features extracted by PointNet.During inference, sampling different latent codes produces different bodies at the same sub-goal.
- Motion synthesis: The motion synthesis stage uses sequential RouteNet and PoseNet networks to predict scene-conditioned intermediate locations, orientations, body poses, and hand poses.RouteNet predicts the route between sub-goals; PoseNet uses that route together with endpoint poses and scene features.
- Optimization: Geometric optimization refines the generated long-term sequence using foot, collision, contact, and smoothness constraints optimized directly over predicted mesh parameters.The optimization combines the error terms with fixed coefficients and uses Adam; Figure 4 illustrates improved floor contact, pivot-foot stability, and vertex smoothing.
4. Experiments
Experiments evaluate reconstruction accuracy, environmental naturalness, and human-rated motion quality against strengthened baselines and ablations. The method achieves strong reconstruction and scene interaction results, while optimization improves physical plausibility and human evaluations favor the proposed approach across motion durations and unseen environments.
- Experimental setup: The experiments use PROX training data, unseen PROX scenes, and direct MP3D testing, with short-term metrics and human evaluations for 2-second, 4-second, and 6-second motions.PROX provides 4.3k bodies and 10k two-second frames for training; evaluation includes 3k sequences in four unseen scenes and 50 sequences per duration for human evaluation.
- Baselines: The comparisons include state-of-the-art motion and affordance baselines, strengthened with modified SMPL-X representations and optimization where needed.Baselines include Route+CVAE, HSNMS, and ablative architectures testing route inputs, rotation prediction, end-to-end prediction, and scene information.
- Reconstruction: The proposed method achieves better location and pose reconstruction accuracy than baselines, while optimization slightly increases error but substantially improves naturalness.It also has the lowest neighbour v2v distance, supporting easier connection of short-term segments into long-term motion.
- Ablation study: Predicted route inputs improve PoseNet accuracy, predicting translation and rotation in RouteNet is more accurate, and scene information improves synthesis accuracy across ablations.A single end-to-end network performs worse than the proposed architecture in the reported ablation comparison.
- Naturalness: In-environment evaluation shows nearly best contact and non-collision performance, and optimization improves the physical plausibility and environmental adaptability of baselines.Qualitative comparisons also report more natural motion, fewer unreasonable movements, and better human-scene interaction.
- Naturalness: Human evaluation gives the proposed method the highest score among baselines for 2-second, 4-second, and 6-second motions, with scores close to pseudo-ground truth.On MP3D, direct testing of the PROX-trained model produces the most realistic results and best environment adaptability despite challenging randomly selected inputs.
5. Conclusion
The paper presents a hierarchical generative framework for long-term human motion in 3D scenes, combining interaction-aware models with optimization. Compared with baselines, it synthesizes more natural and physically plausible motion.
- The framework synthesizes long-term human motion in 3D scenes using a hierarchical generative approach.
- Two deep models generate sub-goal bodies and short-term motions while modeling human-scene interaction.
- An optimization-based method improves realistic synthesis and connects short-term motions into long-term motion.
- Compared with other baselines, the framework synthesizes more natural and physically plausible long-term motion in 3D scenes.
Appendix
The appendix provides additional details on datasets, optimization implementation, other methods, and naturalness evaluation.
- Additional details cover datasets, optimization implementation, comparisons with other methods, and naturalness evaluation.
1. Datasets
The experiments train on selected PROX scenes and test on held-out PROX scenes plus rooms from an MP3D environment. Training data use down-sampled static bodies and motion segments sampled from start/end pairs.
- Training uses eight PROX scenes, while testing uses four held-out PROX scenes and three rooms from MP3D scene 17DRP5sb8fy.
- Sub-goal body synthesis training uses static bodies sampled from original motion sequences every 0.33 seconds.
- Motion synthesis training samples start/end bodies two seconds apart with Euclidean distance greater than 0.5 meters.
2. Implementation details
Optimization balances environmental constraints, motion smoothness, and motion plausibility through two successive stages with different foot-contact weighting.
- The first optimization stage emphasizes environmental constraints and motion smoothness while setting λfoot = 0.
- The second stage improves motion plausibility by setting λfoot = 1 while retaining collision, contact, and smoothness weights.
3. Discussion of other methods
The paper compares its motion synthesis with CVAE interpolation and a prior prediction method, identifying limitations in interpolation and reporting lower errors than the prior method.
- CVAE interpolation: The CVAE interpolation baseline is limited to a few cases and behaves more like averaging for motions spanning a certain distance.
- Comparison with prior work: 8.06 is the reported average path error for the proposed method, compared with the prior method’s reported range of 19.3 to 23.7.
- Comparison with prior work: 219.1 mm is the proposed method’s MPJPE, compared with 237 mm for the prior method.
4. Naturalness evaluation
The evaluation uses a modified contact metric and a controlled human study, while qualitative examples illustrate natural, challenging, and diversified generated motions.
- Evaluation protocol: The modified contact score treats signed-distance values below 0.01 as contact for motion synthesis.
- Evaluation protocol: Human evaluation presents four methods or references with identical inputs and asks users to rate each motion’s naturalness from 1 to 5.
- Evaluation protocol: The controlled evaluation keeps raters and motion inputs aligned across two baselines, the proposed method, and pseudo-ground truth.
- Qualitative results: Qualitative results include walking, sitting down, jumping on a bed, and lying down across different scenes.
- Qualitative results: Generated motions diversify with randomly sampled body shapes and latent variables that produce different motion styles.