Source-linked AI summary
Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis
Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, Bo Dai
TL;DR
Existing scene-aware motion synthesis approaches are constrained by predefined objects or positions, limiting interaction diversity. This paper decomposes motion diversity into anchors, paths, and body movements and proposes a hierarchical framework for modeling them. The authors report improved diversity and naturalness over previous methods.
Problem
Method
A three-stage framework generates diverse action-conditioned interaction anchors, plans diverse paths with Neural Mapper-guided A∗, and completes body movements along those paths.
Source: §1 · p. 2Results
The authors report that their framework surpasses previous approaches in diversity and naturalness across evaluated datasets.
Source: §1 · p. 2, §4.2 · p. 8, §5 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
The ability to synthesize long-term human motion sequences in real-world scenes can facilitate numerous applications. Previous approaches for scene-aware motion synthesis are constrained by pre-defined target objects or positions and thus limit the diversity of human-scene interactions for synthesized motions. In this paper, we focus on the problem of synthesizing diverse scene-aware human motions under the guidance of target action sequences. To achieve this, we first decompose the diversity of scene-aware human motions into three aspects, namely interaction diversity (e.g. sitting on different objects with different poses in the given scenes), path diversity (e.g. moving to the target locations following different paths), and the motion diversity (e.g. having various body movements during moving). Based on this factorized scheme, a hierarchical framework is proposed, with each sub-module responsible for modeling one aspect. We assess the effectiveness of our framework on two challenging datasets for scene-aware human motion synthesis. The experiment results show that the proposed framework remarkably outperforms previous methods in terms of diversity and naturalness.
1. Introduction
Scene-aware motion synthesis needs to capture diversity in interaction locations, paths, and body poses, while prior approaches rely on limited scene representations or manually assigned targets. The paper proposes a three-stage framework to model these diversity levels under target action sequences.
1. Introduction
- Earlier scene-aware methods were limited by 2D scene representations or manually assigned interaction targets, leaving motion diversity incompletely modeled.The cited approaches mainly emphasized body movements while walking to predefined objects or positions.Source: §1 · pp. 1–2
- The framework decomposes scene-aware motion diversity into interaction anchors, paths, and body poses, with a stage for each level.It aims to synthesize motions guided by scene context and target action sequences.Source: §1 · p. 2
- For anchor diversity, the method generates scene-agnostic action-conditioned poses, then places them in the scene with position penalties and a CVAE Place Refiner.The penalty discourages same-action anchors from being placed near one another; the refiner produces diverse offsets.Source: §1 · p. 2
- For path diversity, a scene-conditioned Neural Mapper adds stochastic guidance to A∗ so it can generate diverse, natural paths between anchors.The mapper provides dynamic probabilistic guidance to the planner.Source: §1 · p. 2
- A Transformer-based CVAE completes motion by generating trajectories that loosely follow planned paths, then producing poses conditioned on scene context, actions, and anchors.The network targets long-term motion spanning different actions, rather than single-action synthesis.Source: §1 · p. 2
- The authors report that their framework surpasses previous approaches in diversity and naturalness on datasets including PROX.The introduction also lists action-conditioned pose placing, Neural Mapper, and motion completion as its technique contributions.Source: §1 · p. 2
2. Related Works
Prior scene-aware synthesis methods constrain interaction diversity through predefined targets, while motion-prediction methods address a different task. This paper’s framework combines diverse anchors, paths, and body movements.
2. Related Works
- Earlier motion-synthesis work focused on body poses, while later approaches incorporated actions, scene context, or destinations; some use 2D scenes or specified endpoints.The cited approaches include CVAE and Transformer synthesis, destination-conditioned prediction, and 3D-scene methods with assigned begin-end points or action-specific objects.Source: §2 · pp. 2–3
- The framework has three stages: interaction-anchor generation, diverse path planning, and motion completion guided by anchors and paths.Figure 2 describes scene-agnostic pose synthesis and scene placement, Neural Mapper-assisted A∗ planning, and motion synthesis.Source: Figure 2 · p. 3
- Prior scene-aware methods rely on predefined objects or positions, constraining exploration of interaction diversity in synthesized motions.The proposed framework instead generates diverse anchors, plans paths between them, and completes body movements.Source: §2 · p. 3
- Motion prediction forecasts future dynamics from orientations or prior motion, whereas this paper synthesizes motions without prior-motion knowledge.The related methods include Seq2Seq, LSTM, and graph-convolution approaches.Source: §2 · p. 3
3. Methodology
The method models scene-aware motion through action-conditioned interaction anchors, diverse obstacle-free paths, and motion completion. Its hierarchy links the generated anchors and planned paths to the synthesized body movements.
3.1. Overview
- The framework first generates action-conditioned interaction anchors, then plans obstacle-free paths between them, and finally completes body movements along those paths.Scene context informs pose placement and path planning; motion completion fills gaps between consecutive anchors.Source: §3.1 · p. 3
3.2. Human-Scene Interaction Anchor Synthesis
- Anchor synthesis first generates scene-agnostic poses conditioned on action labels, then places them in scene locations compatible with the pose and interaction semantics.Candidates pair uniform scene grids with sampled ground-parallel orientations and are ranked by affordance and penetration compatibility; a same-action location penalty encourages diversity, and Place Refiner adds offsets.Source: §3.2 · pp. 3–4
3.3. Diverse Path Planning
- A∗ searches discrete 3D grids using movement and target-distance costs, excluding grids where a human-volume cylinder would collide.Neural Mapper replaces naïve random cost noise with a learned feasibility score for neighboring grids, adding 1−m to the cost to guide diverse, reasonable paths.Source: §3.3 · pp. 4–5
3.4. Motion Completion
- Motion completion uses a Path Refiner and Motion Synthesizer, both Transformer-based CVAEs, to produce continuous paths and body poses between anchors.The Path Refiner uses scene context and planned-path positional encoding; the Motion Synthesizer conditions on the refined path, scene context, and anchor action labels and poses.Source: §3.4 · pp. 5–6, Figure 6 · p. 6
4. Experiments
The experiments evaluate interaction-anchor, path, and motion diversity alongside naturalness on PROX, with additional qualitative results on Matterport3D. The reported comparisons favor the proposed framework for diverse, natural motion synthesis.
4.1. Experimental Setting
- The framework is trained on PROX with manually assigned action labels and also evaluated on unseen Matterport3D scenes, which are not used for training.Diversity is measured across anchors, paths, and completed motions; naturalness is evaluated through user study and physical plausibility, while FD measures whole-motion quality.Source: §4.1 · pp. 6–7
4.2. Experimental Results
- For sit examples, position-related sampling produces anchors interacting with different scene objects, while Place Refiner results paired with optimization post-processing yield diverse optimal solutions from diverse translation and orientation initializations.Source: §4.2 · p. 7
- Neural Mapper generates diverse paths with naturalness similar to standard A*, while random-noise methods produce unnatural paths despite greater diversity.The evaluation uses 50 samples per method and compares path distances and user-study naturalness.Source: §4.2 · p. 7, Table 2 · p. 7
- The proposed method achieves the best results against other methods on motion completion, with comparisons reporting both FD and APD.The paper describes the overall comparison as showing more diverse and natural motions than other methods.Source: §4.2 · p. 8, Table 3 · p. 8
- Qualitative results on PROX and Matterport3D show interaction anchors, planned paths, and completed motions for target action sequences.Figure 8 presents outputs from each framework stage and reports diverse scene interactions.Source: §4.2 · p. 8, Figure 8 · p. 8
5. Conclusion
The conclusion presents a hierarchy that addresses interaction-anchor, path, and body-movement diversity in scene-aware motion synthesis. It reports varied motions across all three levels and says the factorized framework can be incorporated into other motion-synthesis frameworks.
5. Conclusion
- The framework decomposes scene-aware motion diversity into action-conditioned interactions, obstacle-free paths, and body movements.Its hierarchy assigns a component to each diversity level.Source: §5 · p. 8
- The authors report varied, vivid motions across all three diversity levels, with improved efficiency and generality.Source: §5 · p. 8
- The factorized design can be incorporated into other human-motion-synthesis frameworks.Source: §5 · p. 8
6. Supplementary Materials
The supplementary material reports results on PROX and Matterport3D, details model training and optimization, and discusses how interaction anchors and planned paths affect synthesis. It also examines the Neural Mapper’s path planning under constraints.
6.1.1 Network Architecture.
- The pose generator, Place Refiner, and Neural Mapper use two-layer MLPs, while the motion completion network uses an eight-layer Transformer encoder and decoder.The CVAE encoders predict 32-dimensional Gaussian parameters; inputs to the motion completion network are encoded into 256-dimensional features.Source: §6.1.1 · p. 9
6.1.2 Training and Inference Details.
- Training uses reconstruction and KL-divergence objectives; inference splits planned paths into equal-length pieces, adds intermediate anchors, and synthesizes and connects motion segments.The Place Refiner learns pose offsets from sampled position and orientation candidates, while the Neural Mapper learns local scene-conditioned moving directions from 60-frame PROX sequences.Source: §6.1.2 · p. 9
6.1.3 Optimization.
- Anchor optimization uses affordance, penetration, and pose-regularization losses, while motion optimization targets foot location, environment, and smoothness.Anchors are optimized for 10 iterations with L-BFGS; motions are optimized for 100 iterations with ADAM.Source: §6.1.3 · p. 10
6.2. Experiments
- The PROX comparisons find more natural motions than other methods, including without optimization post-processing; Path Refiner guidance also improves naturalness.The table evaluates physical plausibility and user-study results; “Ours*” omits the Path Refiner.Source: §6.2 · p. 10, Table 4 · p. 10
- On Matterport3D, the method improves anchor diversity and produces motions with better diversity and physical plausibility than previous methods.The Path Refiner also improves diversity and naturalness on this dataset.Source: §6.2 · p. 10, Table 5 · p. 10, Table 6 · p. 10
6.3. Further Discussion
- The human-centric placing paradigm explicitly relates pose structure to scene structure, unlike prior scene-centric methods that may produce implausible poses when conditioned on action labels.The discussion attributes this limitation to insufficiently modeling the relationship between action and scene context.Source: §6.3 · p. 11
- The Neural Mapper provides data-driven path guidance without complex manual constraints and can still produce diverse, natural paths when regions are manually marked for avoidance.The discussion also reports that random intermediate points can yield obstructed or zigzag paths.Source: §6.3 · p. 11, Figure 11 · p. 11
- Motion synthesis benefits from encoding both translation and orientation of the planned path; omitting path encoding can cause table penetration or unnatural orientations.Using translation alone is insufficient for natural orientations.Source: Figure 12 · p. 11, §6.3 · p. 12