Source-linked AI summary
Stochastic Scene-Aware Motion Prediction
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, Michael Black
TL;DR
Realistically synthesizing diverse human motion that navigates cluttered scenes and interacts with objects remains challenging. SAMP combines stochastic goal-conditioned motion prediction, object-aware goal sampling, and explicit path planning, trained with diverse MoCap data. It demonstrates varied interactions across complex indoor scenes and reports superior performance to existing solutions, while retaining limitations on penetrations and substantially novel object geometries.
Problem
Synthesizing virtual humans that navigate cluttered indoor scenes and naturally interact with objects is difficult because human motion is diverse and scene-adaptive.
Method
SAMP uses MotionNet for stochastic goal-conditioned motion, GoalNet for object-geometry-conditioned contact points and orientations, explicit A* path planning, and a diverse human-scene MoCap dataset.
Results
SAMP generates varied sitting and lying styles, adapts motion to interacting objects, navigates cluttered scenes, and achieves superior performance compared to existing solutions.
Takeaways & Limitations
SAMP makes progress toward lifelike avatars that navigate obstacles, interact with objects in different ways, and produce naturally varying goal-directed motions.
Takeaways & Limitations
SAMP can produce slight character-object penetrations and may not generalize to objects with substantially different geometry from its training examples.
Abstract
from arXiv · showhide
A long-standing goal in computer vision is to capture, model, and realistically synthesize human behavior. Specifically, by learning from data, our goal is to enable virtual humans to navigate within cluttered indoor scenes and naturally interact with objects. Such embodied behavior has applications in virtual reality, computer games, and robotics, while synthesized behavior can be used as a source of training data. This is challenging because real human motion is diverse and adapts to the scene. For example, a person can sit or lie on a sofa in many places and with varying styles. It is necessary to model this diversity when synthesizing virtual humans that realistically perform human-scene interactions. We present a novel data-driven, stochastic motion synthesis method that models different styles of performing a given action with a target object. Our method, called SAMP, for Scene-Aware Motion Prediction, generalizes to target objects of various geometries while enabling the character to navigate in cluttered scenes. To train our method, we collected MoCap data covering various sitting, lying down, walking, and running styles. We demonstrate our method on complex indoor scenes and achieve superior performance compared to existing solutions. Our code and data are available for research at https://samp.is.tue.mpg.de.
1. Introduction
SAMP addresses the largely unsolved problem of synthesizing diverse, realistic human-scene interactions in cluttered 3D environments. It combines stochastic motion generation, object-conditioned interaction goals, path planning, and a new interaction dataset.
- Realistically synthesizing virtual humans that move and interact with 3D scenes remains largely unsolved despite progress in scene understanding, human motion capture, and locomotion modeling.
- A virtual human instructed to sit on a couch must navigate around obstacles, select a plausible contact point and direction, and produce natural motion variations.
- SAMP takes a 3D scene as input, samples valid interaction goals, and generates goal-conditioned, scene-aware motion sequences for dynamic character-scene interactions.
- MotionNet uses an autoregressive cVAE with a random latent vector at each frame to model diverse action styles, while GoalNet predicts plausible contact points and orientations across object geometries.
- SAMP is trained on a new dataset capturing both body motion and object motion across diverse human-scene interactions using high-resolution optical-marker MoCap.
- The contributions include stochastic goal-driven interaction synthesis, action-dependent goal prediction, explicit path planning for cluttered scenes, and a diverse human-scene MoCap dataset.
2. Related Work
Prior work covers static interactions, locomotion, motion synthesis, and stochastic trajectory modeling, but dynamic human-scene interaction synthesis remains insufficiently addressed. Existing approaches face limits in variation, generalization, long-sequence stability, or computational cost.
- Research has explored object functionality, affordance analysis, static human-scene interactions, and locomotion with terrain-responsive foot placement.
- Motion-sequence assembly methods synthesize interactions in new environments or with new characters, but require large databases and expensive nearest-neighbor matching.
- NSM models interactions such as sitting, carrying, and opening, but does not generate interaction variations and often fails to avoid character-object intersections in cluttered scenes.
- An RL-based sitting approach lacks variation in goal positions and directions, while generalizing learned policies to new environments or actions is often challenging.
- Neural motion predictors can generate short sequences yet may converge to mean poses or diverge on long sequences, motivating techniques such as scheduled sampling.
- Stochastic motion research has modeled future trajectories and full-body motion, but previous work had not tackled diverse human-scene interaction generation.
3. Method
SAMP combines stochastic motion prediction, object-specific goal sampling, and explicit path planning to synthesize varied character-scene interactions in cluttered environments.
- SAMP uses MotionNet, GoalNet, and a Path Planning Module to address realistic motion, plausible interaction goals, and obstacle-free navigation.MotionNet predicts character poses, GoalNet predicts interaction goals, and path planning connects the start location to the goal.
- 3.1. MotionNet: MotionNet is an autoregressive cVAE that predicts each next character state from the previous state, interaction-object geometry, and a sampled latent vector.At test time, the decoder samples the latent vector from a standard normal distribution.
- 3.1. MotionNet: MotionNet encodes character state and object geometry, then decodes the latent representation through a prediction network and a gating network.The gating network predicts blending weights for the prediction network's mixture of experts.
- 3.2. GoalNet: GoalNet is a cVAE that predicts plausible goal positions and directions from target-object geometry, allowing multiple valid interaction goals.At test time, GoalNet samples a latent code and decodes varied goals on the object's surface.
- 3.3. Path Planning: An explicit A* planner computes an obstacle-free waypoint path from the character's starting position to the desired interaction goal.The planner supports navigation through cluttered environments while avoiding obstacles.
- 3.4. Training Strategy: Scheduled sampling trains MotionNet with its own predictions as later inputs, addressing error accumulation during autoregressive rollout.This strategy is used because standard supervised training produced poor runtime predictions.
4. Data Preparation
The dataset captures diverse human-scene interactions with synchronized body and object motion, while augmentation expands object variation and preserves contacts.
- 4.1. Motion Capture Dataset: The MoCap dataset records varied interaction styles with seven objects, including sofas, chairs, an armchair, and a table.The setup uses 54 Vicon cameras and also includes walking, running, and idle sequences with varied directions and speeds.
- 4.1. Motion Capture Dataset: Each sequence captures both subject motion and object pose, with CAD models available for the recorded objects.Subjects approach objects from random locations, perform actions, then get up and walk away.
- 4.2. Motion Data Augmentation: Because only seven objects were captured, the augmentation pipeline switches or scales objects and recomputes poses while preserving contacts.Contacts are detected at the pelvis, hands, and feet, projected to replacement objects, and maintained using inverse kinematics.
- 4.3. GoalNet Data: GoalNet training labels one to five plausible positions and directions for objects across five ShapeNet categories.The number of labels depends on how many distinct interactions an object affords.
5. Experiments & Evaluation
SAMP produces diverse, plausible scene-aware motions, predicts varied interaction goals, and navigates cluttered scenes more safely than versions without planning and compared baselines. Evaluation includes qualitative analyses, diversity and goal metrics, penetration rates, and comparisons with MLP, MoE, and NSM.
- Generating Diverse Motion: SAMP generates diverse, realistic sitting and lying styles and adapts motion to objects with different geometries.The action style is conditioned on the interacting object, such as lifting legs for a high chair or extending them for a low table.
- Goal Generation: GoalNet samples multiple valid goal positions and directions that reflect object-specific interaction constraints.For example, sideways sitting is valid on a regular chair but restricted on an armchair.
- Goal Generation: Starting from the same state, different sampled goals produce different paths and final poses during the same couch-sitting action.GoalNet guides the paths, while MotionNet’s stochasticity changes the final pose.
- Goal Generation: GoalNet reconstructs unseen goals with 6.04 cm position error and 2.29 deg orientation error.Generated-goal diversity is 16.42 cm for APD-Pos and 41.27 deg for APD-Rot, compared with 16.18 cm and 90.23 deg for ground truth.
- Comparison to Previous Models: Compared with MLP and MoE, SAMP executes tasks more effectively, while NSM achieves comparable goal completion but lacks stochastic motion and safe navigation.MLP and MoE often fail at lying down and produce implausible poses; NSM fails in complex scenes where SAMP’s path planning succeeds.
- Comparison to Previous Models: SAMP’s evaluation includes diversity, execution time, precision, Fréchet distance, and penetration measurements on unseen ShapeNet test objects.The tables report diversity, execution time, positional and rotational precision, Fréchet distance, and SAMP-versus-NSM comparisons.
- Limitations and Future Work: Generated motions can still exhibit slight penetration with interacting objects.The paper suggests post-processing pose optimization as a potential remedy.
6. Conclusion
SAMP combines navigation, stochastic goal selection, and incremental motion generation to create lifelike avatars in previously unseen and complex environments.
- SAMP integrates path planning, stochastic interaction-goal generation, and autoregressive motion synthesis for human-object interactions.GoalNet samples interaction locations and directions, while MotionNet generates body poses from past motion and the goal.
- The method is trained on a motion-capture dataset containing diverse human-object interactions.
- SAMP is designed to help characters navigate while avoiding obstacles, interact with objects in different ways, and produce naturally varying goal-directed motions.
A.1. Motion Data
The motion data appendix documents diverse captured action styles, scene objects, and object-specific interaction goals used to support SAMP’s training data.
- Motion Data: The MoCap data includes different sitting and lying-down styles, with the action breakdown reported in Table S.1.
- Motion Data: The captured interactions use multiple scene objects, with the objects shown in Figure S.2.
- Motion Data: GoalNet data covers sofas, L-shaped sofas, chairs, armchairs, and tables selected from ShapeNet.
- Motion Data: Each selected object instance receives 1–5 manually labeled interaction goals, depending on the distinct goals its geometry affords.L-shaped sofas receive five goals in the example, compared with three for chairs.
B. Training Details
Training details specify SAMP’s network components, optimization settings, baseline architectures, scheduled sampling procedure, and path-planning setup.
- Network Architecture: MotionNet uses a 64-dimensional latent code and 12 experts, with three-layer fully connected sub-networks using ELU activations.
- GoalNet: GoalNet uses a three-layer Interaction Encoder with shape {512, 512, 64} and a 3-dimensional latent vector Z_goal.
- Schedule Sampling: Scheduled sampling trains 60-frame roll-outs by feeding the ground-truth first frame and then sequentially predicting subsequent frames.The schedule uses C1 = 30 and C2 = 60, with clips divided into equal-length windows.
- Baselines: The baselines include a feedforward MLP and a Mixture of Experts model built from specified encoder, gating, and prediction networks.
- Schedule Sampling: Schedule Sampling is essential for successfully reaching goals and executing actions; without it, the model may diverge, get stuck, or take very long to arrive.
- Path Planning: The Path Planning Module computes a navigation mesh from the character collider and scene geometry, then uses A* to find a shortest path between mapped polygons.
F. Data Augmentation Details
When an object is replaced, SAMP transfers contacts to the new surface, interpolates a new motion curve, and reconstructs the full-body pose with CCD IK.
- Data Augmentation Details: Object replacement projects each original contact onto the closest point on the new object’s surface.
- Data Augmentation Details: The augmented motion curve is interpolated and the full-body pose is computed using a CCD IK solver.The procedure does not guarantee smoothness but was found stable in practice.
G. Interaction Encoder Ablation:
The ablation evaluates the Interaction Encoder’s effect on goal-reaching precision, while noting limits in direct comparison and adaptation to substantially different object geometries.
- Ablation result: 6.09 cm and 3.55 deg with the Interaction Encoder versus 14.82 cm and 3.65 deg without it.Removing the encoder worsens both positional and directional goal-reaching precision.
- Method context: SAMP samples object-specific goals, plans an obstacle-free path, and synthesizes diverse motion sequences until the goal is executed.A direct comparison with Cao et al. is difficult because their formulation differs significantly from SAMP’s.
- Failure case: SAMP might not adapt well to objects with significantly different geometry from those seen during training.The authors identify object-geometry encoding as a direction for future work.